FAST Search Server 2010 for SharePoint

Published: May 12, 2010

Microsoft FAST Search Server 2010 for SharePoint includes all the search features and integration capabilities of SharePoint Server 2010 and adds deep platform flexibility and scale and enhanced content processing capabilities.

You can deploy FAST Search Server 2010 for SharePoint across multiple servers to meet demanding requirements for redundancy, performance and capacity. It can be scaled among the following axes: document volume, query volume and processing power for content, query, and results. Deployment, configuration and management take place through user interfaces, Windows PowerShell cmdlets, XML configuration files and command-line operations.

In this section:

  • Newly published content for FAST Search Server 2010 for SharePoint

  • Getting started (FAST Search Server 2010 for SharePoint)

  • Planning and architecture (FAST Search Server 2010 for SharePoint)

  • Deployment for FAST Search Server 2010 for SharePoint

  • Operations (FAST Search Server 2010 for SharePoint)

  • Security and protection (FAST Search Server 2010 for SharePoint)

  • Troubleshooting (FAST Search Server 2010 for SharePoint)

  • Technical reference (FAST Search Server 2010 for SharePoint)


Newly published content for FAST Search Server 2010 for SharePoint

This article contains a list of new or updated content for Microsoft FAST Search Server 2010 for SharePoint. This article is updated regularly and is also available in an RSS feed RSS feed (http://sxp.microsoft.com/feeds/3.0/msdntn/FASTSearchServer2010NewContent).

In this article:

April 23, 2012 | April 16, 2012 | August 22, 2011 | June 27, 2011 | June 13, 2011 | June 06, 2011 | May 02, 2011 | March 28, 2011 | March 14, 2011 | February 14, 2011 | February 07, 2011 | December 13, 2010 | December 06, 2010 | November 29, 2010 | November 1, 2010 | October 25, 2010 | September 13, 2010

April 23, 2012

Administration pages: Cannot access the pages (FAST Search Server 2010 for SharePoint) This article explains how to troubleshoot problems with accessing the administration pages for Central Admin, Site Settings and Site Collections pages for keywords and managed properties.

April 16, 2012

New articles

How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper) This white paper describes how MSIT planned and deployed enterprise search to improve index capacity, crawl speed, query response, relevance, and availability.

August 22, 2011

New articles

About backup and restore scripts (FAST Search Server 2010 for SharePoint) This article describes the available options for the FAST Search Server 2010 for SharePoint backup and restore scripts.

June 27, 2011

New articles

Install a software update (FAST Search Server 2010 for SharePoint) This article describes how to install a software update for FAST Search Server 2010 for SharePoint.

June 13, 2011

New articles

Test results: Extra-large scenario (FAST Search Server 2010 for SharePoint) This article describes the set up and test results for an extra large FAST Search Server 2010 for SharePoint test scenario with up to 500 million items. The article describes disk usage, crawling, indexing and query performance under load.

June 06, 2011

New articles

Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint) This article describes how to manage a backup indexer server failure if you are using a backup indexer row.

May 02, 2011

New articles

Manage the Search Center Home page (FAST Search Server 2010 for SharePoint) This article gives an introduction on how to change properties for the Search Box Web Part and how to use search scopes to narrow a search to particular locations or content on the Search Center Home page.

Manage the Search Results page (FAST Search Server 2010 for SharePoint) This article gives an introduction on how to change how search results are displayed, how to add sort options to search results, how to display information form external sources and how to make it easy to find people on the Search Results Page.

Configure additional features for the Search Core Results Web Part (FAST Search Server 2010 for SharePoint) This article explains how to configure additional features for the Search Core Results Web Part.

Manage hit highlighted summary for a managed property by using Windows PowerShell (FAST Search Server 2010 for SharePoint) This article explains how to manage hit highlighted summary for a managed property.

Set a rank profile as sort option on the Search Results page via the graphical user interface (FAST Search Server 2010 for SharePoint) This article explains how to set a rank profile as sort option on the Search Results page.

Set a managed property as sort option on the Search Results page (FAST Search Server 2010 for SharePoint) This article explains how to set a managed property as sort option on the Search Results page.

Manage refiners and refiner settings (FAST Search Server 2010 for SharePoint) This article explains how to manage refiners and refiner settings.

March 28, 2011

New articles

Plan the relevance of search results (FAST Search Server 2010 for SharePoint) This article describes the features involved when improving the relevance of search results.

How ranking and sorting of search results are calculated (FAST Search Server 2010 for SharePoint) This article describes how ranking and sorting of search results are calculated.

The index schema and its features (FAST Search Server 2010 for SharePoint) This article describes the index schema and its features.

Which relevance factors can you tune and what is the impact (FAST Search Server 2010 for SharePoint) This article gives an overview of the different factors you can tune to influence search relevance.

Static rank (quality rank) (FAST Search Server 2010 for SharePoint) This article describes the different factors you can tune to influence the static rank (quality rank) component of search relevance.

Dynamic rank (FAST Search Server 2010 for SharePoint) This article describes the different factors you can tune to influence the dynamic rank component of search relevance.

Keyword rank (FAST Search Server 2010 for SharePoint) This article describes the different factors you can tune to influence the keyword rank component of search relevance.

Linguistic features (FAST Search Server 2010 for SharePoint) This article describes the different linguistic features you can tune to influence search relevance.

March 14, 2011

New articles

Installation: Configuration wizard fails (FAST Search Server 2010 for SharePoint) This article describes how to troubleshoot problems with the post-configuration wizard for FAST Search for Sharepoint 2010.

Reset the content index (FAST Search Web crawler) This article describes how to remove content that was placed in the index by the FAST Search Web crawler.

Reset the content index (FAST Search database connector) This article describes how to remove content that was placed in the index by the FAST Search database connector.

Reset the content index (FAST Search Lotus Notes connector) This article describes how to remove content that was placed in the index by the FAST Search Lotus Notes content connector.

Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint) This article describes the set up and test results for the extra-small FAST Search Server 2010 for SharePoint test scenario with up to 8 million items.

Test results: Medium scenario (FAST Search Server 2010 for SharePoint) This article describes the set up and test results for the medium FAST Search Server 2010 for SharePoint test scenario with up to 40 million items.

Test results: Large scenario (FAST Search Server 2010 for SharePoint) This article describes the set up and test results for the large FAST Search Server 2010 for SharePoint test scenario with up to 100 million items.

Recommendations: Content volume capacity (FAST Search Server 2010 for SharePoint) This article describes content volume capacity recommendations for FAST Search Server 2010 for SharePoint.

Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint) This article describes redundancy and availability recommendations for FAST Search Server 2010 for SharePoint.

Recommendations: Content freshness (FAST Search Server 2010 for SharePoint) This article describes content freshness recommendations for FAST Search Server 2010 for SharePoint.

Recommendations: Query throughput (FAST Search Server 2010 for SharePoint) This article describes query throughput recommendations for FAST Search Server 2010 for SharePoint.

Recommendations: Storage (FAST Search Server 2010 for SharePoint) This article describes storage recommendations for FAST Search Server 2010 for SharePoint.

Recommendations: Virtualization (FAST Search Server 2010 for SharePoint) This article describes virtualization recommendations for FAST Search Server 2010 for SharePoint.

February 14, 2011

New articles

Indexer server: Disk usage increases unexpectedly (FAST Search Server 2010 for SharePoint) This article describes how to troubleshoot an issue with unexpected increased disk usage on a FAST Search Server 2010 for SharePoint indexer server.

Updated articles

Manage certificates (FAST Search Server 2010 for SharePoint) The article has been updated with information on how to add a password for the general purpose FAST Search certificate.

Create and view search scopes (FAST Search Server 2010 for SharePoint) The article has been updated with information on how to display scopes on the Search Center site.

February 07, 2011

New articles

Performance and capacity planning (FAST Search Server 2010 for SharePoint) This article provides guidance on how to plan performance and capacity of a FAST Search Server 2010 for SharePoint deployment.

Performance and capacity testing (FAST Search Server 2010 for SharePoint) This article provides guidance on how to test performance and capacity of a FAST Search Server 2010 for SharePoint deployment.

Performance and capacity monitoring (FAST Search Server 2010 for SharePoint) This article provides guidance on how to monitor performance and capacity of a FAST Search Server 2010 for SharePoint deployment.

Performance and capacity tuning (FAST Search Server 2010 for SharePoint) This article provides guidance on how to tune performance and capacity of a FAST Search Server 2010 for SharePoint deployment.

December 13, 2010

New articles

Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint) This article describes considerations for configuring FAST Search Server 2010 for SharePoint to index up to 40 million items per column.

Add an index column (FAST Search Server 2010 for SharePoint) This article describes how to increase indexing capacity by adding an index column.

Extend content capacity for a new deployment (FAST Search Server 2010 for SharePoint) This article describes how to support up to 40 million items per index column for a new FAST Search Server 2010 for SharePoint deployment.

Extend content capacity for an existing deployment (FAST Search Server 2010 for SharePoint) This article describes how to support up to 40 million items per index column for an existing FAST Search Server 2010 for SharePoint deployment.

Handle software updates when you have extended the content capacity (FAST Search Server 2010 for SharePoint) This article describes additional steps during a software update if you have configured for up to 40 million items per index column for a FAST Search Server 2010 for SharePoint deployment.

Increase query capacity (FAST Search Server 2010 for SharePoint) This article describes how to handle more queries per second and provide query fault-tolerance.

Add a backup indexer row (FAST Search Server 2010 for SharePoint) This article describes how to add a backup indexer row for indexer fault-tolerance.

Verify the status of the backup indexer (FAST Search Server 2010 for SharePoint) This article describes how you can verify that the backup and primary indexer are in sync.

Synchronize the primary and the backup indexer servers (FAST Search Server 2010 for SharePoint) This article describes how to handle a situation when the primary and backup indexer gets out of sync with respect to number of items indexed.

Replace a failed primary indexer server (FAST Search Server 2010 for SharePoint) This article describes how to manage primary or backup indexer failure when using a backup indexer row.

December 06, 2010

New articles

Configure people search (FAST Search Server 2010 for SharePoint) This article describes how to enable and configure people search for FAST Search Server 2010 for SharePoint.

Crawl the user profile store (FAST Search Server 2010 for SharePoint) This article describes how to add a content source for people and explains how to crawl the user profile store using the FAST Search Query Search Service Application.

Manage people search (FAST Search Server 2010 for SharePoint) This article describes how to enable and configure people search and how to crawl people profiles.

Updated article

Configure and use the Lotus Notes connector (FAST Search Server 2010 for SharePoint) Updated with information about the client and server versions that will work with the Lotus Notes connector.

November 29, 2010

indexerinfo XML status report syntax (FAST Search Server 2010 for SharePoint) This article describes the syntax of the XML-formatted status report for the indexerinfo status command.

searchinfo XML status report syntax (FAST Search Server 2010 for SharePoint) This article describes the syntax of the XML-formatted status report for the searchinfo status command.

IFilter and file type reference (FAST Search Server 2010 for SharePoint) This article lists the file types that are supported for crawling out-of-the-box by FAST Search Server 2010 for SharePoint. The article also lists the file types that are not crawled by default.

Default indexing connector protocols (FAST Search Server 2010 for SharePoint) This article lists the indexing connector protocols that are installed by default in Microsoft SharePoint Server 2010 and are used when you crawl content with the FAST Search Content Search Service Application or people profiles with the FAST Search Query Search Service Application.

November 1, 2010

New Resource Center

Advanced training for enterprise search Provides dozens of videos, presentations, quizzes, and links to other resources for learning about Microsoft Enterprise Search products.

October 25, 2010

New downloadable book

Deployment guide for FAST Search Server 2010 for SharePoint Downloadable and printable content (available in .doc, .pdf, or .xps file formats) that contains information about how to deploy FAST Search Server 2010 for SharePoint.

New Resource Center

Updates for Enterprise Search Products Find resources for installing software updates on Microsoft Search Server 2010, Microsoft Search Server 2010 Express, and Microsoft FAST Search Server 2010 for SharePoint.

October 04, 2010

Manage federated locations (FAST Search Server 2010 for SharePoint) Describes how to configure federated locations in the FAST Search Query Search Service Application to display query results from locations outside your internal network.

October 04, 2010

Content sources that require additional configuration (FAST Search Server 2010 for SharePoint) Updated indexing connector protocols list.

September 13, 2010

East Asian language specific tokenization tasks by using Windows PowerShell (FAST Search Server 2010 for SharePoint) Describes how you can tune relevancy for East Asian languages by influencing the default tokenization using linguistic tokenization, substring tokenization or a combination of substring tokenization and linguistic tokenization.


Downloadable content for FAST Search Server 2010 for SharePoint

Published: June 24, 2010

The content in this technical library is augmented by content available for download.

Content | Description
FAST Search Server 2010 for SharePoint evaluation guide (white paper) | The overall goal of this guide is to help technical decision makers, IT professionals, and developers perform a thorough and effective evaluation of the search features provided by Microsoft FAST Search Server 2010 for SharePoint.
Deployment guide for FAST Search Server 2010 for SharePoint (http://go.microsoft.com/fwlink/p/?LinkId=204984) | Downloadable and printable content (available in .doc, .pdf, or .xps file formats) that contains information about how to deploy FAST Search Server 2010 for SharePoint.
Microsoft FAST Search Server 2010 for SharePoint Technical Library in Compiled Help format (http://www.microsoft.com/downloads/details.aspx?FamilyID=ead44c66-7d02-4edf-9e56-2f56c6f59f22&DisplayLang=en) | The entire library for FAST Search Server 2010 for SharePoint is now available as a downloadable .chm file.


Getting started (FAST Search Server 2010 for SharePoint)

Use the downloadable white papers and the links below to get started with Microsoft FAST Search Server 2010 for SharePoint.

  • Getting started with enterprise search in SharePoint 2010 Products

  • FAST Search Server 2010 for SharePoint evaluation guide (white paper)

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • To plan how to install and deploy FAST Search Server 2010 for SharePoint, see the topics in Planning and architecture (FAST Search Server 2010 for SharePoint).

  • To install FAST Search Server 2010 for SharePoint, see the topics in Deployment for FAST Search Server 2010 for SharePoint.

  • To find articles about how to crawl content, how to perform backup and recovery, optimize search relevance, and manage FAST Search Server 2010 for SharePoint on a daily and weekly basis, see Operations (FAST Search Server 2010 for SharePoint).

  • For information about security, see Security and protection (FAST Search Server 2010 for SharePoint).

  • For reference information about the FAST Search Server 2010 for SharePoint Windows PowerShell cmdlets, command-line tools, configuration files, and overviews of the WMI classes and performance counters for monitoring, see Technical reference (FAST Search Server 2010 for SharePoint)


Getting started with enterprise search in SharePoint 2010 Products
See Also

Summary: Learn how to get started matching the features and capabilities in Microsoft SharePoint 2010 products with the search requirements of your organization or enterprise.

Applies to: Microsoft SharePoint Foundation 2010, Microsoft Search Server 2010 Express, Microsoft Search Server 2010, Microsoft SharePoint Server 2010, Microsoft FAST Search Server 2010 for SharePoint

Contents

  • Introduction

  • Capabilities Comparison

  • Feature Comparison

  • First Steps

  • Evaluation Guides and Further Reading

Introduction

Search connects people to the information that they must have to do their jobs. General productivity search solutions increase employee efficiency by connecting a broad set of people to a broad set of information. High-value search applications drive measurable returns on investment (ROI) by helping a specific set of people extract the most from a specific set of information. Until now, organizations have had to purchase multiple products to address both scenarios; no single search solution delivered the correct functionality at the correct price, which forced organizations to incur the following additional costs:

  • Software costs — for example, licenses, support, and maintenance costs for separate products.

  • Headcount costs — for example, dedicated positions required to design, deploy, and manage different search solutions.

  • Opportunity costs — for example, mediocre general productivity search and limited deployment of high-value search applications.

When you assess the enterprise search offerings from Microsoft, review the capabilities of the various products and match them to the needs of your organization. You will find a broad set of enterprise search capabilities that are based on SharePoint 2010 Products and that serve the enterprise as growth, maturity, and changing business needs redefine your organizational enterprise search requirements.

SharePoint Foundation 2010 and Search Server 2010 Express are free to use on licensed Windows servers. They are entry-level products that crawl and index content. They also have a user interface that you can use to execute queries against the catalog of crawled content.

SharePoint Server 2010 and Search Server 2010 have improved scalability, and options to index content from many different sources. They include the configurability, manageability, and reporting features that your organization requires. SharePoint Server 2010 has a great platform that you can use to create general productivity search applications.

FAST Search Server 2010 for SharePoint is new, and it delivers the power of FAST with the simplicity and total cost of ownership (TCO) benefits that are typically associated with Microsoft products. FAST Search Server 2010 for SharePoint delivers the best general productivity search on the market and makes it easy and economical to build high-value search applications. FAST Search Server 2010 for SharePoint further improves the crawling, content processing, user interface, and scale of SharePoint Server 2010.

Capabilities Comparison

The following table compares general enterprise search capabilities that were in Office SharePoint Server 2007 with the features that are in SharePoint Server 2010 and FAST Search Server 2010 for SharePoint.

Note:

The table uses the following values:

  • X = available in 2007

  • O = available in 2010

  • † = improved in 2010

Features and Capabilities Office SharePoint Server 2007 SharePoint Server 2010 FAST Search Server 2010 for SharePoint

People and expertise search

Capture knowledge not found in documents by searching for people and expertise using SharePoint products.

X

O†

O†

SharePoint 2010 connector framework

Securely connect to content in SharePoint sites and from sources across your enterprise.

Use the Business Data Catalog to easily create your own connectors that work exactly like those available out of the box.

X

O†

O†

100m content volume with sub-second query response time

Meet the scale and performance needs of your entire organization or the specialized needs of individual departments.

X

O†

O†

Search from Windows 7 & Windows Mobile

Search beyond the search center. Conduct searches from the Windows 7 desktop and on your Windows mobile device.

X

O†

O†

Taxonomy tag integration

Bring the power of taxonomy into search. Tag metadata is shown in results, and users can refine by taxonomy-based tags.

O

O

Metadata-driven refinement panel

With the new refinement panel in SharePoint Server 2010 and FAST Search Server 2010 for SharePoint, users can narrow the results of their searches and navigate to the correct content faster.

O

O

Relevance improves with social behavior

The click-through behavior of similar search queries affects the rank that documents receive.

The more users click a certain item, the higher its ranking for related queries.

O

O

Phonetic and nickname search

Confidently search for a person's name as it sounds - without worrying about the exact spelling.

O

O

Contextual search

Tailor different results and refinement options, depending on the profile of the user or audience.

O

Thumbnails and previews

Thumbnails and previews make the results of a search query visual, which enable users to recognize the correct content quickly.

O

> 500m content volume with sub-second query response time

Scale to extremes with FAST Search Server 2010 for SharePoint while maintaining sub-second query response times.

O

Advanced content processing with advanced linguistics

Extract and create metadata latent in documents to improve search results, sorting capabilities, and the refinement panel.

O

Search-driven applications

Meet all the search application needs you have across your business.

Common examples include 360° Customer Insight, Research and Development Innovation Portal, and Product Support.

O

Business Intelligence Indexing Connector

Crawl Microsoft Excel 2010 workbooks and Reporting Services Reports with improved results, descriptions, thumbnails, and refiners.

Discover your business intelligence (BI) assets quickly and easily, navigate not only the document, but also the data behind the scenes, and access the information that you need quickly and easily.

O

Feature Comparison

Use the following table to make quick comparisons of the enterprise search features that each SharePoint product provides.

Note:

The table uses the following values:

  • * Specifies NTLM environments

  • ** Specifies FAST Search Server 2010 for SharePoint provides enhanced capabilities in these areas

  • *** Specifies a subset of the supported languages

Feature | SharePoint Foundation 2010 | Search Server 2010 Express | Search Server 2010 | SharePoint Server 2010 | FAST Search Server 2010 for SharePoint
Basic site search | Y | Y | Y | Y | Y
Best Bets |  | Y | Y | Y | Y
Visual Best Bets |  |  |  |  | Y
Similar Results |  |  |  |  | Y
Duplicate Results | Y | Y | Y | Y | Y
Search Scopes |  | Y | Y | Y | Y
RSS Feeds for Search Results | Y | Y | Y | Y | Y
Alerts for Search Results | Y* | Y* | Y* | Y* |
Advanced Search Page |  | Y | Y | Y | Y
Search Enhancement based on User Contexts |  |  |  |  | Y
Crawled and Managed Properties |  | Y | Y | Y | Y**
Entity Extraction |  |  |  |  | Y
Query Federation |  | Y | Y | Y | Y
Query Suggestions |  | Y | Y | Y | Y
Sort Results on Managed Properties or Rank Profiles |  |  |  |  | Y
Relevancy Tuning by Document or Site Promotions |  | Y | Y | Y | Y**
Shallow Results Refinement |  | Y | Y | Y | Y
Deep Results Refinement |  |  |  |  | Y
Document Preview |  |  |  |  | Y
Windows 7 Federation |  | Y | Y | Y | Y
People Search |  |  |  | Y | Y
Phonetic Name Search*** |  |  |  | Y | Y
Nickname Search*** |  |  |  | Y | Y
Self Search |  |  |  | Y | Y
Social Search |  |  |  | Y | Y
Taxonomy Integration |  |  |  | Y | Y
Multi-Tenant Hosting |  |  |  | Y |
Rich Web Indexing Support |  |  |  |  | Y
Note:

For details about a specific feature, review the evaluation guides for the products.

First Steps

There are some general tasks that you must perform when you evaluate search products or implement enterprise search solutions that meet your organization's requirements. Use the following sections to get started.

Identify your content

Before you take any action, identify the content in your organization that you want to index. Content from SharePoint sites is automatically included in all of the SharePoint search products, but you can also index content from file shares, Microsoft Exchange public folders, external Web sites, Lotus Notes databases, and your line-of-business applications.

You can also use SharePoint Designer 2010 or Visual Studio 2010 to create Business Data Catalog applications that enable custom databases, Web services, and other custom solutions to be indexed.

Note:

SharePoint Foundation 2010 can index content in SharePoint sites only. All other enterprise search products from Microsoft can index content in the content repositories mentioned earlier by using the connector framework.

FAST Search Server 2010 for SharePoint also has improved capabilities that enable you to index business intelligence data. You can obtain the Business Intelligence Indexing Connector to provide better search and analysis for business intelligence assets.

Crawl your content

The next step is to crawl the content in the content repositories that you identified. You can implement both full and incremental crawl schedules to ensure that indexed data is as fresh as possible.

The new index partitioning capabilities in Search Server 2010, SharePoint Server 2010, and FAST Search Server 2010 for SharePoint enable you to balance the indexing load (and query load) across dedicated servers that provide search for large amounts of content.

Scale your search infrastructure

One of the design goals for Search Server 2010 and SharePoint Server 2010 is to provide sub-second query response times for up to 100 million indexed documents. FAST Search Server 2010 for SharePoint provides sub-second query response times for more than 1 billion indexed documents. To achieve sub-second query response times for your organization, you might have to scale out your server farm to include multiple query servers. Additionally, to make sure that that indexed data is as fresh as possible when you index many documents, you might have to include multiple crawl components. The highly componentized architecture of the enterprise search products enables you to do that. For example, you can implement multiple crawl components to meet crawling demands, and you can include multiple query components to balance the query processing load at search time.

Customize the user search experience

Optionally, you can provide a customized user interface that information workers can use to search for information and manipulate results in the most efficient way for your business. The Enterprise Search Center for SharePoint Server 2010 and the FAST Search Center are highly customizable, and do not require developer effort. For example, you can create search scopes, change properties of search Web Parts, apply XSLT styles, and apply themes to search centers with minimal effort by using the Web browser or by using SharePoint Designer 2010.

If you want to further customize the search experience, developers can create a completely customized user interface by creating new Web Parts that inherit functionality from the existing Search Web Parts. Also, the search API is completely open. Therefore, developers can create new paradigms and ways of working with search — for example, by creating new Web Parts or even Silverlight applications that run searches and then render results.

Customize the content processing pipeline

One of the key differentiators between FAST Search Server 2010 for SharePoint and other search products is the ease with which you can customize content processing at indexing time. FAST Search Server 2010 for SharePoint enables you to create your own metadata extractors to use in the content processing pipeline. This enables you to define how managed properties are populated from crawled content, and you can then use that metadata in deep refiners. That is, you can include these organization-specific vocabularies in the metadata refinement panel that enables you to provide search solutions in the terms and language of your business.

Evaluation Guides and Further Reading

For more information about enterprise search and SharePoint Products and Technologies, see the following:

  • Microsoft Enterprise Search Center

  • TechNet Enterprise Search Technical Center

  • Microsoft Enterprise Search Blog

  • Microsoft SharePoint Server 2010 Search TechCenter

  • Microsoft SharePoint Server 2010 Enterprise Search Evaluation Guide

  • Enterprise search administration (SharePoint Server 2010)

  • Manage search topology (SharePoint Server 2010)

  • Microsoft FAST Search Server 2010 for SharePoint TechCenter

  • Microsoft FAST Search Server 2010 for SharePoint Evaluation Guide

  • FAST Search Server 2010 for SharePoint in the TechNet library

See Also

Other Resources

Resource Center: Architecture Design for SharePoint Server 2010
Resource Center: Enterprise Search in SharePoint Server 2010


FAST Search Server 2010 for SharePoint evaluation guide (white paper)

The overall goal of this guide is to help technical decision makers, IT professionals, and developers perform a thorough and effective evaluation of the search features provided by Microsoft FAST Search Server 2010 for SharePoint. This guide includes the following:

  • Descriptions of the enterprise search features and technologies provided by FAST Search Server 2010 for SharePoint.

  • Details of the additional enterprise features that FAST Search Server 2010 for SharePoint provides.

  • Details on the indexing and query architecture implemented by FAST Search Server 2010 for SharePoint.

  • Tours and walkthroughs of the main search features provided by FAST Search Server 2010 for SharePoint.

Note:

FAST Search Server 2010 for SharePoint shares much in common with the enterprise search features of SharePoint Server 2010. If you are not already familiar with the enterprise search features of SharePoint Server 2010, you should also review the SharePoint Server 2010 enterprise search evaluation guide (white paper).

To download the guide in the format that you want, click the following links:

  • FAST Search Server 2010 for SharePoint Evaluation Guide (Microsoft Office Word) (http://go.microsoft.com/fwlink/p/?LinkId=189428)

  • FAST Search Server 2010 for SharePoint Evaluation Guide (PDF) (http://go.microsoft.com/fwlink/p/?LinkId=189429)

  • FAST Search Server 2010 for SharePoint Evaluation Guide (XPS) (http://go.microsoft.com/fwlink/p/?LinkId=189430)


How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

Published: April 16, 2012

Summary: Learn how MSIT planned and deployed enterprise search to improve index capacity, crawl speed, query response, relevance, and availability.

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

Situation

To perform their day-to-day responsibilities, Microsoft employees frequently use enterprise search to find information on the corporate intranet. Microsoft IT used Microsoft Office SharePoint Server 2007 to provide enterprise search for employees until 2009, when the content index for the enterprise Search Center approached the software capacity of 50 million items and crawl and query performance decreased. To improve performance, MSIT configured the search system so that it no longer crawled certain content repositories, but then employees were no longer able to get complete search results from the enterprise Search Center.

Solution

Microsoft IT migrated to Microsoft FAST Search Server 2010 for SharePoint to provide a more powerful enterprise search solution. This provided the following benefits:

  • Centralized search management

  • Content index completeness

  • Huge content index capacity

  • Faster crawls for fresher search results

  • Faster query responses

  • More customizable search experiences

  • More relevant search results

  • High availability

  • Increased user satisfaction

Organization of this white paper

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Executive summary (Enterprise search at Microsoft, white paper)

Published: April 16, 2012

This is the second of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper).

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

Microsoft employees create and store large amounts of information on the Microsoft corporate intranet every day. Employees require a powerful enterprise search solution that enables them to find items on the intranet quickly and easily amid the huge amount of stored data.

The Microsoft information Technology (MSIT) group manages all aspects of the enterprise search implementation at Microsoft. In 2006, MSIT deployed Microsoft Office SharePoint Server 2007 to provide a new enterprise search solution for employees. The solution was geographically distributed to provide fast crawl and query performance. It also provided an enterprise Search Center that all employees could access to perform queries on all intranet content.

By 2009, the content index for the enterprise Search Center approached the Office SharePoint Server 2007 maximum capacity of 50 million items, and the large crawl load caused crawl and query performance to decrease. At that point, MSIT evaluated options for migrating to an enterprise search solution that would provide benefits such as centralized management, greater scalability for large capacity, improved crawl and query performance, high availability, and advanced features such as fine adjustments for ranking of search results. MSIT chose Microsoft FAST Search Server 2010 for SharePoint because it satisfied all these demands.

MSIT upgraded to FAST Search Server 2010 for SharePoint in late 2009, some months before the public release of the software. This white paper describes the experience of MSIT as an early adopter of FAST Search Server 2010 for SharePoint. The articles describe how MSIT deployed and configured FAST Search Server 2010 for SharePoint, how MSIT administers the search system, and how the solution meets the need for an advanced enterprise search solution at Microsoft by providing fresh, complete, and useful search results in a robust, highly customizable deployment.

In comparison to a SharePoint Server 2007 search solution, MSIT found that it takes additional hardware, skills, and people to plan, deploy, operate, and support a large deployment such as the MSIT search solution that uses FAST Search Server 2010 for SharePoint. However, automation of certain maintenance tasks, together with careful planning and governance, helped MSIT simplify and speed up deployment and management of the FAST Search Server 2010 for SharePoint solution.

Audience

The primary audience for this white paper includes enterprise-level business decision makers, technical decision makers, FAST Search Server 2010 for SharePoint server farm administrators, and search-service application administrators. SharePoint site collection administrators and site owners might also find this white paper useful.

It is assumed that readers of this white paper are familiar with the purpose and business value of enterprise search. It is also assumed that readers are knowledgeable about SharePoint search technologies, specifically about the search features and functionality of FAST Search Server 2010 for SharePoint, SharePoint Server 2010, and SharePoint Server 2007. For information about search features and functionality in these products, see the following resources:

  • FAST Search Server 2010 for SharePoint Resource Center

  • Enterprise Search in SharePoint Server 2010

  • SharePoint Server 2007

Non-goals of this white paper

This white paper is not a procedural guide. Because each IT environment is unique, each organization should implement enterprise search according to its particular needs. However, many of the considerations, practices, and lessons that this paper describes can be relevant to the implementation of enterprise search in other large-scale IT environments. For information about how to deploy an enterprise search solution, see the following articles:

  • Deployment for FAST Search Server 2010 for SharePoint

  • Deployment for SharePoint Server 2010

  • Best practices for search in SharePoint Server 2010

Also, this white paper does not discuss the implementation of SharePoint services at Microsoft other than the FAST Search Server 2010 for SharePoint search service and the SharePoint Server 2007 search service. For information about using other SharePoint services with the FAST Search Server 2010 for SharePoint search service, see Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper).

Finally, this white paper does not compare the enterprise search capabilities of Microsoft products. For a technical diagram that compares enterprise search capabilities in current Microsoft releases, see Search Technologies for SharePoint 2010 Products, which is available in the following formats:

  • Visio

  • PDF

  • XPS

Note:

For security reasons, names in this paper that refer to Microsoft-internal resources are for illustration only and do not represent actual names.

Other articles in this white paper

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Background on the use of enterprise search at Microsoft

Published: April 16, 2012

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the third of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper). In this article:

  • The need for enterprise search at Microsoft

  • The MSIT implementation of enterprise search that used SharePoint Server 2007

  • Requirements for an improved search solution

  • The solution: FAST Search Server 2010 for SharePoint

The need for enterprise search at Microsoft

Every day, thousands of Microsoft employees, contractors, and vendors create and store large amounts of digital information on servers in many locations across the worldwide Microsoft corporate enterprise. Content is in the form of Microsoft Office documents, graphics, videos, and other formats. Because Microsoft workers use the collaborative features of SharePoint Server extensively, MSIT must manage more than 34 terabytes of content on SharePoint intranet sites. This includes content such as SharePoint Server libraries, lists, blogs, wikis, and My Sites.

MSIT hosts and manages all intranet sites that consume the enterprise search service. The following table provides examples of these sites.

Table 1. Examples of sites that use the enterprise Search service

Site or sites Description

MSW

MSW is one of the primary enterprise portals at Microsoft. It provides daily news, executive updates, and other information. In addition, employees frequently go to this site to get to other intranet sites, such as sites for human resources, legal affairs, and technical research. The MSW farm also hosts the enterprise Search Center.

Division-level portals

Examples of division-level portals that use the enterprise search service include the following:

  • ITWeb, the portal for technical support

  • FinWeb, the finance portal

  • LCAWeb, the portal for the Legal and Corporate Affairs group

  • Infopedia, the portal for sales and marketing information, which provides content related to engaging with customers and partners

  • Office, the portal for employees in the Microsoft Office Division

Department-level portals

Employees can use these portals to create collaboration or portal sites that host business-critical information.

Team sites

Employees can use these portals to create team sites for collaboration. On team sites, employees can also use the latest SharePoint features and test proof-of-concept solutions.

My Sites

Each employee has a My Site to store business and personal information and documents. Employees can restrict access to items on their My Site.

However, many other groups and individuals in the company create and host their own sites. MSIT does not permit these sites to consume the enterprise search service, but the site owners can ask MSIT to configure the enterprise search service to crawl their sites. Other important corporate data repositories include file shares and structured data sources in line-of-business applications. MSIT must provide an enterprise search solution that enables the members of the Microsoft work force to quickly and easily find up-to-date, relevant information in all of these large and diverse repositories.

The MSIT implementation of enterprise search that used SharePoint Server 2007

MSIT has used SharePoint technologies to provide enterprise search capability to Microsoft employees for a number of years. MSIT used Microsoft Office SharePoint Portal Server 2003 to provide enterprise search from 2003 until late 2006, at which point that solution no longer scaled to company requirements. MSIT then upgraded to an enterprise search solution that used Microsoft Office SharePoint Server 2007.

As with the previous solution, in the Office SharePoint Server 2007 solution, MSIT maintained three separate Shared Services Providers (SSPs), each in a separate Microsoft data center. One SSP was in Singapore, the second SSP was in Dublin, Ireland, and the third SSP was in Redmond, Washington. The search service in the Dublin SSP crawled only content in the Europe, Middle East, and Africa (EMEA) region (and therefore provided search results for that region only), while the search service in the Singapore SSP crawled content in the Asia-Pacific region (and therefore provided search results for that region only). For a time, however, the Redmond SSP search service crawled content in all three regions and provided worldwide search results that all users could access by going to the enterprise Search Center.

Figure 1 indicates the region or regions that the search service of each SSP crawled in the initial state of the SharePoint Server 2007 search solution.

Figure 1. Content crawled by each search service in the initial state of the SharePoint Server 2007 search solution

Initial state of SharePoint Server 2007 solution

MSIT took this regional approach to enterprise search because search services were not fully supported over the corporate wide area network (WAN). Therefore, each region required a separate server-farm deployment for search. The deployment that hosted the Redmond search service used one dedicated index server, three dedicated query servers, and two database servers. The EMEA and Asia-Pacific search deployments each used one dedicated index server, two servers configured with both the query server role and the web server role, and one database server. Thus, a total of 14 computers in three different data centers handled the worldwide enterprise search solution.

An advantage of this approach for users in the EMEA and Asia-Pacific regions was that search results were relatively fresh, because crawling and indexing were relatively fast on the regional scale. Query responses were also relatively fast in those regions for a similar reason.

The main disadvantage of this approach was that there were three separate search services for MSIT to manage, so there were three separate content indexes, one corresponding to each search service. By searching from a site in the Asia-Pacific region, users could get search results for sites in that region only. Similarly, by searching from a site in the EMEA region, users could get search results for sites in that region only. However, because the Redmond search service was crawling all three regions, users in any region could still get search results for content worldwide by searching from the enterprise Search Center.

By late 2009, the Redmond SSP search service was crawling and indexing more than 13 terabytes of data that corresponded to more than 40 million items. Incremental crawl times for some content sources had increased to seven days or more, so that search results were frequently not fresh. Moreover, query latency in the Redmond search service had increased to eight seconds or more in some cases. Therefore, MSIT needed to reduce the amount of content that the Redmond search service was crawling.

For this reason, MSIT configured the Redmond search service so that it crawled only the Americas region and no longer crawled the EMEA and Asia-Pacific regions. The search service in each region was then crawling only regional data. However, users became dissatisfied because they could no longer get search results for content worldwide by searching from the enterprise Search Center.

Figure 2 indicates the region that the search service of each SSP crawled in the final state of the SharePoint Server 2007 search solution.

Figure 2. Content crawled by each search service in the final state of the SharePoint Server 2007 search solution

Final state of SharePoint Server 2007 solution

To improve performance of the Redmond search service even more, MSIT removed two other content sources from that service: Team (the content source for crawling SharePoint team sites) and Office (the content source for crawling the site collection of the Microsoft Office Division). The drawback of this change was that the only way to get search results from either of those site collections after that was to search directly from that site collection.

Another issue was that users in various locations were sometimes reporting that they were not getting helpful search results. Search results were sometimes out of date or not ranked appropriately as to relevance. In some cases, the search service took a long time to report that there were no search results at all.

In summary, due to the increased number of users, the distribution of users in different geographical locations, and the growth in content, the MSIT deployment of SharePoint Server 2007 no longer provided a suitable enterprise search solution to Microsoft workers. Thus, the need arose for a more powerful enterprise search solution.

Requirements for an improved search solution

MSIT identified the following business, technical, and administrative requirements for an enterprise-level search service:

  • Centralized management. The search solution must be hosted and manageable in a single deployment instead of in deployments in multiple locations.

  • Completeness of content index. There must be a single content index for all crawled content. This includes all site content in the three geographic regions (Americas, Asia-Pacific, and EMEA), so that all users can get search results from the same set of content.

  • High capacity. The content index must be able to accommodate projected growth to 200 million items in 2013 or later.

  • Significantly reduced crawl times. Incremental crawls of all high-priority content sources must take less than 24 hours so that all search results are as fresh as possible for the most important content.

  • Significantly reduced query latency. Users must be able to get search results in less than two seconds.

  • Increased ability to find relevant content. Search results must be ranked according to relevance--for example, based on authority of sites and pertinence of metadata--so that users can quickly find information that they are looking for.

  • Uniform search experience. All users worldwide must have access to an enterprise search center that provides the same search experience.

  • Significantly increased user satisfaction. Users must be pleased with the increase in performance and effectiveness of the search system and with the overall search experience.

The solution: FAST Search Server 2010 for SharePoint

In 2008, Microsoft acquired FAST Search and Transfer, a leading enterprise search vendor. Microsoft merged search technologies from that acquisition with SharePoint search technologies to create a new product, FAST Search Server 2010 for SharePoint.

MSIT evaluated the search features and functionality of SharePoint Server 2010 and FAST Search Server 2010 for SharePoint to determine which product would best meet the enterprise search requirements. FAST Search Server 2010 for SharePoint satisfied all of the demands. MSIT adopted FAST Search Server 2010 for SharePoint primarily because it is highly scalable and can support high capacity and performance for a large enterprise deployment. FAST Search Server 2010 for SharePoint also provides many advanced options, such as the ability to tune the ranking of search results.

Other articles in this white paper

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Microsoft enterprise search teams

Published: April 16, 2012

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the fourth of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper). In this article:

  • Enterprise search positions in MSIT

  • Enterprise search product group

Enterprise search positions in MSIT

Among the responsibilities of MSIT is the deployment and operation of SharePoint shared services for the enterprise. These shared services include the Managed Metadata service, the User Profile service, Business Connectivity Services, and the enterprise Search service.

MSIT manages all aspects of the enterprise search implementation at Microsoft, which include planning, deployment, operation, and support. MSIT administers the search service, deploys and maintains related hardware and software, provides the search experience on the enterprise Search Center site, and helps other site owners provide a search experience that meets their requirements and the requirements of their users.

The following positions in MSIT have responsibility for enterprise search:

  • Search service manager. This is a program manager position that has the following responsibilities:

    • Business ownership, which includes gathering business requirements from site owners and users, specifying solutions for the end-user experience, managing the budget, and acting as liaison to the search service operations team and the enterprise search product group.

    • Planning and deployment of the FAST Search Server 2010 for SharePoint solution.

    • Configuration of the search service to make sure that published content is easily discoverable, such as by configuring managed properties and global scopes for the search service and by managing Best Bets in the enterprise Search Center.

    • Management and support of search service operations.

    • Training and support for owners of sites that consume the enterprise search service. This includes working with site owners to help them create the search experience that they want for their users. It also includes helping site owners make their content more easily discoverable by using search features such as scopes, managed properties, and Best Bets.

    • Collecting and analyzing end-user feedback and updating the configuration of the search service to meet end-user needs.

    • Training for Helpdesk personnel who provide end-user support.

    • Planning and vision for the MSIT implementation of enterprise search and the user search experience beyond the current deployment.

  • Search service engineer. Search service engineers plan, design, test, deploy, operate, and support the search solution. They have primary responsibility for ensuring the continuity and responsiveness of the enterprise search service. This includes server infrastructure and topology (logical and physical components), operations monitoring, incident management, configuration changes, application of updates, and documentation of certain search issues.

  • Search analyst. Search analysts create, maintain, and review Best Bets and associated keywords for the enterprise Search Center every day based on user feedback and based on their analysis of search usage reports such as query logs and click-through logs. As the analysts identify new queries and new trends, they can create Best Bets for content that is not appearing high enough in search results or for content that is not in the content index. For more information, see Best Bets and URL promotion.

  • Software development engineer. This developer migrated customizations from the SharePoint Server 2007 search solution and customizes certain aspects of the end-user search experience on the enterprise Search Center.

  • UI designer. The designer creates standards for MSW and other sites and is responsible for the visual design of the search experience.

  • Support engineer. This person responds to technical or operational issues with the search service. For example, a site owner might report that new content is not yet showing in search results as expected, or that content that was removed from a site is still appearing in search results. In such cases, the support engineer can check whether a crawl has finished since the given content was added or removed. As another example, a site owner might report that search is not working on a particular site, and the support engineer might verify that search is not available and can notify a search service engineer of the problem. The support engineer also responds to end-user issues and feedback, such as questions about query syntax or content availability in search results.

The following table shows how MSIT allocated these resources during the planning and deployment phases of FAST Search Server 2010 for SharePoint, and how the resources are allocated during the operations and support phase. To help indicate the kinds of duties of each position, the table includes information about SharePoint permissions and local server permissions.

MSIT team members

Table 1. Search team member permissions and resource allocations by phase

Team member position Permissions for the enterprise search deployment Full-time planning and deployment resources per life-cycle phase Full-time operations and support resources per life-cycle phase

Search Service Manager

  • Farm administrator

  • Search service application administrator

1

1

Search Service Engineer

  • Farm administrator

  • Search service application administrator

  • Member of the Administrators group on each farm server

  • Microsoft SQL Server 2008

    • db_owner fixed database role

    • dbcreator fixed server role

1.5

2

Search Analyst

Site collection administrator for the enterprise Search Center

0

1.5

Software Development Engineer

Site collection administrator for the enterprise Search Center

.5

1

UI Designer

Site collection administrator for the enterprise Search Center

.5

1

Support Engineer

  • Farm administrator

  • Search service application administrator

0

1

Enterprise search product group

The enterprise search product group at Microsoft is responsible for program management, development, and testing of SharePoint Server 2010 and FAST Search Server 2010 for SharePoint features and functionality. MSIT collaborates with this group to exchange information and recommendations regarding the MSIT enterprise search solution.

MSIT deployed FAST Search Server 2010 for SharePoint to production at Microsoft in late 2009, before public release of the product in the second quarter of 2010. MSIT provided feedback to the enterprise search product group at that time about the MSIT experience with product deployment and operations. This included feedback that MSIT received from users. This helped the product group make sure that the software was thoroughly tested in an enterprise production environment before it was delivered to customers.

MSIT continues to provide feedback to the product group about its use of FAST Search Server 2010 for SharePoint. This feedback helps improve the product. The product group, in turn, provides occasional guidance to MSIT about configuration and use of certain features and functionality of the software.

Other articles in this white paper

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Background on the use of enterprise search at Microsoft

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Migration to FAST Search Server 2010 for SharePoint

Published: April 16, 2012

Summary: Learn how Microsoft IT upgraded from Office SharePoint Server 2007 enterprise search to FAST Search Server 2010 for SharePoint.

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the fifth of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper).

To migrate from the Office SharePoint Server 2007 search solution to the FAST Search Server 2010 for SharePoint solution, MSIT performed the following steps:

  1. Built a new FAST Search Server 2010 for SharePoint deployment dedicated to search. The new deployment is described in the next section, FAST Search Server 2010 for SharePoint solution: environment and topology.

  2. Migrated SSP-level settings from the Redmond SharePoint Server 2007 search solution to the new FAST Search Server 2010 for SharePoint deployment as follows:

    1. Reproduced content sources from the SharePoint Server 2007 search solution by using an internal script.

    2. Migrated managed properties by using SharePoint Search Property Creator, an internal tool.

    3. Migrated global scopes by using Windows PowerShell scripts.

  3. From the FAST Search Server 2010 for SharePoint deployment, completed a full crawl of all content sources. The full crawl took approximately 10 days and indexed about 90 million items.

  4. Selected a SharePoint Server 2007 consumer farm (a farm such as http://teamsites that was consuming a SharePoint Server 2007 search service) that had to be upgraded to Microsoft SharePoint Server 2010.

  5. For the SharePoint Server 2007 consumer farm that was selected in step 4, did the following:

    1. Ensured that all local search settings (such as Best Bets and local scopes) in the farm were exported to a file. (To do this, Search service application administrators can use the SharePoint Enterprise Search Migration Tool for SharePoint Server 2010.)

    2. Ensured that the farm was upgraded to SharePoint Server 2010. For information about how to perform this process, see Connect to a service application on a remote farm (SharePoint Server 2010).

  6. From the new SharePoint Server 2010 farm (the farm mentioned in step 5), set up a federated connection to the FAST Search Server 2010 for SharePoint deployment (so that the new SharePoint Server 2010 farm was consuming search from the FAST Search Server 2010 for SharePoint deployment.)

  7. Ensured that all local search settings that were exported to a file from the SharePoint Server 2007 consumer farm were imported into the new SharePoint Server 2010 consumer farm (by using the SharePoint Enterprise Search Migration Tool for SharePoint Server 2010).

  8. Repeated steps 4 through 7 as necessary to connect SharePoint Server 2010 farms to the FAST Search Server 2010 for SharePoint deployment.

At this point, all participating SharePoint sites were consuming search from the FAST Search Server 2010 for SharePoint deployment instead of from a SharePoint Server 2007 search service. The whole migration process took about 20 weeks—6 weeks for planning and 14 weeks to set up the new deployment, migrate and configure search settings, and deploy the search service.

MSW was one of the SharePoint Server 2007 consumer farms that was upgraded to SharePoint Server 2010 as part of this procedure. After the upgrade, MSIT created a new enterprise Search Center and customized the search experience for that new Search Center. For more information, see Enterprise search center (Enterprise search at Microsoft, white paper).

Other articles in this white paper

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


FAST Search Server 2010 for SharePoint solution: environment and topology

Published: April 16, 2012

Summary: Learn about the environment and topology for the Microsoft IT deployment of FAST Search Server 2010 for SharePoint.

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the sixth of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper). In this article:

  • Environment and topology overview

  • SharePoint Server 2010 server farm

  • FAST Search Server 2010 for SharePoint server farm

  • Database servers

Environment and topology overview

MSIT operates and maintains the Microsoft FAST Search Server 2010 for SharePoint deployment as a centralized search solution. This solution provides enterprise search capability for all of the Microsoft SharePoint Server 2010 sites that MSIT hosts and manages.

Enterprise search is a resource-intensive service, especially when it is deployed on a large scale. Therefore, to provide high capacity and performance, and to speed up system administration and maintenance, the FAST Search Server 2010 for SharePoint deployment is dedicated to search. The deployment does not host any content and does not host any SharePoint services other than the search service.

The FAST Search Server 2010 for SharePoint deployment has the following topology:

  • A SharePoint Server 2010 farm that contains four servers

  • A FAST Search Server 2010 for SharePoint farm that contains 30 servers

  • Two database clusters that contain a total of four servers

Figure 1 shows the FAST Search Server 2010 for SharePoint deployment, and farms and sites that consume the search service from the deployment.

Figure 1. MSIT enterprise search topology in the FAST Search Server 2010 for SharePoint solution

FAST Search Server 2010 for SharePoint solution

To download Figure 1 as a full-size Microsoft Visio drawing, go to the Microsoft Download Center (http://go.microsoft.com/fwlink/p/?LinkId=248254).

The following sections describe the topology of the FAST Search Server 2010 for SharePoint deployment in more detail.

SharePoint Server 2010 server farm

The SharePoint Server 2010 farm consists of the following four servers:

  • Two servers that host crawl components that crawl all content other than user profiles. Each server hosts two search service application (SSA) crawl components for content. The four crawl components can crawl content at the same time.

  • Two servers that host crawl components that crawl only user profiles. One server is primary. The additional server is a backup for high availability. Each server hosts one query SSA crawl component. Only one query SSA crawl component crawls user profiles at a time. In addition, these servers connect to the consumer farms to serve search queries.

FAST Search Server 2010 for SharePoint server farm

The associated FAST Search Server 2010 for SharePoint farm consists of the following 30 servers:

  • Six servers that perform one or more tasks such as the following: distribute crawled items to the index, extract properties, maintain managed properties, perform indexing, and match queries.

  • 24 servers in a configuration of two rows with 12 columns each. The primary task of the servers in the index row is to index content. The primary task of the servers in the search row is to query the index. To provide high availability, if one of the servers in a column becomes unavailable, the other server in the column can perform both the index and the query tasks. The content index is distributed across the 12 columns so that each column has a partition of approximately equal size. Each column or partition has a capacity of 15 million items, so that the total capacity of the content index is 200 million items. (The solution is designed to meet enterprise search requirements at Microsoft until the index grows to 200 million items in about 2013 or later.) For indexing efficiency, each index partition has four sub-partitions.

Database servers

There are two database clusters associated with the FAST Search Server 2010 for SharePoint deployment. Each cluster contains two database servers. The clusters store the following databases:

  • Cluster 1:

    • Content crawl database 1

    • Content property database

    • People property database

    • Content SSA database

  • Cluster 2:

    • Content crawl database 2

    • SharePoint configuration database

    • People crawl database

    • FAST administration database

    • Query SSA database

Other articles in this white paper

For more information, see the other articles in this series:

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Measurable gains from implementing FAST Search Server 2010 for SharePoint

Published: April 16, 2012

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the seventh of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper).

The new FAST Search Server 2010 for SharePoint search solution immediately provided measurable gains in many areas, such as those shown in tables 1, 2, and 3.

Table 1 shows how the scale of the Microsoft FAST Search Server 2010 for SharePoint solution is much greater than the scale of the search solution that the Microsoft Office SharePoint Server 2007 Redmond SSP provided.

Table 1. Scale comparison

Characteristic SharePoint Server 2007 solution (approximate, 2009) FAST Search Server 2010 for SharePoint solution (approximate, August 2011)

Amount of content crawled on sites hosted by MSIT

13 terabytes

34.5 terabytes

Number of sites and subsites crawled

350,000

683,050

Number of items in content index

43 million

90 million

Size of content index

300 gigabytes (GB)

6 terabytes

Database sizes

  • Redmond SSP search database: 340 GB

  • Redmond SSP profiles database: 70 GB

  • Crawl database 1: 427.3 GB

  • Crawl database 2: 237.1 GB

  • People crawl database: 17.3 GB

  • Search administration database: 2.15 GB

  • Content (FAST connector) property database: 0.19 GB

  • People (SharePoint connector) property database: 5.42 GB

Query volume per month

500,000

5.45 million

Query volume per second

.1929

2.033

Table 2 shows how much SharePoint content MSIT hosted for each region, and that the FAST Search Server 2010 for SharePoint solution crawled, as of August 2011.

Table 2. SharePoint content crawled by region

Region | Number of sites and subsites crawled | Corpus size (terabytes)
Americas (content hosted in Redmond data center) | 128,435 sites together with 333,831 subsites | 23.34
EMEA (content hosted in Dublin data center) | 35,932 sites together with 89,591 subsites | 6.6
Asia (content hosted in Singapore data center) | 33,360 sites together with 61,901 subsites | 4.53
Total | 197,727 sites together with 485,323 subsites | 34.47

In addition to the 34.47 terabytes of hosted SharePoint content, the search service crawls a large amount of other content, which includes the following:

  • Sites that are not hosted by MSIT

  • File shares

  • Structured data sources

Table 3 shows average incremental crawl times for some content sources with the SharePoint Server 2007 search solution compared to average incremental crawl times with the FAST Search Server 2010 for SharePoint solution.

Table 3. Crawl performance comparison

Content source | Average incremental crawl time with SharePoint Server 2007 solution (late 2009) | Average incremental crawl time with FAST Search Server 2010 for SharePoint solution
MSW | 21 hours | 40 minutes
SharePoint sites, excluding My Sites | 145 hours | 9 hours
My Sites | 18 hours | 2 hours
Redmond portals | 35 hours | 4 hours
User profiles | 7 hours | 10 minutes
Host headers | 19 hours | 1 hour
Team | Deleted in 2009 to reduce crawl load | 2 hours
Office | Deleted in 2009 to reduce crawl load | 4 hours

Other articles in this white paper

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Administration at the search service application level

Published: April 16, 2012

Summary: Learn how Microsoft IT configures crawls, content sources, and scopes, and monitors the FAST Search Server 2010 for SharePoint deployment.

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the eighth of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper). In this article:

  • Configuring crawls

  • Configuring other search settings

  • Monitoring the search system

Configuring crawls

In addition to setting up content sources for crawls, MSIT uses crawl rules and dedicated crawl targets.

Content sources

MSIT reviews content sources on a quarterly schedule to make sure that that crawls are appropriately configured. Also, MSIT is often asked to add or remove a site from crawls.

To reduce the complexity of scheduling and managing crawls, MSIT consolidates content sources and start addresses based on similarity and priority when possible. For example, MSIT decreased the number of content sources from 25 in the Microsoft Office SharePoint Server 2007 search solution to 14 in the FAST Search Server 2010 for SharePoint solution.

Table 1 shows the number of each kind of content source in the previous and the current enterprise search solutions. Some of the content sources contain multiple start addresses. For example, a content source that contains sites that MSIT does not host (SharePoint sites such as http://sqlserversites and http://devdivsites, and other websites) has more than 150 start addresses.

Table 1. Comparison of content sources

Type of content source | Number of content sources (SharePoint Server 2007 search solution) | Number of content sources (FAST Search Server 2010 for SharePoint solution)
SharePoint sites hosted by MSIT (including the user profile store | 13 | 11
Sites not hosted by MSIT (SharePoint sites and other websites)* | 6 | 1
File shares** | 1 | 1
Microsoft Exchange public folders | 1 | 0
Business Data Catalog | 3 | 1
Custom | 1 | 0

*These sites host content that the search service crawls, but the sites do not consume the search service.

**The search service currently crawls nine file shares.

Full crawls

MSIT performs a full crawl of all content sources only when it is necessary, as in the following situations:

  • After deployment of a crawl-related hotfix or a schema configuration change.

  • After addition of managed properties to the search system. In this case, it is only necessary to perform a full crawl of content sources to which the managed properties might apply.

MSIT tries to minimize the impact of full crawls on search performance by crawling content sources in order of priority, so that crawls are performed in a staggered manner.

As of August 2011, the search service was crawling about 90 million items on the Microsoft intranet worldwide. Table 2 shows the number of items that each major content source was crawling at that time.

Table 2. Items crawled during full crawl of each content source

Content source | Number of items crawled
Academy portal | 72,619
Asia content (includes team sites, My Sites, and portal sites) | 9,344,940
EMEA content (includes team sites, My Sites, and portal sites) | 15,798,592
File shares | 55,407
Infopedia portal | 69,967
MSLibrary portal | 91,202
MSW portal | 292,751
My Sites | 6,131,717
Sites not hosted by MSIT (SharePoint sites and other websites) | 2,715,466
Office portal | 8,717,300
Primus | 25,168
Redmond custom portals | 5,099,814
SharePoint | 29,495,749
Team | 6,367,784

Incremental crawls

MSIT determines the priority of a content source primarily according to the degree of dependence that consumer sites have on the content source for search, and according to how frequently the associated content is updated. MSIT schedules incremental crawls according to four levels of priority of content sources. The content sources that have priorities 1, 2, and 3 are for repositories that MSIT hosts and manages. Table 3 shows how the frequency of incremental crawls depends on the priority of content sources.

Table 3. Incremental crawl frequencies

Content source priority | Crawl frequency | Content sources (examples)
1 | Four times a day | MSW, Infopedia, user profiles
2 | Two times a day | Academy, Team, My Sites
3 | Once a day | SharePoint, MSLibrary
4 | Three times a week | Sites that MSIT does not manage

Crawl rules

The current search solution uses about 120 crawl rules. When MSIT builds a new deployment, it uses a custom Windows PowerShell script to create some of these crawl rules. Some of the crawl rules specify how to crawl complex URLs. Other crawl rules specify certain sites or site collections to exclude from crawling because they contain confidential information or information that should not be included in the content index. For example, although the FAST Search Server 2010 for SharePoint deployment can crawl content on the Internet, MSIT uses a crawl rule that excludes the path http://*.*. This rule prevents crawlers from going outside the firewall.

MSIT tries to limit the number of crawl rules that it uses, and it encourages site owners to set permissions on content locally to include or exclude content from crawls. For more information, see Plan site permissions (SharePoint Server 2010).

Dedicated crawl targets

MSIT uses at least one web server as a dedicated crawl target in each farm that it crawls. MSIT uses two web servers as dedicated crawl targets for each farm that hosts a large amount of content (relative to the amount of content in other farms) and for each farm that hosts content that must be crawled frequently due to requirements for freshness of search results.

For information about how to use dedicated crawl targets, see Manage crawl load (SharePoint Server 2010).

Crawler impact rules

MSIT used crawler impact rules at one point as a temporary solution to reduce the load on two sites that had performance issues. The issues were not related to crawler impact and were resolved. Currently, MSIT is not using any crawler impact rules.

For more information, see Manage crawler impact rules (FAST Search Server 2010 for SharePoint).

Configuring other search settings

MSIT configures other search settings such as global scopes and removal of items from the content index, and it manages other global operations such as backups.

Global scopes

Table 4 shows the global scopes that the FAST Search Server 2010 for SharePoint deployment provides. Site owners can use these scopes on their sites.

Table 4. Global scopes

This global scope | Provides results from this content
Intranet | All SharePoint sites (preconfigured scope)
MSLibrary | http://mslibrary
People | User profiles and My Sites (preconfigured scope)
EnterpriseMedia | All SharePoint content that has content type Video. (This scope is used by the Video tab in the enterprise Search Center.)
News | MSW
Webcasts | MSW
MSArchives | MSW
ImageOnly | MSW

Removal of items from the content index

In rare cases, MSIT must immediately remove items from the content index because of business or legal requirements. To do this, MSIT uses Windows PowerShell. MSIT then creates a crawl rule to exclude the content from future crawls.

For more information, see Remove URLs from search results (SharePoint Server 2010).

Time-out settings, IFilters, and file types

MSIT uses the default time-out settings for connection time and request-acknowledgement time. In addition, MSIT uses only the default IFilters and the default file-type inclusions and exclusions list.

For more information, see the following:

  • Configure search time-out settings (SharePoint Server 2010)

  • IFilter and file type reference (FAST Search Server 2010 for SharePoint)

Thesaurus settings and stop word files

MSIT uses the default configuration for thesaurus files and stop word files. During product development, MSIT provided input to the product team about the default configuration for these files, and helped validate the usefulness of the default settings before the software was released.

For more information, see the following:

  • Manage thesaurus files (SharePoint Server 2010)

  • Manage stop word files (SharePoint Server 2010)

Backups

MSIT performs weekly backups of the FAST Search Server 2010 for SharePoint configuration. This includes the administration database, which contains settings such as managed properties, Best Bets, and global scopes. For more information, see Configuration backup and restore (FAST Search Server 2010 for SharePoint).

As part of regular SQL Server backups, MSIT also backs up the SharePoint configuration database and the other SQL Server databases that are part of the FAST Search Server 2010 for SharePoint deployment.

Monitoring the search system

MSIT uses various tools to monitor the health and use of the search system.

Search health monitoring

To monitor the health of the search system, MSIT uses the following FAST Search Server 2010 for SharePoint tools:

  • Crawl logs. MSIT reviews the crawl logs every day to find errors and troubleshoot issues. For example, site owners or users sometimes report that content is missing from search results. By reviewing the crawl logs, MSIT can determine the time of the last successful crawl of each content source, and can determine whether crawled content was successfully added to the index, whether it was excluded because of a crawl rule, or whether indexing failed because of an error. In the case of a complex issue, MSIT sometimes works with the enterprise search product group, such as when an item is in the content index but does not appear as it should in search results.

    For more information, see Best practices for using crawl logs (FAST Search Server 2010 for SharePoint).

  • Search administration reports. MSIT reviews search administration reports for the following details about crawl and query performance:

    • Crawl rate per content source

    • Crawl rate per type of crawl transaction

    • Overall query latency

    • SharePoint back-end query latency

    • Crawl processing per activity in the crawl pipeline

    • Crawl processing per component in the crawl pipeline

    • Crawl queue

    • Query latency trend

      High query latency might indicate delays in server rendering, for example, or in query matching or other back-end processing.

      For more information, see the following articles:

      • Use search administration reports (FAST Search Server 2010 for SharePoint)

      • View search administration reports (FAST Search Server 2010 for SharePoint)

MSIT also uses the following tools to monitor the search system:

  • Microsoft System Center Operations Manager 2007 R2 with the FAST Search Server 2010 for SharePoint management pack. MSIT uses System Center Operations Manager to monitor the status of each server and service in the FAST Search Server 2010 for SharePoint deployment. For example, System Center Operations Manager obtains monitoring data and performance counters from the FAST Search Server 2010 for SharePoint Monitoring Service, which runs on each server in the farm. System Center Operations Manager helps identify issues and can provide alerts when performance bottlenecks occur. For more information, see Monitor FAST Search Server 2010 for SharePoint with SCOM.

  • Custom SQL Server tool for monitoring crawls. MSIT developed a custom tool that helps monitor crawl progress. The tool runs a job on each computer that hosts a crawl database. This helps monitor crawl latencies and can identify when a crawl hangs. The tool writes an entry with a time stamp to a SQL Server table for each item that is crawled. The SQL Server job sends a message if more than a 30-minute interval appears between successive time stamps.

  • Windows Server 2008 R2 performance counters. MSIT uses Windows Server 2008 R2 performance counters to monitor the following:

    • Crawl and query progress

    • Disk, memory, and application performance

Search usage monitoring

To understand the effectiveness of the enterprise search service and provide the best experience for users, MSIT regularly analyzes use of the service. MSIT uses the results of this analysis to make adjustments to improve search results.

To monitor search usage, MSIT uses the following FAST Search Server 2010 for SharePoint tools:

  • Click-through logs. These logs provide information about click-through rates on search results, which indicate how users browse through results.

  • Query logs. Query logs provide the following information about search queries that users submit:

    • Queries submitted during the previous 30 days

    • Queries submitted most frequently during the previous 30 days

    • Queries submitted most frequently from each site collection during the previous 30 days

    • Queries submitted per scope during the previous 30 days

    • Queries submitted during the previous 12 months

    • Queries that returned zero results

    • Destination pages reached most frequently from search results

      This information helps MSIT configure the search service to provide more useful search results, such as by adjusting Best Bets.

MSIT constantly strives to improve the search experience for employees, and it uses the following resources to understand the user experience:

  • Feedback Tool for search. The Feedback Tool is a Microsoft-internal tool that was developed with the help of the enterprise search product team. This tool provides a Feedback button on the enterprise Search Center results page that invites the search user to explain the goal of the most recent search, rate the search experience and usefulness of the results, report search problems, and suggest improvements. When the user clicks Send, the tool sends an email message to an alias that monitors the feedback. The message captures the user's query, the URL where the user submitted the query (which indicates the scope that was used), and the search results, with an indication of which results were Best Bets, if any. If information that a user was looking for did not appear in the search results, MSIT works with site owners to determine where the associated content is available so that links to the content can be returned in search results. If the information that a user is looking for is in the search results but the ranking is unsatisfactory, MSIT can adjust the Best Bets or use URL promotion or demotion accordingly. For more information, see Best Bets and URL promotion.

  • Annual user survey regarding the MSW enterprise portal. This survey includes several questions to help measure user satisfaction with the enterprise search service. Users can provide feedback on the effectiveness of the search service and on what kind of content is difficult to find.

MSIT also uses a third-party tool that monitors and tracks website usage. The tool collects and reports data on the number of visits to MSW and the queries that were performed there.

Other articles in this white paper

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Collaboration with site owners (Enterprise search at Microsoft, white paper)

Published: April 16, 2012

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the ninth of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper).

Upon request, MSIT collaborates with owners of sites that consume the search service to help them customize the search experience and make sure that the content on their sites is discoverable. For example, in response to requests from site owners, MSIT creates managed properties and helps site owners configure relevance.

Creating managed properties

The Microsoft FAST Search Server 2010 for SharePoint deployment includes more than 600 managed properties. For the deployment, MSIT started with the default managed properties and the managed properties that MSIT migrated from the Microsoft Office SharePoint Server 2007 search solution by using the internal SharePoint Search Property Creator tool, as described in Migration to FAST Search Server 2010 for SharePoint, earlier in this series of articles.

Addition of new managed properties is one of the most common change requests that MSIT receives. Therefore, MSIT has a governance policy to keep the process orderly. In accordance with the policy, MSIT tracks requests for new managed properties, requires requests to be in a certain format, automates the creation of managed properties to help guarantee that the properties have certain attributes, and creates the managed properties on a regular schedule.

To request a new managed property, a site owner fills out a template that MSIT provides. When site owners use the template, it encourages them to plan thoroughly before submitting a request. This helps MSIT avoid having managed properties that use different names that serve the same purpose, for example. The request template is a comma-separated value (CSV) file that includes the following attributes of the proposed new managed property: Name, Description, Type (text, decimal, or date/time), StemmingEnabled, SummaryType, MergeCrawledProperties, SortableType, Queryable (whether the property can be used in queries), Refinement (whether the property can be used as a refiner), Crawled Properties, and IsExisting. If MSIT approves the request, it uses a Windows PowerShell script that it created that takes the CSV file as input, creates the managed property with the appropriate attribute values, and maps the managed property to one or more crawled properties.

In general, MSIT adds new managed properties only one time per month, because adding a new managed property requires a full crawl of all content sources to which the managed property might apply.

Configuring relevance

For most intranet sites that it hosts, such as the enterprise Search Center, MSIT uses the default rank profile and the default content index. The following are examples of the way MSIT works with site owners to adjust search relevance by providing custom rank profiles and custom full-text indexes.

Rank profiles

In response to a request from the Infopedia site owner to rank results differently than the default rank profile, MSIT provided a custom rank profile for the site. The new profile caused topic pages to be displayed first, and then blogs, documents, and document sets. The topic pages (usually .aspx pages) are articles with named owners to which other users can contribute notes or attach other documents.

To implement a new rank profile, MSIT created a script that copied the default rank profile into a new rank profile, added new ranking attributes and settings, and updated weights of various criteria. MSIT then ran the script in a pre-production environment that was crawling production content. The site owner created test cases to determine how the new rank profile affected the search results. MSIT and the site owner compared the search results visually, because there were no tools to automate the comparison. The site owner provided feedback to MSIT regarding the results, and MSIT updated the script based on the feedback. MSIT then ran the script again. MSIT and the site owner repeated this procedure until they achieved the desired search results. At that point, MSIT ran the script in the production deployment. Although the process was iterative and time-consuming, in the end it yielded better search results for users.

Rank profiles are global to a search service application. Therefore, any site in the web application can use any custom rank profile. After MSIT created the custom rank profile, the Infopedia site owner added it to the sort drop-down list for the site's search results page. The name of the custom rank profile then appeared among the managed properties, and the site owner was able to set it as the default sort option for that search results page.

For more information, see the following articles:

  • Manage rank profiles (FAST Search Server 2010 for SharePoint)

  • Dynamic rank (FAST Search Server 2010 for SharePoint)

Custom full-text indexes

Another requirement from certain site owners was to provide keyword-based search, that is, to provide a search experience based on metadata associated with items in the content index. MSIT satisfied this requirement by using a Windows PowerShell script that it created to provide custom full-text indexes.

Each custom full-text index takes advantage of content classification through the Managed Metadata service to provide more relevant search results by targeting keyword searches to a limited set of properties. For example, for a full-text search (the default) on the term "FAST," a search returned several thousand results. However, for the same term, a tag-based search (which uses a custom full-text index) returned only slightly more than 100 results—only items that were tagged with the term "FAST." In either case, the user was able to narrow the results by using custom refiners (such as Industries, Partners, and Products) that MSIT created.

Other articles in this white paper

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Enterprise search center (Enterprise search at Microsoft, white paper)

Published: April 16, 2012

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the tenth of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper). In this article:

  • Introduction to the enterprise Search Center

  • Getting to the enterprise Search Center

  • Enterprise Search Center tabs

  • Best Bets and URL promotion

Introduction to the enterprise Search Center

After the migration to Microsoft FAST Search Server 2010 for SharePoint, MSIT created a new enterprise Search Center that used the FAST Search Center template. The enterprise Search Center in FAST Search Server 2010 for SharePoint includes many default features that were not available by default in Microsoft Office SharePoint Server 2007, such as search refiners, document previews for Microsoft PowerPoint 2010 presentations, and a Web Part for related searches.

For the new Search Center, MSIT provided customizations to match the visual design of the MSW enterprise portal, and provided customizations for the search tabs and search results page. MSIT uses the default full-text index and the default rank profile for search results on the enterprise Search Center, but adjusts relevance by using Best Bets and URL promotion.

MSIT also upgraded custom search features from the SharePoint Server 2007 search solution, such as the Glossary Web Part, so that the features worked with FAST Search Server 2010 for SharePoint. To reduce page rendering and refresh times, MSIT combined multiple cascading style sheets (CSS) files into one CSS file, and configured the Glossary Web Part to render and refresh asynchronously with respect to the rest of the search results page.

Getting to the enterprise Search Center

Employees often start their searches for information on the MSW enterprise portal, which is one of the most heavily used intranet sites at Microsoft. The portal is shown in Figure 1.

Figure 1. MSW enterprise portal

MSW enterprise portal

When a user types a query in the search box on MSW, the search service displays the results on the enterprise Search Center results page. Employees can also get to the enterprise Search Center by typing a search query in a search box on certain other intranet sites. Employees can even get to the enterprise Search Center from Windows Explorer on a computer that is running Windows 7 or Windows Server 2008. For more information, see Search the enterprise from Windows 7.

Enterprise Search Center tabs

The FAST Search Center template includes two search tabs: All Sites and People. MSIT changed this configuration so that the enterprise Search Center contains three tabs: Intranet, People, and Video.

Intranet tab

MSIT renamed the default All Sites tab to Intranet to help clarify the purpose of the tab. The Intranet tab uses the default All SharePoint Sites search scope. When employees conduct a general search, they often use this tab because it returns the most comprehensive search results. However, the Intranet tab does not include people search results or results from structured data sources.

The Intranet tab includes the following preconfigured refiners when applicable: result type, site, author, modified date, and company. (In FAST Search Server 2010 for SharePoint, company is a preconfigured managed property for property extraction.) Web Parts in the right zone include Related Searches, Glossary, People Matches, and MSDN/TechNet Results. Related Searches, Glossary, People Matches, and MSDN/TechNet Results are default Web Parts. The MSDN/TechNet Results Web Part displays links to federated results from MSDN or TechNet and a View More Results link that starts the Bing search engine.

The Glossary custom Web Part draws on a corporate glossary that MSIT created by using the Managed Metadata service. When a user types a term in the search box on the Intranet tab, the Glossary Web Part accesses the corporate glossary by sending a query to the Managed Metadata service via the Managed Metadata web service. The Glossary Web Part then displays a list of terms and definitions that the web service returns. For example, if a user searches on DOS, the Web Part displays definitions for "denial of service attacks" and "disk operating system."

Figure 2 shows a portion of the enterprise Search Center results page after a user searches on windows phone 7 on the Intranet tab. The first search result is a Visual Best Bet graphic that MSIT configured as a hyperlink.

Figure 2. Search results page for the Intranet tab

Search results page for the Intranet tab

People tab

People is a default tab. A search on the People tab provides search results based on information that is in employee user profiles and My Sites. (MSIT imported user profiles from Active Directory Domain Services, and SharePoint Server uses the information for the creation of My Sites.) The People tab includes the following refiners where applicable: job title, organization, ask me about, interests, skills, past projects, and schools. The job title and organization refiners use managed properties that MSIT created based on fields in the user profiles.

Video tab

MSIT added the Video tab in response to popular demand. For the Video tab, MSIT created a "video" content type. The Video tab uses a global scope that displays only results that have that content type. (A search from the Video tab does not extract data from audio or video.)

The Video tab includes custom refiners in the left zone based on metadata for Channel, Published Date, and Tags. Channel refiners enable users to sort results by subject matter, and Tags refiners enable users to sort results according to metadata tags that site owners have applied to the items. Other metadata such as title, description, and length are displayed in the search results under a thumbnail of a frame of the associated video. To customize the search experience for the Video tab so that it differs from the Intranet tab and the People tab, MSIT used XSLT and CSS.

The Video tab search results page can display results in a grid view or a list view. Figure 3 shows a list view of search results on the Video tab.

Figure 3. Search results page for the Video tab in list view

Search results page for the Video tab in list view

Best Bets and URL promotion

For Best Bets for the enterprise Search Center, MSIT selects keywords and synonyms from the Microsoft corporate vocabulary that MSIT created by using the Managed Metadata service. Addition of new Best Bets is another one of the most common change requests that MSIT receives from site owners and users. In addition, MSIT search analysts regularly review user feedback, query logs, and click-through logs to make sure that that Best Bets are available, up-to-date, and relevant. Based on their reviews, the search analysts can create new Best Bets as appropriate.

In the MSIT SharePoint Server 2007 search solution, some keywords had as many as 10 Best Bets. However, in general, the fewer the Best Bets, the more authoritative and effective they are. Therefore, for the enterprise Search Center in the FAST Search Server 2010 for SharePoint solution, MSIT tries to maintain fewer than five Best Bets per keyword and uses the URL promotion feature to boost the rank of any additional results that are highly relevant. With the URL promotion feature, the appearance of a particular keyword in a search query causes a specified site or document to rank higher in search results, regardless of other factors.

MSIT provides Visual Best Bets for the most popular search results of the most common queries, and for search results about some Microsoft events that have worldwide participation. The Visual Best Bets are graphical hyperlinks. For more information, see Add, remove, and display Visual Best Bets for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

For more information, see the following articles:

  • Tune keyword rank (FAST Search Server 2010 for SharePoint)

  • Promoting or Demoting Sites

  • Manage FAST Search keywords

Other articles in this white paper

For more information, see the other articles in this series:

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

Published: April 16, 2012

Summary: Learn what best practices Microsoft IT used when it upgraded the Microsoft enterprise search solution to FAST Search Server 2010 for SharePoint.

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the eleventh of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper).

In comparison to a Microsoft Office SharePoint Server 2007 search solution, MSIT found that it takes additional hardware, skills, and people to plan, deploy, operate, and support a large deployment such as the MSIT search solution that uses Microsoft FAST Search Server 2010 for SharePoint. Accordingly, MSIT followed these best practices to avoid issues and keep the environment available and performing well:

  • Plan and design the search architecture carefully. MSIT considered many factors when planning its FAST Search Server 2010 for SharePoint solution. This included the location and amount of content to be crawled, the number of users, the anticipated query load, user requirements for results freshness, requirements for search availability, and the expected growth in the amount of content and number of user queries in connection with the anticipated longevity of the farm. Understanding these factors helped MSIT determine the number of physical and logical components that the new farm required. Certain aspects of a FAST Search Server 2010 for SharePoint farm cannot be easily changed after the initial deployment, so it was important to plan carefully to avoid having to deploy the farm again if the initial design was not viable or optimal.

    For more information, see the following articles:

    • Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)

    • Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)

  • Plan content types and managed properties. MSIT had to rapidly deploy FAST Search Server 2010 for SharePoint to production at Microsoft due to a requirement to test the software before it was released to the public. Therefore, MSIT did not have time to plan all content types and managed properties before it deployed the search service, and it had to create some content types and managed properties after the deployment was in operation. In some cases, site owners then had to adjust search refiners or rank profiles to accommodate changes in content types and managed properties. In addition, end users had to make adjustments in the way they used search. For example, when a managed property changed, a user might not be able to use it in searches anymore, or might get different results by using it. This experience confirmed to MSIT that it is best to plan and create content types and managed properties before an organization deploys a search solution. This enables site owners to create and configure more permanent search refiners and rank profiles at the beginning of a deployment, and it results in a more consistent search experience for users.

  • Automate search configuration changes. MSIT automates search configuration changes whenever possible by using Windows PowerShell. Automation of changes, such as addition of managed properties to the system, saves time and provides uniformity in the change process. This helps MSIT ensure that it makes changes in similar ways each time.

  • Provide governance policies for search configuration changes. MSIT receives many requests for changes to the search configuration, such as requests for new managed properties. Without governance policies, change processes in such a large deployment could quickly become unmanageable. Therefore, MSIT enforces governance policies for certain configuration changes, which provides order and helps in change tracking.

  • Integrate with other SharePoint services. To maximize the benefits of using FAST Search Server 2010 for SharePoint, MSIT invested resources in configuring and using other SharePoint services on which search has dependencies. For example, the search service has the following dependencies:

    • User Profile service and My Sites for results for people searches

    • Managed Metadata service for managed properties that can be used for refiners, ranked results, custom full-text indexes, and property queries

    • Microsoft Business Connectivity Services for the operation of connectors to crawl repositories of structured data

    Investing in these other areas helped MSIT create richer search experiences and improve information discovery.

    For information about deploying and operating these other services, see the following articles:

    • User Profile service application overview (SharePoint Server 2010)

    • Managed Metadata and Taxonomy in SharePoint Server 2010

    • Business Connectivity Services in SharePoint Server 2010

  • Gather user feedback. User feedback is invaluable to MSIT for understanding user problems and improving the enterprise search system. For more information, see Monitoring the search system, earlier in this series of articles.

Other articles in this white paper

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Conclusion (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Conclusion (Enterprise search at Microsoft, white paper)

Published: April 16, 2012

Applies to: Microsoft FAST Search Server 2010 for SharePoint, Microsoft Office SharePoint Server 2007

This is the last of 12 articles that compose How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper).

When the enterprise search solution that used Microsoft Office SharePoint Server 2007 was no longer practical, MSIT reviewed business, technical, and user requirements for an improved solution and chose to migrate to Microsoft FAST Search Server 2010 for SharePoint. The new deployment met all of the requirements and continues to provide the following benefits:

  • Centralized management. The search solution is hosted in a single, central deployment.

  • Content index completeness. All users can get search results from a single content index.

  • Huge index capacity. The content index holds all content items from the crawls of all repositories for which MSIT wants to provide search coverage. The content index can accommodate projected growth to 200 million items.

  • Faster crawls for fresher search results. An incremental crawl of each high-priority content source takes much less than 24 hours.

  • Faster query responses. For most queries, users get search results in less than two seconds.

  • More customizable search experience for more relevant search results. Search results are ranked according to relevance so that users can quickly find information that they need.

  • High availability. FAST Search Server 2010 for SharePoint redundancy and failover configuration provides continuous search availability.

  • Uniform search experience. Through the enterprise Search Center, all users worldwide have access to the same search experience.

  • Significantly increased user satisfaction. As measured by the annual survey on MSW, user satisfaction with the performance and effectiveness of the search system and with the overall search experience increased 20 percent in 2010 and 26 percent in 2011.

The FAST Search Server 2010 for SharePoint deployment provides a robust enterprise search solution that empowers employees to find up-to-date, relevant information that is important for performing their day-to-day job functions. The solution helps connect employees with the work of colleagues all over the world, which enables them to collaborate more easily and work more productively than ever before.

Other articles in this white paper

For more information, see the other articles in this series:

  • How Microsoft IT deployed FAST Search Server 2010 for SharePoint (white paper)

  • Executive summary (Enterprise search at Microsoft, white paper)

  • Background on the use of enterprise search at Microsoft

  • Microsoft enterprise search teams

  • Migration to FAST Search Server 2010 for SharePoint

  • FAST Search Server 2010 for SharePoint solution: environment and topology

  • Measurable gains from implementing FAST Search Server 2010 for SharePoint

  • Administration at the search service application level

  • Collaboration with site owners (Enterprise search at Microsoft, white paper)

  • Enterprise search center (Enterprise search at Microsoft, white paper)

  • Lessons learned and MSIT best practices (Enterprise search at Microsoft, white paper)

To view the white paper as a single article on TechNet, or to download it, see Improving enterprise search at Microsoft: How FAST Search Server 2010 for SharePoint Powers Worldwide Intranet Search at Microsoft (http://technet.microsoft.com/en-us/library/bb735129.aspx).


Planning and architecture (FAST Search Server 2010 for SharePoint)

This section helps you develop conceptual, logical, and physical designs for configuring Microsoft FAST Search Server 2010 for SharePoint features, servers, and topologies. This section also provides recommendations for system designs based on customer scenarios and includes information to help you design a highly reliable, consistently available, and scalable system.

In this section:

  • Plan the search solution (FAST Search Server 2010 for SharePoint)

  • Plan search topology (FAST Search Server 2010 for SharePoint)


Plan the search solution (FAST Search Server 2010 for SharePoint)

Planning search is an important part of planning any deployment of Microsoft FAST Search Server 2010 for SharePoint. If you devote time to plan how search is implemented, you can save time in the future and can reinforce the effectiveness of other business processes.

It is important to understand that search needs may change and grow over time. Planning for change and growth is essential and should include considerations for additional sites and content and additional search hardware, such as server and client computers.

In this section:

  • Plan for crawling and federation (FAST Search Server 2010 for SharePoint)

  • Plan the relevance of search results (FAST Search Server 2010 for SharePoint)

  • Security considerations for indexing (FAST Search Server 2010 for SharePoint)

  • Plan to monitor the health of the search system (FAST Search Server 2010 for SharePoint)


Plan for crawling and federation (FAST Search Server 2010 for SharePoint)
See Also

This article discusses how to plan for crawling and federation by helping you understand how Microsoft FAST Search Server 2010 for SharePoint federates, crawls and indexes content.

Before end-users can use the enterprise search functionality in FAST Search Server 2010 for SharePoint, you must crawl or federate the content that you want to make available for users to search.

This article separates between three types of indexing connectors:

  1. The Microsoft SharePoint Server 2010 indexing connectors and crawling framework

    Most content sources can be crawled using this framework, either by using the integrated indexing connectors or Business Connectivity Services. You use the SharePoint Server 2010 Central Administration for most configuration and operation tasks.

  2. Federated search connectors

    Federated search connectors enable you to pass a query to a target system and display results returned from that system without actually crawling that content. You use the SharePoint Server 2010 Central Administration for most configuration and operation tasks.

  3. The FAST Search Server 2010 for SharePoint specific indexing connectors

    FAST Search Server 2010 for SharePoint offers three additional indexing connectors to crawl web, database and Lotus Notes content. These indexing connectors are configured mainly by editing XML files and Windows PowerShell cmdlets and you operate them by using the command line.

In this article:

  • Identify content sources and determine which indexing connector to use

    This section will help you to determine which indexing connector to use.

  • Plan to use indexing connectors integrated in SharePoint Server 2010

    This section helps you plan to use the SharePoint Server 2010 indexing connector framework and the FAST Search Content Search Service Application.

    • Plan content sources

    • Plan content sources for business data

    • Plan indexing connector protocols

    • Plan file-type inclusions and IFilters

    • Plan crawl schedules, crawl rules and manage the impact of crawling

    • Plan for authentication

  • Plan for federation

    This section helps you plan how to include federated search results for end-user queries using the SharePoint Server 2010 framework and the FAST Search Query Search Service Application.

  • Plan to use the FAST Search Server 2010 for SharePoint indexing connectors

    This section helps you plan to use the FAST Search Server 2010 for SharePoint specific connectors.

    • About the FAST Search Web crawler

    • About the FAST Search database connector

    • About the FAST Search Lotus Notes connector

    • Include or exclude content from a crawl

    • Set crawl schedules

Identify content sources and determine which indexing connector to use

FAST Search Server 2010 for SharePoint uses different indexing connectors for different content sources. The choice of indexing connector is influenced by the kind of content that you want to crawl, by personal preference and by specific needs of your organization.

Most content sources can be crawled using the various indexing connectors offered through Microsoft SharePoint Server 2010. In the Central Administration user interface, the collection of these indexing connectors is referred to as the FAST Search connector. This is not one separate indexing connector, but rather a collection of several indexing connectors. The FAST Search connector is associated with one or more content sources (and therefore indexing connectors) through the FAST Search Content Search Service Application (Content SSA). The Content SSA also connects the Microsoft SharePoint Server 2010 front-end with the FAST Search Server 2010 for SharePoint back-end.

When you install FAST Search Server 2010 for SharePoint, you also have access to three FAST Search Server 2010 for SharePoint specific indexing connectors. These connectors can feed web, database and Lotus Notes content to the index. The table summarizes the available indexing connectors and their recommended use cases.

Type of content Indexing connector Recommended use case

SharePoint

SharePoint indexing connector

Use in all use cases.

File shares

File share indexing connector

Use in all use cases.

Exchange public folders

Exchange indexing connector

Use in all use cases.

People profiles

User profiles indexing connector

Use in all use cases.

This kind of content is crawled through the FAST Search Query Search Service Application.

Web sites

Web site indexing connector

Use when you have a limited amount of Web sites to crawl, without dynamic content.

FAST Search Web crawler

Use when you have many web sites to crawl.

Use when the web site content contains dynamic data, including JavaScript.

Use when the organization needs access to advanced web crawling, configuration and scheduling options.

Use when you want to crawl RSS web content.

Use when the web site content uses advanced logon options.

Database

Business Data Catalog-based indexing connectors

Use if the preferred configuration method is using the Microsoft SharePoint Designer 2010.

Use when you want to use time stamp based change detection for incremental database crawls.

Use when the preferred operation method is using the Microsoft SharePoint Server 2010 Central Administration.

Use when you want to enable crawling based on the change log. This can be achieved by directly modifying the connector model file and creating a stored procedure in the database.

FAST Search database connector

Use when the preferred configuration method is using SQL queries.

Use when you want advanced data joining operation options through SQL queries.

Use when you want to use advanced incremental update features. FAST Search database connector uses checksum based change detection for incremental crawls if there is no update information available. The connector also supports time stamp based change detection and change detection based on update and delete flags.

Lotus Notes

Lotus Notes indexing connector

Use when the preferred operation method is using the Microsoft SharePoint Server 2010 Central Administration.

FAST Search Lotus Notes connector

Use when full Lotus Notes security support is required, including support for Lotus Notes roles.

Use when you want to crawl Lotus Notes databases as attachments.

Line of Business Data

Business Data Catalog-based indexing connectors

Use when the data in your content source contains data in line of business applications.

Use when you want to enable crawling based on the change log. This can be achieved by directly modifying the connector model file and creating a stored procedure in the database.

About crawling and indexing content

The result of successfully crawling content is that the individual files or pieces of content that you want to make available to search queries are accessed and read by the indexing connector. By crawling the content, you create a set of crawled properties for those items. These crawled properties are mapped to managed properties that are stored in the search index, also known as the index.

Note:

The indexing connectors do not change the files on the host servers. The files on the host servers are only accessed and read, they are not modified. In some cases, the last accessed date on files that have been crawled may be updated, as the indexing connectors read the content on the host server. This applies only to some servers that host certain content sources.

Plan to use indexing connectors integrated in SharePoint Server 2010

Most content sources can be crawled using the indexing connectors integrated in SharePoint Server 2010. You use the SharePoint Server 2010 Central Administration for most configuration and operation tasks.

These indexing connectors are set up by configuring the FAST Search Content Search Service Application (Content SSA). Among other things, the Content SSA enables communication with the FAST Search Server 2010 for SharePoint back-end. Within the Content SSA you specify the location of the content source(s), the crawl schedule and other information. The Content SSA feeds to the (default) content collection that is named sp.

The FAST Search connector crawls:

  • SharePoint sites

  • Web sites

  • File shares that contain content such as Microsoft Office documents

  • Exchange public folders

  • Line of business data, for example content from databases

  • Custom repositories, accessed with a custom built connector

Plan content sources

A content source in the FAST Search Content Search Service Application (Content SSA) is a set of options that you can use to specify what kind of content is crawled, what URLs to crawl, and how deep and when to crawl. The default content source is Local SharePoint sites. You can use this content source to specify how to crawl all content in all web applications that are associated with a particular Content SSA. By default, for each web application that uses a particular Content SSA, FAST Search Server 2010 for SharePoint adds the start address of the top-level site of each site collection to the default content source.

Some organizations can use the default content source to satisfy their search requirements. However, many organizations have to have additional content sources. Plan additional content sources when you have to do the following:

  • Crawl different kinds of content — for example, SharePoint sites, file shares, and business data.

  • Crawl some content on different schedules than other content.

  • Limit or increase the quantity of content that is crawled.

  • Set different priorities for crawling different sites.

You can create up to 500 content sources in the Content SSA, and each content source can contain as many as 500 start addresses. To keep administration as simple as possible, we recommend that you limit the number of content sources that you create.

Plan to crawl different kinds of content

You can only crawl one kind of content per content source. That is, you can create a content source that contains start addresses for SharePoint sites and another content source that contains start addresses for file shares. However, you cannot create a single content source that contains start addresses to both SharePoint sites and file shares. The following table lists the kinds of content sources that you can configure.

Use this kind of content source For this content

SharePoint sites

SharePoint sites from the same farm or different Microsoft SharePoint Server 2010, Microsoft SharePoint Foundation 2010, or Microsoft Search Server 2010 farms

SharePoint sites from the same farm or different Microsoft Office SharePoint Server 2007, Windows SharePoint Services 3.0, or Microsoft Search Server 2008 farms

SharePoint sites from Microsoft Office SharePoint Portal Server 2003 or Windows SharePoint Services 2.0 farms

Note:

Unlike crawling SharePoint sites on SharePoint Server 2010, SharePoint Foundation 2010, or Search Server 2010, the SharePoint Server 2010 crawler cannot automatically crawl all subsites in a site collection from previous versions of SharePoint Products and Technologies. Therefore, when crawling SharePoint sites from previous versions, you must specify the start address of each top-level site and the URL of each subsite that you want to crawl.

Web sites

Other web content in your organization that is not located in SharePoint sites

Content on Web sites on the Internet

File shares

Content on file shares in your organization

Exchange public folders

Microsoft Exchange Server content

Lotus Notes

E-mail messages stored in Lotus Notes databases

Note:

Unlike all other kinds of content sources, the Lotus Notes content source option does not appear in the user interface until you have installed and configured the appropriate prerequisite software. For more information, see Configure and use the Lotus Notes connector (FAST Search Server 2010 for SharePoint).

Business data

Business data that is stored in line-of-business applications

Plan content sources for business data

Business data content sources require that the applications hosting that data are specified in an Application Model in a Business Data Connectivity service application. You can create one content source to crawl all applications that are registered in the Business Data Connectivity service, or you can create separate content sources to crawl individual applications.

Often, the people who plan for integration of business data into site collections are not the same people involved in the overall content planning process. Therefore, include business application administrators in content planning teams so that they can advise you how to integrate the business application data into content and effectively present it in the site collections.

About Business Connectivity Services models

To crawl certain repositories, for example databases or Web services, you need the SharePoint Server 2010 search connector framework. This framework enables you to use a Business Connectivity Service (BCS) model to crawl external data sources. These model files define the connection details and structure of the external content source that you plan to crawl. The BCS models are imported into the Business Connectivity Service. You will point to this model and service when you set up the Line of Business Data or Custom Repository type content source.

There are several predesigned BCS models you can use for database content and Web services (WCF). In addition, you can also create your own, custom BCS model and custom connector using the connector framework and BCS models.

To build on the SharePoint Server 2010 search connector framework, you must use either SharePoint Designer or Microsoft Visual Studio 2010, depending on your specific requirements and goals.

Use SharePoint Designer to:

  • Create BCS models that are needed to crawl out of the box supported external content sources such as databases and Web services.

  • Import/export models between BCS applications

Use Microsoft Visual Studio to:

  • Implement methods for .NET BCS Connector

  • Write a custom connector for your repository

Multiple content sources can all pull from the same Business Connectivity Service (BCS), and you can point different Search Service Applications to the same model in a shared BCS.

For more information about the SharePoint Server 2010 connector framework, Business Connectivity Services and creating custom connectors, refer to SharePoint Server Search Connector Framework (MSDN).

Plan indexing connector protocols

All content that is crawled requires you to use a connector to gain access to that content. FAST Search Server 2010 for SharePoint (through the SharePoint Server 2010 connector framework) provides connectors for all common Internet protocols. However, if you want to crawl content that requires a connector that is not installed with SharePoint Server 2010, you must install the third-party or custom connector before you can crawl that content. For a list of indexing connector protocols that are installed by default, see Default indexing connector protocols (FAST Search Server 2010 for SharePoint). You can install additional indexing connectors and protocols to crawl content created by other Microsoft products or third-party software. For more information, see Content sources that require additional configuration (FAST Search Server 2010 for SharePoint).

Plan file-type inclusions and IFilters

FAST Search Server 2010 for SharePoint crawls and extracts metadata and content from most common file types. Several file types and IFilters are included automatically during initial installation. When you plan for content sources in your initial deployment, determine whether content that you want to crawl uses file types that are not included. If file types are not included, you must add those file types either by enabling the Advanced Filter Pack or by installing and registering a third-party IFilter to support that file type.

If you want to exclude certain file types from being crawled, you can add the file name extension for that file type to the file type exclusions list. Doing so excludes file names that have that extension from being crawled. For a list of file types and IFilters that are supported or excluded by default, see IFilter and file type reference (FAST Search Server 2010 for SharePoint).

Plan crawl schedules, crawl rules and manage the impact of crawling

Several factors determine whether and how many crawl schedules and crawl rules to plan for and the extent to which you have to manage the impact of crawling when you use the Content SSA and the SharePoint crawler to crawl content.

Note:

Before you can start incremental crawls of one or more content sources, the system must first complete a full crawl.

Example reasons that require you to configure crawl schedules, crawl rules and/or manage the impact of crawling are:

  • To accommodate down times and periods of peak usage.

  • To more frequently crawl content that is frequently updated.

  • To crawl content that is located on slower servers separately from content that is located on other servers.

  • To exclude content from crawls that is likely to be less relevant.

  • To decrease or increase the request frequency for particular (external) Web sites or content servers.

  • To crawl content with a different account than the default content access account.

For more detailed information and additional considerations, refer to the relevant sections in the SharePoint Server 2010 topic Plan for crawling and federation (SharePoint Server 2010). For more information about crawling content on different schedules, see the sections Considerations for planning crawl schedules and Reasons to do a full crawl. Read the section Plan to manage the impact of crawling if you want to learn more about crawler impact rules.

Plan for authentication

When the SharePoint Server 2010 crawler accesses the start addresses that are listed in content sources, the SharePoint Server 2010 crawler must be authenticated by, and granted access to, the servers that host that content. This means that the domain account that is used by the SharePoint Server 2010 crawler must have at least read permissions on the content.

By default, the system uses the default content access account. Or, you can use crawl rules in the FAST Search Content SSA to specify a different content access account to use when crawling particular content. Whether you use the default content access account or a different content access account specified by a crawl rule, the content access account that you use must have read permissions on all content that is crawled. If the content access account does not have read permissions, the content is not crawled, is not indexed, and therefore not available to queries.

For more information, see the section Plan for authentication in the in the SharePoint Server 2010 topic Plan for crawling and federation.

Plan for federation

Federated search is the concurrent querying of multiple web resources or databases to generate a single search results page for end-users. In FAST Search Server 2010 for SharePoint, you configure federated locations in the FAST Search Query Search Service Application so that end-users can search for and retrieve content that has not been crawled by servers in the local system. Federated locations enable queries to be sent to remote search engines and feeds. Accordingly, the system renders the results to end-users as if the federated content were part of the crawled content.

FAST Search Server 2010 for SharePoint, through SharePoint Server 2010, supports the following kinds of federated locations:

  • Search index on this server. You can use a local index in your organization that has a server that is running SharePoint Server 2010 as a federated location. For example, imagine that a SharePoint site on a Human Resources server in your company is the only available source of employee contact information. Even if the site is not part of your crawl scope, you can configure a federated location for it so that users who start a search from your Search Center site can retrieve employee contact information results that they are authorized to see. The following conditions apply:

    1. The location is set to Search index on this server.

    2. No query template is required. SharePoint Server 2010 uses the object model to query a location.

    3. Default server authentication is used.

    4. Advanced search queries are not supported.

  • FAST Search index.

    Use this option if you want to federate results from a local FAST Search Server 2010 for SharePoint index into a Search Center or a FAST Search Center.

  • OpenSearch 1.0 or 1.1. You can use any public Web site that supports the OpenSearch standard as a federated location. An example of such a location is an Internet search engine such as Bing, or a search results page that supports RSS or Atom protocols. For example, imagine that you want users who search your internal sites for proprietary technical research to also see related research information from public Web sites. By configuring a federated location for a Bing search query, web search results will be automatically included for users. The following conditions apply:

    1. Queries can be sent to a search engine as a URL, such as http://www.example.com/search.aspx?q=TEST.

    2. Search results are returned in RSS, Atom, or another structured XML format.

    3. Location capabilities, query templates, and response elements are part of an OpenSearch description (.osdx) file that is associated with the location.

    4. Extensions to OpenSearch that are specific to FAST Search Server 2010 for SharePoint enable you to include triggers and the ability to associate XSL code with search results.

    5. The choice of metadata to display in the search results is determined by the OpenSearch location.

    For more information about OpenSearch, visit http://www.opensearch.org.

For more information about federation and about planning authentication types for federation, consult the section Plan for federation in the SharePoint Server 2010 topic Plan for crawling and federation.

An overview of federated search connectors that you can use to import a federated location can be found on the Federated Search Connector Gallery on the Enterprise Search Tech Center.

Plan to use the FAST Search Server 2010 for SharePoint indexing connectors

In addition to the indexing connectors integrated in Microsoft SharePoint Server 2010, FAST Search Server 2010 for SharePoint offers additional content indexing connectors for web, Lotus Notes and database content.

These indexing connectors are configured mainly by editing XML files and Windows PowerShell cmdlets and you operate them by using the command line.

About the FAST Search Web crawler

The FAST Search Web crawler is a customizable indexing connector used to crawl Web site content. The FAST Search Web crawler can scale up to large environments, for example when your organization is crawling many external Web sites. In addition, the FAST Search Web crawler can crawl dynamic web content, such as Web sites that contain JavaScript.

The FAST Search Web crawler collects content from a set of defined Web sites, which can be internal or external. The configuration of the FAST Search Web crawler is done by editing a copy of an XML file. You can operate the FAST Search Web crawler through several command line tools.

The FAST Search Web crawler is typically a component inside a FAST Search Server 2010 for SharePoint installation. Internally, the FAST Search Web crawler is organized as a collection of processes and logical entities, which in most cases run on a single server. When the number of Web sites or total number of pages to be crawled is large, the FAST Search Web crawler can be scaled up by distributing these processes across multiple hosts. This requires additional configuration.

The FAST Search Web crawler can crawl HTTP, HTTPS and FTP content and supports NTLM version 1 (and to a limited extend version 2), Digest, basic auth and form based logon authentication. RSS scheduling is supported and you can tag linked documents from the feed.

About the FAST Search database connector

The FAST Search database connector is a specialized indexing connector that collects content from database content sources.

The indexing connector is configured by using an XML template. You operate the connector by using the command-line options from the jdbcconnector.bat file. After running the configured connector, you map crawled properties to managed properties in the SharePoint Server 2010 Central Administration to enable and customize search on the content collected by the connector.

The connector uses an SQL statement to run against the crawl database. This statement is completely customizable. The FAST Search database connector uses checksum based change detection for incremental crawls if there is no update information available. The connector also supports time stamp based change detection and change detection based on update and delete flags. Also, you can indicate pre and post operation procedures that have to be done to the database before it is crawled, which can be an advantage in certain use cases.

About the FAST Search Lotus Notes connector

The FAST Search Lotus Notes connector is a specialized indexing connector that consists of two parts: a user directory connector and a content connector. The content connector collects content from a Lotus Notes content source. The user directory connector ensures that the end-users can only search Lotus Notes content that they have access to. The user directory connector maps the Active Directory user directory with the Lotus Notes user accounts and is closely integrated with FAST Search Authorization.

The connector is configured by using two XML templates, one for the user directory connector and one for the content connector. You operate the connector by using the command-line options from the lotusnotesconnector.bat and lotusnotessecurity.bat files. After running the configured content connector, you map crawled properties to managed properties in the SharePoint Server 2010 Central Administration to enable and customize search on the content collected by the content connector.

The FAST Search Lotus Notes connector supports Lotus Notes version 6.5.6, 7.x and 8.x and Lotus Domino version 6.5, 7.x and 8.x.

The connector fully supports Lotus Notes security, including roles, and can index Lotus Notes databases as attachments.

Include or exclude content from a crawl

The FAST Search Server 2010 for SharePoint specific connectors each have parameters in their respective configuration files to indicate include and exclude rules.

Important:

Do not overload any content sources that you plan to crawl.

For content within your organization that other administrators are crawling, you can coordinate with those administrators to set impact rules based on the performance and capacity of the servers. For most external sites, this coordination is not possible. Requesting too much content on external servers or making requests too frequently can cause administrators of those sites to limit your future access if your crawls are using too many resources or too much bandwidth. Therefore, the best practice is to crawl more slowly. In in this manner, you can reduce the risk of losing access to crawl the relevant content.

With the FAST Search Web crawler, you can control the crawl rate by setting a request delay, set a maximum to the number of concurrent requests that are sent to the same Web site at the same time or enable or disable concurrent crawling of an IP address where multiple sites are hosted. You can also limit the bandwidth of the FAST Search Web crawler by limiting the number of concurrent Web sites to crawl at the same time.

Set crawl schedules

The FAST Search Lotus Notes connector and the FAST Search database connector use the Windows Task Scheduler to schedule crawls. Scheduling crawls for the FAST Search Web crawler is possible by setting parameters in the XML configuration file.

Tip:

It is recommended to complete a manually started full crawl cycle before scheduling additional crawls. This to find out how long it takes to run a full crawl and to avoid starting a new or incremental crawl before the initial crawl has finished.

See Also

Concepts

Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint)


Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint)

Before you start crawling content with the FAST Search specific connectors for Lotus Notes, database or Web content, there are a number of considerations to take into account. By first identifying the search needs and the organization of the content sources you plan to crawl, and collect important information, you will be able to set up the connectors more efficiently.

Consider creating several content collections

FAST Search Server 2010 for SharePoint uses a content collection to feed crawled content to the FAST Search Server 2010 for SharePoint index. You can choose to feed all the content to the same content collection, which defaults to sp.

However, sometimes it is necessary to remove content from a particular FAST Search specific connector configuration from the content collection, which requires that the content collection is cleared. This is done by running a Windows PowerShell command that clears all content from the particular content collection. In that case you will have to re-crawl all the content sources that feed to that particular content collection, which can be time consuming.

Consider creating separate content collections for your FAST Search specific connector configuration if you expect that the content from a particular content collection will have to be deleted at some point in the future. If you create different content collections for each or some FAST Search specific connector configuration(s), you avoid losing content from the index and prevent having to re-crawl any other content sources that feed to the same collection.

Important:

Creating an additional content collection uses memory. You should consider an additional memory consumption of 500 MB per content collection per running indexer node. For example, in an environment with two indexers and two content collections, the total memory consumption will be 2x(2X500MB)=2000MB, or 2 GB, divided across all indexers.

Before implementing the FAST Search Lotus Notes connector, consider:

  • Which Domino servers hold data to be crawled and indexed.

  • Which database(s) on each server should be crawled and indexed.

  • Which database view (if any) should be used to select which Lotus Notes documents are to be made searchable.

  • Which attachment types (for example: pdf, txt, doc, nsf, ppt) should or should not be made searchable.

  • Which accounts can be used to access the data with sufficient permissions.

  • What database metadata should be made searchable.

  • What the total number of Lotus Notes documents to be crawled and indexed is.

    • The total number of Lotus Notes documents to be crawled affects the type of SQL Server database that should be used to track the state data for the connector. The maximum size for a SQL Server Express database is about 4 GB. This amounts to circa 2 million Lotus Notes items. If you are planning to crawl more items, use SQL Server 2008 Enterprise (or newer).

    • Instructions on how to change to a different state tracking database can be found in Configure the FAST Search Lotus Notes connector.

  • How often the connector should run, for example daily, weekly or monthly.

    • Different types of Lotus Notes content may have different demands for freshness. For example, e-mail databases may have to be crawled daily whereas archive databases can be crawled on a weekly (or even monthly) basis. In that case, you should create separate configurations for each of these content sets.

  • If several separate connector configurations must be created. There can be various reasons to split the crawl into two or more configurations and then schedule each configuration separately:

    • Different demands for freshness;

    • Different include/exclude rules for different content. For example, do not index Lotus Notes databases attached to e-mails, but index Lotus Notes databases attached to project databases;

    • Different property mappings for different parts of the content. For example, the property "last modified" in e-mails may be a time stamp indicating WHEN it was last modified while the property "last modified" in a project database may be a string indicating WHO modified the document last. These two properties cannot be indexed in the same managed property, because they are of different type. Therefore, a separate property mapping is needed for the two. This is done by configuring different configurations.

    • The need to feed different content to separate content collections in the search engine. If it is likely that you will delete one configuration (and the corresponding contents) sometime in the future, use a separate content collection for that configuration. This lets you delete all the indexed content for that configuration by deleting the content collection, without losing any other crawl data for different configurations. Refer to the section Considerations when modifying the filters in the article Manage crawl rules (FAST Search Lotus Notes connector) to decide whether you want to use separate content collections or not.

  • Whether you want to use item status tracking or not.

    • The item status tracking feature monitors when each item was last crawled and the status of that crawl, including any error messages. This information is stored in a database table that is not emptied automatically. Depending on the number of items in each crawl and how often a crawl is run, this table can become quite big.

      You can choose to turn off item status tracking by setting the FAST Search Lotus Notes content connector configuration file parameter ConnectorExecution/EnableStatustracker to false. Alternatively, you can manually delete the contents from the connectors.statustracker table at regular intervals when the connector is not running.

Before implementing the FAST Search database connector, consider:

  • Which servers contain databases with data to be crawled and indexed.

  • Which databases/tables hold data to be crawled and indexed.

  • Be prepared to formulate (or have a database owner formulate) an SQL query for each data source that is to be crawled and indexed. Database client tools can be used as an aid to this.

  • For each SQL query, determine which column(s) should serve as the document identifier.

  • Determine if the result from the SQL query will return rows with the same identifier that should be merged into one FAST Search Lotus Notes connector document.

  • How incremental updates should be configured. Find out if the database includes a time stamp column that can be used to find updated rows or if such a column can be added. If this column is not present and/or you cannot add such a column, you can use a checksum or flag based approach.

  • Whether you want to use item status tracking or not.

    • The item status tracking feature monitors when each item was last crawled and the status of that crawl, including any error messages. This information is stored in a database table that is not emptied automatically. Depending on the number of items in each crawl and how often a crawl is run, this table can become quite big.

      You can choose to turn off item status tracking by setting the FAST Search database connector configuration file parameter ConnectorExecution/EnableStatusTracker to false. Alternatively, you can manually delete the contents from the connectors.statustracker table at regular intervals when the connector is not running.

Before implementing the FAST Search Web crawler, consider:

  • How many crawler configurations you intend to use. Consider using multiple FAST Search Web crawler configurations when you have multiple independent Web sites to crawl that require different crawl configuration rules. Note that in many cases it may be possible to use sub collections instead.

  • How many FAST Search Web crawler servers you intend to use. Consider using multiple FAST Search Web crawler servers (known as a multi-node Web crawler) when you have a large number of Web sites to crawl. Note that a single DNS domain (for example *.contoso.com) can only be crawled by a single FAST Search Web crawler server.

  • Which Web sites hold the data to be crawled and indexed.

  • Think about the number of Web sites to be crawled, how quickly content should be requested and how often the Web sites should be re-visited.

  • What Web sites or part of Web sites should not be indexed. For example, CVS repository front ends or other systems that have a Web front-end that may pollute the index with unwanted data, or put a load on the Web server that it is not scaled to handle.

  • Which Web sites require authentication, what kind of authentication is required, and what accounts can be used to access the data with sufficient permissions.


Plan the relevance of search results (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

One of the key features in a search solution is the ability to return relevant information. Relevance describes how well a given search satisfies a user's information needs. Relevance includes which documents are returned in the results (recall) and the order of those documents in the results (ranking). FAST Search Server 2010 for SharePoint provides good search relevance out of the box in addition to many possibilities for custom tuning. Optimizing search for relevance includes refining content for searching, tuning sort order and ranking. It is a task that should be done by advanced search administrators and developers.

There is no single best relevance solution that satisfies all users and different search applications. This is why FAST Search Server 2010 for SharePoint provides advanced possibilities for relevance tuning. Optimal relevance must be planned for and tuned to suit specific needs.

To get the best possible results from FAST Search Server 2010 for SharePoint, you should invest time in relevance tuning as an on-going process. As a search administrator or developer, you should be familiar with the content to be indexed.

  • Know the content

    • Work with colleagues who know the content to be represented. This should be part of structuring the SharePoint Server installation for optimized search.

  • Know the end-users of the search solution

    • Understand what the user defines as "good relevancy".

    • Verify your assumptions with facts: talk to end-users of the system and examine the log files.

  • Involve the user

    • Involve all possible business users to agree on one set of rules and examples.

    • Define representative test cases based on agreed upon rules and examples. Define search words and phrases particular to your organization and their expected or desired search results.

    • Perform thorough benchmark testing based on standard or initial tuning.

  • Test the results of each tuning iteration.

    • Perform the same tests every time for comparable results. Use the defined search input and expected or desired search results to test your changes, both in the development phase of the project and after upgrades or tuning of the content.

    • Ensure that the tuning had no negative side-effects.

Important:

Define the results that you want before spending time on the technical aspects of relevancy tuning.

In this section:

  • Terminology for search relevance (FAST Search Server 2010 for SharePoint)

  • How ranking and sorting of search results are calculated (FAST Search Server 2010 for SharePoint)

  • Which relevance factors can you tune and what is the impact (FAST Search Server 2010 for SharePoint)

  • How to optimize content for search (FAST Search Server 2010 for SharePoint)


Terminology for search relevance (FAST Search Server 2010 for SharePoint)

Relevance describes how well a given search satisfies a user's information needs. The following terms are important to search relevance:

Term | Definition
Crawled properties | Crawled properties are metadata extracted from content sources when you run a crawler or connector. Metadata can be structured content such as the title or the author from a Word document, or unstructured content based on the content of an item such as a detected language or extracted keywords. Typically, crawled properties include column data for SharePoint list items, document properties for Microsoft Office or other binary file types, and HTML metadata in web pages.
Managed properties | Administrators map crawled properties to managed properties in order to make metadata searchable and provide useful search experiences. For example, an administrator might create a managed property named Client that maps to various crawled properties called Customer, Client, and Cust from different content repositories. Managed properties can then be used across the search solution, such as in defining search scopes and in applying search filters.
Full-text index | Managed properties are mapped to a full-text index. This enables you to search across several properties at the same time. When a search is performed, it searches in a full-text index.
Index schema | The index schema controls which content parts are indexed, searchable, and available for viewing. It also manages the mapping of managed properties into full-text indexes.
Search item | A searchable item in the index. This can be a document, a list, a list item, a database record, or any other kind of content that is indexed and available for search.
Recall | The number of search items found in the index related to a search. High recall means that many search items are returned.
Rank points | Points that are given to search items based on the importance of each item. The more rank points a search item receives, the higher up it appears in the search results list.
Dynamic rank points | Rank points that are added to search items at search time. The number of points that are given is based on a combination of the search words used and several boost values in the rank profile.
Quality rank points | Also known as Static rank points, are points added to a search items before it is indexed. The number of points given are based on boost values that are stored in predefined quality rank components.
Keyword rank | Rank points that are added to a search item when a search word matches a specified keyword. Each keyword can include a list of associated synonyms, and recall is improved by also searching for these associated synonyms.
Rank score | The sum of all rank points given to a search item during query processing. A search item with a high rank score will appear in the upper end of the search results list.


How ranking and sorting of search results are calculated (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

Different features within the Index schema are used to rank and sort search results. In addition, you can also use the FAST Query Language (FQL) to create queries programmatically, for example narrow the scope of your search.

In this section:

  • The index schema and its features (FAST Search Server 2010 for SharePoint)

  • FAST Query Language (FQL) (FAST Search Server 2010 for SharePoint)


The index schema and its features (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

The index schema controls which content parts are indexed, searchable, and available for viewing. The predefined index schema (delivered out of the box) has been tested and tuned to suit many common deployments. However, depending on the content you want to index and make available for viewing in your web front-end, you should consider the index schema strategy before you deploy a full-scale FAST Search Server 2010 for SharePoint farm. Make sure that you plan the overall index schema strategy before you start indexing large amounts of content. If not, you may have to re-index all the content for the changes to take full effect. It is possible to make incremental changes to the mapping without any service interruption or search downtime. However, it is very inconvenient to apply major changes after having indexed large amount of content.

If your deployment will index many million documents, we recommend that you first tune the index schema and the associated end-user search features on a smaller test installation that has a relevant subset of the content that you want to index.

The index schema

FAST Index Schema

The following features are handled in the index schema:

  • Crawled properties

  • Managed properties

  • Full-text index

  • Rank profile

Crawled properties

Crawled properties are metadata that is extracted from content sources when you run a crawler or connector. Metadata can be structured content such as the title or the author from a Word document, or unstructured content based on the content of a search item such as a detected language or extracted keywords. A crawled property is uniquely defined by Name, Propset and VariantType. Crawled properties get sorted into categories, which is a high-level grouping based on the iFilter and Protocol Handler (given by the Indexing connector used and data source) used to extract the metadata from the content. Examples of categories are as follows:

  • Business Data - Metadata associated with content in the Business Data Catalog.

  • Mail - Metadata associated with Microsoft Exchange Server.

  • Office - Metadata contained in Microsoft Office documents such as Word, Excel and PowerPoint.

  • People - Metadata associated with the people profiles in SharePoint. The majority of these are also mapped to various managed properties from Active Directory and SharePoint information.

  • Web - HTML metadata associated with web pages.

A number of crawled properties contain metadata that is irrelevant or may adversely affect the search relevance. It generally does not make sense to add numeric values to the full-text index. You would also not include crawled properties that have sensitive content, or crawled properties that have content that contains words and phrases that occur in most of the documents in the full-text index. These should be excluded from searching. The default index schema already provides several excluded crawled properties that work well for common content formats. However, because the discovery of crawled properties is content driven, additional properties will be discovered that you must consider when you tune the index schema for relevance.

There are several reasons to for excluding the content of crawled properties to improve both relevance and recall. Some crawled properties might be searchable through a managed property mapping, and do not have to be searchable through the default full-text index also.

For more information about how to exclude content of crawled properties, see Exclude crawled properties from searching by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

Managed properties

To make metadata searchable, you map one or more crawled properties to a managed property. The crawled properties will contain a large amount of different metadata properties. A key phase of your deployment planning is to determine the mapping of these crawled properties to managed properties. The default index schema provides default mappings that are adapted to common content formats. As you optimize the system for relevance, look at the quality of the content in managed properties, determine whether there are other crawled properties that have better quality for your content, and if so, update the mappings. To make it easier to test your changes, perform an initial tuning of the crawled property mapping on a test installation that has a limited amount of content.

For more information about how to map crawled properties, see Map a crawled property to a managed property by using Windows PowerShell.

In the simplest form, a search index can contain the searchable representation of the body and title of a document. However, you will quickly experience the power of mapping and indexing the various metadata of your content sources. By using the FAST Search Server 2010 for SharePoint schema administration services, you can explore the actual crawled properties of the content sources and decide a mapping to managed properties. You will then be able to assign features to the managed properties that enhances the end-user experience when they make their queries, such as defining refiners and alphabetical sorting of the search results.

For more information about how to define a refiner, see Define a refiner for a managed property by using Windows PowerShell.

For more information about how to sort search results alphabetically, see Set a managed property as sort criteria by using Windows PowerShell.

Full-text index

A full-text index will typically contain a set of managed properties that represents the content of the item that you are querying. This includes the body of the item, the title, the URL, and so on. The default full-text index is named content.

When a search is performed, it searches in a full-text index. Managed properties are mapped to a full-text index to enable you to search across several managed properties at the same time. The index schema manages the mapping of managed properties into full-text indexes. With this mapping, the managed properties are assigned an importance level between 1 and 7, where 7 is the most important. The importance level represents the perceived importance of the managed property within the full-text index as related to drilling. Drilling works with a StopWordThreshold to ensure that the most relevant search items are returned first when the StopWordThreshold is reached on a search against a full-text index (see later in this article for more information about the StopWordThreshold).

For example, the StopWordThreshold is set at 2,000,000, and you have the following managed properties with the different importance levels in parenthesis:

  • Title (7)

  • DocKeywords (5)

  • urlkeywords (4)

  • description (3)

  • Author (2)

  • body (1)

When a search is performed, the first level that is used in rank computation includes all managed properties from levels 1 and up to 7. So in our example, all the listed managed properties would be included. If the number of matching search items is larger than 2,000,000 (the StopWordThreshold), the system automatically retries the search by drilling to the next importance level. By drilling to importance level 2, only managed properties from levels 2 and up will be used in the rank computation. In our example, this would mean that the managed property body is not included. If, after drilling to importance level 2, the number of matching search items is still larger than 2,000,000, the system will drill to level 3. In our example, the managed properties body and Author would not be included in rank computation. It is important to be aware that all matching search items will appear in the search results list. However, search items that are not part of the rank computation will appear in the search results list after all search items included in the rank computation.

You can tune and improve relevance by separating managed properties that map to the same full-text index into different importance levels. For more information about how to do this task, see Map a managed property to a full-text index at a specific importance level by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

In certain cases you may want to define multiple full-text indexes for different kinds of queries or for different applications. Although this gives a large amount of flexibility, it will have a certain performance cost for disk space and use of system resources such as file descriptors. We therefore recommend that you do not define more than 10 full-text indexes inside an index schema.

The index schema discovers new crawled properties and automatically make their content available for searching. The contents of new crawled properties are mapped into the default full-text index at the lowest importance level (1). Therefore, you do not have to have prior knowledge about the crawled properties or create mappings to managed properties to make the contents of newly crawled properties searchable.

Rank profile

A rank profile is a sort criterion that is used by FAST Search Server 2010 for SharePoint. Rank configuration is stored in a rank profile. Compared to sort criteria like name or date, the rank profile is a more complex structure used to calculate the rank score. The rank profile is configured to calculate the rank score with input from quality rank components (also known as static rank components) and dynamic rank components. You can create new rank profiles or modify the default rank profile to optimize search results based on your kind of content. You can also set the different rank profiles as sort criteria on the search results page. This gives the user more options for filtering the search result. The default rank profile is named default. It is also used as the default sort criteria (Sort by Relevance) on the search result page. The final rank score calculation is complex. But the ability to configure rank profiles gives you a way to control the rank factors that you consider important for the system. Customizing the rank profiles and creating new rank profiles will have small effect on static system resources like disk and memory.

For more information about the rank profile, see Manage rank profiles (FAST Search Server 2010 for SharePoint).

Rank profile changes will mainly affect query performance as outlined in the following list.

  • Stop Word Threshold. This is an important parameter to avoid that queries for very common words take too much resources to evaluate. The number of matching search items used in the rank computation is determined by the StopWordThreshold attribute in the rank profile. If the number of matching items for a search item is larger than the StopWordThreshold for the rank profile being used, the FAST Search Server 2010 for SharePoint system automatically retries the search by limiting the managed properties searched. This process is known as drilling.

    The value of the StopWordThreshold is relative to a full-text index size of 10,000,000 items. In the default rank profile the StopWordThreshold value is 2,000,000, which means that drilling is performed when a search word is found in more than 20% of items in the full-text index (2,000,000 divided by 10,000,000).

    For more information about how to change the Stop Word Threshold, see Change the Stop Word Threshold by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

  • Managed property field boost. This is an efficient way to achieve targeted relevance boost for documents that have managed properties that have certain values. For example, if you want to give additional rank points to indexed items of a specific type, such as Word documents, you would give search items with the file extension .docx additional rank points. Each managed property field boost setting will add to the evaluation time for all queries. Therefore, be careful not to define too many such boosts within the same rank profile. It is better to define multiple rank profiles with targeted managed property boost setting.

    For more information about how to change the Managed property field boost, see Change the managed property field boost weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint).


FAST Query Language (FQL) (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

FAST Query Language (FQL) is a query language intended for creating queries programmatically. It is a structured language that is not intended to be exposed to end-users. FQL can be used only for solutions developed for FAST Search Server 2010 for SharePoint.

User Query Options
The user query is a sequence of query terms typed in a single query box by a user. In most cases the user types one or more words. However, the user query may also include special characters such as the plus ("+") and minus ("-") signs. The user query is usually treated as a string that is passed transparently by the query client. You can pass the user query on the query client interface in two main ways:

  • Pass the user query as a Keyword Query Syntax query string, which enables advanced users to express a more specific query by using a set of basic query controls that are exposed by the Keyword Query Syntax. By default, a standard Search Center Web application uses the keyword query language

  • Pass the user query as an FQL Simple Query Language string within an FQL query expression. You encapsulate the end-user query string in FQL, and can add any other property filter or other query constraints in the FQL query expression.

Advanced ranking and sorting options
Using FQL enables you to do the following:

  • Sort by rank: Enables you to impact rank calculation in the query string.

  • Sort by managed property: Enables you to sort the query result based on the value of one or more managed properties.

  • Sort by formula: Enables you to sort the query result by a formula specified in a query result.

For more information, see Ranking and sorting


Which relevance factors can you tune and what is the impact (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

Ranking and sorting of search results are determined by the relevance score, which is quantified in a rank score for each searchable item. By using FAST Search Server 2010 for SharePoint you can tune relevance in several ways. Predefined relevance (delivered out of the box) was tested and tuned against large sets of different data. However, relevance is not absolute, and because perceived relevance differs greatly among users, organizations, and locations, it is important to tune relevancy according to your organization's specific needs.

You can tune the following relevance factors:

  • Static rank (quality rank)

  • Dynamic rank

  • Keyword rank

  • Linguistic features

In this section:

  • Static rank (quality rank) (FAST Search Server 2010 for SharePoint)

  • Dynamic rank (FAST Search Server 2010 for SharePoint)

  • Keyword rank (FAST Search Server 2010 for SharePoint)

  • Linguistic features (FAST Search Server 2010 for SharePoint)


Static rank (quality rank) (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

Static rank points, also known as quality rank points, are added to search items before it is indexed. The number of points given is based on boost values that are stored in predefined static/quality rank components. You can decide which of these components to use at search time so that you only get the rank points from the components that you specify. You can control the effect of the static rank points by modifying the weight of the different components. Static rank points assigned to an indexed item are independent of the search words that are used, so static rank only indicates something about the general importance of an item. Adding static rank points is efficient from a search performance point of view because it does not add any complexity to the search evaluation.

Static rank points are derived from multiple managed properties. The following sets of managed properties are predefined for static ranking:

  • Urldepthrank: Rank points given to boost shorter URLs.

  • Docrank: Rank points given based on the number of and relative importance of links pointing to an item.

  • Siterank: Rank points given based on the number of and relative importance of links pointing to the items on a site.

  • Hwboost: FAST Search Server 2010 for SharePoint placeholder for generic usage of quality rank points.

In addition to these managed properties, you can add any custom integer managed property to the list of static/quality rank properties.

See also

For information about how to tune static rank, see Tune static rank (quality rank) (FAST Search Server 2010 for SharePoint).


Dynamic rank (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

Dynamic rank is calculated at search time. The dynamic rank score is calculated based on a combination of the input search words and the rank profile configuration. You can control the effect of the dynamic rank points by modifying the weight of different components in the rank profile.

You can modify the following components in the rank profile to effect dynamic rank points:

  • Freshness: Rank points given based on the age (last modified) of the item compared to the time of the search. The resolution used (second/minutes/hour/day/year) is important when calculating rank, because a lower resolution implies that smaller time differences will have a larger effect on ranking.

  • Context: Rank points given based on where in the search item the search words match. The different managed properties such as title and body are mapped to different importance levels. This makes it possible to give more rank points for a search word match in a title than in the body text of the search item. Context boost is assigned as an overall weight and as a weight for a specific managed property.

  • Proximity: Rank points given depending on the distance between the search words and where these words are located in the search item. For example, if two search words appear in the same managed property within the full-text index, they will get more rank points.

  • Managed Property Field Boost: Rank points given based on a specific item type of a managed property. For example, if you want to give additional rank points to indexed items of a specific type such as Word documents, you can give search items with the file extension .docx additional rank points.

  • Authority (anchor text): Rank points given based on when a search word occurs in the link text (anchor text) of a link pointing to the search item. This rank boost has two components:
    Partial match - the link text is Microsoft Word, and the search word is Microsoft.
    Complete match - additional score if there is a match between the search word and link text.

  • Query authority (click-through): Rank points given based on when a search word matches search items associated with previously performed searches (click-through). In the search front-end, all of a user's searches and the search items that the user clicks on that are associated with those searches, are stored and processed. For example, if a user searched for vacation and clicked on the vacation request template in the search result (for example http://myserver/template/vacation.aspx), this URL is associated with the search word and obtains additional rank points when any new search with the same search word is performed. Processing occurs one time per day. The last 30 days of clicks are kept for processing. This rank boost has two components:
    Partial match - the initial search words that resulted in a click to this item was Microsoft Word, and the search word now is Microsoft.
    Complete match - an additional score if there is a match between the search word and link text.

For search performance reasons, the calculated dynamic rank score is based on different importance levels in the full-text index. Drilling, or going to the next level, occurs when the number of matching search items for a search word is larger than the StopWordThreshold for the rank profile that is being used. Drilling works with the StopWordThreshold to ensure that the most relevant items are returned first when the stop word threshold is reached.

See also

For information about how to tune dynamic rank, see Tune dynamic rank (FAST Search Server 2010 for SharePoint)


Keyword rank (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

A keyword is an entry point for improving search. The keyword can be a single word or a phrase. When a search word matches a keyword, the search experience is improved by searching for the additional synonyms, incorporating promotions to move relevant search results to the top, and displaying additional relevant information as Best Bets and Visual Best Bets.

The following components are involved when tuning keyword rank:

  • Keywords are words or phrases that you have identified as typical terms within your organization. Each keyword entry can include a list of synonyms, which is used to improve recall for the keyword. By identifying keyword terms, you can provide a "glossary" of names, processes, and concepts that are unique to your organization. When a search includes a synonym for a keyword, search items that contain the related keyword are also returned. Furthermore, if a search includes a keyword, search items that contain the synonym are also returned, regardless of whether they contain the keyword.

  • Best Bets are recommended results for a particular keyword and its synonyms. Best Bets can link to recommended websites, data stores or documents. When a search include a keyword or one of its synonyms associated with a Best Bet, links to the associated Best Bets will be displayed in a prominent position on the search results page.

  • Visual Best Bets are manually created content that is associated with a particular keyword and its synonyms. A Visual Best Bet is not limited to text or links, but can be visual elements such as image banners or HTML. When a search include a keyword or one of its synonyms associated with a Visual Best Bet, the associated Visual Best Bets will be displayed above all other search results on the search results page.

  • You can use Document Promotions to indicate that a certain document is a highly relevant result for a particular keyword and its synonyms. When a search include a keyword or one of its synonyms associated with a document promotion, the promoted document will appear higher in the search result list. You can promote multiple documents for each keyword. You can also apply Global Document Promotions, where boost points are given to a specified document regardless of the search words entered by the user. Reversely you can use Documents Demotions to indicate documents that are less relevant for a particular keyword and its synonyms. When a search include a keyword or one of its synonyms associated with a document emotion, demoted documents will appear somewhat lower on the search result list. You can demote multiple documents each keyword.

  • You can use Site Promotions to add rank points to all documents that belong to a specific site. This causes those documents to appear somewhat higher on the search results page. You can also use Global Site Promotions to add rank points to all documents within the specified site regardless of the search words entered by the user. It often makes sense to apply a user context to the site promotion, for example, promote the marketing collateral site for people who belong to the marketing organization. Reversely documents that belong to less relevant sites can be given a Site Demotion that will cause the documents to appear towards the end on the search results page. A Global Site Demotion is applied regardless of the search words entered by the user. It can make sense to apply a user context to the site demotion, for example, demote the R&D wiki for people who belong to the sales organization.

  • User contexts enable targeted communication and targeted search improvements by displaying Best Bets, Visual Best Bets, document promotions and demotions to a specific user group only. User contexts match the properties that are defined on the user's SharePoint User Profile page. For example, a Visual Best Bet banner can be configured to be displayed only for employees who belong to the sales organization and who are located in Germany. The different search settings - Best Bets, Visual Best Bets, document promotions, document demotions, site promotions and site demotions - can have one or more user contexts associated with them.

See also

For information about how to tune keyword rank, see Tune keyword rank (FAST Search Server 2010 for SharePoint).


Linguistic features (FAST Search Server 2010 for SharePoint)

Published: March 31, 2011

FAST Search Server 2010 for SharePoint has many linguistic features that help improve search relevancy. Some features can be tuned, and other features have a default behavior that cannot be changed. The table here describes the different linguistic features and their effect on relevance and recall.

Linguistic feature Impact on relevance Impact on recall Description

Synonyms

No

Yes

Synonyms are a list of words attached to a keyword. Keywords are words or phrases that you have identified as typical terms within your organization. You attach synonyms to keywords to increase recall. When a search includes a synonym for a keyword, search items that contain the related keyword are also returned. Furthermore, if a search includes a keyword, search items that contain the synonym are also returned, regardless of whether they contain the keyword. Please be aware that this only applies when there is a complete match between the search words and any defined keywords or synonymous terms.

Stemming

Yes

Yes

Words can have multiple forms but basically mean the same thing. For example, the verb "to write" includes forms such as writing, wrote and writes. Similarly, nouns usually include singular and plural versions, such as book and books. The stemming feature in FAST Search Server 2010 for SharePoint can increase recall of relevant documents by mapping one form of a word to its variants.

For languages that have many forms of the same word, stemming is very important to achieve sufficient recall. Stemming is applied to the content of managed properties where stemming is enabled. You cannot tune the stemming dictionaries.

Spell checking and spell tuning

Yes

No

The spell checking feature improves the quality of searches by comparing the search words against language specific dictionaries and identifying misspelled terms. If the dictionary contains a closely matching word with a significantly higher frequency, that word is suggested through the Did you mean? feature. You can fine-tunes the spell checking dictionaries to make sure that they are aligned with the frequency of words in the processed documents. Users will only get spell checking suggestions that are relevant within the processed content.

You can also define spell checking exceptions. These are words that are not found in the default spell checking dictionary but that are still valid words. When a user types a search word that is included in the Spell checking exception list, the, Did you mean? feature will not suggest a correction for that word.

The spell checking dictionary contributes to both increased recall and relevance because the feature prevents usage of misspelled words.

Anti-phrasing

Yes

No

Anti-phrasing refers to phrases for which there is no value in indexing. "Where can I find information about" is a typical anti-phrase for English. You cannot tune the anti-phrasing dictionaries.

Tokenization

Yes

Yes

The tokenization process splits a stream of text into individual words (tokens) that can be indexed. Spaces, tabs, periods, commas, dashes, question marks and quotation marks are considered delimiting characters. For East Asian languages, that is Chinese (Simplified and Traditional), Japanese and Thai, where spaces are not consistently used to separate words, tokenization is especially important for relevancy.

Tokenization is performed on text content both during item processing and search processing. The tokenization process in FAST Search Server 2010 for SharePoint consists of three stages:

  • Language-independent input normalization, where input text is transformed into a unified format. This includes replacing complex characters such as ligatures with their canonical forms, and replacing less-used Unicode characters by compatible characters or sequences of characters (Trademark sign with tm or full width Latin characters with half-width equivalents).

  • A language-specific tokenization engine based on the document language splits the text into individual words/tokens based on word breaker tokenizers.

  • Indexed tokens are normalized according to language-independent rules to ensure cross-language retrieval. The normalization reduces the complexity of a character by changing it or removing parts of it. In FAST Search Server 2010 for SharePoint, all characters are lowercased and accented characters are reduced to their unaccented base characters.

Automatic language and encoding detection

During item processing, FAST Search Server 2010 for SharePoint automatically recognizes more than 80 different languages in all common encodings. The text language and encoding can be defined in the metadata of a document, or it can be determined by an automatic process during item processing. The information is used to select the appropriate language-specific dictionaries and algorithms during item processing.

Property extraction

FAST Search Server 2010 for SharePoint provides advanced, language-specific property extractors for person names, company names and geographic names/locations.

For more information, see Manage property extraction (FAST Search Server 2010 for SharePoint).

Offensive content filtering

FAST Search Server 2010 for SharePoint can provide filtering against offensive content for many language. Offensive content filtering is not provided out of the box, but can be configured.

Optimizing relevancy for East Asian languages

As there is no fixed standard in different languages for what forms a separate token, different speakers have different ideas of what a token may be. For example, some users may consider 富士山(Mount Fuji) to be one token and other users may regard it as two tokens, 富士 (Fuji) and 山 (Mount).

The inconsistencies between what users consider being one token, and what the tokenizer module actually identifies as one token, can cause lower precision or lower recall, for example:

  • The Simplified Chinese tokenizer module splits the name 萨斯喀彻温 (Saskatchewan) into these tokens: 萨 (Sa), 斯 (s), 喀 (ka), 彻 (tche), 温 (wan).

    A search for the name "Saska", 萨 (Sa), 斯 (s), 喀 (ka) will also retrieve a document that contains "Saskatchewan", which means that the precision is lower than may be expected.

  • The Japanese tokenizer module marks "サスカチュワンサスカトゥーン" (Saskatchewan Saskatoon), as one token.

    A search for "Saskatchewan" will not retrieve a document that contains "Saskatchewan Saskatoon". Therefore, recall will be decreased.

FAST Search Server 2010 for SharePoint performs language specific tokenization based on automatic language detection for the indexed items and the end-user's locale setting. However, you can influence the default tokenization using two methods: linguistic tokenization and substring tokenization.

Linguistic tokenization means that a string of text is split into individual tokens based on language-specific rules. For East Asian languages, you can influence tokenization by creating custom dictionaries. If words are missing from the system dictionary provided by FAST Search Server 2010 for SharePoint, for example technical terms, person names or company names, or if the default tokenization is incorrect, you can add words to the custom dictionary to ensure that they are tokenized as required.

Substring tokenization, also known as N-gram tokenization, is typically applied to managed properties that are considered difficult to tokenize automatically. Substring tokenization removes all spaces from the text and then splits it into bigrams (overlapping two character long tokens). For example, "アメリカ" (America) is split up into: ア,アメ,メリ,リカ (a, ame, meri, ca). Without substring tokenization enabled, a CJK query may, in certain cases, be tokenized incorrectly and therefore return a meager or empty result list. This will never occur if substring search is used, as all N-gram substrings of each token will be indexed, and also N-grams spanning token boundaries.

Substring tokenization is especially useful for applications where recall (the overall number of documents retrieved) is considered much more important than precision (high relevancy of the results). By using this feature, you will improve the recall, but may also reduce the precision and return more items than desired. Note that substring tokenization will have a significant effect on the size of the index for these managed properties. It is therefore not recommended that you use the feature on free-text, but may be considered for metadata that contains domain-specific product names, codes, and so on. To minimize the drop in precision, you can use a combination of substring tokenization and linguistic tokenization.

See also

For information about how to tune linguistic features, see Tune linguistic relevance (FAST Search Server 2010 for SharePoint)


How to optimize content for search (FAST Search Server 2010 for SharePoint)

After installation of a new SharePoint Server, not all content is equally relevant for search. Content has different levels of importance, and some content should not be indexed at all. The search engine must be informed of this content differentiation so that the most relevant items are returned when users search.

The search engine must contain the important and relevant information for relevancy tuning to have any effect. It is also important to prevent indexing of information that has no value for search. Removing unwanted information from the search engine helps prevent the return of unwanted and low-value results. The following sections include some guidelines that should be established for content owners and site administrators when organizing the data:

  • Structure your sites

  • Use short URLs

  • Use simple and descriptive link texts

  • Add metadata to your content

  • Prevent indexing of irrelevant content

  • Encourage archiving and deleting old content

  • Review your security settings

Structure your sites

Decide which sites are (or will be) the most important and reserve them for premium content. These sites can also be given additional rank points through site promotions. For more on site promotions, see Add, remove and display site promotions for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

Use short URLs

Add rank points to pages that have short URLs, for example giving the page http://myserver.com/ additional rank points compared to the page http://myserver.com/site. General and important information should therefore be part of a page that has a short URL. In the FAST Search Server 2010 for SharePoint solution, you can give additional rank points to sites that have short URLs by adjusting the static/quality rank component Urldepthrank. For more on how to adjust this component, see Change the weight, remove or add a static rank component by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

You can also nest less important sites, use natural hierarchies, and avoid flat structures.

For example:

Use | Versus
http://sales |
http://sales/china | http://china
http://sales/japan | http://japan
http://products |
http://products/proda | http://proda
http://products/prodb | http://prodb

Use simple and descriptive link texts

The search engine contains a module that contributes to dynamic rank points based on the link text (anchor text) and static rank points based on links that are created between items in the SharePoint Server installation and other content that you plan to index.

Dynamic rank points are added when your search words match the words used in the link text (anchor text). A search hit in the link text is considered of high quality, because it contains words that other users have used to describe a certain page. For more see Change the authority (anchor text) weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

In addition, FAST Search Server 2010 for SharePoint will add static rank points based on the link structure, that is the number of links pointing to a site or an item. The static rank that is given by the created links in the SharePoint Server installation is handled through the static/quality rank components SiteRank and DocRank. The rank score is based on a link graph analysis of the indexed content. The idea is that the more links that point to a page or an item, the higher the rank score. For more on how to adjust these components, see Change the weight, remove or add a static rank component by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

Additional tips:

  • Remember that using hyperlinks affects the importance of pages.

  • When using links, use link text (anchor text) that gives a good description of the referred page and avoid using link text like click, here, and see.

  • For image links, ensure that you add relevant "alt" text in the link.

Add metadata to your content

Encourage all content providers to annotate content with meaningful metadata. Metadata is used by the search engine both for initial recall when you search and for refining search results by drilling down based on provided or extracted metadata. Metadata is either added to the document properties or updated directly in SharePoint Server for the document. The most important metadata are as follows:

Metadata | Description
Title | Used as a high quality part of information for search. When searching, a match in a title gives additional rank points compared to a match in the document body.
Dates | Used for freshness relevance. Equally important items are sorted by the age of the document.
Authors | Generally important for recall when you search on names, in addition to being an important document refiner.
Description/comments | Generally important for recall, but also important for presentation of the search result as it is used as a short description on the search result page.

All other metadata is also important for relevance and search recall and for use as custom refiners.

When importing a larger document set into the SharePoint Server installation, verify that metadata for the items such as title and author is correct and not a template title or author. Instead of using template values, you should remove these values during the import operation to prevent false rank and use of wrong values when presenting the search results.

Prevent indexing of irrelevant content

By excluding content that is of no interest for search, you prevent irrelevant content from appearing in the search result.

To prevent indexing of content on a SharePoint site, use the SharePoint Server feature Site Action> Site Settings > Search and Offline Availability and set Indexing Site Content to No.

If a Web Part contains different levels of security for example reference to items with both general and admin permissions, the content of the web part will not be available for search. This behavior can be changed in Site Action > Site Settings > Search and Offline Availability by changing the Indexing ASPX Page Content setting.

For web content outside SharePoint, you can use tags to prevent indexing of that content. User the tag <meta name="robots" content="noindex"/> to prevent indexing of a whole page, and the tag <div class ="noindex"> for excluding individual items. By using these tags, only relevant content will be indexed and used for search recall. Examples of using these tags could be for menu, footer, and header information that is repeated on several pages and not something that you want to search on. Common header/footer information is often set in the master page of the SharePoint Server Web Content Management (WCM) or in the page layout templates.

You can also exclude content of crawled properties from searching. For more, see Exclude crawled properties from searching by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

Encourage archiving and deleting old content

Archive or delete data that no longer has any interest to the organization. Keeping the SharePoint Server installation clean from historical data and junk that is of no interest is good for both the maintenance of SharePoint Server and for the search solution.

Review your security settings

FAST Search Server 2010 for SharePoint can index intranet content that is protected by ACLs, and will help ensure that users are only able to see results that they have permission to see. As long as the permission settings are correct, this will not cause any issues.

The main challenge when indexing many content repositories on your intranet is that any incorrect permission settings will be visible to the users, as the content is suddenly available via one query. Before indexing the content, these permission inconsistencies may not have been visible, as users may not have direct access to these repositories.


Security considerations for indexing (FAST Search Server 2010 for SharePoint)
See Also

Before reading this article, review the information in FAST Search Authorization (FSA) overview and FAST Search Authorization (FSA) data flow.

Most FAST Search Server 2010 for SharePoint systems include several content sources for searching, which may include both public documents and proprietary items. For example, you may have an internal human resources database that requires restricted access. Or you may have a products database that is available for public consumption. This article discusses the implementation and management of content security at the item, or document, level.

When planning for item level security, consider the following questions:

  • What content will be indexed?

  • Is the content public or proprietary?

  • What organizational security policies must be implemented?

  • What user and group permissions are required?

  • How does item level security affect the search experience?

Use the information in this article to make decisions about secure content:

  • About item level security

  • Gather information about your organization's security policies

  • Plan to index content sources

  • Plan to assign user and group permissions

  • Plan to index Lotus Notes databases

About item level security

When source content is crawled and indexed, authorization information is added to each item's managed properties (the item's ACL, or access control list), identifying users and groups that are granted or denied access to the item. Item ACLs help set up item level security by tagging each item with access permissions.

When a user submits a query and the indexes find search results, the query processing service rewrites the user's query so that the user only sees items that he is authorized to see. This security trimming is performed with the help of a user search security filter, which specifies content permissions for the user. By checking item ACLs against a user's search security filter, unsuitable search results are filtered out, narrowing query results to only those items that the user is allowed to view.

FAST Search Authorization (FSA) is the security module in FAST Search Server 2010 for SharePoint that enforces security trimming. FSA limits user access to items in search results that they are authorized to see. When you install FAST Search Server 2010 for SharePoint, FSA is installed automatically.

A user store is a logical grouping of users, groups, and content permissions that serves as a security gateway to a third-party content repository to help protect its content from unauthorized access. During installation, a default claims user store is created for Microsoft repositories (Outlook, Word documents, SharePoint databases, etc.). (Claims-based authentication is a set of operations that establishes trust relationships between claims providers and applications.) When you have content from several different providers, you may have multiple security user stores (for example, you may need a Lotus Notes user store to cache credentials for Lotus Notes users). To create a user search security filter, FSA consults the user stores to determine which groups a user belongs to.

Gather information about your organization's security policies

Work with colleagues who play a role in your organization's security team to understand security policies that relate to your content databases.

  • Is the content public?

  • Is the content proprietary?

  • Is there a subset of items that contain confidential content?

  • Should item level security be defined for access by specific users and groups?

  • Can you map the users/groups in content databases with Windows users/groups?

Consult your search administrator to determine what guidelines must be followed to implement item level security. See Security considerations for the server farm (FAST Search Server 2010 for SharePoint) for more security planning guidelines.

Plan to index content sources

Planning for item level security should be an important part of your content planning. As you consider which content sources will be crawled and indexed, consider the following:

  • What are the sources of the content repositories (Microsoft, Lotus Notes, external database, etc.)?

  • What repositories are public? Which are proprietary? Which include confidential items?

  • Which users and groups have permissions to access content source?

If you are not a subject matter expert, ask for help from content owners/administrators in your organization.

Caution:

Be careful not indexing new content sources before you have verified that the security permissions on the content repository are correct. This is particularly important for content sources that are not easily accessible via a browser, such as databases and file shares. Incorrect security permissions become very visible when the content has been indexed.

See Plan for crawling and federation (FAST Search Server 2010 for SharePoint) for planning guidance on identifying, crawling, and indexing content sources.

Plan to assign user and group permissions

Content permissions control access to your sites and indexed content. For Microsoft content, you will use Microsoft SharePoint Server 2010 groups, which control membership and fine-grained permissions to help provide item level security. See Plan security for sites and content (SharePoint Server 2010) for more information about the following considerations:

  • Planning site permissions

  • Determining permission levels and groups

  • Choosing security groups

  • Choosing administrators and owners

Each content source should have a user store created, to serve as a security gateway to the permissions on content in the different content sources. However, when you plan to index Microsoft content sources, you do not have to create a user store. By default, a claims user store is created and implemented for Microsoft content when you install FAST Search Server 2010 for SharePoint.

FSA generates user search security filters by accessing permissions in your user stores.

Plan to index Lotus Notes databases

If you plan to crawl and index Lotus Notes content with the FAST Search Lotus Notes connector, additional security measures (beyond those planned for indexing Microsoft content) must be considered:

  • To include Lotus Notes content in a FAST Search Server 2010 for SharePoint index, you must first create a Lotus Notes user store. See Prepare FAST Search Authorization for use with the FAST Search Lotus Notes connector for detailed information.

  • When you run the FAST Search Lotus Notes connector, plan to generate the ssomapping.xml. This file creates principal aliasing mapping for FSA between Windows user claim values and their equivalent IDs in the Lotus Notes system. Principal aliasing is required to create comprehensive and accurate user security filters. See Configure the FAST Search Lotus Notes connector for more information.

Note:

If you plan to use the Lotus Notes indexing connector, these planning guidelines do not apply. The Lotus Notes indexing connector maps Lotus Notes users/groups to Windows users/groups and feeds NTACL information.

See Also

Concepts

FAST Search Authorization (FSA) overview
FAST Search Authorization (FSA) data flow
About principal aliasing
Plan for crawling and federation (FAST Search Server 2010 for SharePoint)
Configure the FAST Search Lotus Notes connector
Prepare FAST Search Authorization for use with the FAST Search Lotus Notes connector

Other Resources

Plan security for sites and content (SharePoint Server 2010)
Plan authentication methods (SharePoint Server 2010)


Plan to monitor the health of the search system (FAST Search Server 2010 for SharePoint)
See Also

System Center Operations Manager (SCOM) 2007 R2 is recommended to monitor Microsoft FAST Search Server 2010 for SharePoint.

Microsoft System Center Operations Manager (SCOM) 2007 provides a comprehensive monitoring suite for monitoring and managing a complete range of IT infrastructure and applications that include, but are not limited to, FAST Search Server 2010 for SharePoint, Microsoft SQL Server 2008, Windows Server 2008, IIS 7.0, and RAID/SAN hardware. The management pack for FAST Search Server 2010 for SharePoint supports SCOM 2007 SP1 and SCOM 2007 R2.

If you do not use a SCOM 2007 solution, you can gather monitoring data from monitoring interfaces like the Windows event log, performance counters, Windows Management Instrumentation (WMI) and the SharePoint Server Unified Logging Service (ULS). These tools also provide information to monitor system performance and health.

Plan to deploy SCOM

This article contains recommendations for a basic SCOM 2007 server setup that is most suitable for testing. Consult the Operations Manager 2007 Deployment Guide for extensive information on how to deploy and configure SCOM 2007.

There are several considerations and pre-requisites that you should be aware of before you install SCOM 2007.

Service packs and hot fixes

When you use SCOM 2007 SP1, it is important that you apply hotfixes from http://support.microsoft.com/kb/953141. Install these hotfixes on all managed servers and SCOM 2007 servers that run on Windows Server 2008. Service Pack 2 is required for Windows Server 2008. Use Windows Update to apply the latest security updates and other important updates before installation.

SQL Server instance

SCOM 2007 requires a SQL Server instance. The database connection is the main performance bottleneck, so plan for deployment accordingly. Both SQL Server 2008 and SQL Server 2005 are supported for use with SCOM 2007 SP1. SQL Server Express is not a supported configuration.

SCOM 2007 and FAST Search Server 2010 for SharePoint co-existence

SCOM 2007 and FAST Search Server 2010 for SharePoint can co-exist on the same server. For production systems, one or more separate SCOM 2007 servers are recommended.

The SCOM 2007 server and managed servers must be in the same Active Directory domain if you are using the most basic Single Server, Single Management Group deployment scenario. For more advanced deployments, consult the Operations Manager 2007 Deployment Guide.

Refer to System Center Operations Manager TechCenter for more general information on SCOM 2007.

Important:

For testing or lab deployments of SCOM 2007, the built-in account Local System (NT AUTHORITY\SYSTEM) can be used for most services. This is not recommended for production environments. More flexible and secure deployment types use multiple domain user accounts and global security groups. For more information, see the article Security Considerations in the Operations Manager 2007 Deployment Guide.

See Also

Concepts

Deploy System Center Operations Manager (SCOM) 2007 (FAST Search Server 2010 for SharePoint)
Monitor FAST Search Server 2010 for SharePoint with SCOM


Plan search topology (FAST Search Server 2010 for SharePoint)
See Also

This section helps you plan the Microsoft FAST Search Server 2010 for SharePoint topology, including the integration with the Microsoft SharePoint Server farm.

In this section:

  • Topology Overview

  • Deployment Options

  • Plan search service applications (FAST Search Server 2010 for SharePoint)

  • Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)

  • Performance and capacity management (FAST Search Server 2010 for SharePoint)

  • Performance and capacity test results (FAST Search Server 2010 for SharePoint)

  • Performance and capacity recommendations (FAST Search Server 2010 for SharePoint)

  • System requirements (FAST Search Server 2010 for SharePoint)

  • Security considerations for the server farm (FAST Search Server 2010 for SharePoint)

Topology Overview

The following figure shows the FAST Search Server 2010 for SharePoint system as part of an overall Microsoft SharePoint Server deployment.

FAST Search Server in an overall SharePoint Server farm topology

FAST Search Server in an overall SharePoint Server

FAST Search Server 2010 for SharePoint provides enterprise search infrastructure for the SharePoint Server farm infrastructure. The search solution is hosted on two farm topologies:

  • SharePoint Server farm The parent SharePoint farm that hosts the query interfaces and content collection (crawling) interfaces for the search solution.

  • FAST Search Server 2010 for SharePoint farm The server farm infrastructure that provides the back-end indexing and query matching capabilities, and the administration of these features.

    For further details on this farm topology, see Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint).

    Important:

    In order to provide search services, the FAST Search Server 2010 for SharePoint farm must be connected to a parent SharePoint Server farm. For more information, see Plan search service applications (FAST Search Server 2010 for SharePoint).

The FAST Search Server 2010 for SharePoint farm must have access to a SQL Server host that is used to store configuration information. Note that FAST Search Server 2010 for SharePoint will not store any data related to the indexed content in the database. It is therefore recommended that you use a SQL Server database server already available in the parent SharePoint farm for this purpose.

Deployment Options

When you install FAST Search Server 2010 for SharePoint you must select between standalone deployment and multiple server installation. The main differences are indicated below.

Standalone deployment

A standalone deployment is useful when you want:

  • A small-scale deployment with minimal administrative overhead

  • A development or test environment

  • A deployment for product evaluation

Important:

Even for a standalone deployment, you have to deploy the Query SSA and Content SSA separately. These components must be installed on an application server within the associated SharePoint Server 2010 farm, and not on the server where you plan to install FAST Search Server 2010 for SharePoint.

It is possible to change a standalone deployment into a multiple server deployment at a later stage.

Multiple server deployment

A multiple server deployment is recommended for most production environments, except small deployments without any fault-tolerance requirements.

When performing a multiple server deployment, you first install FAST Search Server 2010 for SharePoint on all servers to be used in the deployment. After that you deploy the different components to the servers based on your selected topology. For more information, see Configure a stand-alone deployment or a multiple server deployment (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Plan for crawling and federation (FAST Search Server 2010 for SharePoint)


Plan search service applications (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

A Search Service Application (SSA) is a logical grouping of search service components within the Microsoft SharePoint Server 2010 farm that can be consumed by remote SharePoint farms. This means that remote farms may utilize search using an SSA on a parent farm.

In a Microsoft FAST Search Server 2010 for SharePoint deployment you will find two different SSAs on the parent SharePoint farm, a FAST Query Search Service Application (SSA) and a FAST Content Search Service Application (SSA). The parent SharePoint farm interfaces with the FAST Search Server 2010 for SharePoint farm through these two SSAs.

Important:

You must deploy the SSAs on different servers than FAST Search Server 2010 for SharePoint farm.

FAST Query Search Service Application (SSA)

This SSA provides the query-side integration between the FAST Search Server 2010 for SharePoint farm and the parent SharePoint farm.

The Query SSA provides the following functionality:

  • Serves as a gateway between a local or remote front-end Web server and the FAST Search Server 2010 for SharePoint farm

  • Contains query and crawl components for the people search functionality

Important:

People Search queries are evaluated within the Query SSA. All other queries are evaluated within the FAST Search for SharePoint farm.

You install and deploy the Query SSA on a SharePoint Application server within the parent SharePoint Server farm.

Important:

You must only deploy one Query SSA associated with your FAST Search Server 2010 for SharePoint farm. You can scale out the Query SSA across multiple servers for performance and failover.

For information about deployment and configuration of the FAST Query SSA, see Create and set up the Query Search Service Application (FAST Search Server 2010 for SharePoint).

FAST Content Search Service Application (SSA)

This SSA enables retrieving (crawling) content for indexing from content repositories. The Content SSA represents the default indexing connector for your FAST Search Server 2010 for SharePoint deployment.

The Content SSA can retrieve content from almost any content source, such as SharePoint farms, Web servers, file shares and Exchange public folders.

Important:

You must only deploy one Content SSA associated with your FAST Search Server 2010 for SharePoint farm. You can scale out the Content SSA across multiple servers for performance and failover.

For information about deployment and configuration of the FAST Content SSA, see Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint).

Important:

FAST Search Server 2010 for SharePoint provides a set of additional indexing connectors for advanced content retrieval use cases. For more details, see Plan for crawling and federation (FAST Search Server 2010 for SharePoint).

Differences from SharePoint Server Search

There are two alternatives for enterprise search in SharePoint Server 2010.

  • SharePoint Server 2010 search is an integrated search solution within the SharePoint Server 2010 farm, and is deployed as a single Search Service Application within the farm. In this case the query and crawl components are running within the same SSA.

    Note:

    If you use the single-server SharePoint Server 2010 installation option, SharePoint Server 2010 search will automatically be deployed on your farm. If you later deploy FAST Search Server 2010 for SharePoint, this SSA will continue to appear on the SharePoint Server 2010 Search Administration page, but is not used.

  • FAST Search Server 2010 for SharePoint replace the standard SharePoint Server 2010 search, and will appear on the SharePoint Server 2010 Search Administration page as two SSAs. In this case key parts of the search functionality take place in the FAST Search Server 2010 for SharePoint farm, but certain parts are performed in the Query SSA and content SSA as described earlier in this article.

See Also

Concepts

Plan search topology (FAST Search Server 2010 for SharePoint)
Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Performance and capacity planning (FAST Search Server 2010 for SharePoint)
System requirements (FAST Search Server 2010 for SharePoint)
Security considerations for the server farm (FAST Search Server 2010 for SharePoint)
Plan for crawling and federation (FAST Search Server 2010 for SharePoint)


Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
See Also

This topic describes the Microsoft FAST Search Server 2010 for SharePoint farm topology, including the various components that can be scaled out on multiple servers for performance and fault-tolerance reasons.

For more information on the overall Microsoft SharePoint Server 2010 farm topology, see Plan search topology (FAST Search Server 2010 for SharePoint).

In this article:

  • The content flow

  • Components within the FAST Search Server 2010 for SharePoint farm

The content flow

FAST Search Server 2010 for SharePoint retrieves content for indexing using one or more of the supported indexing connectors. The FAST Content Search Service Application (SSA) is the default indexing connector and retrieves content from various content sources such as SharePoint content repositories, Web servers, Exchange folders, line of business data and file servers. You can use other FAST Search Server 2010 for SharePoint indexing connectors for more specific content retrieval scenarios. For more information about the indexing connector options, see Plan for crawling and federation (FAST Search Server 2010 for SharePoint).

Item processing extracts searchable content from retrieved documents, and processes the items based on the written language.

The Indexing component converts the searchable content into inverted indexes that are in turn used by the query matching.

Query processing processes user queries by performing query transformations, such as synonym expansion, before the actual query matching against the index.

Query matching uses the search indexes to return items that match a user query. The items are returned in a query hit list that is sorted by the relevancy to the specified query.

FAST Search Server 2010 for SharePoint interacts with Active Directory and claims infrastructure to resolve permissions and group memberships. It then only returns items the current user is allowed to see, according to the settings of the content source.

Components within the FAST Search Server 2010 for SharePoint farm

FAST Search Server 2010 for SharePoint can run on a single node. Or, you can scale out to run one or more of the components on multiple nodes. In that case the system can index a larger number of items, handle more item updates, reduce indexing latency, or respond to more queries per second.

The following figure shows the main components of the FAST Search Server 2010 for SharePoint.

FAST Search for SharePoint farm topology

FAST Search Server Farm Topology

The following subsections describe the functionality for each component.

Item Processing

The item processing component receives items to be indexed from indexing connectors. and process the items according to the given configuration. It then sends the processed items to the indexing service.

Key features of the item processing service are:

  • Mapping from crawled properties to managed properties. Managed properties contain the content that will be indexed including metadata associated with the items.

    First, you will discover crawled properties on an initial set of crawled items. Based on this, you can change the mapping to managed properties.

  • Parsing of document formats such as Word, Excel and PDF. This includes extracting searchable text and metadata from these formats.

  • Extracting properties from crawled content. Property extraction detects various properties such as names and dates, and maps them into managed properties. In this manner you can query these properties, and also apply query refinement based on these properties. Key extracted properties are company names, people names, locations, and dates.

    It is also possible to create custom property extractors using, for example, a dictionary of product names relevant to your organization.

  • Linguistic processing of items before indexing. In search, linguistics is defined as the use of information about the structure and variation of languages so that users can more easily find relevant information. The item's relevancy with regard to a query is not necessarily decided based on words common to both query and document, but instead the extent that its content satisfies the user's need for information.

    The linguistic processing includes detection of the written language and linguistic normalization of the content according to the given language. Linguistic normalization includes character normalization and normalization of stemming variations.

FAST Search Server 2010 for SharePoint enables you to customize how items are processed—for example, by specifying what kinds of properties to extract and how they can be queried.

The content distributor communicates with the indexing connectors and organizes the feeding of documents from indexing connectors to the indexing service. You can set up a primary and a backup content distributor for fault-tolerance.

You can set up multiple item processing nodes for fault-tolerance and performance. Certain item processing operations are processing-intensive and will require more than one item processing node to handle the feeding rate.

Web link analysis (Web Analyzer)

The Web Analyzer has two main functions: It analyzes search clickthrough logs and hyperlink structures. Both contribute to better ranked search results.

Items that show many clicks in the search clickthrough log are popular and therefore receive better rank scores than less-viewed items. Items that are linked to from many other items are also perceived to be more relevant for the user and therefore receive better rank scores.

The Web Analyzer improves search relevancy by analyzing the link graph and adding anchor texts and a query independent rank boost based on link cardinality to the items in the index. Anchor texts describe the items they refer to and will improve recall and relevancy when a query term matches the anchor text. Items with many links pointing to them will be ranked higher

The Web Analyzer may scale out to many nodes to reduce the total time that is needed for the analysis. This is done by adding dedicated lookup database components and link processing components that are used during the link analysis.

The link processing component receives tasks from the Web Analyzer during link processing. Large scale installations use multiple link processors.

The lookup database component represents a key/value lookup server that retrieves the link information generated by the link processing. The item processing looks up the link information for an item using the URL as key. Large scale installations use multiple lookup database components.

Search Cluster

The search cluster provides the main topology for indexing and query matching. These components require their own scaling models using a matrix of servers in a row/column configuration. The following figure shows the key concepts used in a search cluster topology. The figure illustrates a deployment with four rows: two pure indexer rows and two pure search rows. In real deployments, it is common to combine indexer and search rows in order to reduce hardware footprint.

FAST Search Cluster Architecture

FAST Search Cluster Architecture

  • Index column The complete searchable index can be split into multiple disjoint index columns when the complete index is too large to reside on one server. A query will be evaluated against all index columns within the search cluster, and the results from each index column are merged into the final query hit list.

  • Search row A set of search nodes that contain all items indexed within the search cluster. A search row consists of one indexing node for each index column within the search cluster. You use multiple search rows to provide performance load sharing and fault-tolerance. Search rows can be combined with both types of indexer rows (see below).

  • Primary and backup indexer You can configure a backup indexer node for fault tolerance. The two indexers produce the same set of indexes, but only the primary indexer distributes the indexes to the search rows. Indexer rows cannot be combined on the same indexing nodes.

Important:

The primary and backup indexer nodes are specified as indexer rows in the deployment configuration file (deployment.xml). Search rows and indexer rows use the same row numbering in the deployment configuration file.

In the deployment configuration file (deployment.xml), a search row corresponds to searchcluster/row/search="true". An indexer row can have the roles primary, backup, and none. An indexer row can also be a search row. Searchcluster/row/index="none" is a pure search row. Searchcluster/row/search="false" is a pure indexer row.

Indexing

The indexing component creates inverted indexes, based on the items that it receives. The indexing component sends these inverted indexes to the query matching component for later use during query evaluation.

The indexing service consists of two components, the indexing dispatcher component and the indexing component. If the indexing service is deployed on multiple nodes, instances of these components will also be deployed on multiple nodes.

If you have more than one index column, you must combine the indexes to yield consistent search results. In this case, you have to deploy one indexing node for each index column. The indexing dispatcher manages the routing of processed items to the correct column.

The indexing service scales out according to the number of items. If the indexing service runs on a single node, both the number of items it can handle per second and the total number of items it can include in the index are limited. To scale out the indexing service, you can deploy it across more than one index column. Each index column will contain one part of the index, and the combined set of index columns will form the complete index. In this case, each indexing node will handle only a part of the whole index, and so that it scales out both the number of items that can be indexed per second and the total number of items. Additionally, backup indexing nodes can provide fault tolerance.

It is also possible to use multiple indexing dispatchers for both fault tolerance and performance reasons. You normally deploy the indexing dispatcher to the same node as the primary indexing node.

Query Matching

The query matching service uses the inverted indexes created by the indexing service to retrieve the items that match a query and then return these items as a query hit list. A query usually contains several terms combined with query operators, such as AND and OR. The query matching service looks up each term in the index and retrieves a list of items in which that term appears. In the case of an AND operator, for example, the query hit list will consist of the set of items that contain all the terms. The order of the returned items is based on the requested sorting mechanism, which is usually a complex ranking that is calculated from various item properties or a sort based on one or more of the item properties.

The query matching service can also return a hit highlighted summary for each item in the query hit list. A hit highlighted summary consists of a fragment of the original item in which the matching query terms are highlighted.

The query matching service is responsible for the deep refinement that is associated with query results. Query refinement enables drilling down into a query result by using aggregated statistical data that was computed for the query result. The query matching service maintains aggregation data structures to enable deep refinement across large result sets.

You can deploy the query matching service in a row/column setup to achieve fault-tolerance and scaling in content and query volume. Index columns provide ways to scale out for content volume, by partitioning the overall index into a set of disjoint columns. Search rows provide ways to scale out for query volume, by duplicating the same partition of the index across more than one query matching node.

Important:

The number of columns in the query matching service always equals the number of columns in the indexer service. The reason is that the index columns represent a partitioning of the index, and each query matching node can handle only one such partition of the index.

Search rows and indexer rows scale out independently. A search row duplicates another search row to provide fault tolerance and an increased capacity for queries. An indexer row serves as a backup mechanism for fault tolerance purposes during indexing.

Query Processing

The query processing component performs pre-processing of queries and post-processing of results. Query processing includes query-language parsing, linguistic processing, and item-level security processing. Result processing includes merging the results from multiple index columns, formatting the query hit list, formatting the query refinement data, and removing duplicates.

The query processing component interacts with the FAST Search Authorization (FSA) component to make sure that the user performing a query sees only the results that he or she is authorized to see. The query processing service therefore validates the user's permissions and rewrites the incoming query with an access filter that corresponds to the current user and group membership.

The query processing service can be scaled out across multiple nodes to handle fault-tolerance and more queries per second. In this case, all the nodes have to be set up in the same manner.

System Administration

The SharePoint Server 2010 Central Administration and site collection user interfaces provide the administrative interfaces for managing the FAST Search Server 2010 for SharePoint deployment and features. Common system administration services include UI and cmdlet based system and feature configuration, logging, index schema administration and search authorization.

Important:

Certain administrative operations can only be performed by using Windows PowerShell cmdlets or by using command line tools.

The administration component contains functionality to control the search experience, such as determining how to perform property extraction, ascertaining which synonyms to use, and determining which items to use as best bets.

The FSA manager is a part of the administration service that manages user authorization for indexed content. This ensures that only items that a user is entitled to read appear in the query results. The FSA manager communicates with Claims services, Active Directory services or other LDAP based directory services to manage the authorization process.

Index Schema administration

A key part of the administration service is the index schema administration. The index schema contains all the configuration entities that are needed to generate the configuration files that are related to the index schema for all the other services in the system.

The index schema controls which managed properties of an item will be indexed, how the properties will be indexed and which properties can be returned in the query hit list.

The rank profile is a part of the index schema that controls how the query hit list will be sorted by relevancy. You can configure the relevance calculation using a set of rank profile parameters.

FAST Search Authorization (FSA)

The FAST Search Authorization (FSA) manager is a part of the administration service that manages user authorization for indexed content.

The FSA manager grants users access to indexed items based on user's read permissions on the content source repositories. This ensures that only items that a user is entitled to read appear in the query results.

The FSA manager communicates with Active Directory services or other LDAP based directory services to manage the authorization process.

FAST Search Web Crawler

The FAST Search Web crawler is an optional indexing connector that can be used for complex Web crawl scenarios involving a mix of Internet and Intranet sites.

You can find more information about the differences between crawling Web sites with the FAST Search Content SSA and with the FAST Search Web crawler in Plan for crawling and federation (FAST Search Server 2010 for SharePoint).

The FAST Search Web crawler reads Web pages and follows links on the pages to process a complete Web of items. It then passes the retrieved items to the item processing service.

For further architecture details on the FAST Search Web crawler, see Crawling Web content with the FAST Search Web crawler.

See Also

Concepts

Plan search topology (FAST Search Server 2010 for SharePoint)
Plan for crawling and federation (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Performance and capacity planning (FAST Search Server 2010 for SharePoint)
The index schema and its features (FAST Search Server 2010 for SharePoint)
Performance and capacity test results (FAST Search Server 2010 for SharePoint)


Performance and capacity management (FAST Search Server 2010 for SharePoint)
See Also

Published: February 10, 2011

The articles in this section provide guidance on how to effectively manage performance and capacity in a Microsoft FAST Search Server 2010 for SharePoint farm deployment.

In this section:

  • Performance and capacity planning (FAST Search Server 2010 for SharePoint)

  • Performance and capacity testing (FAST Search Server 2010 for SharePoint)

  • Performance and capacity monitoring (FAST Search Server 2010 for SharePoint)

  • Performance and capacity tuning (FAST Search Server 2010 for SharePoint)

For more information about capacity management for Microsoft SharePoint Server 2010 in general, refer to Capacity management and sizing for SharePoint Server 2010.

For an overview of FAST Search Server 2010 for SharePoint as part of an overall SharePoint Server 2010 farm topology, refer to Plan search topology (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Performance and capacity test results (FAST Search Server 2010 for SharePoint)
Performance and capacity recommendations (FAST Search Server 2010 for SharePoint)


Performance and capacity planning (FAST Search Server 2010 for SharePoint)
See Also

Updated: August 13, 2013

When planning performance and capacity of a Microsoft FAST Search Server 2010 for SharePoint deployment, you must consider several aspects of both your own business environment and the system architecture.

  • Business environment considerations

  • System architecture considerations

For more information about how to plan the performance and capacity of a Microsoft SharePoint Server 2010 farm overall, refer to Capacity planning for SharePoint Server 2010.

Note:

This article assumes that you use the SharePoint Server 2010 crawler, indexing connector framework and the FAST Search Server 2010 for SharePoint Content Search Service Application (Content SSA) to crawl content.

Business environment considerations

In your environment, define the following:

Content volume capacity | How much content has to be searchable? The total number of items should include all objects: documents, web pages, list items, and so on.
Redundancy and availability | What are the redundancy and availability requirements? Do customers need a search solution which can survive the failure of a particular server? Note that there are two main levels of availability: Availability of query matching and availability of indexing.
Content Freshness | How "fresh" must the search results be? How long after the customer modifies the data do you expect searches to provide the updated content in the results? How often do you expect the content to change?
Query throughput | How many people will be searching over the content at the same time? This includes people typing in a query box and other hidden queries like web-parts automatically searching for data, or Microsoft Outlook 2010 Social Connectors requesting activity crawls that contain URLs which need security trimming from the search system.

System architecture considerations

In the system architecture perspective, make sure that you understand the indexing and query evaluation process in addition to the effect of various topology choices and the network traffic they generate. In addition, you may also want to pay attention to the dimensioning of the web analyzer component as the performance of this component depends on the number of indexed items and whether the items contain hyperlinks.

  • Impact of indexing on performance and capacity

  • Impact of query evaluation on performance and capacity

  • Impact of topology on performance and capacity

  • Impact of network traffic on performance and capacity

  • Impact of web link analysis on performance and capacity

Impact of indexing on performance and capacity

A FAST Search Server 2010 for SharePoint farm move through multiple stages as content is crawled: Index acquisition, index maintenance and index cleanup.

Index acquisition

The index acquisition stage is characterized by full, possibly concurrent, crawls.

When adding new content, crawling performance is mainly determined by the configured number of item processing components. Both the number of CPU cores and the speed of each of them will affect the results. As a first order approximation, a 1GHz CPU core will be able to process one average size Office document (around 250 kB) per second. For example, a scenario with 48 CPU cores for item processing, each being 2.26GHz, provides a total estimated throughput of 48 × 2.26 ≈ 100 items per second on average.

Note:

The actual throughput may deviate from the estimated throughput, depending on the size and type of the indexed items. Test your installation to ensure that it meets your performance and capacity expectations.

Index maintenance

The index maintenance stage is characterized by incremental crawls of all content, detecting new and changed content. Typically, when you crawl a SharePoint Server 2010 content source, most of the encountered changes are related to access rights.

Incremental crawls can consist of various operations:

  • Access right (ACL) changes and deletes: These require near zero item processing, but high processing load in the indexer. Crawl rates will be higher than for full crawls.

  • Content updates: These require full item processing in addition to more processing by the indexer compared to adding new content. Internally, such an update corresponds to a delete of the old item, and an addition of the new content.

  • Additions: Incremental crawls may also contain newly discovered items. These have the same workload as index acquisition crawls.

Depending on the kind of operation, an incremental crawl may be faster or slower than an initial full crawl. It will be faster if there are mainly ACL updates and deletes, and slower if there are mainly updated items.

In addition to updates from content sources, internal operations such as link analysis, click-through log analysis and reorganization of the index partitions, also alter the index. The FAST Search Server 2010 for SharePoint link analysis and click-through log analysis generate additional internal updates to the index. For example, a hyperlink in one item will lead to an update of the anchor text information associated with the referenced item. Such updates have a similar load pattern as the ACL updates. The indexer regularly performs internal reorganization of index partitions and data defragmentation. It starts defragmentation every night at 3 am and redistribution across partitions when it is needed. Both of these internal operations imply that you may see indexing activity also outside intervals with ongoing content crawls.

Index cleanup

The index cleanup stage occurs when you delete a content source or start address, or both, from a search service application. It can also occur when the content indexing connector cannot find a host supplying content: The indexing connector will look for the host during three consecutive crawls, but if the host is not found it will delete the content source and cause the index to enter the cleanup stage.

Impact of query evaluation on performance and capacity

The overall index has two levels of partitioning: index columns and index partitions.

When the complete index is too large to reside on one server, you can split it into multiple disjoint index columns. The query matching component will then evaluate the queries against all index columns within the search cluster, and merge the results from each index column into the final query hit list. Within each index column the indexer uses a partitioning of the index in order to handle large number of indexed items with low indexing and query latency. This partitioning is dynamic and handled internally on each index server. When the query matching component evaluates a query, each partition runs in a separate thread. The default number of partitions is 5. In order to handle more than 15 million items per server (column), you must configure a larger number of partitions.

Evaluation of a single query is schematically illustrated in the following figure.

Single query illustration

CPU processing (light blue) is followed by waiting for disk access cycles (white) and actual disk data read transfers (dark blue); repeated in the order of 2-10 times per query. This implies that the query latency depends on the speed of the CPU in addition to the I/O latency of the storage subsystem. The query matching component evaluates each single query separately, and in parallel, across multiple index partitions in all index columns. In the default five-partition configuration, this means that each query is evaluated in five separate threads within every column.

When query load increase, the query matching component evaluates multiple queries in parallel as indicated in the following figure.

Multiple queries illustration

As different phases of the query evaluation occurs at different times, concurrent I/O accesses are unlikely to become a bottleneck. CPU processing shows considerable overlap, which will be scheduled across the available CPU cores of the server. In all scenarios tested, the query throughput reaches its maximum when all available CPU cores are 100% utilized. This happens before the storage subsystem becomes saturated. More and faster CPU cores will increase the query throughput, and eventually make disk accesses the bottleneck. For more information about the tested scenarios, refer to Performance and capacity test results (FAST Search Server 2010 for SharePoint).

Note:

In larger deployments with many index columns the network traffic between query processing and query matching components may also become a bottleneck, and you might consider increasing the network bandwidth for this interface.

Query latency is somewhat independent of query load up to the CPU starvation point at maximum throughput. The latency for each query is a function of the number of items in the largest index partition. Note that when you apply query load after an idle period, query latency is slightly elevated because of caching effects. Ongoing crawl gives some degradation of query latency. But if you have a search row with backup indexer, the effect is much less than in systems with search running on the same servers as indexer and item processing.

Impact of topology on performance and capacity

As indexing and query matching both use CPU resources, deployments with indexing and query matching on the same row will show some degradation in query performance during content crawls. Single row deployments are likely to have indexing, query matching and item processing all running on the same servers.

You can deploy a dedicated search row to isolate query traffic from indexing and item processing. This requires two times the number of servers in the search cluster, which provides better and more consistent query performance. Such a configuration will also provide query processing and query matching redundancy. A dedicated search row implies some additional traffic during crawling and indexing when the indexer creates a new index generation (a new version of the index for a given partition). The indexer component passes the new index data over the network to the query matching component. Given a suitable storage subsystem, the main effect on query performance is a small degradation when new generations arrive because of cache invalidation.

You can also deploy a backup indexer in order to handle non-recoverable errors on the primary indexer. You will typically co-locate the backup indexer with a search row. For this scenario, you should not deploy item processing to the combined backup indexer and search row. The backup indexer increase the I/O load on the search row because there will be additional housekeeping communication between the primary and backup indexer to keep the index data on the two servers in sync. This also includes additional data storage on disk for both servers. Make sure that you dimension your storage subsystem to handle the additional load.

Impact of network traffic on performance and capacity

With increased CPU performance on the individual servers, the network connection between the servers can become a bottleneck. As an example, even a small FAST Search Server 2010 for SharePoint farm with four servers can process and index more than 100 items per second. If the average item is 250 Kbytes, this represents around 250 Mbit/s average network traffic. Such a load may saturate even a 1Gbit/s network connection.

The network traffic generated by content crawling and indexing can be decomposed as follows:

  • The indexing connector within the Content SSA retrieves the content from the source

  • The Content SSA (within the SharePoint Server 2010 farm) passes the retrieved items in batches to the content distributor component in the FAST Search Server 2010 for SharePoint farm

  • The content distributor sends each item batch to an available item processing component, typically located on another server

  • After processing, the item processing component passes each batch to the indexing dispatcher that splits the batches according to the index column distribution

  • The indexing dispatcher distributes the processed items to the indexers of each index column

  • If there are multiple search rows, the indexer copies the binary index to the additional search rows

The accumulated network traffic across all servers and components can be more than five times higher than the content stream itself in a distributed system. You must have a high performance network switch to interconnect the servers in such a deployment.

High query throughput also generates high network traffic, especially when you are using multiple index columns. Make sure that you define the deployment configuration and network configuration to avoid too much overlap between network traffic from queries and network traffic from content crawling and indexing.

Impact of web link analysis on performance and capacity

Performance dimensioning of the web analyzer component depends on the number of indexed items and whether the items contain hyperlinks. Items that contain hyperlinks, or items that are linked to, represents the main load. Database-type content does typically not contain hyperlinks. Intranet content (including SharePoint site content) will often contain HTML with hyperlinks. External web content is almost exclusively HTML documents that have many hyperlinks.

Although the number of CPU cores is important for performance dimensioning of the web analyzer component, determining the time that it must have to update the index with anchor text and rank data, disk space is the most important. The web analyzer component will only perform link, anchor text or click through log analysis if sufficient disk space is available. The following table specifies rule-of-thumb dimensioning recommendations for the web analyzer component.

Content type | Number of items per CPU core | GB disk per million items
Database | 20 million | 2
SharePoint Server 2010 / Intranet | 10 million | 6
Public web content | 5 million | 25
Note:

The table provides dimensioning rules for the whole farm. If you distribute the web analyzer component over two servers the requirement per server will be half of the given values.

The amount of memory that you must have available is the same for all kinds of content, but depends on the number of cores you use. We recommend 30 MBytes per million items plus 300 MBytes per CPU core. If the installation contains different kinds of content, the best capacity planning strategy is to use the most demanding content type as the basis for the dimensioning. For example, if the system contains a mix of database and SharePoint site content we recommend that you dimension the system as if it only contains SharePoint site content.

See Also

Concepts

Performance and capacity management (FAST Search Server 2010 for SharePoint)
Performance and capacity testing (FAST Search Server 2010 for SharePoint)
Performance and capacity monitoring (FAST Search Server 2010 for SharePoint)
Performance and capacity tuning (FAST Search Server 2010 for SharePoint)

Change History

Date | Description
August 13, 2013 | Emphasized that for crawling performance the actual throughput may deviate from the estimated throughput.
February 10, 2011 | Initial publication


Performance and capacity testing (FAST Search Server 2010 for SharePoint)
See Also

Published: February 10, 2011

When testing the performance and capacity of a Microsoft FAST Search Server 2010 for SharePoint deployment, consider the following:

  • Understand how to measure for your testing goal

    If you are measuring for search performance then you will want to measure crawl time, document indexing rates and query matching rates.

  • Identify your test content

    Determine the test content sources that you want to crawl, and verify that you have the required user account and network access to crawl them.

    If you create specific test content, you must ensure that you use unique content. A common mistake is to upload the same document, maybe hundreds or even thousands of times, to different document libraries with different names. That can affect search performance because the query processor will spend time doing duplicate detection that it would not otherwise have to in a production environment that uses real content.

  • Identify a suitable set of sample search keywords and phrases

For more information about how to test performance and capacity of Microsoft SharePoint Server 2010 in general, refer to Performance testing for SharePoint Server 2010.

FAST Search Server 2010 for SharePoint uses the storage subsystem extensively. You can test the raw I/O performance to get an early verification of having sufficient performance. Refer to the following section for more information.

Analyzing raw I/O performance

You can analyze raw I/O performance using a tool named SQLIO (http://www.microsoft.com/downloads/details.aspx?familyid=9a8b005b-84e4-4f24-8d65-cb53442d9e19).

After you have installed SQLIO, the first step is to obtain or generate a suitable test file. The following tests include write operations. Therefore, the content of this file will be partly overwritten. The size of the file should be much larger than the available system memory (by a factor of 10) to avoid most caching effects. The test file can also be generated by SQLIO itself, although not directly for very large file sizes. We recommend that you generate a 1 GB file by using the command "sqlio.exe -t32 -s1 -b256 1g" which will create a file that is named "1g" in the current directory. You can then concatenate this file to a sufficiently large file like 256GB, by the command "copy 1g+1g+1g+…..+1g testfile". To ensure that caching during the test file preparation do not skew the results, you should restart the server before continuing with the tests.

The following set of commands is representative for the most performance critical disk operations in FAST Search Server 2010 for SharePoint:

Test number | Scope | Command
1 | 1kB read [IOPS] | sqlio.exe -kR -t4 -o25 -b1 -frandom -s300 testfile
2 | 32kB read [IOPS] | sqlio.exe -kR -t4 -o25 -b32 -frandom -s300 testfile
3 | 32kB write [IOPS] | sqlio.exe -kW -t4 -o25 -b32 -frandom -s300 testfile
4 | 100MB read [MB/s] | sqlio.exe -kR -t1 -o1 -b100000 -frandom -s300 testfile
5 | 100MB write [MB/s] | sqlio.exe -kW -t1 -o1 -b100000 -frandom -s300 testfile

All the commands assume that a file "testfile" exists in the current directory, which you should locate on the disk that will host FAST Search Server 2010 for SharePoint. Each test runs for 300 seconds. The first test measures the maximum number of I/O operations per second for small read transfers. The second and third tests measure the performance for medium sized random accesses. The two last tests measures read and write throughput for large transfers. The following table shows some example results, with minimum recommendations during ordinary operation in its topmost row.

Disk layout | Test number 1 | Test number 2 | Test number 3 | Test number 4 | Test number 5
Recommended minimum | 2000 | 1800 | 900 | 500 | 250
16x SAS 10k RPM 2.5" drives RAID50 in two parity groups | 2952 | 2342 | 959 | 568 | 277
22x SAS 10k RPM 2.5" drives RAID50 in two parity groups | 4326 | 3587 | 1638 | 1359 | 266
22x SAS 10k RPM 2.5" drives RAID50 in two parity groups with drive failure | 3144 | 2588 | 1155 | 770 | 257
12x SAS 7200 RPM 3.5" drives RAID50 in two parity groups | 1844 | 1315 | 518 | 677 | 780
12x SAS 7200 RPM 3.5" drives RAID50 in two parity groups with drive failure | 1424 | 982 | 531 | 220 | 477
12x SAS 7200 RPM 3.5" drives RAID10 | 1682 | 1134 | 1169 | 762 | 692
12x SAS 7200 RPM 3.5" drives RAID10 with drive failure | 1431 | 925 | 1154 | 213 | 220
12x SAS 15k RPM 3.5" drives RAID50 in two parity groups | 4533 | 3665 | 848 | 501 | 235
2x ZeusIOPS 400GB MLC 2.5" drives RAID0 | 52709 | 14253 | 27171 | 360 | 122
1x ioDrive 640GB MLC | 83545 | 21875 | 17687 | 676 | 533
1x ioDrive Duo 1280GB MLC | 160663 | 42647 | 32574 | 1309 | 664
3x ioDrive Duo 1280GB MLC RAID0 | 162317 | 83661 | 44420 | 2382 | 1412
3x ioDrive Duo 1280GB MLC RAID0 Non-default option: 4kB block size | 1815932 | 86396 | 47423 | 2340 | 1631
3x ioDrive Duo 1280GB MLC RAID5 | 188284 | 87270 | 11800 | 2459 | 545
3x ioDrive Duo 1280GB MLC RAID5 with card failure | 126469 | 48564 | 10961 | 716 | 202

1This is the average IOPS over the standard 300 second test period. It although starts out as ~3500 IOPS, degrading to a sustained ~1700 IOPS after 3-4 minutes.

2Tested with 4kB block reads due to the different block size formatting.

Note:

These results are to a large degree dependent on the disk controller and spindles used. In addition, the numbers in the table reflect a deployment where the disk subsystem is at least 50% utilized in capacity before you add the test file. Note that if you test on empty disks you can get elevated results as the test file is then put in the most optimal tracks across all spindles (short-stroking). This can give 2-3x increased performance. Finally, notice that the numbers in rows highlighted with bold are measured with a forced drive failure.

RAID50 provides better performance during ordinary operation than RAID10 for most tests apart from small writes. RAID10 has less decrease in performance if a drive should fail. We recommend that you use RAID50 for most deployments, as 32kB writes is the least important of the five tests. RAID50 provides near two times the storage capacity compared to RAID10 on the same number of disks. If you deploy a backup indexer, 32kB writes are more frequent. This is because a large amount of pre-index storage files (FiXML) are passed from the primary to the backup indexer component. This may in certain cases lead to a performance improvement by using RAID10.

See Also

Concepts

Performance and capacity management (FAST Search Server 2010 for SharePoint)
Performance and capacity planning (FAST Search Server 2010 for SharePoint)
Performance and capacity monitoring (FAST Search Server 2010 for SharePoint)
Performance and capacity tuning (FAST Search Server 2010 for SharePoint)


Performance and capacity monitoring (FAST Search Server 2010 for SharePoint)
See Also

Published: February 10, 2011

When monitoring the performance and capacity of a Microsoft FAST Search Server 2010 for SharePoint deployment, there are two main areas of analysis:

  • Crawling and indexing performance analysis

  • Query performance analysis

For more information about how to monitor a FAST Search Server 2010 for SharePoint farm in general, refer to Monitor FAST Search Server 2010 for SharePoint. For more information about how to monitor a Microsoft SharePoint Server 2010 farm overall, refer to Monitoring and maintaining SharePoint Server 2010.

For an overview of the relevant performance counters for FAST Search Server 2010 for SharePoint, refer to Performance counters (FAST Search Server 2010 for SharePoint).

Note:

This article assumes that you use the SharePoint Server 2010 crawler, indexing connector framework and the FAST Search Server 2010 for SharePoint Content Search Service Application (Content SSA) to crawl content.

Crawling and indexing performance analysis

Content processing chain

The item processing chain in FAST Search Server 2010 for SharePoint consists of the following components that may run on separate servers:

  • Crawler(s) Any component pushing content into FAST Search Server 2010 for SharePoint

  • Content distributor(s) Receives content in batches and distributes the batches to item processing and indexing

  • Document processors (item processing) Converts items to a unified internal format

  • Indexing dispatcher(s) Distributes processed item batches to index columns

  • Primary indexer Generates the searchable index

  • Backup indexer Persists a backup of the information in the primary indexer

Content flows as indicated by arrows 1-5 in the item processing chain figure. The last flow from primary to backup indexer is an optional deployment choice. Asynchronous callbacks for completed processing are propagating in the other direction as indicated by arrows 6 through 9. Crawlers throttle the crawl rate based on the callbacks (9) it receives for document batches (1). The slowest component in this chain determines the overall crawl rate.

Each FAST Search Server 2010 for SharePoint farm has one or more content distributors. These components receive all content in batches, which they pass on to the item processing components. You can ensure good performance by verifying that the following conditions are true:

  • Item processing components are effectively utilized

  • Incoming content batches are quickly distributed for processing

You can only achieve maximum throughput when the Content SSA has a constant queue of batches to submit. Each item processing component will use 100% of a CPU core when busy. You can scale item processing components up to one per CPU core.

Indexers are the most write intensive component in a FAST Search Server 2010 for SharePoint installation, and you must ensure that you have high disk performance. High indexing activity can also affect query matching operations when it runs on the same row. Indexers distribute the items across several partitions. Partition 0, and up to three of the other partitions, can have ongoing activity at the same time. During redistribution of items among partitions, one or more partitions might be waiting for other partitions to reach a specific checkpoint.

Tip:

Analysis of crawling and indexing performance can be done through several tools; for example the "Performance monitor" of Windows Server 2008 [R2], or on Systems Center Operations Manager (SCOM). In addition, you can retrieve the indexer status using the indexerinfo command, for example indexerinfo -a status.

Crawling performance counters

The following table shows the most important performance counters for the Content SSA. Note that you find these performance counters on the server(s) hosting the Content SSA crawl components, under "OSS Search FAST Content Plugin", and not in the FAST Search Server 2010 for SharePoint farm.

Performance counter | Description
Batches ready | The number of batches that the Content SSA has retrieved from the content sources, and that are ready for passing on to the content distributor. When zero, the FAST Search Server 2010 for SharePoint farm back-end is processing content faster than the Content SSA can crawl.
Batches submitted | The number of batches that the Content SSA has sent to FAST Search Server 2010 for SharePoint, and for which a callback is still pending. When zero, nothing was sent to the FAST Search Server 2010 for SharePoint farm back-end for processing.
Batches open | The total number of batches in some stage of processing.
Items Total | The total number of items crawled by the Content SSA since last service restart.
Available Mbytes | The total amount of available memory on the computer. By default, the Content SSA will stop aggregating batches ready when 80% system memory is used
Processor time | Overall CPU usage on the computer. High CPU load could limit the throughput of the Content SSA.
Bytes Total/sec | Overall network usage on the computer. High network load might become a bottleneck for the rate of data that the Content SSA can crawl and push to the FAST Search Server 2010 for SharePoint farm.

Content distributors and item processing performance counters

The following table shows the performance counters for the content distributors and item processing. For a total overview of the system, you must sum up these performance counters across all content distributors.

Performance counter | Description
Document processors | The number of item processing components registered with each content distributor. When having multiple content distributors, the item processing components will be evenly distributed across the content distributors.
Document processors busy | The number of item processing components that are currently working on a content batch. This should be close to the total number of item processing components under maximum load.
Average dispatch time | The time that the content distributor uses to send a batch to an item processing component. This should be less than 10ms. Higher values indicate a congested network.
Average processing time | The time that an item processing component uses to process a batch. This time can vary depending on content types and batch sizes, but would typically be less than 60 seconds.
Available Mbytes | The total amount of available memory on the computer. Each item processing component might need up to 2GB of memory. Memory starvation will affect processing throughput.
Processor time | Overall CPU usage on the computer. Item processing components are CPU intensive. Typically, the CPU utilization is high during crawls but item processing has reduced priority and will yield CPU resources to other components when it is required.
Bytes Total/sec | Overall network usage on the computer. High network load might become a bottleneck for the rate of data that can be processed by the FAST Search Server 2010 for SharePoint servers.

Indexing dispatcher and indexers performance counters

The following table shows the most important performance counters for the indexing dispatcher and indexers.

Performance counter | Description
Current queue size | Indexers queue incoming work under high load. This is common, especially for partial updates. If API queues never (intermittently) reaches zero, the indexer is the bottleneck. The crawler will pause when the queue reaches 256MB in one of the indexers. This can occur if the storage subsystem is not sufficiently powerful. It will also occur during large redistribution of content between partitions, which temporarily blocks more content from being indexed.
FiXML fill rate | FiXML files (internal item storage in the indexers) are compacted regularly, by default between 3am and 5am every night. Low FiXML fill rate (<70%) will lead to inefficient operation.
Active documents | Partitions 0 and 1 should have less than 1 million items each, preferably even less in order to keep indexing latency low. In periods with high item throughput the indexing latency will be higher because partition 0 and 1 will be larger as this is more optimal for overall throughput. The query matching component will automatically rearrange the items into the higher numbered partitions during periods with lighter load.
% Idle Time | Low disk idle time suggest a saturated storage subsystem.
% Free space | Indexers need space for both the index generation currently used by the query matching component and new index generations that are under processing. On a fully loaded system, disk usage will vary between 40% and near 100% for the same number of items, depending on the status of the indexer.

Query performance analysis

SharePoint Server 2010 administrative reports provide useful statistics for query performance from an end-to-end perspective. These reports are effective for tracing trends over time and identifying where you should investigate when performance is not optimal.

Query latency can occur in three different areas: Web server rendering, Query SSA processing and back-end processing. In general, latency due to web server page rendering and Query SSA latency occur on the servers that are running SharePoint Server 2010 (Query SSA). These latencies are also dependent on the performance of the SQL server(s) backing the SharePoint Server 2010 installation. The back-end latency is within the FAST Search Server 2010 for SharePoint servers:

  • Query processing

  • Query matching

Queries are sent from the Query SSA to the FAST Search Server 2010 for SharePoint query processing component ("query" in the deployment file). In the reports, query processing appears as "QRproxy", "QRserver" and "Fdispatch". The Query SSA and the query processing component are likely to represent a bottleneck. Any difference between the two is because of communication delays or query processing.

The query dispatcher (known as "Fdispatch" in the reports) distributes queries across index columns. There is also a query dispatcher located on each query matching server, distributing queries across index partitions. Both query dispatchers may be a bottleneck when there are large amounts of data in the query results. This leads to network saturation. We recommend that you use a separate network switch for the communication between the query processing and query matching components.

The query matching component (known as "Fsearch" in the reports) is responsible for performing the actual matching of queries against the index, computing query relevancy and performing deep refinement. For each query, it reads the required information from the indices generated by the indexer. Information that is likely to be reused will be kept in a memory cache. Good query matching performance relies on a powerful CPU and low latency from small random disk reads (typically 16-64 kB).

Query processing performance counters

The following table shows the performance counters that can be helpful for correlating the back-end latency reported by the Query SSA and the query processing component.

Performance counter | Description
# Queries/sec | Current number of queries per second
# Requests/sec | Current number of requests per second. In addition to the query load, one internal request is received every second to check that QRserver is alive.
Average queries per minute | Average query load
Average latency last - ms | Average query latency
Peak queries per sec | Peak query load

Query matching performance counters

The following table shows the performance counters that can be helpful for analyzing a server that runs the query matching component.

Performance counter | Description
% Idle Time | Low disk idle time suggest a saturated storage subsystem.
Avg. Disk sec/Read | Each query will need a series of disk reads. An average read latency of less than 10 ms is desirable.
Avg. Disk Read Queue Length | On a saturated disk subsystem, read queues will build up. Queues will affect query latency. An average queue length smaller than 1 is desirable for any server that runs query components. This will typically be exceeded in single row deployments during indexing, adversely affecting search performance.
Processor time | CPU utilization is likely to become the bottleneck for high query throughput. When query matching has high processor time (near 100%), query throughput will be unable to increase more.

See Also

Concepts

Performance and capacity management (FAST Search Server 2010 for SharePoint)
Performance and capacity planning (FAST Search Server 2010 for SharePoint)
Performance and capacity testing (FAST Search Server 2010 for SharePoint)
Performance and capacity tuning (FAST Search Server 2010 for SharePoint)


Performance and capacity tuning (FAST Search Server 2010 for SharePoint)
See Also

Published: February 10, 2011

This article describes various methods of tuning your Microsoft FAST Search Server 2010 for SharePoint farm to achieve extended performance and capacity targets.

  • Crawler capacity tuning

  • Index capacity tuning

Caution:

We recommend that you only use these methods when you fulfill the conditions specified for each tuning option. Ensure also that your deployment has no other performance and capacity issues before tuning it. Refer to Performance and capacity monitoring (FAST Search Server 2010 for SharePoint) for more information.

Note:

This article assumes that you use the SharePoint Server 2010 crawler, indexing connector framework and the FAST Search Server 2010 for SharePoint Content Search Service Application (Content SSA) to crawl content.

Crawler capacity tuning

Only tune the crawler capacity if your deployment:

  • Doesn't have a backup indexer row but more than 100 item processing component instances deployed per crawl component in the Content SSA

  • Does have a backup indexer row and more than 50 item processing component instances deployed per crawl component in the Content SSA

  • Does have more than 3 index columns per crawl component in the Content SSA

For large deployments, you must dimension the number of crawl components within the Content SSA to avoid network bottlenecks. Note that this scaling will often eliminate the need for additional configuration tuning. But for deployments with backup indexer rows, the batches submitted by the Content SSA typically exceed the number of item processing components. These "additional" batches are content that is processed but not yet been persisted in both indexer rows. By default, the Content SSA will throttle crawls in order to avoid more than 100 "batches submitted". For large installations, you should consider adjusting the throttling limits to allow for more batches to be processed at the same time.

You can adjust the throttling limits by using the MaxSubmittedBatches and MaxSubmittedPUDocs properties. The default value is 100 for MaxSubmittedBatches and 1000 for MaxSubmittedPUDocs. The throttling limits apply for each crawl component within the Content SSA. If you use two crawl components, the maximum total number of batches submitted will be two times the configured value. Use the following formula to calculate the new throttling limits:

Tuning crawler processing formula

For example, a scenario with a=1, b=48, c=3, s=2 will result in MaxSubmittedBatches = 54 and MaxSubmittedPUDocs = 5400. The default value for MaxSubmittedBatches is not altered in this case. If the crawling and indexing performance is limited by a high rate of ACL changes, you can increase MaxSubmittedPUDocs (the maximum number of items that have ACL changes submitted).

The MaxSubmittedBatches and MaxSubmittedPUDocs properties are configurable through the Microsoft SharePoint Server 2010 Management Shell on the farm hosting the Content SSA. The following commands set the default values:

Copy Code
$ssa = Get-SPEnterpriseSearchServiceApplication -Identity <nameOfContent SSA>
$ssa.ExtendedConnectorProperties["MaxSubmittedBatches"] = 100
$ssa.ExtendedConnectorProperties["MaxSubmittedPUDocs"] = 1000
$ssa.Update()

where the <nameOfContentSSA> is the name of your Content SSA.

Note:

Increasing the throttling limits will increase the load on the item processing and indexing components. When these consume more of the farm resources, it affects query performance. This is less of an issue when you are running without a dedicated search row. Increasing MaxSubmittedPUDocs will increase the I/O load on primary and backup indexers.

Index capacity tuning

Only tune the index capacity if your deployment:

  • Has high content volumes where the number of content changes over time is low

  • Has low query throughput requirements (maximum 5-10 queries per second, depending on CPU performance of the servers)

  • Doesn't require low indexing latency (there will be a longer delay from an item is updated until it is searchable)

FAST Search Server 2010 for SharePoint has a default configuration that is optimized for handling up to 15 million items per index column, with a hard limit of 30 million items. For scenarios with low content update rates and moderate query volume, you may want to index more items per column in order to save hardware cost. You can configure the FAST Search Server 2010 for SharePoint indexer to handle up to 40 million items per column. This is known as extended capacity configuration. The extended content capacity configuration has more index partitions within each server. In this manner, you can maintain low query latency at the expense of reduced maximum QPS.

There are some tradeoffs by extending the capacity: Query throughput (QPS) is reduced. Query latency (while not exceeding the throughput limitation) is less affected. You can compensate for query throughput reductions with multiple search rows, but then the reduction in server count is diminishing. Indexing will require more resources, and also more disk accesses. More items per column require more storage space per server. The total storage space across the farm is mainly the same. There are fewer servers for distributing item processing components. Initial crawl rate will be reduced, as the crawl rate is mainly dependent on the number of available CPU cores. Incremental crawls will also have lower throughput as each index column has more work. You can temporarily speed up initial pre-production bulk crawls by adding item processing components to eventual search rows or additional servers temporarily assigned to the cluster. Finally, you must have more hardware resources per server. We recommend that you do not use the extended settings on a server that has less than 16 CPU cores/threads (it should be 24 or more). We recommend 48 GB RAM and a high-performance storage subsystem.

Note:

When estimating the change rate that a farm must be able to consume, remember that any content change implies a load to the system. This includes ACL changes. ACL changes may appear for many items at a time when there are permission changes to document libraries or sites. This results in high peak update rates.

Important:

To configure the indexer, you must modify a configuration file that also contains several other important parameters for the indexer. This configuration file may be changed by a future software update. You must re-apply the configuration change after the software update.

Refer to Extend content capacity for the indexer (FAST Search Server 2010 for SharePoint) for the operational details.

See Also

Concepts

Performance and capacity management (FAST Search Server 2010 for SharePoint)
Performance and capacity planning (FAST Search Server 2010 for SharePoint)
Performance and capacity testing (FAST Search Server 2010 for SharePoint)
Performance and capacity monitoring (FAST Search Server 2010 for SharePoint)


Performance and capacity test results (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

The articles in this section provide the test results for typical deployments of Microsoft FAST Search Server 2010 for SharePoint. The articles include the following information for each sample deployment:

  • Test environment specifications, such as hardware, farm topology and configuration

  • Workload specifications that include the number and kind of users and farm usage characteristics

  • Test farm dataset specifications that include search indexes and external data sources

  • Test environment health and performance data

In this section:

Test scenario | Description | Link
Extra-small | A farm with up to 8 million items | Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint)
Medium | A farm with up to 40 million items | Test results: Medium scenario (FAST Search Server 2010 for SharePoint)
Large | A farm with up to 100 million items | Test results: Large scenario (FAST Search Server 2010 for SharePoint)

For more information about Microsoft SharePoint Server 2010 test results in general, refer to Performance and capacity test results and recommendations (SharePoint Server 2010)

For an overview of FAST Search Server 2010 for SharePoint as part of an overall SharePoint Server 2010 farm topology, refer to Plan search topology (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Performance and capacity management (FAST Search Server 2010 for SharePoint)
Performance and capacity recommendations (FAST Search Server 2010 for SharePoint)


Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

With the extra-small Microsoft FAST Search Server 2010 for SharePoint test scenario we targeted a small test corpus with high query rates. The scenario had no off-business hours with reduced query load, and crawls were likely to occur at any point in time. The amount of content for the scenario was up to 8 million items. We measured query performance at 1M, 5M and 8M content volume.

We set up the parent Microsoft SharePoint Server 2010 farm with four front-end web servers, one application server and one database server, and arranged them as follows:

  • We used the SharePoint Server 2010 crawler, indexing connector framework and the FAST Content Search Service Application (Content SSA) to crawl content, and the crawl component of the Content SSA was running on the application server

  • The application server also hosted the Central Administration for the farm

  • The database server hosted the crawl databases, the FAST Search Server 2010 for SharePoint administration databases and the other SharePoint Server 2010 databases

We used no separate data storage because the application server and web front-end servers only needed space for the operating system, application binaries and log files.

In this article:

  • Test deployments

  • Test characteristics

  • Test results

Test deployments

Within the extra-small scenario, we tested the following FAST Search Server 2010 for SharePoint deployments:

Name | Description
XS1 | A stand-alone server that hosted all the FAST Search Server 2010 for SharePoint components using regular disk drives
XS2 | Same as XS1, but deployed on a virtual (HyperV) machine
XS3 | Multi-node deployment using four virtual computers that were running on the same physical server
XS4 | Same as XS1, with the addition of a dedicated search row
XS5 | Same as XS1, but with storage on SAS SSD drives

Test characteristics

This section provides detailed information about the hardware, software, topology and configuration of the test environment.

Hardware/Software

We tested all the specified deployments on similar hardware and software. The main difference was that we used virtualization for some setups and solid state disk (SSD) storage for other setups.

FAST Search Server 2010 for SharePoint servers

  • Windows Server 2008 R2 x64 Enterprise Edition

  • 2x Intel L5520 CPUs with Hyper-threading and Turbo Boost switched on

  • 24 GB memory

  • 1 Gbit/s network card

  • Storage subsystem:

    • OS: 2x 146GB 10k RPM SAS disks in RAID1

    • Application: 7x 146 GB 10k RPM SAS disks in RAID5. Total formatted capacity of 880 GB.

    • Disk controller: HP Smart Array P410, firmware 3.30

    • Disks: HP DG0146FARVU, firmware HPD6

Variations:

Deployment Variation description

XS2/XS3

Virtualized servers running under Hyper-V:

  • 4 CPU cores

  • 8 GB memory

  • 800 GB disk on servers with index component

XS5

Storage subsystem:

  • Application: 2x 400 GB SSD disks in RAID0. Total formatted capacity of 800 GB.

  • SSD disks: Stec ZeusIOPS MLC Gen3, part Z16IZF2D-400UCM-MSF

SharePoint Server 2010 servers:

  • Windows Server 2008 R2 x64 Enterprise edition

  • 2x Intel L5420 CPUs

  • 16 GB memory

  • 1 Gbit/s network card

  • Storage subsystem for OS/Programs: 2x 146GB 10k RPM SAS disks in RAID1

SQL servers:

Same specification as for SharePoint Server 2010 servers, but with additional disk RAID for SQL data with 6x 146GB 10k RPM SAS disks in RAID5.

Topology

This section provides the deployment files (deployment.xml) we used to set up the test deployments.

XS1

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="XS1"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>XS1</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=XS1.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <content-distributor />
      <indexing-dispatcher />
      <searchengine row="0" column="0" />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="12" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
   </searchcluster>

</deployment>

XS2

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="XS2"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>XS2</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=XS2.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <content-distributor />
      <indexing-dispatcher />
      <searchengine row="0" column="0" />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="1"/>
      <document-processor processes="4" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
   </searchcluster>

</deployment>

Note that this is the default deployment.xml file.

XS3

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="XS3"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>XS3</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=XS3.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <document-processor processes="4" />
   </host>

   <host name="fs4sp2.contoso.com">
      <indexing-dispatcher />
      <searchengine row="0" column="0" />
   </host>

   <host name="fs4sp3.contoso.com">
      <content-distributor />
      <document-processor processes="4" />
   </host>

   <host name="fs4sp4.contoso.com">
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4" />
      <document-processor processes="4" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
   </searchcluster>

</deployment>

XS4

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="XS4"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>XS4</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=XS4.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <content-distributor />
      <indexing-dispatcher />
      <searchengine row="0" column="0" />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="16" />
   </host>

   <host name="fs4sp2.contoso.com">
      <query />
      <searchengine row="1" column="0" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="none" search="true" />
   </searchcluster>

</deployment>

XS5

Same as XS1

Dataset

This section describes the test farm dataset: The database content and sizes, search indexes and external data sources.

The following table shows the overall metrics.

Object | Value
Search index size (# of items) | 5.4 M
Size of crawl database | 16.7 GB
Size of crawl database log file | 1.0 GB
Size of property database | < 0.1 GB
Size of property database log file | < 0.1 GB
Size of SSA administration database | < 0.1 GB

The next table shows the content source types we used to build the index. The numbers in the table reflect the total number of items per source. Note that the difference between the total number of items and the index size can have two reasons:

  • Items may have been disabled from indexing in the content source, or

  • The document format type could not be indexed.

For SharePoint Server 2010 sources, the size of the respective content database in SQL represents the raw data size.

Content source | Items | Raw data size | Average size per item
HTML 1 | 1.1 M | 8.8 GB | 8.1 kB
SharePoint 1 | 4.5 M | 2.0 TB | 443 kB
HTML 2 | 3.2 M | 137 GB | 43 kB
Total | 8.8 M | 2.2 TB | 246 kB
Note:

The extra-small test scenario did not include people search data.

Test results

This section provides data that show how the various deployments performed under load: Crawling and indexing performance, query performance and disk usage.

Crawling and indexing performance

All the tested deployments had the same CPU resources available, except for XS2. XS2 was running on a single virtual machine and was therefore limited to four CPU cores as opposed to sixteen for the other deployments. The following graph shows the average number of items per second for the different content sources during a full crawl.

Full crawl performance graph

Overall, XS2 showed 65-70% decrease in performance compared to the deployments running on physical hardware. XS3, running four VMs and thus having the same hardware footprint as XS1, results in 35-40% degradation compared to running directly on the host computer. The major degradation for XS3 stems from the lower IO performance when a deployment runs on a virtual machine using a fixed size VHD file. The split of XS3 resources across four virtual machines also resulted in more server to server communication.

Query performance

The following sub sections describe how the deployment configurations and varying content volume affect query performance. We also tried tuning XS5 to make the most out of its high performance storage. The last sub section describes the results compared to XS1 and XS5 without tuning.

Impact of deployment configuration

The following graph shows the query performance when crawling is idle.

Impact of deployment configuration (graph 1)

XS1 and XS5 showed only minor differences, with a slightly better performance for the SSD based XS5 (running with two SSDs versus 7 regular SAS spindles for XS1). As you can expect, the additional search row in XS4 did not improve query performance under idle crawl conditions. XS4 had the same throughput as XS1/XS5 under high load, but with slightly increased latency. The increased latency was caused by queries being directed to both search rows, implying a lower cache hit ratio and server to server communication.

The virtualized deployments (XS2 and XS3) had a significantly lower query performance with more variation than the non-virtualized options. This reduction was related to the storage performance in addition to the search components having maximum four CPU cores at disposal.

The following graph shows the query performance during a full crawl.

Impact of deployment configuration (graph 2)

The XS1 deployment had a reduction in query performance under concurrent crawling and indexing. XS5 was less affected due to the improved storage performance, but we did still see CPU congestion between item processing and query components. XS4 was the least affected, as this deployment had a dedicated search row. XS4 results varied more at concurrent high query and crawl load because of competition for network resources.

The virtualized deployments did not reach 10 QPS during the full crawl. XS1 (native hardware) and XS3 (virtualized) used the same hardware, with the non-virtualized deployment having more than five times the throughput. The difference was caused by virtualization overhead, especially storage performance, and the limitation of four CPU cores per virtual machine. Under high query load, the query components could use all sixteen CPU cores in XS1, whereas they were restricted to maximum four CPU cores in XS3.

Impact of varying content volume

Even though the deployments in this scenario were sized for 8M documents, we also tested query performance when 1M and 5M items were indexed. The following graph shows how the content capacity affected query performance in the XS1 deployment.

Impact of varying content volume graph

The solid lines show that maximum query capacity improved with less content, with maximum 90 QPS at 1M items, 80 QPS at 5M items, and 64 QPS at 8M items. During crawling and indexing, the 1M index could still sustain more than 40 QPS, although with lots of variance. The variance was due to the total index size being fairly small and most of it being able to fit inside application and OS level caches. Both the 5M and 8M indexes had a lower maximum query performance during crawling and indexing, in the 25-30 QPS range.

Impact of tuning for high performance storage

Even if the XS5 deployment demonstrated improved performance over XS1 with default settings, tuning for high performance storage enabled better usage of the higher IOPS potential in SSDs. Such tuning spreads the workload across multiple smaller partitions and allows for more parallel query execution at the expense of more disk operations.

The following graph shows the result of this tuning at full capacity (8M items). The tuning enabled the SSD based XS5 scenario to serve up to 75 QPS, and also reduced the response time under light query load. For example, the response time at 40 QPS at idle crawl was reduced from 0.4 to 0.2 seconds. Further, the response time during crawls was better and more consistent with this tuning. The tuned XS5 scenario was able to deliver around 40 QPS with sub-second latency during crawls, whereas XS1 only delivered 15 QPS with the same load and latency requirements.

Impact of high performance storage graph

In general, high performance storage provides improved query performance, especially during concurrent content crawls, and reduces or even eliminates the performance driven need to run search on dedicated rows. SSDs also provide sufficient performance with a smaller number of disks. In the extra-small test scenario, two SSDs outperformed seven SAS spindles. This is attractive where power, or space restrictions, does not allow for a larger number of disks, for example for blade servers.

Disk usage

The following table shows the combined increase in disk usage on all servers after the various content sources were indexed. Note that deployments providing redundancy (multiple search/indexer rows) require additional disk space because they replicate FiXML and index data.

Content source | Items | FiXML data size | Index data size | Other data size
HTML 1 | 1.1 M | 6 GB | 20 GB | 4 GB
SharePoint1 | 4.5 M | 41 GB | 108 GB | 15 GB
HTML 2 | 3.2 M | 27 GB | 123 GB | 22 GB
Total | 8.8 M | 74 GB | 251 GB | 41 GB

See Also

Concepts

Performance and capacity test results (FAST Search Server 2010 for SharePoint)
Test results: Medium scenario (FAST Search Server 2010 for SharePoint)
Test results: Large scenario (FAST Search Server 2010 for SharePoint)


Test results: Medium scenario (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

With the medium Microsoft FAST Search Server 2010 for SharePoint test scenario we targeted a moderate test corpus. To meet freshness goals, incremental crawls were likely to occur during business hours. The amount of content for the scenario was up to 40 million items.

We set up the parent Microsoft SharePoint Server 2010 farm with two front-end web servers, two application servers and one database server, and arranged them as follows:

  • We used the SharePoint Server 2010 crawler, indexing connector framework and the FAST Content Search Service Application (Content SSA) to crawl content. We distributed two crawl components for the Content SSA across the two application servers, mainly to accommodate I/O limitations in the test setup (1 Gbit/s network), where a single network adapter would have been a bottleneck

  • One of the application servers also hosted Central Administration for the farm

  • The database server hosted the crawl databases, the FAST Search Server 2010 for SharePoint administration databases and the other SharePoint Server 2010 databases.

We used no separate data storage because the application servers and front-end web servers only needed space for the operating system, application binaries and log files.

In this article:

  • Test deployments

  • Test characteristics

  • Test results

Test deployments

Within the medium scenario, we tested the following FAST Search Server 2010 for SharePoint deployments:

Name | Description
M1 | One combined administration and web analyzer server, and three index column servers that had default configuration (4 servers in total)
M2 | Same as M1, but with SAN storage
M3 | A stand-alone high capacity server that hosted all FAST Search Server 2010 for SharePoint components
M4 | Same as M1, with the addition of a dedicated search row (7 servers in total)
M5 | Same as M3, with the addition of a dedicated search row (2 servers in total)
M6 | Same as M4, but where the search row included a backup indexer
M7 | Same as M5, but where the search row included a backup indexer
M8 | Same as M3, but using solid state drives
M9 | Same as M3, but on more powerful hardware
M10 | Same as M1, but using solid state drives for the indexer and search servers
Note:

We configured the index capacity of the M3, M5, M7, M8 and M9 deployments, allowing for up to 40 million items indexed on one server. For more information, refer to Performance and capacity tuning (FAST Search Server 2010 for SharePoint).

Test characteristics

This section provides detailed information about the hardware, software, topology and configuration of the test environment.

Hardware/Software

We tested the specified deployments using the following hardware and software.

FAST Search Server 2010 for SharePoint servers

  • Windows Server 2008 R2 x64 Enterprise Edition

  • 2x Intel L5520 CPUs with Hyper-threading and Turbo Boost switched on

  • 24 GB memory

  • 1 Gbit/s network card

  • Storage subsystem:

    • OS: 2x 146GB 10k RPM SAS disks in RAID1

    • Application: 18x 146 GB 10k RPM SAS disks in RAID50 (two parity groups of 9 drives each). Total formatted capacity of 2 TB.

    • Disk controller: HP Smart Array P410, firmware 3.00

    • Disks: HP DG0146FARVU, firmware HPD5

Variations:

Deployment Variation description

M2

  • Application hosted on 2TB partitions on a SAN

  • SAN used for test:

    • 3Par T-400

    • 240 15k RPM spindles (450GB each)

    • Dual ported FC connection to each application server that uses MPIO without any FC switch. MPIO enabled in the operating system.

M3/M5

  • 48 GB memory

  • Application: 22x300GB 10k RPM SAS drives in RAID50 (two parity groups of 11 spindles each). Total formatted capacity of 6TB.

M7

  • 2x Intel L5640 CPUs with Hyper-threading and Turbo Boost switched on

  • 48 GB memory

  • Dual 1 Gbit/s network card

  • Storage subsystem:

    • Application: 12x 1TB 7200 RPM SAS drives in RAID10. Total formatted capacity of 6TB.

    • Disk controller: Dell PERC H700, firmware 12.0.1-0091

    • Disks: Seagate Constellation ES ST31000424SS, firmware KS65

M8

  • 2x Intel L5640 CPUs with Hyper-threading and Turbo Boost switched on

  • 48 GB memory

  • Dual 1 Gbit/s network card

  • Storage subsystem:

    • Application: 3x 1280 GB SSD cards in RAID0. Total formatted capacity of 3.6 TB.

    • SSD cards: Fusion-IO ioDrive Duo 1.28 TB MLC, firmware revision 43284, driver 2.2 build 21459

M9

  • 2x Intel X5670 CPUs with Hyper-threading and Turbo Boost switched on

  • 48 GB memory

  • Dual 1 Gbit/s network card

  • Storage subsystem:

    • Application: 12x 600GB 15k RPM SAS drives in RAID50. Total formatted capacity of 6TB.

    • Disk controller: LSI MegaRAID SAS 9260-8i, firmware 2.90-03-0933

    • Disks: Seagate Cheetah 15K.7 ST3600057SS, firmware ES62

M10

  • 2x Intel L5640 CPUs with Hyper-threading and Turbo Boost switched on

  • 48 GB memory

  • Dual 1 Gbit/s network card

  • Storage subsystem (indexing and query matching servers only):

    • Application: 1x Fusion-IO ioDrive Duo 1.28 TB MLC SSD card, firmware revision 43284, driver 2.2 build 21459

SharePoint Server 2010 servers

  • Windows Server 2008 R2 x64 Enterprise edition

  • 2x Intel L5420 CPUs

  • 16 GB memory

  • 1 Gbit/s network card

  • Storage subsystem for OS/Programs: 2x 146GB 10k RPM SAS disks in RAID1

SQL servers

Same specification as for SharePoint Server 2010 servers, but with additional disk RAID for SQL data with 6x 146GB 10k RPM SAS disks in RAID5.

Topology

This section describes the topology of all the test deployments.

Note:

We used the same SharePoint Server 2010 and database server configuration as shown for M1/M2/M10 for all the tested deployments, but for convenience we only show the FAST Search Server 2010 for SharePoint farm topology in the other deployment descriptions.

M1/M2/M10

The M1, M2 and M10 deployments were similar except for the storage subsystem. We ran M1 on local disk, for M2 we used SAN storage and for M10 we used solid state storage for the search server farm. All three deployments had three index columns and one search row. They all had a separate administration server that also included the web analyzer components. We spread item processing out across all servers.

None of these three deployments had a dedicated search row. This implied that there would be a noticeable degradation in query performance during content crawling and indexing. This effect could be reduced by crawling and indexing in off-peak hours, or by limiting the number of item processing components to reduce the maximum crawl rate.

The following figure shows the M1 deployment. M2 and M10 had the same configuration.

M1 topology illustration

We used the following deployment.xml file to set up M1, M2 and M10.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="M1"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>M1</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=M1.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="12" />
   </host>

   <host name="fs4sp2.contoso.com">
      <content-distributor />
      <searchengine row="0" column="0" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp3.contoso.com">
      <content-distributor />
      <searchengine row="0" column="1" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp4.contoso.com">
      <indexing-dispatcher />
      <searchengine row="0" column="2" />
      <document-processor processes="12" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
   </searchcluster>

</deployment>

M3

The M3 deployment combined all components on one server. Running concurrent crawling, indexing and query load had the same impact as for M1/M2/M10. In addition, the reduced number of servers implied fewer items processing components, and therefore lower crawl rate.

The following figure shows the M3 deployment.

M3 topology illustration

We used the following deployment.xml file to set up the deployment.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="M3"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>M3</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=M3.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <content-distributor />
      <indexing-dispatcher />
      <searchengine row="0" column="0" />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="12" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
   </searchcluster>

</deployment>

M4

The M4 deployment had the same configuration as M1/M2/M10 with the addition of a dedicated search row. A dedicated search row adds query throughput capacity, introduces query redundancy, and provides better separation of query load and crawling and indexing load. Each of the three servers that were running the dedicated search row also included a query processing component (query). In addition, the deployment included a query processing component on the administration server (fs4sp1.contoso.com). The Query SSA does not use the latter query processing component during ordinary operation. But it may use it as a fallback to be able to serve queries if you take down the whole search row for maintenance.

The following figure shows the M4 deployment.

M4 topology illustration

We used the following deployment.xml file to set up the deployment.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="M4"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>M4</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=M4.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="12" />
   </host>

   <host name="fs4sp2.contoso.com">
      <content-distributor />
      <searchengine row="0" column="0" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp3.contoso.com">
      <content-distributor />
      <indexing-dispatcher />
      <searchengine row="0" column="1" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp4.contoso.com">
      <indexing-dispatcher />
      <searchengine row="0" column="2" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp5.contoso.com">
      <query />
      <searchengine row="1" column="0" />
   </host>

   <host name="fs4sp6.contoso.com">
      <query />
      <searchengine row="1" column="1" />
   </host>

   <host name="fs4sp7.contoso.com">
      <query />
      <searchengine row="1" column="2" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="none" search="true" />
   </searchcluster>

</deployment>

M5

The M5 deployment had the same configuration as M3 with the addition of a dedicated search row that provided the same benefits as M4 compared to M1/M2/M10.

The following figure shows the M5 deployment.

M5 topology illustration

We used the following deployment.xml file to set up the deployment.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="M5"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>M5</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=M5.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <content-distributor />
      <indexing-dispatcher />
      <searchengine row="0" column="0" />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="16" />
   </host>

   <host name="fs4sp2.contoso.com">
      <query />
      <searchengine row="1" column="0" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="none" search="true" />
   </searchcluster>

</deployment>

M6

The M6 deployment had the same configuration as M4 but with an additional backup indexer enabled on the search row. We deployed the backup indexer by modifying the M4 deployment.xml file as follows.

Copy Code
…
<searchcluster>
   <row id="0" index="primary" search="true" />
   <row id="1" index="secondary" search="true" />
</searchcluster>
…

M7

The M7 deployment had the same setup as M5, with an additional backup indexer enabled on the search row. M7 was also running on servers that had more CPU cores and enabled us to increase the number of item processing components in the farm.

We used the following deployment.xml to set up the deployment.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="M7"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>M7</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=M5.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <content-distributor />
      <indexing-dispatcher />
      <searchengine row="0" column="0" />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="20" />
   </host>

   <host name="fs4sp2.contoso.com">
      <query />
      <searchengine row="1" column="0" />
      <document-processor processes="8" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="secondary" search="true" />
   </searchcluster>

</deployment>

M8/M9

Similar to M3, the M8/M9 deployments combined all components on one server. The differences were that M8/M9 were running on hardware with better performance, especially for the disk subsystem, and that they had an increased number of CPU cores that enabled more item processing components.

M8 used solid state storage. M9 had more CPU power (X5670 vs. L5520/L5640 that we used to test most of the other medium deployments) and the fastest disk spindles readily available (12x 15k RPM SAS disks).

We used the following deployment.xml file to set up both M8 and M9.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="M8"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>M8</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=M8.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <content-distributor />
      <indexing-dispatcher />
      <searchengine row="0" column="0" />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="20" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
   </searchcluster>

</deployment

Dataset

This section describes the test farm dataset: The database content and sizes, search indexes and external data sources.

The following table shows the overall metrics.

Object | Value
Search index size (# of items) | 42.7 million
Size of crawl database | 138 GB
Size of crawl database log file | 11 GB
Size of property database | <0.1 GB
Size of property database log file | 0.3 GB
Size of SSA administration database | <0.1 GB

The next table shows the content source types we used to build the index. The numbers in the table reflect the total number of items per source and include replicated copies. Note that the difference between the total number of items and the index size can have two reasons:

  • Items may have been disabled from indexing in the content source, or

  • The document format type could not be indexed.

For SharePoint sources, the size of the respective content database in SQL represents the raw data size.

Content source | Items | Raw data size | Average size per item
File share 1 (2 copies) | 1.2 M | 154 GB | 128 kB
File share 2 (2 copies) | 29.3 M | 6.7 TB | 229 kB
SharePoint 1 | 4.5 M | 2.0 TB | 443 kB
SharePoint 2 | 4.5 M | 2.0 TB | 443 kB
HTML 1 | 1.1 M | 8.8 GB | 8.1 kB
HTML 2 | 3.2 M | 137 GB | 43 kB
Total | 43.8 M | 11 TB | 251 kB

To reach sufficient content volume in the testing of the medium scenario, we added two replicas of the file shares. Each copy of each document then appeared as a unique item in the index, but they were treated as duplicates by the duplicate trimming feature. From a query matching perspective the load was similar to having all unique documents indexed, but any results from these sources triggered duplicate detection and collapsing in the search results.

Note:

The medium test scenario did not include people search data.

Test results

This section provides data that shows how the various deployments performed under load: Crawling and indexing performance, query performance and disk usage.

Crawling and indexing performance

The following diagram shows the average number of items each deployment processed per second during a full crawl.

Full crawl performance graph
Note:

We have only included test results from M1 through M6 in the diagram because the other deployments did not show significantly different crawling and indexing performance than their corresponding counterparts.

For full crawls, the item processing components represented the bottleneck. The limiting factor was the CPU processing capacity.

M1, M2 and M4 had similar performance characteristics because they had the same number of item processing components available. The same applied to M3 and M5, but notice that M1, M2 and M4 had four times increased crawl performance during a full crawl. The reason for this was that they had four times the item processing capacity compared to M3 and M5. If you compare M6 to M4 you can also see that running with backup indexers incurs a performance overhead due to the additional synchronization work it requires. Typically, an installation without backup indexers, such as M4, will outperform one with backup, such as M6.

The following diagram shows the average number of items each deployment processed per second during an incremental crawl.

Incremental crawl performance graph

Incremental crawls are faster than full crawls, from slightly faster up to a factor of 2 or 3. This is mainly because incremental crawls mostly consist of partial updates, which only updates metadata. This also implies that the crawling and indexing performance is largely the same for all content types.

For incremental crawls, the indexers were the bottleneck because the item processing load was limited. Typically, disk I/O capacity is the limiting factor. During an incremental update, FAST Search Server 2010 for SharePoint fetches the old version of the item from disk, modifies it, persist it to disk and then indexes it. This is more expensive than a full crawl operation where the item is only persisted and indexed.

Query performance

The following sub sections describe how the deployment configurations affect query performance.

M1

The following diagram shows the query latency as a function of QPS for the M1 deployment.

M1 query performance (graph 1)

An idle indexer provided the best query performance, with an average latency less than 0.7 until approximately 21 QPS. During full crawl the limit was 10 QPS. During incremental crawl it was 15 QPS.

Note that the latency was not affected by higher QPS until we reached max system capacity. This occurred at the point where the curve starts bending "backward". The figure shows that QPS decreased and latency increased when we applied more query load after reaching the maximum capacity of the system. In the M1 deployment, the peak QPS was about 28 with idle indexers. The bottleneck in this deployment was the amount of CPU resources. The behavior is also illustrated in the next diagram where you can observe that performance decreased when we had more than 40 concurrent user agents and an idle indexer.

M1 query performance (graph 2)

Because the amount of CPU resources was the bottleneck, we tested the impact of using hyper-threading in the CPU. By enabling hyper-threading we allowed for more threads to execute in (near) parallel, at the expense of slightly reduced average performance for single-threaded tasks. Note that the query matching components ran in a single thread when QPS was low and no other tasks were running on the server.

M1 query performance (graph 3)

Hyper-threading performed better for all the three crawling and indexing cases in the M1 deployment. In the deployments with dedicated search rows, we observed a small reduction (around 150ms) in query latency when they were running at very light query load. In general, hyper-threading reduces query latency and allows for higher QPS, especially when having multiple components on the same server. Disabling hyper-threading only provides a small improvement under conditions where the performance already is good.

M2

The following diagram shows the query latency as a function of QPS for the M2 deployment.

M2 query performance (graph 1)

Note that the latency was not affected by higher QPS until we reached max system capacity. When the indexers were idle, there was a slow latency increase until the deployment reached the saturation point at approximately 20 QPS. For full and incremental crawls the latency increased as indicated in the graph. The test did not include data to indicate exactly when the query latency saturation occurred during the crawl.

The following diagram shows the same test data as user agents versus latency.

M2 query performance (graph 2)

The next diagram shows the query latency as a function of QPS for M1 versus M2.

M2 query performance (graph 3)

The main conclusion is that M1 performed somewhat better than M2. M1 was able to handle about 3 QPS more than M2 before reaching the saturation point. The SAN disks we used for M2 should have been able to match M1's locally attached disks with regards to I/O operations per second. However, the bandwidth towards the disks was somewhat lower with the SAN configuration. For full crawl and incremental crawl the performance was comparable during the light load tests. During heavy load, ongoing indexing had less impact on M2 as the SAN provided more disk spindles to distribute the load.

M3

The following diagram shows the query latency as a function of QPS for M3 versus M1.

M3 query performance graph

The M3 deployment was able to handle about 10 QPS with idle indexers. One characteristic of a stand-alone server deployment is that the query latency fluctuates more when you move close to the saturation point. Under low query load, M3 was almost able to match the performance of M1, but during increased load the limitations became apparent. M1 had three times the number of query matching components and the peak QPS capacity was close to three times as high, 28 versus 10 QPS.

M4

M4 was an M1 deployment with an additional dedicated search row. The main benefit of a dedicated search row is that the index and the search processes are not directly competing for the same resources, primarily disk and CPU.

The following diagrams show that the added search row in M4 provided a 5 QPS gain versus M1. In addition, the query latency was improved by about 0.2-0.4 seconds.

M4 query performance (graph 1) M4 query performance (graph 2)

Adding search rows will in most cases improve query performance, but at the same time, it introduces additional network traffic. The query performance may be decreased if you add search rows without sufficient network capacity. The additional network traffic is generated by the indexers copying large index files to the additional query matching servers. The index file copying may also affect indexing performance.

With the M4 deployment, we also tested the impact of document volume. The following diagram shows the query latency as a function of QPS for M4 with 18, 28 and 43 million documents indexed.

M4 query performance (graph 3)

The graph shows that the document volume affects the maximum QPS the system can deliver. Adding ~10 million documents gave a ~5 max QPS reduction. Below 23 QPS the document volume had low impact on the query latency.

M5

The following diagram shows user agents versus query latency for M5 and M3.

M5 query performance

As illustrated when we compared M1 and M4, adding a dedicated search row improves query performance. Comparing M3 and M5 shows that the same is the case when you are adding a query matching server to a stand-alone deployment.

M6

The following diagram shows the query latency as a function of QPS for M6 versus M4.

M6 query performance graph

The difference between M6 and M4 was the addition of a backup indexer row. In general, backup indexers will compete with query matching for available resources, and may decrease query performance. However, in this specific test, that was not the case. The reason was that the hardware we used had enough resources to handle the additional load during regular operations. Note that the backup indexers use significantly less resources than the primary indexer. This is because it is the primary indexer that performs the actual indexing and distributes the indexes to the search rows and backup indexer row.

Note:

All indexers perform regular optimization tasks of internal data structures between 03.00 AM and 05.59 AM every night. These tasks may, depending on the crawling and indexing pattern, be I/O intensive. Testing of M6 showed that you may see a significant reduction in query performance during indexer optimization processes. The more update and delete operations the indexer handles, the more optimization is required.

M7

The following diagram shows the query latency as a function of QPS for M7 compared to M5.

M7 query performance graph

M7 was similar to M5, but it was running on servers that had more powerful CPUs and more memory. On the other hand, it had a disk subsystem that could not handle the same amount of I/O operations per second (IOPS); the M7 storage subsystem had more bulk capacity but less performance compared to the M5 storage subsystem.

The main difference in results compared to M5 was a slightly increased QPS rate before the system became saturated. M5 was saturated around 10 QPS, whereas M7 provided approximately 12 QPS. This was due to the increased CPU performance and added memory, although partly counterbalanced by the weaker disk configuration.

M8

The M8 deployment used solid state storage with much higher IOPS and throughput capabilities. This deployment was only limited by the available CPU processing power. A more powerful CPU configuration, for example with quad CPU sockets, should be able to achieve linearly performance improvements with the added CPU resources.

Query performance results for M8 are discussed together with M9.

M9

The M9 deployment had improved CPU performance (X5670) and high-end disk spindles (15k RPM SAS). M9 is an example of the achievable performance gains by using high-end components, keeping regular disk spindles for storage.

The improved CPU performance implied 20-30% increased crawl speeds for M9 over M8 (both with twenty item processing components), and even more compared M3 (which had twelve item processing components). Note that M3 ran with a less powerful server for the content sources, and was more often limited by the sources than M8 and M9. M9 achieved more than 50 documents per second for all content sources.

The following diagram shows the query latency as a function of QPS for M8 and M9 under varying load patterns, compared to M3 and M5:

M9 query performance graph

Both M8 and M9 performed better during idle crawling and indexing than M5. M5 performance is only shown during crawling and indexing. Because M5 had a dedicated search row, the query performance was fairly constant regardless of ongoing crawling and indexing. The main contribution to peak query rate improvements were the additional CPU resources M8 and M9 had compared to M5.

During idle crawling and indexing, M9 had slightly better QPS than M8 due to the more powerful CPU. Under overload conditions (more than 1 second latency), M9 decreased in performance because of an overloaded storage subsystem (exactly like M5), while M8 sustained the peak rate with its solid state storage.

M3, which is a M5 deployment without the search row, saturated already at 5 QPS. M9 provided higher QPS rates. M9 had higher latency than M3 at low QPS during crawling and indexing. The reason for this was that M9 had more than 50% higher crawl rates than M3 because the deployment had more item processing components and faster CPU. If we reduced the crawl rates to M3 levels, M9 would have given better query performance than M3 also at low QPS.

During crawling and indexing, M8 query performance decreased less than 20%, dominantly because of CPU congestion. This means that the storage subsystem on M8 made it possible to maintain good query performance during crawling and indexing without doubling the hardware footprint with a search row. Adding more CPU resources would have allowed for better query performance, as the storage subsystem still had spare resources in the given deployment.

On M9, query latency approximately doubled during crawling and indexing. The deployment could still deliver acceptable performance under low QPS loads, but was much more affected than M8. This was due to the storage subsystem on M9 having slower read accesses when we combined it with write traffic from crawling and indexing.

M10

The M10 deployment had the same configuration as M1 and M2 but used solid state storage. The deployment had the same amount of storage as M8, but it was spread across three servers to obtain more CPU power.

It is most interesting to compare M10 to M4, as we in both these deployments tried to achieve a combination of high crawl rate and query performance at the same time. In M4, we did this by splitting the application storage across two search rows, each with three columns with 18 SAS disk per server. M10 only had a single row and was replacing the application disk spindles with solid state storage. The search server farm totals were:

  • M4: 6 servers, 108 disk spindles

  • M10: 3 servers, 3 solid state storage cards

Both deployments had an additional administration server.

The following diagram shows the query latency as a function of QPS for M10 versus M4:

M10 query performance graph

With idle crawling (solid lines), M4 achieved around 23 QPS before degrading to around 20 QPS under overload conditions. IO became the bottleneck. M10 was able to deliver 30 QPS at which point it became limited by the throughput CPU. By using faster or more CPUs we could have increased this benefit even more than the measured 30% gain.

During crawling, M4 had no significant changes in query performance compared to idle. This was because of its dedicated search row. M10 had some degradation in query performance because content processing and queries competed for the same CPU resources. But notice that M10 still achieved the same 20 QPS as M4 under the highest load conditions. Also notice that the crawling rate for M10 is 20% higher than for M4. The reason for the latter was that the increased IO performance allowed for better handling of concurrent operations.

Disk usage

The following table shows the combined increase in disk usage on all servers after the various content sources were indexed.

Content source | Raw source data size | FiXML data size | Index data size | Other data size
File share 1 (2 copies) | 154 GB | 18 GB | 36 GB | 5 GB
File share 2 (2 copies) | 6.7 TB | 360 GB | 944 GB | 10 GB
SharePoint 1 | 2.0 TB | 70 GB | 220 GB | 13 GB
SharePoint 2 | 2.0 TB | 66 GB | 220 GB | 17 GB
HTML 1 | 8.8 GB | 8 GB | 20 GB | 8 GB
HTML 2 | 137 GB | 31 GB | 112 GB | 6 GB
Total | 11 TB | 553 GB | 1.6 TB | 56 GB

The following table shows the disk usage for the web analyzer component in a mixed content scenario, where the data is both file share content and SharePoint items.

Number of items in index | 40,667,601
Number of analyzed hyperlinks | 119,672,298
Average number of hyperlinks per items | 2.52
Peak disk usage during analysis (GB) | 77.51
Disk usage between analysis (GB) | 23.13
Disk usage per 1 million items during peak (GB) | 1.63
Note:

In this scenario, the disk usage values for the web analyzer component were somewhat lower than the recommended dimensioning values. The reason was that the URLs in the test dataset were fairly short.

The average number of links per item was low compared to pure web or SharePoint content sources. For example, in a pure web content installation the average number of links can be as high as 50. Since the web analyzer component only stores document IDs, hyperlinks and anchor texts, the number of links is the dominant factor determining the disk usage.

See Also

Concepts

Performance and capacity test results (FAST Search Server 2010 for SharePoint)
Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint)
Test results: Large scenario (FAST Search Server 2010 for SharePoint)


Test results: Large scenario (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

With the large Microsoft FAST Search Server 2010 for SharePoint test scenario we targeted a large test corpus. To meet freshness goals, incremental crawls were likely to occur during business hours. The amount of content for the scenario was up to 100 million items.

We set up the parent Microsoft SharePoint Server 2010 farm with two front-end web servers, two application servers and one database server, and arranged them as follows:

  • We used the SharePoint Server 2010 crawler, indexing connector framework and the FAST Content Search Service Application (Content SSA) to crawl content. We distributed two crawl components for the Content SSA across the two application servers, mainly to accommodate I/O limitations in the test setup (1 Gbit/s network), where a single network adapter would have been a bottleneck

  • One of the application servers also hosted Central Administration for the farm

  • The database server hosted the crawl databases, the FAST Search Server 2010 for SharePoint administration databases and the other SharePoint Server 2010 databases

We used no separate data storage because the application servers and front-end web servers only needed space for operating system, application binaries and log files.

In this article:

  • Test deployments

  • Test characteristics

  • Test results

Test deployments

Within the large scenario, we tested the following FAST Search Server 2010 for SharePoint deployments:

Name | Description
L1 | Single row, six column deployment, with an additional administration node (7 servers in total)
L2 | Same as L1, with the addition of a dedicated search row (13 servers in total)
L3 | Same as L2, but where the search row included a backup indexer row

Test characteristics

This section provides detailed information about the hardware, software, topology and configuration of the test environment.

Hardware/Software

We tested the specified deployments using the following hardware and software.

FAST Search Server 2010 for SharePoint servers

  • Windows Server 2008 R2 x64 Enterprise Edition

  • 2x Intel L5520 CPUs with Hyper-threading and Turbo Boost switched on

  • 24 GB memory

  • 1 Gbit/s network card

  • Storage subsystem:

    • OS: 2x 146GB 10k RPM SAS disks in RAID1

    • Application: 12x 146 GB 10k RPM SAS disks in RAID50 (two parity groups of 6 drives each). Total formatted capacity of 2 TB.

    • Disk controller: HP Smart Array P410, firmware 3.30

    • Disks: HP DG0146FARVU, firmware HPD6

SharePoint Server 2010 servers

  • Windows Server 2008 R2 x64 Enterprise edition

  • 2x Intel L5420 CPUs

  • 16 GB memory

  • 1 Gbit/s network card

  • Storage subsystem for OS/Programs: 2x 146GB 10k RPM SAS disks in RAID1

SQL servers

Same specification as for SharePoint Server 2010 servers, but with additional disk RAID for SQL data with 6x 146GB 10k RPM SAS disks in RAID5.

Topology

This section describes the topology of all the test deployments.

L1

L1 was a single row, six column deployment with an additional administration node. We used the following deployment.xml file to set up L1.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="L1"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>L1</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=L1.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="12" />
   </host>

   <host name="fs4sp2.contoso.com">
      <query />
      <searchengine row="0" column="0" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp3.contoso.com">
      <content-distributor />
      <searchengine row="0" column="1" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp4.contoso.com">
      <content-distributor />
      <searchengine row="0" column="2" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp5.contoso.com">
      <content-distributor />
      <searchengine row="0" column="3" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp6.contoso.com">
      <indexing-dispatcher />
      <searchengine row="0" column="4" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp7.contoso.com">
      <indexing-dispatcher />
      <searchengine row="0" column="5" />
      <document-processor processes="12" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
   </searchcluster>

</deployment>

L2

L2 had the same configuration as L1 with the addition of a dedicated search row.

The search row added query throughput capacity, introduced query redundancy, and provided better separation of query and content processing load. Three of the servers that ran in the dedicated search row also included a query processing component (query). The deployment also included a query processing component on the administration server (fs4sp1.contoso.com). The Query SSA does not use the latter query processing component during ordinary operation. But it may use it as a fallback to be able to serve queries if you take down the whole search row for maintenance.

We used the following deployment.xml file to set up L2.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>

<deployment version="14" modifiedBy="contoso\user"

   modifiedTime="2009-03-14T14:39:17+01:00" comment="L2"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

   <instanceid>L2</instanceid>

   <connector-databaseconnectionstring>
      [<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=L2.jdbc]]>
   </connector-databaseconnectionstring>

   <host name="fs4sp1.contoso.com">
      <admin />
      <query />
      <webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="4"/>
      <document-processor processes="12" />
   </host>

   <host name="fs4sp2.contoso.com">
      <searchengine row="0" column="0" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp3.contoso.com">
      <content-distributor />
      <searchengine row="0" column="1" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp4.contoso.com">
      <content-distributor />
      <searchengine row="0" column="2" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp5.contoso.com">
      <content-distributor />
      <searchengine row="0" column="3" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp6.contoso.com">
      <indexing-dispatcher />
      <searchengine row="0" column="4" />
      <document-processor processes="12" />
   </host>

   <host name="fs4sp7.contoso.com">
      <indexing-dispatcher />
      <searchengine row="0" column="5" />
      <document-processor processes="12" />
   </host>

   <host name=" fs4sp8.contoso.com ">
      <query />
      <searchengine row="1" column="0" />
   </host>

   <host name="fs4sp9.contoso.com ">
      <query />
      <searchengine row="1" column="1" />
   </host>

   <host name="fs4sp10.contoso.com ">
      <query />
      <searchengine row="1" column="2" />
   </host>

   <host name="fs4sp11.contoso.com ">
      <searchengine row="1" column="3" />
   </host>

   <host name="fs4sp12.contoso.com">
      <searchengine row="1" column="4" />
   </host>

   <host name="fs4sp13.contoso.com"
      <searchengine row="1" column="5" />
   </host>

   <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="none" search="true" />
   </searchcluster>

</deployment>

L3

L3 had the same configuration as L2 with an additional backup indexer enabled on the search row. We deployed the backup indexer by modifying the L2 deployment .xml file as follows.

Copy Code
…
<searchcluster>
   <row id="0" index="primary" search="true" />
   <row id="1" index="secondary" search="true" />
</searchcluster>
…

Dataset

This section describes the test farm dataset: The database content and sizes, search indexes and external data sources.

The following table shows the overall metrics.

Object | Value
Search index size (# of items) | 103 million
Size of crawl database | 358 GB
Size of crawl database log file | 65 GB
Size of property database | <0.1 GB
Size of property database log file | 0.6 GB
Size of SSA administration database | <0.1 GB

The next table shows the content source types we used to build the index. The numbers in the table reflect the total number of items per source and include replicated copies. Note that the difference between the total number of items and the index size can have two reasons:

  • Items may have been disabled from indexing in the content source, or

  • The document format type could not be indexed.

For SharePoint sources, the size of the respective content database in SQL represents the raw data size.

Content source | Items | Raw data size | Average size per item
File share 1 (4 copies) | 2.4 M | 308 GB | 128 kB
File share 2 (4 copies) | 58.6 M | 13.4 TB | 229 kB
SharePoint 1 (4 copies) | 18.1 M | 18.1 M | 443 kB
SharePoint 2 (3 copies) | 13.6 M | 6.0 TB | 443 kB
HTML 1 (3 copies) | 3.2 M | 26 GB | 8.1 kB
HTML 2 (3 copies) | 9.5 M | 411 GB | 43 kB
Total | 105.5 M | 28 TB | 268 kB

To reach sufficient content volume in the testing of the large scenario, we added replicas of the data sources. Each copy of each document then appeared as a unique item in the index, but they were treated as duplicates by the duplicate trimming feature. From a query matching perspective the load was similar to having all unique documents indexed, but any results from these sources triggered duplicate detection and collapsing in the search results.

Note:

The large test scenario did not include people search data.

Test results

This section provides data that shows how the various deployments performed under load: Crawling and indexing performance, query performance and disk usage.

Crawling and indexing performance

All the large scenario deployments were limited by the bandwidth of the content sources; L1 through L3 all achieved around 200 items per second crawl rates.

Query performance

L1, L2 and L3 were scaled up versions of M1, M4 and M6 in the medium test scenario (refer to Test results: Medium scenario (FAST Search Server 2010 for SharePoint) for more information). To make the large deployments able to index more content while maintaining the query performance, we added more columns to the medium deployments. The following diagram shows the query latency as a function of QPS for L1 through L3, with and without ongoing crawls.

Query performance graph

L2 and L3 had approximately the same performance, and were also similar to the corresponding smaller scale M4 and M6.

L1 showed a slightly different performance pattern compared to M1. Given 1 second maximum approved latency, M1 achieved 27 QPS with idle crawl and 16 QPS during crawling. The same numbers for L1 was 18 and 9 QPS. The additional columns in L1 compared to M1 inferred that more servers were involved, where the slowest one for any given query at any time was the determining factor for the query latency. This effect was much less visible for the L2 and L3 deployments. The latter deployments had dedicated search rows and the servers did not have other components competing for resources.

Disk usage

The following table shows the combined disk usage on all servers in the L1 deployment. L2 and L3 used additional disk space for replication of FiXML and index files on the second row.

Content source | Raw source data size | FiXML data size | Index data size | Other data size
Total | 28 TB | 1.1 TB | 3.8 TB | 104 GB

See Also

Concepts

Performance and capacity test results (FAST Search Server 2010 for SharePoint)
Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint)
Test results: Medium scenario (FAST Search Server 2010 for SharePoint)


Test results: Extra-large scenario (FAST Search Server 2010 for SharePoint)

Published: June 16, 2011

With the extra-large Microsoft FAST Search Server 2010 for SharePoint test scenario, we targeted an extra-large test corpus. For this scenario, redundancy is crucial, because with this many computers, hardware failures are likely to occur. The content volume for the scenario was up to 500 million items. The content volume means that crawls will have to run during business hours.

We set up the parent Microsoft SharePoint Server 2010 farm with two front-end web servers, four application servers, and two database servers, and arranged them as follows:

  • We used the SharePoint Server 2010 crawler, indexing connector framework, and the FAST Content Search Service Application (Content SSA) to crawl content. We distributed four crawl components for the Content SSA across the four application servers. This accommodates I/O limitations in the test setup (1 gigabit per second network), where four nodes provide a theoretically maximum crawl rate of 4 gigabits per second.

  • One of the application servers also hosted Central Administration for the farm.

  • One database server hosted the crawl databases.

  • One database server hosted the FAST Search Server 2010 for SharePoint administration databases and the other SharePoint Server 2010 databases.

We did not use any separate data storage, because the application servers and front-end web servers only needed space for operating system, application binaries and log files.

In this article:

  • Test deployment

  • Test characteristics

  • Test results

Test deployment

Within the extra-large scenario, we tested one FAST Search Server 2010 for SharePoint deployment, with service pack 1 (SP1):

Name | Description
XL1 | Two rows, twelve columns setup, with one additional administration node (25 servers in total). The extended capacity mode can support up to 40 million items per server.

Test characteristics

This section provides detailed information about the hardware, software, topology and configuration of the test environment.

Hardware/Software

We tested the specified deployment by using the following hardware and software, although the administration node does not need this much disk space.

FAST Search Server 2010 for SharePoint servers

  • Windows Server 2008 R2 x64 Enterprise Edition

  • 2x Intel L5640 CPUs with Hyper-threading and Turbo Boost switched on

  • 48 GB memory

  • 1 gigabits per second network card

  • Storage subsystem:

    • OS: 2x 146GB 10k RPM 2.5" SAS disks in RAID1

    • Application: 12x 1 terabyte 7200 RPM 6 gigabits per second 3.5" SAS disks in RAID10. Total formatted capacity of 5.5 terabytes.

    • Disk controller: Dell PERC H700 Integrated, firmware 12.10.0-0025

    • Disks: Seagate Constellation ES ST31000424SS, firmware KS68

SharePoint Server 2010 servers

  • Windows Server 2008 R2 x64 Enterprise edition

  • 2x Intel L5420 CPUs

  • 16 GB memory

  • 1 gigabit per second network card

  • Storage subsystem for OS/Programs: 2x 146GB 10k RPM SAS disks in RAID1

SQL servers

  • General SQL server: Same specification as for SharePoint Server 2010 servers, but with additional disk RAID for SQL data with 6x 146GB 10k RPM SAS disks in RAID5.

  • Crawl database SQL server:

    • Windows Server 2008 R2 x64 Enterprise edition

    • 2x Intel X5670 CPUs with Hyper-threading and Turbo Boost switched on

    • 48 GB memory

    • 1 gigabit per second network card

    • Storage subsystem:

      • OS: 2x 146GB 10k RPM 2.5" SAS disks in RAID1

      • Application: 12x SAS 15k RPM 600GB 3.5" SAS disks in RAID50. Total formatted capacity 5.6 terabytes.

Topology

This section describes the topology of all the test deployments.

XL1

XL1 is a setup with two rows and twelve columns, with an additional administration node. The second row adds query throughput capacity, query redundancy, and gives better separation of query and feeding load. The deployment includes three query processing components across the administration node (fsadmin.contoso.com), and two query processing components in the second row (fsr1coo.contoso.com and fsr1c01.contoso.com). The Query SSA is configured to only use the query processing components on the search row during normal operation. The query component on the administration node may be reconfigured for use as a fallback. It can then serve queries in conjunction with the first search row if the second search row is taken down for maintenance.

The following figure shows the XL deployment.

XL topology illustration

We used the following deployment.xml file to set up XL1.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>
<deployment version="14" modifiedBy="
  <deployment version="14"
   modifiedBy="contoso\user" modifiedTime="2011-01-01T12:00:00+00:00" comment="XL1"
   xmlns="http://www.microsoft.com/enterprisesearch"
   xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
   xsi:schemaLocation="http://www.microsoft.com/enterprisesearch deployment.xsd">

  <instanceid>XL1</instanceid>
  <connector-databaseconnectionstring>[<![CDATA[jdbc:sqlserver://sqlbox.contoso.com\sql:1433;DatabaseName=XL1.jdbc]]></connector-databaseconnectionstring>

  <!-- Admin -->
  <host name="fsadmin.contoso.com">
   <admin />
 <query />
<content-distributor />
    <indexing-dispatcher />
<document-processor processes="8" />
  </host>

  <!-- Row 0 -->
  <host name="fsr0c00.contoso.com">
<content-distributor />
<searchengine row="0" column="0" />
<document-processor processes="12" />
  </host>

  <host name="fsr0c01.contoso.com">
<content-distributor />
    <searchengine row="0" column="1" />
<document-processor processes="12" />
  </host>

  <host name="fsr0c02.contoso.com">
<content-distributor />
    <searchengine row="0" column="2" />
<document-processor processes="12" />
  </host>

  <host name="fsr0c03.contoso.com">
    <searchengine row="0" column="3" />
<document-processor processes="12" />
<webanalyzer server="true" link-processing="true" lookup-db="true" max-targets="2" redundant-lookup="true" />
  </host>

  <host name="fsr0c04.contoso.com">
    <searchengine row="0" column="4" />
<document-processor processes="12" />
<webanalyzer server="false" link-processing="true" lookup-db="true" max-targets="2"/>
  </host>

  <host name="fsr0c05.contoso.com">
    <searchengine row="0" column="5" />
<document-processor processes="12" />
<webanalyzer server="false" link-processing="true" lookup-db="true" max-targets="2"/>
  </host>

  <host name="fsr0c06.contoso.com">
    <searchengine row="0" column="6" />
<document-processor processes="12" />
<webanalyzer server="false" link-processing="true" lookup-db="true" max-targets="2"/>
  </host>

  <host name="fsr0c07.contoso.com">
    <searchengine row="0" column="7" />
<document-processor processes="12" />
<webanalyzer server="false" link-processing="true" lookup-db="true" max-targets="2"/>
  </host>

  <host name="fsr0c08.contoso.com">
    <searchengine row="0" column="8" />
<document-processor processes="12" />
<webanalyzer server="false" link-processing="true" lookup-db="true" max-targets="2"/>
  </host>

  <host name="fsr0c09.contoso.com">
    <indexing-dispatcher />
    <searchengine row="0" column="9" />
<document-processor processes="12" />
<webanalyzer server="false" link-processing="true" lookup-db="true" max-targets="2"/>
  </host>

  <host name="fsr0c10.contoso.com">
    <indexing-dispatcher />
    <searchengine row="0" column="10" />
<document-processor processes="12" />
<webanalyzer server="false" link-processing="true" lookup-db="true" max-targets="2"/>
  </host>

  <host name="fsr0c11.contoso.com">
    <indexing-dispatcher />
    <searchengine row="0" column="11" />
<document-processor processes="12" />
<webanalyzer server="false" link-processing="true" lookup-db="true" max-targets="2"/>
  </host>

  <!-- Row 1 -->
  <host name="fsr1c00.contoso.com">
<query />
<searchengine row="1" column="0" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c01contoso.com">
   <query />
    <searchengine row="1" column="1" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c02.contoso.com">
    <searchengine row="1" column="2" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c03.contoso.com">
    <searchengine row="1" column="3" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c04.contoso.com">
    <searchengine row="1" column="4" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c05.contoso.com">
    <searchengine row="1" column="5" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c06.contoso.com">
    <searchengine row="1" column="6" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c07.contoso.com">
    <searchengine row="1" column="7" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c08.contoso.com">
    <searchengine row="1" column="8" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c09.contoso.com">
    <searchengine row="1" column="9" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c10.contoso.com">
    <searchengine row="1" column="10" />
<document-processor processes="8" />
  </host>

  <host name="fsr1c011.contoso.com">
    <searchengine row="1" column="11" />
<document-processor processes="8" />
  </host>

  <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="secondary" search="true" />
  </searchcluster>

</deployment>

Dataset

This section describes the test farm dataset: The database content and sizes, search indexes, and external data sources. The Content SSA was configured to use 12 crawl databases.

The following table shows the overall metrics.

Object | Value
Search index size | 503 million items
Size of crawl database | 2.8 terabytes
Size of crawl database log file | 1.6 terabytes
Size of property database | 0.1 terabytes
Size of property database log file | 2 gigabytes (GB)
Size of SSA administration database | 35 gigabytes (GB)

The next table shows the content source types we used to build the index. The numbers reflect the total number of items per source and include replicated copies. The difference between the total number of items and the index size can have two reasons:

  • Items may have been disabled from indexing in the content source, or

  • The document format type could not be indexed.

For SharePoint sources, the size of the respective content database in SQL represents the raw data size.

Content source | Items | Raw data size | Average size per item
File share 1 (12 copies) | 7.2 million | 924 gigabytes (GB) | 128 kilobytes (KB)
File share 2 (12 copies) | 176 million | 40 terabytes | 229 kilobytes (KB)
SharePoint 1 (12 copies) | 54 million | 24 terabytes | 443 kilobytes (KB)
SharePoint 2 (12 copies) | 54 million | 24 terabytes | 443 kilobytes (KB)
SharePoint 3 (12 copies) | 54 million | 24 terabytes | 443 kilobytes (KB)
HTML 1 (12 copies) | 13 million | 105 gigabytes (GB) | 8.1 kilobytes (KB)
HTML 2 (12 copies) | 38 million | 1.7 TB terabytes | 43 kilobytes (KB)
HTML 3 (12 copies) | 122 million | 5.9 terabytes | 49 kilobytes (KB)
Total | 518 million | 121 terabytes | 233 kilobytes (KB)

To reach sufficient content volume in the testing of the large scenario, we added replicas of the data sources. Each copy of each document appeared as a unique item in the index, but they were treated as duplicates by the duplicate trimming feature. From a query matching perspective, the load was similar to having all unique documents indexed, but any results from these sources triggered duplicate detection and collapsing in the search results.

Note:

The extra-large test scenario did not include people search data.

Test results

This section describes how the various deployments performed under load: Crawling and indexing performance, query performance and disk usage.

Crawling and indexing performance

The extra-large scenario deployment was limited by the bandwidth of the content sources. Crawl rates were averaging around 250 items per second. This would have corresponded to 25 days of constant crawling. In testing, crawls were spilt up in blocks of around 43 million items, allowing for intermediate query testing at less than full capacity. Crawl pauses were also needed for maintenance periods, for example Windows Update and replacement of failed hardware.

Query performance

The graph below shows the query performance under various conditions. At full capacity of 500 million items, and with no ongoing indexing, the farm was able to sustain about 15 queries per second (QPS) with less than 1 second average latency before being limited by the CPU resources on the search row. During incremental crawls and full crawls, this number dropped to 12.5 and 10 QPS respectively. Additional testing was done before reaching full capacity. The red lines in the graph are measured when the system was half full. Without any feed, 27 QPS is possible. Under feed, there are fewer differences between the 250 million and 500 million item states. This is expected, because incoming feeds require certain amounts of network, CPU and disk capacity regardless of how many items that are already in the index.

XL Query Performance

The next graph shows the query latency during a computer failure, taking one of the columns in the search row offline. Query load is around 10 QPS, with latency plotted as a function of time. The primary row will automatically provide failover query matching for the failed column, but with a performance impact since it also needs to serve ongoing indexing. During this degraded period, query latency is on average around 1.4 seconds, recovering to around 1 second halfway through the plot when the failed computer is brought back online.

XL Query Latency

Disk usage

The following table shows the combined disk usage on all nodes in the deployment.

Content source | FiXML data size | Index data size | Other data* size | Total disk usage
Administration node | 0 | 0 | 33 gigabytes (GB) | 33 gigabytes (GB)
Row 0 (across 12 computers) | 6.2 terabytes | 15.8 terabytes | 40 gigabytes (GB) | 22.0 terabytes
Row 1 (across 12 computers) | 6.4 terabytes | 15.8 terabytes | 40 gigabytes (GB) | 22.2 terabytesTB
Total | 12.6 terabytes | 31.6 terabytes | 113 gigabytes (GB) | 44.2 terabytes

* Logs and intermediate data for handling fault recovery on computer failures


Performance and capacity recommendations (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

This section provides performance and capacity recommendations for Microsoft FAST Search Server 2010 for SharePoint within the following areas:

  • Recommendations: Content volume capacity (FAST Search Server 2010 for SharePoint)

  • Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)

  • Recommendations: Content freshness (FAST Search Server 2010 for SharePoint)

  • Recommendations: Query throughput (FAST Search Server 2010 for SharePoint)

  • Recommendations: Storage (FAST Search Server 2010 for SharePoint)

  • Recommendations: Virtualization (FAST Search Server 2010 for SharePoint)

Important:

FAST Search Server 2010 for SharePoint contains a rich set of features and a flexible topology model. Before you use this architecture you must carefully consider the effect upon your farm's overall capacity and performance.

For more information about performance and capacity management for SharePoint in general, refer to Performance and capacity management (SharePoint Server 2010).

For an overview of FAST Search Server 2010 for SharePoint as part of an overall SharePoint Server 2010 farm topology, refer to Plan search topology (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Performance and capacity management (FAST Search Server 2010 for SharePoint)
Performance and capacity test results (FAST Search Server 2010 for SharePoint)


Recommendations: Content volume capacity (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

This article describes content volume capacity recommendations for Microsoft FAST Search Server 2010 for SharePoint. For a complete recommendations overview, refer to Performance and capacity recommendations (FAST Search Server 2010 for SharePoint).

As a general guideline you should deploy one index column per 15 million items to index.

The content basis for this general guideline is a mix of 70% SharePoint items and 30% items from file shares where the source item size is between 10 and 500 kilobytes (100-250 on average).

You can add content volume capacity in two ways:

  • By adding additional index columns

  • By indexing more items per column

You can scale out by adding additional index columns on new sets of servers. For more information about redundancy and availability, refer to Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint).

Note:

When you add more columns to a running installation, you must re-index all content in order to re-partition the index columns. A complete re-build of a multi-column index may take several days.

You can scale up by indexing more items per column. The default indexer configuration for each column has five partitions, each with a maximum of 6 million items. In order to keep index freshness high, the two first partitions should preferably have very little content. Effectively, the default configuration is suitable for up to 15 million items, with the three large partitions holding close to the maximum of 6M items each. Exceeding 15 million items will make indexing progressively slower until you meet the default configuration's absolute limit of 30 million items. You can tune the column capacity up to 40 million items per column by increasing the number of partitions to ten. This will yield eight large partitions with 6 million items each. You may want to do this tuning, and save hardware cost, for scenarios with few content updates and moderate query volume or in case the size of the searchable content is very small. An example of the latter may be PowerPoint presentations with lots of graphics. They may be large in source volume size but contain very little searchable content. For more information about index capacity tuning, refer to Performance and capacity tuning (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Recommendations: Content freshness (FAST Search Server 2010 for SharePoint)
Recommendations: Query throughput (FAST Search Server 2010 for SharePoint)
Recommendations: Storage (FAST Search Server 2010 for SharePoint)
Recommendations: Virtualization (FAST Search Server 2010 for SharePoint)


Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

This article describes redundancy and availability recommendations for Microsoft FAST Search Server 2010 for SharePoint.

Any recommendation on redundancy and availability of FAST Search Server 2010 for SharePoint will depend on your business environment. When sizing your deployment, we recommend that you evaluate the scenarios and deployments we have tested, copy the topology of the deployment with the most relevant characteristics and then further develop the topology to fit your environment.

Each part of the system has its own redundancy and availability possibilities. These are described in the following sections:

  • Content SSA redundancy and availability

  • Query SSA redundancy and availability

  • Indexing redundancy and availability

  • Search redundancy and availability

  • SQL Server database redundancy and availability

  • Storage subsystem redundancy and availability

  • Front-end web server redundancy and availability

Note:

In FAST Search Server 2010 for SharePoint, the people search functionality is provided by the Query SSA. It includes a crawl component that retrieves and indexes user profiles, and then stores the metadata properties related to the user profiles in a SQL Server database. For people search redundancy and availability, you may want to add an additional crawl component within the Query SSA.

For a complete recommendations overview, refer to Performance and capacity recommendations (FAST Search Server 2010 for SharePoint).

For more information about the test scenarios, refer to Performance and capacity test results (FAST Search Server 2010 for SharePoint).

Content SSA redundancy and availability

The FAST Search Content SSA enables you to crawl and index content. This SSA represents the default indexing connector for your FAST Search Server 2010 for SharePoint deployment, and you will typically deploy the Content SSA on the parent SharePoint Server 2010 farm. We recommend that you scale out the Content SSA when your business environment requires a certain level of crawling and indexing redundancy and throughput. But do not scale out the Content SSA if there is a chance that the feeding chain will freeze on a partial update. For example, if you don't have a backup indexer.

You can scale out the Content SSA by adding additional crawl components to the SSA. For more information about how to add a crawl component, see Multiple server deployment of the Content SSA (FAST Search Server 2010 for SharePoint).

Important:

Do not deploy more than one Content SSA associated with your FAST Search Server 2010 for SharePoint farm.

For general information about redundancy and availability for the application servers inside a SharePoint Server 2010 farm, see Plan for availability (SharePoint Server 2010).

Query SSA redundancy and availability

The FAST Search Query SSA provides the query-side integration between the FAST Search Server 2010 for SharePoint farm and the parent SharePoint Server 2010 farm. You must deploy the Query SSA on the parent SharePoint Server 2010 farm. You can scale out the Query SSA for query availability by adding additional query components to the SSA. For more information about how to add a query component, see Multiple server deployment of the Query SSA (FAST Search Server 2010 for SharePoint).

Important:

Do not deploy more than one Query SSA associated with your FAST Search Server 2010 for SharePoint farm.

For general information about redundancy and availability for the application servers inside a SharePoint Server 2010 farm, see Plan for availability (SharePoint Server 2010).

Indexing redundancy and availability

The following indexing components within the FAST Search Server 2010 for SharePoint farm support redundancy and availability:

Component Description

Content distributor

A content distributor is a stateless component that handles crawling flow control. If a content distributor server fails, the outstanding item batches associated with this content distributor will fail too. If you have redundant content distributors, the flow control protocol will ensure that the indexing connector resubmits the subset of item batches associated with the failing content distributor.

Item processing

An item processing component handles a given set of item batches. If a server that is running item processing fails, the outstanding item batches associated with the item processing instances running on this server will fail too. If you distribute item processing to one or more servers in the farm, each running one processor thread, the flow control protocol will ensure that the indexing connector resubmits the subset of item batches associated with the failing item processing server.

Link analysis

Link analysis consists of database lookup and link processing, and is performed by the web analyzer component. The web analyzer component operates in a batch processing mode, and can distribute link analysis jobs to multiple servers that are running link analysis subcomponents.

If a server that is running link analysis fails with unrecoverable disk errors, you must re-establish the web analyzer link database from the latest backup. If no backup exists, the ranking based on link analysis will be incomplete until you complete a re-crawl. If a server performing database lookup fails, and there is no redundancy, this blocks crawling. You can configure redundancy for the database lookup component during deployment, by modifying the deployment configuration file.

Indexing dispatcher

An indexing dispatcher is a stateless component that handles indexing flow control. If an indexing dispatcher server fails, the outstanding item batches associated with this indexing dispatcher will fail too. If you have redundant indexing dispatchers, the flow control protocol will ensure that the indexing connector resubmits the subset of item batches associated with the failing indexing dispatcher.

Indexer

There is one active indexer per index column. You can scale out the indexing component by defining multiple index columns. Within each index column, you can set up a backup indexer for high availability. If an indexer server fails with unrecoverable disk errors, you must either recover the server from the latest backup, or manually enable a backup indexer to be the new primary indexer.

Full indexing redundancy requires a backup indexer on a separate row, which will give you an increased server count and additional storage volume requirements. Full redundancy provides the quickest recovery path from hardware failures, but other options might be more attractive when hardware outages are infrequent:

  • You can run full re-crawl of all the content sources after recovery. Depending on the deployment this may take several days. If you have a separate search row, you can perform the re-crawl while keeping the old index searchable.

  • You can run regular backup of the index data.

FAST Search specific indexing connectors

There are three FAST Search specific connectors:

  • FAST Search Web crawler

  • FAST Search Lotus Notes connector

  • FAST Search database connector

The FAST Search Web crawler is an alternative indexing connector that we recommend for large-scale web crawl use cases. You can scale out this component by deploying multiple node schedulers that will handle crawl scheduling of different parts of the overall crawl.

The FAST Search Lotus Notes and FAST Search database connectors are stand-alone components, each with one or more associated content repositories, and you can scale out the system by deploying multiple instances of the indexing connectors.

Search redundancy and availability

The following components within the FAST Search Server 2010 for SharePoint farm support redundancy and availability:

Component | Description
Query matching | There is at least one query matching component per index column. You can scale out the query matching component by defining multiple search rows. If a query matching server fails, the outstanding queries dispatched to the search row that contains the server will fail too. Subsequent queries will be handled by another search row in the given index column.
Query processing | Query processing is a stateless component that handles query processing flow control. If a query processing server fails, the remaining queries associated with this query processing server will fail too. If you have multiple query processing components, the flow control protocol will ensure that subsequent queries will be handled by another query processing server.
Note:

The system will in most cases provide search availability during crawling or indexing failure recovery. But be aware of the following:

If you must recover the system from backup, search will be down for the time that is required to complete the recovery. Depending on the size of the backup, this may take significant time.

If you exclude the binary index from the backup, the time to recover the index becomes significantly longer. The reason is that you must rebuild the index from the pre-index item store (FiXML files). In such situations, you can improve availability for search by keeping the latest available index (before the error situation) on non-affected search rows available for queries until the index is rebuilt from the backup, at the cost of a longer freshness delay before new content is made searchable.

If you do not have end-to-end fault-tolerance in the crawling and indexing chain, and you do not take data backups, you will have to re-crawl and re-index all content from the source repositories. During the re-crawl only content that was indexed after you started the re-crawl will be available for search.

SQL Server database redundancy and availability

A FAST Search Server 2010 for SharePoint farm must have access to a Microsoft SQL Server host that stores configuration information; typically you use an existing SQL Server host within the parent SharePoint Server 2010 farm

For more information about redundancy and availability for the SQL Server within the SharePoint Server farm, refer to Storage and SQL Server capacity planning and configuration (SharePoint Server 2010). Note that the reference to "search" in that topic refers to SharePoint Server 2010 search, which also uses the database as a property store for the index.

Note:

The FAST Search Server 2010 for SharePoint farm does not use the SQL Server database for indexing metadata properties. This differs from the default SharePoint Server 2010 search.

Storage subsystem redundancy and availability

The storage subsystem for a FAST Search Server 2010 for SharePoint farm must have some redundancy because loss of storage even in a redundant setup will lead to decreased performance during a recovery period that can last for days. We recommend that you use a redundant RAID disk set like RAID1, RAID10, RAID5 or RAID50 for your storage subsystem, preferably also with hot spares.

Front-end web server redundancy and availability

You can deploy front-end web servers in the parent SharePoint Server 2010 farm, or in child farms. In the latter case, you connect the front-end web servers via a SSA Proxy in the parent farm. You should plan availability for the front-end web servers according to the guidelines in Plan for availability (SharePoint Server 2010).

See Also

Concepts

Recommendations: Content volume capacity (FAST Search Server 2010 for SharePoint)
Recommendations: Content freshness (FAST Search Server 2010 for SharePoint)
Recommendations: Query throughput (FAST Search Server 2010 for SharePoint)
Recommendations: Storage (FAST Search Server 2010 for SharePoint)
Recommendations: Virtualization (FAST Search Server 2010 for SharePoint)


Recommendations: Content freshness (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

This article describes our recommendations for achieving content freshness in a Microsoft FAST Search Server 2010 for SharePoint environment. For a complete recommendations overview, refer to Performance and capacity recommendations (FAST Search Server 2010 for SharePoint).

Content freshness depends on how fast your deployment can crawl and process content. You should first perform some benchmark testing of your deployment to determine how long it takes to crawl, process and index content, and then evaluate whether it interferes with your target user response times.

If you want to increase the content freshness you have the following options:

  • You can scale out the indexing connector. For more information about how to increase content freshness by redundancy, refer to Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint).

  • If you are using the FAST Search Content SSA, you can increase the number of concurrent crawling requests.

    The Content SSA is based on a content pull approach, where the length of each crawl cycle determines the average time to discover changed content that must be indexed. You can increase the number of concurrent requests that the crawler generates when it crawls using crawler impact rules. The more concurrent requests, the faster you crawl. What you want to achieve is that all changes in the content repositories result in a re-indexed document. As long as you dimension the connector to be able to catch all updates, the load on the item processing and indexing will not increase even if you reduce the time that is required to crawl all content sources. For more information about crawler impact rules, refer to Manage crawler impact rules (FAST Search Server 2010 for SharePoint).

  • You can deploy the item processing component to all servers in your deployment to minimize the overhead related to document format parsing and extraction of searchable content and metadata. Crawling performance is mainly determined by the item processing capacity. Therefore, it is important that you deploy the item processing components in a way that utilizes spare CPU capacity across all servers.

    Note:

    For the item processing component, do not deploy more than 20 item processors (<document-processor processes="20" /> in deployment.xml).

See Also

Concepts

Recommendations: Content volume capacity (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Recommendations: Query throughput (FAST Search Server 2010 for SharePoint)
Recommendations: Storage (FAST Search Server 2010 for SharePoint)
Recommendations: Virtualization (FAST Search Server 2010 for SharePoint)


Recommendations: Query throughput (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

This article describes our recommendations achieving high query throughput in a Microsoft FAST Search Server 2010 for SharePoint environment. For a complete recommendations overview, refer to Performance and capacity recommendations (FAST Search Server 2010 for SharePoint).

There are two main parameters that you must consider when dimensioning for query performance:

  • Maximum number of queries

  • Query latency

You measure the maximum number of queries served in queries per second (QPS). For intranet search solutions this is usually not a limiting factor. But if you must handle peak query rates of more than approximately 5 QPS, you should consider scaling out the search solution for this dimensioning parameter. You can scale out for more QPS by adding more search rows to your deployment. You can also add more query components to the FAST Search Query SSA.

Tip:

In SharePoint Server 2010, you can have one parent farm serve queries for multiple child farms. You connect the front-end web servers in the child farm to the FAST Search Query SSA in the parent farm via the SSA Proxy.

The query latency is the average round-trip delay from the point where you issue a query until you get a query result in return. You can reduce this latency by limiting the number of items per column or deploy one or more additional search rows, or both. By adding search rows, you avoid that the indexing load affects the query latency, and you will also achieve increased query availability. We recommend that you deploy a separate search row for deployments where the query latency must be continuously low.

You can also combine a separate search row with a backup indexer. This will provide short recovery time in case of a non-recoverable disk error, with some loss of query performance and incremental update rates. For the highest query performance requirements, we recommend a pure search row.

Note:

If you run indexer, item processing and query matching on the same server, you should schedule all crawling outside periods with high query load to avoid decrease in query performance during crawling.

For more information about how to add search rows and scaling out the FAST Query SSA, refer to Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)

The following table summarizes the performance effect of FAST Search Server 2010 for SharePoint search related features. Use these values as a rule-of-thumb but be aware that there are also inter-dependencies between the features that we have not covered in this table.

Feature | Item processing | Indexing | Query matching | Query processing | RAM - Query matching | Disk access | Disk space | Net/IO
Deep refiners |  | M | L | L | H1 |  |  | L-M2
Shallow refiners |  |  | L | H |  |  |  | H3
Property extraction | M |  |  |  |  |  | L |
Trim duplicates |  |  | M |  |  |  |  | L-H4
Full-text sorting |  |  | L | L | H1 |  |  |
Hit highlighted summary |  |  | M |  |  |  |  |
Complex queries (many terms) |  |  | M | M |  | H |  |
Substring search |  |  |  |  |  |  | L-M5 |
Stemming | L | L | L | L |  |  |  |
Spell check |  |  | L | L |  | L |  |
Synonyms |  |  | L | L |  | L |  |
High stop-word threshold6 |  |  | H | H |  | H |  |
Managed property boost |  |  | L | L |  | L |  |

H=High, M=Medium and L=Low. When no letter is specified, enabling the feature has insignificant effect on the corresponding resource compared to disabling it.

Notes:

  1. The memory usage pattern is similar for deep refiners and full-text sorting. The query matching component keeps aggregation data for the associated managed properties in main memory. The memory usage effect is proportional with the number of items per column and the number of unique values for the associated managed property.

  2. Deep string refiners with many unique values in the index will have good I/O performance effect on the interface between query matching and query processing nodes. The I/O load performance effect is proportional with number of columns and the QPS in the farm.

  3. We recommend that you use deep refiners for query refinement in most cases, but using shallow string refiners has good I/O performance effect on the interface between query matching and query processing components. The I/O load performance effect is proportional with the average size of the associated managed property, the number of columns and the QPS in the farm.

  4. Duplicate trimming may in certain cases have good I/O performance effect on the interface between query matching and query processing components. The I/O load performance effect is proportional with the average number of duplicates per query result, the number of columns and the QPS in the farm.

  5. If you apply substring search to large managed properties (e.g. the body), this has good effect on the index disk usage.

See Also

Concepts

Recommendations: Content volume capacity (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Recommendations: Content freshness (FAST Search Server 2010 for SharePoint)
Recommendations: Storage (FAST Search Server 2010 for SharePoint)
Recommendations: Virtualization (FAST Search Server 2010 for SharePoint)


Recommendations: Storage (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

This article describes storage recommendations for Microsoft FAST Search Server 2010 for SharePoint. For a complete recommendations overview, refer to Performance and capacity recommendations (FAST Search Server 2010 for SharePoint).

In general, we recommend high performance storage. High performance storage provides improved query performance, especially during concurrent content crawls, and reduces or even eliminates the performance driven need to run search on dedicated rows.

Deployments on storage area networks (SAN)

FAST Search Server 2010 for SharePoint can use SAN storage instead of local disks if necessary for operational reasons. Testing has shown that a sufficiently powerful SAN will not be a bottleneck. Although the actual workload is scenario dependent, the following parameters can be used as an initial estimation of the required SAN resources for a server in the FAST Search Server 2010 for SharePoint farm:

  • 2000 - 3000 I/O operations per second (IOPS)

  • 50 - 100 kB average block size

  • Less than 10 ms average read latency

For example, for a farm setup with 7 servers, the SAN must be able to serve 15.000 - 20.000 IOPS to the FAST Search Server 2010 for SharePoint farm regardless of any other traffic the same storage system may serve.

Deployments on solid state disks (SSD)

FAST Search Server 2010 for SharePoint can take advantage of the increased IO performance of solid state disks. Deployments with low content volume, high QPS and regular spindles will typically require many disks, whereas testing has shown that if you are using solid state disks, it can be sufficient with only a couple. This allows for using blade servers that have local disks on the blade itself, and still having sufficient IO performance.

SSDs make it possible to have high query performance without a dedicated search row, and let you cut the server count. But be aware that removing the search row also removes redundancy and reduces availability in case of server failures.

See Also

Concepts

Recommendations: Content volume capacity (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Recommendations: Content freshness (FAST Search Server 2010 for SharePoint)
Recommendations: Query throughput (FAST Search Server 2010 for SharePoint)
Recommendations: Virtualization (FAST Search Server 2010 for SharePoint)


Recommendations: Virtualization (FAST Search Server 2010 for SharePoint)
See Also

Published: March 17, 2011

This article describes virtualization recommendations for Microsoft FAST Search Server 2010 for SharePoint. For a complete recommendations overview, refer to Performance and capacity recommendations (FAST Search Server 2010 for SharePoint).

FAST Search Server 2010 for SharePoint supports virtualization, but for larger deployments, we recommend that you only use virtualization for your test and development environments- not the production environment. Our rationale is as follows:

  • FAST Search Server 2010 for SharePoint is a heavy user of CPU and IO

  • Virtual machines only have access to a limited number of CPU cores (Hyper-V = 4, VM ware = 8)

  • Virtual machines will give 30-40% decrease in overall performance

For smaller deployments, virtualization might be a good alternative for production environments even though the performance decreases. We tested virtualization within the extra-small test scenario (with up to 8 million items) and got the following results:

Overall, the virtual deployments showed 35-70% decrease in crawling and indexing performance compared to the deployments running on physical hardware. Even when crawls were not running, the virtualized deployments had a significantly lower query performance with more variation than the non-virtualized options. During a full crawl, the virtualized deployments had over 10 QPS. The non-virtualized deployments had more than five times the throughput. The performance reduction was related to the storage performance in addition to the search components having a limited number of CPU cores at disposal.

For more information of the test environment, refer to Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint).

Tip:

FAST Search Server 2010 for SharePoint can be installed on virtual environments that use task offloading. Task offloading reduce load on the CPU caused by the construction, framing and validating of network packets and maintaining network sessions (TCP), but we recommend that you disable this feature because it causes decreased crawl performance.

See Also

Concepts

Recommendations: Content volume capacity (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Recommendations: Content freshness (FAST Search Server 2010 for SharePoint)
Recommendations: Query throughput (FAST Search Server 2010 for SharePoint)
Recommendations: Storage (FAST Search Server 2010 for SharePoint)


System requirements (FAST Search Server 2010 for SharePoint)
See Also

Before you install Microsoft FAST Search Server 2010 for SharePoint, you must ensure that you have installed all required hardware and software. To effectively plan your deployment, you must understand the level of support provided for the Web browsers that you will be using in your environment.

This article provides references to articles describing system requirements for the FAST Search Server 2010 for SharePoint farm and the parent Microsoft SharePoint Server farm.

For an overview of the farm topology, see Plan search topology (FAST Search Server 2010 for SharePoint).

System Requirements for SharePoint Server farm

System requirements for the parent SharePoint Server farm can be found here:

  • Determine hardware and software requirements (SharePoint Server 2010)

    This article describes the hardware and software requirements that you must meet to successfully install SharePoint Server.

  • Plan browser support (SharePoint Server 2010)

    This article describes levels of support for Web browsers to use with SharePoint Server.

System Requirements for FAST Search for SharePoint farm

System requirements for the FAST Search Server 2010 for SharePoint farm can be found here:

  • Review hardware and software requirements (FAST Search Server 2010 for SharePoint)

    This article describes the hardware and software requirements that you must meet to successfully install and deploy FAST Search Server 2010 for SharePoint.

See Also

Concepts

Plan search topology (FAST Search Server 2010 for SharePoint)


Security considerations for the server farm (FAST Search Server 2010 for SharePoint)
See Also

When planning a Microsoft FAST Search Server 2010 for SharePoint system, consider the following server farm security issues:

  • Certificates

  • Communication between the FAST Search Server farm and the SharePoint farm

  • Protecting administrative interfaces

  • Protecting communication within the farm

  • Proxy settings

  • Anti-virus configuration

  • Required user accounts

  • User authentication

For item level security trimming specific to FAST Search Server 2010 for SharePoint, see Security considerations for indexing (FAST Search Server 2010 for SharePoint).

Certificates

FAST Search Server 2010 for SharePoint uses certificates for:

  • Authentication and encryption

  • Secure Sockets Layer (SSL) communication between FAST Search Server 2010 for SharePoint and Microsoft SharePoint Server

  • Communication between servers in a multiple server FAST Search Server 2010 for SharePoint environment

Each server in a FAST Search Server 2010 for SharePoint system may have up to three certificates, fulfilling the following functions:

  • General purpose FAST Search certificate: for internal communications, administrative services, and feeding SharePoint Server. The general purpose FAST Search certificate must also be password-protected. You will choose a password during FAST Search Server 2010 for SharePoint deployment.

  • Claims certificate: to enable queries from the SharePoint Server search application to FAST Search Server 2010 for SharePoint

  • Server-specific certificate: for example, to help secure query traffic using HTTPS (optional)

Important:

When you install FAST Search Server 2010 for SharePoint, a self-signed certificate is created. This default general purpose certificate has a one year expiration date and is only useful for test environments. You should replace self-signed certificates in your production environment with certificates that are signed by a common certification authority. For more information, see Manage certificates (FAST Search Server 2010 for SharePoint).

Communication between the FAST Search Server farm and the SharePoint farm

All internal communication within the FAST Search Server 2010 for SharePoint farm uses Internet Protocol Security (IPsec). You can find details about required open ports and protocols for the communication between the FAST Search Server 2010 for SharePoint farm and the Search Service Applications (SSA) in the file <FASTSearchFolder>\Install_Info.txt (where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch).

By default, all query traffic from the FAST Search Query Search Service Application (SSA) to the FAST Search Server 2010 for SharePoint farm is sent via HTTP. This non-encrypted information transmits faster than HTTPS. However, to help provide more security for queries on sensitive content, you can enable an HTTPS communication channel that uses SSL certificates. See Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint) for more information.

Protecting administrative interfaces

By default, the Administration Service, which configures FAST Search Server 2010 for SharePoint, uses Windows Communication Foundation (WCF) with HTTP. To provide more protection, you can use HTTPS for this traffic. See Enable Administration Service over HTTPS (FAST Search Server 2010 for SharePoint) for information.

Default authentication for the administrative interfaces (e.g. Add Best Bets) uses NTLM out-of-the-box. If you want an additional level of security, you can change this to Kerberos authentication. See Configure Kerberos authentication (SharePoint Server 2010) for more information.

Protecting communication within the farm

By default, all internal communication within the FAST Search Server 2010 for SharePoint farm uses Internet Protocol Security (IPsec) without encryption. To help protect sensitive content, you can enable IPsec encryption on internal interfaces.

Proxy settings

HTTP communications are used in multiple server FAST Search Server 2010 for SharePoint farms and between query traffic from the FAST Search Query Search Service Application (SSA) to the FAST Search Server 2010 for SharePoint farm. HTTP communication must be enabled between all servers and the network proxy configuration on each server must be set correctly. See Review hardware and software requirements (FAST Search Server 2010 for SharePoint) for detailed information.

Anti-virus configuration

When you install FAST Search Server 2010 for SharePoint on a server with anti-virus software installed, you should exclude the <FASTSearchFolder> directory from virus scanning. See Review hardware and software requirements (FAST Search Server 2010 for SharePoint) for more information.

Required user accounts

A multiple server installation of FAST Search Server 2010 for SharePoint requires credentials for certain user accounts to install, administer, and operate FAST Search Server 2010 for SharePoint. Plan for the following permissions:

  • The user who runs the Prerequisite Installer and the FAST Search Server 2010 for SharePoint installer must be a member of the Administrators group.

  • An authenticated domain user must run FAST Search Server 2010 for SharePoint. This user should not be a local administrator or a site administrator.

  • The FAST Search Server 2010 for SharePoint user must have dbcreator permissions in Microsoft SQL Server to access the FAST Search Server 2010 for SharePoint administration database. See Configure a stand-alone deployment or a multiple server deployment (FAST Search Server 2010 for SharePoint) for more information.

See Review hardware and software requirements (FAST Search Server 2010 for SharePoint) for more information.

User authentication

FAST Search Authorization (FSA) provides item level security for FAST Search Server 2010 for SharePoint systems by implementing security trimming. However, FSA does not authenticate users. Authentication is performed by the SharePoint Server search front-end. See Plan authentication methods (SharePoint Server 2010) for more information.

See Also

Concepts

Security considerations for indexing (FAST Search Server 2010 for SharePoint)
Manage certificates (FAST Search Server 2010 for SharePoint)
Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint)
Enable Administration Service over HTTPS (FAST Search Server 2010 for SharePoint)
Review hardware and software requirements (FAST Search Server 2010 for SharePoint)
Configure a stand-alone deployment or a multiple server deployment (FAST Search Server 2010 for SharePoint)

Other Resources

Plan authentication methods (SharePoint Server 2010)


Deployment for FAST Search Server 2010 for SharePoint
See Also

To install Microsoft FAST Search Server 2010 for SharePoint as a stand-alone deployment (single server deployment) or multiple server deployment, perform the tasks outlined in this overview.

Before installing FAST Search Server 2010 for SharePoint, review the information in in Planning and architecture (FAST Search Server 2010 for SharePoint).

Install FAST Search Server 2010 for SharePoint

  1. Review hardware and software requirements (FAST Search Server 2010 for SharePoint)

  2. Install Microsoft SharePoint Server 2010 and Microsoft SQL Server 2008 (FAST Search Server 2010 for SharePoint)

  3. Install FAST Search Server 2010 for SharePoint:

    1. Download the installation package

    2. Open the FAST Search Server 2010 for SharePoint Start page

    3. Install prerequisites

    4. Enable Windows PowerShell scripts to run

    5. Install FAST Search Server 2010 for SharePoint

    6. Install updates

    7. Set up permissions for Microsoft SQL Server

  4. Configure a stand-alone deployment or a multiple server deployment (FAST Search Server 2010 for SharePoint):

    1. Configure a stand-alone deployment:

      1. Configure a stand-alone deployment

      2. Verify that all modules are running

      or

    2. Configure a multiple server deployment:

      1. Prepare the deployment file

      2. Configure the administration server in a multiple server deployment

      3. Verify that all modules are running on the administration server

      4. Configure the non-administration servers in a multiple server deployment

      5. Verify that all modules are running on the non-administration servers

  5. Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint):

    1. Create and set up the FAST Search Content SSA in SharePoint Server Central Administration

    2. Set up SSL on each server in the SharePoint farm that runs a Content SSA crawl component

    3. Add a content source

  6. Create and set up the Query Search Service Application (FAST Search Server 2010 for SharePoint)

    1. Create and set up the FAST Search Query SSA in SharePoint Server Central Administration

    2. Verify user access to FAST Search Server 2010 for SharePoint administrative services

    3. Connect the FAST Query SSA to your Web application(s)

  7. Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint):

    1. Configure claims authentication

    2. Optional: Enable HTTPS

  8. Create a FAST Search Center site (FAST Search Server 2010 for SharePoint)

  9. Verify successful installation (FAST Search Server 2010 for SharePoint)

See Also

Concepts

Planning and architecture (FAST Search Server 2010 for SharePoint)
Advanced installation and configuration options (FAST Search Server 2010 for SharePoint)
Uninstall FAST Search Server 2010 for SharePoint (FAST Search Server 2010 for SharePoint)
User roles required for deployment tasks


Review hardware and software requirements (FAST Search Server 2010 for SharePoint)
See Also

Before you start the installation of FAST Search Server 2010 for SharePoint, make sure that the system meets the hardware and software requirements as mentioned in this article.

In this article:

  • Hardware requirements

  • Software requirements

Hardware requirements

The following is the minimum and recommended hardware requirements to deploy FAST Search Server 2010 for SharePoint:

Minimum hardware requirements | Recommended hardware requirements
4 GB RAM | 16 GB RAM
4 CPU cores, 2.0GHz CPU | 8 CPU cores, 2.0GHz CPU
50 GB disk | 1 terabyte disk space on RAID across 6 spindles or more. In general, RAID5 configurations with 128 KB stripe size provide a good combination of performance and reliability as content storage for FAST Search Server 2010 for SharePoint. Different RAID controllers have varying sweet spots for operation. SAS disks are recommended over SATA. Within a fixed power and cost budget, performance will be better with a larger number of 10 K RPM disks compared to a smaller number of 15 K RPM disk.

Software requirements

Install FAST Search Server 2010 for SharePoint on a server that has a clean installation of one of the following:

  • Windows Server 2008 SP2 x64. All regular Standard/Enterprise/Datacenter versions are supported.

  • Windows Server 2008 R2 x64

The server must be fully updated with the Service Pack and updates and restarted before you install FAST Search Server 2010 for SharePoint.

Do not install FAST Search Server 2010 for SharePoint on a server that runs a Domain Controller.

Important:

Use the server for FAST Search Server 2010 for SharePoint only. Do not install FAST Search Server 2010 for SharePoint on the same server as SharePoint Server 2010.

Make sure that the FAST Search Server 2010 for SharePoint server name is not more than 15 characters long.

The Prerequisite Installer installs the required software components, including Microsoft .NET Framework version 3.5 SP1.

Note:

.NET Framework version 4 is not supported.

Create users and define permissions

  1. Ensure that the user who runs the installer is a member of the local Administrators group.

    Note:

    The user or group must have the Allow log on locally permission. This permission is assigned by default during installation, and should not be removed.

  2. Create a domain user for running FAST Search Server 2010 for SharePoint, for example fastuser. Do not run FAST Search Server 2010 for SharePoint as a local administrator or a domain administrator.

  3. To simplify the process of adding more search administrators, create a domain group. You will later add this group to a local group named FASTSearchAdministrators on every server in the FAST Search Server 2010 for SharePoint deployment.

    Add all users who will administer the FAST Search Server 2010 for SharePoint installation to this group.

Verify that Windows Firewall is turned on on all servers in the FAST Search Server 2010 for SharePoint farm

Verify that all servers in the FAST Search Server 2010 for SharePoint farm have Windows Firewall turned on.

In Control Panel, under System and Security, click Check firewall status and verify that Windows Firewall state is set to On.

Make sure that IP addresses are static

Make sure that all FAST Search Server 2010 for SharePoint servers have static IP addresses to enable the installer to automatically configure IPsec (Internet Protocol security) in the firewall during the installation.

When you configure a multiple server deployment, you may receive an error message that states that the non-administration servers cannot connect to the administration server. For more information, see Troubleshoot communication issues between servers.

Make sure that the firewall is configured correctly

The FAST Search Server 2010 for SharePoint installer uses the computer's default settings for Internet Protocol security (IPsec) key exchange, data protection, and authentication.

If you do not use the default settings for IPsec, for example because of domain group policies or the computer's local policies, make sure that the custom global firewall IPsec settings provide sufficient security. Also, make sure that the global IPsec settings are the same for all servers in the deployment.

Turn off automatic Windows updating

On all servers in the FAST Search Server 2010 for SharePoint farm, turn off automatic updating in Windows Update. Plan to install updates after a controlled shutdown of FAST Search Server 2010 for SharePoint.

Disable automatic adjustment of daylight saving time (DST) and synchronize clocks between servers

Disable automatic adjustment of daylight saving time for the servers in the FAST Search Server 2010 for SharePoint farm. If DST adjustment is enabled, query timeouts may occur for a brief period around midnight on the date of DST adjustment.

Verify that the clocks on the servers in the FAST Search Server 2010 for SharePoint farm and the SharePoint Server 2010 farm are synchronized on minute level. However, the daylight saving time (DST) setting on the servers in the SharePoint Server 2010 farm may differ from the DST setting on the servers in the FAST Search Server 2010 for SharePoint farm.

Configure anti-virus configuration

We strongly recommend that you exclude the FASTSearch folder from anti-virus scanning to prevent the anti-virus software from falsely identifying FAST Search Server 2010 for SharePoint data files as malware. If the anti-virus software allows it, we also recommend that you exclude the following FAST Search Server 2010 for SharePoint processes from scanning. All files are in the FASTSearch\bin\ folder:

  • cobra.exe

  • contentdistributor.exe

  • create_attribute_files.exe

  • docexport.exe

  • FASTSearch.exe

  • FASTSearchExec.exe

  • fdispatch.exe

  • fdmworker.exe

  • fixmlindex.exe

  • fsearch.exe

  • fsearchctrl.exe

  • ifilter2html.exe

  • indexer.exe

  • indexingdispatcher.exe

  • jsort2.exe

  • make_pu_diff.exe

  • MonitoringService.exe

  • pdftotext.exe

  • procserver.exe

  • qrserver.exe

  • spelltuner.exe

  • truncate_anchorinfo.exe

  • walinkstorerreceiver.exe

  • walookupdb.exe

  • webanalyzer.exe

Configure proxy settings

Note:

This section is only relevant if a proxy server is used for HTTP communication.

HTTP communication is used in multiple server FAST Search Server 2010 for SharePoint farms, and also between SharePoint Server 2010 and FAST Search Server 2010 for SharePoint. This requires that HTTP communication is enabled between all servers in both SharePoint Server 2010 and FAST Search Server 2010 for SharePoint.

As network proxy configuration typically determines how HTTP communication is routed from each server, the network proxy configuration on each server must be set correctly. You can configure the proxy server manually or by using automatic configuration.

You can review and edit the relevant LAN settings on each server by opening Internet Options as the FASTSearch user:

  1. Open a Command Prompt window (Click Start, click All Programs, click Accessories, and then click Command Prompt).

  2. At the command prompt, type the following command:

    Copy Code
    runas /user:<domain>\<user> "cmd /c c:\windows\system32\inetcpl.cpl"

    Where <domain> and <user> are the FASTSearch user domain and user name.

  3. Click Connections and then click LAN settings.

If you are using Automatic configuration, make sure that all servers in both FAST Search Server 2010 for SharePoint and SharePoint Server 2010 can be reached.

If you have manually configured a proxy server, verify that Bypass proxy servers for local addresses is selected.

Next step

When you have verified and set up the required hardware and software, follow the steps in Install Microsoft SharePoint Server 2010 and Microsoft SQL Server 2008 (FAST Search Server 2010 for SharePoint)

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint


Install Microsoft SharePoint Server 2010 and Microsoft SQL Server 2008 (FAST Search Server 2010 for SharePoint)
See Also

Before you install FAST Search Server 2010 for SharePoint, you must install Microsoft SharePoint Server 2010 and Microsoft SQL Server 2008.

For information about how to install SharePoint Server 2010 and SQL Server 2008, refer to Deployment for SharePoint Server 2010.

Also, make sure that the system meets the requirements in Review hardware and software requirements (FAST Search Server 2010 for SharePoint).

SharePoint Server 2010

  1. Do not install SharePoint Server 2010 and FAST Search Server 2010 for SharePoint on the same server.

  2. Both standalone and server farm installations of SharePoint Server 2010 can be used with FAST Search Server 2010 for SharePoint.

  3. To enable FAST Search functionality on SharePoint Server 2010, you must have an Enterprise License on SharePoint Server 2010.

  4. To enable document preview in search results in FAST Search Server 2010 for SharePoint, you must install and configure Microsoft Office Web Apps on the SharePoint Server 2010 farms that you index content from.

    Note:

    Document preview is only available for Word and PowerPoint documents that are stored in SharePoint Server 2010 farms.

    Refer to Microsoft Office Web Apps Resource Center for information about how to install and configure Office Web Apps.

    The document previews may take some time to render the first time that you present them in search results.

SQL Server 2008

The FAST Search Server 2010 for SharePoint deployment must have access to a SQL Server host that is used to store configuration information. FAST Search Server 2010 for SharePoint will not store any data related to the indexed content in the database. It is therefore recommended that you use a SQL Server database server that is already available in the parent SharePoint Server 2010 farm for this purpose.

Next step

When you have installed SharePoint Server 2010 and SQL Server 2008, follow the steps in Install FAST Search Server 2010 for SharePoint.

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint

Other Resources

Deployment for SharePoint Server 2010


Install FAST Search Server 2010 for SharePoint
See Also

This article describes how to install Microsoft FAST Search Server 2010 for SharePoint and the required software prerequisites. As part of this process, you will also download the installation package, enable Windows PowerShell scripts to run and set up permissions for Microsoft SQL Server.

Before you start the installation, make sure that the system meets the requirements in Review hardware and software requirements (FAST Search Server 2010 for SharePoint) and Install Microsoft SharePoint Server 2010 and Microsoft SQL Server 2008 (FAST Search Server 2010 for SharePoint).

In this article:

  1. Download the installation package

  2. Open the FAST Search Server 2010 for SharePoint Start page

  3. Install prerequisites

  4. Enable Windows PowerShell scripts to run

  5. Install FAST Search Server 2010 for SharePoint

  6. Install updates

  7. Set up permissions for Microsoft SQL Server

Note:

Perform the steps in the following sections as a user who is a member of the local Administrators group.

Download the installation package

Log in to MSDN Subscriber Downloads, expand the Servers menu, and then click FAST Search Server 2010 to download FAST Search Server 2010 for SharePoint. The installation package contains a self-extracting exe file that will open a Start page (splash.hta) and extract the installation files to the following folder: \Program Files (x86)\MSECache\FastSearchServer2010\

(Disregard the fact that the files are extracted to Program Files (x86). FAST Search Server 2010 for SharePoint is still a 64 bit system.)

Open the FAST Search Server 2010 for SharePoint Start page

When you download the installation package, the FAST Search Server 2010 for SharePoint Start page opens automatically. The Start page provides links to product documentation and to the installation wizards.

However, after any restarts of the server during the installation process, the Start page does not open automatically. Double-click the splash.hta to reopen the FAST Search Server 2010 for SharePoint Start page.

Install prerequisites

The Prerequisite Installer checks your server for required components and updates, and then downloads and installs the required components and updates if they are not already present on the system. Which prerequisites are required depends on your operating system.

Note:

  • Verify that the server has Internet access and Windows Firewall open on port 80. The Prerequisite Installer downloads components from the Microsoft Download Center. If the server does not have Internet access, see Run the Prerequisite Installer in offline mode (FAST Search Server 2010 for SharePoint).

  • The Windows Update service must be running when you run the Prerequisite Installer. If not, the Prerequisite Installer will fail.

To run the Prerequisite Installer

  1. On the FAST Search Server 2010 for SharePoint Start page, click Install software prerequisites.

  2. On the Welcome to the FAST Search for SharePoint prerequisite installer page, click Next.

  3. On the License page, verify that you accept the license terms and then click Next.

    The prerequisites are installed.

    Note:

    The Prerequisite Installer may require restarting the server during installation.

  4. On the Installation Complete page, click Finish.

  5. Restart the server.

For a multiple server deployment, repeat the procedure for all servers that will run FAST Search Server 2010 for SharePoint.

Tip:

The Prerequisite Installer adds the Web Server (IIS) role to the server, but without the IIS Management Console service. To make system management easier, we recommend that you enable the IIS Management Console in Server Manager.

Enable Windows PowerShell scripts to run

After the server restart in the previous procedure, change the policy for running Windows PowerShell scripts remotely (execution policy). The post-setup configuration wizard that you will run later must be able to run scripts during configuration.

To enable the scripts to run:

  1. From the Start menu, search for Windows PowerShell.

  2. Right-click the Windows PowerShell icon (do not use the one with x86) and select Run as administrator.

  3. At the command prompt, type the following command to enable scripts to run:

    Set-ExecutionPolicy RemoteSigned

    Answer Yes if you are asked whether you want to change the policy.

For a multiple server deployment, repeat the procedure for all servers that will run FAST Search Server 2010 for SharePoint.

Install FAST Search Server 2010 for SharePoint

This procedure describes how to install FAST Search Server 2010 for SharePoint by running the installation wizard.

Before you run the installation wizard, the Prerequisite Installer must be completed.

To install FAST Search Server 2010 for SharePoint

  1. Go to \Program Files (x86)\MSECache\FASTSearchServer2010 and double-click the splash.hta to reopen the Start page after the server restart.

  2. On the FAST Search Server 2010 for SharePoint Start page, click Install FAST Search Server 2010 for SharePoint.

  3. On the Welcome page, click Next.

  4. On the License Terms page, accept the license and then click Next.

  5. On the Destination Folder page, select the folder where you want to install FAST Search Server 2010 for SharePoint.

    Click Next.

  6. On the Ready to install page, click Install.

  7. On the page that says that the wizard is completed, click Finish.

For a multiple server deployment, repeat the procedure for all servers that will run FAST Search Server 2010 for SharePoint.

Install updates

After you have run the installation wizard, make sure that the product is up to date:

  1. On the FAST Search Server 2010 for SharePoint Start page, click Visit Windows update.

  2. Download and install all relevant updates.

For a multiple server deployment, repeat the procedure for all servers that will run FAST Search Server 2010 for SharePoint.

Set up permissions for Microsoft SQL Server

You must install SQL Server 2008 before you install FAST Search Server 2010 for SharePoint. When you run the post-setup configuration wizard to configure the server(s), FAST Search Server 2010 for SharePoint creates a database in SQL Server and needs access to the database server. Therefore, before you run the post-setup configuration wizard, make sure that the FAST Search Server 2010 for SharePoint user (the user who is entered in the wizard) has permissions to create the database.

To set up permissions for SQL Server:

  1. If you have installed SharePoint Server in stand-alone mode, you must install SQL Server Management Studio. For more information, see http://msdn.microsoft.com/library/ms365247.aspx

  2. In the SQL Server Installation Center, click Installation, and then click New SQL Server standalone installation or add features to an existing installation. Installation type is Perform a new installation of SQL Server 2008.

  3. On the server where SQL Server is installed, open SQL Server Management Studio.

  4. In the Object Explorer, move to Security. Right-click Logins and select New Login…

  5. In the Login - New dialog box, on the General page, enter the FAST Search Server 2010 for SharePoint user as the Login name.

  6. In the Select a page pane, select Server Roles

  7. Select the DBCreator role.

  8. Click OK.

After you have run the post-setup configuration wizard, you can remove the DBCreator role. However, make sure that the FAST Search user can still read and write to the database that was created.

In SQL Server Configuration Manager, also follow these steps:

  1. Verify that the main SQL Server service and the SQL Server Browser service are running.

    If the services are not running, start them by using Control Panel. Click Turn Windows features on or off, click Configuration, and then click Services.

  2. In SQL Server Network Configuration, click Protocols for <instance name>, and verify that TCP/IP is enabled.

  3. Follow the steps in: How to: Configure a Server to Listen to a Specific TCP Port.

Next step

When you have performed the tasks in this article, you must configure either a stand-alone deployment (single server deployment) or a multiple server deployment. Follow the steps in Configure a stand-alone deployment or a multiple server deployment (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint
Overview of installation files
Review hardware and software requirements (FAST Search Server 2010 for SharePoint)
Install Microsoft SharePoint Server 2010 and Microsoft SQL Server 2008 (FAST Search Server 2010 for SharePoint)


Configure a stand-alone deployment or a multiple server deployment (FAST Search Server 2010 for SharePoint)
See Also

This article describes how to configure either a stand-alone deployment (single server deployment) or a multiple server deployment by running the post-setup configuration wizard.

Before you can run the post-setup configuration wizard to do the initial configuration of the servers, follow the steps in Install FAST Search Server 2010 for SharePoint.

Select one of the following:

  1. Configure a stand-alone deployment (single server deployment)

    or

  2. Configure a multiple server deployment

  • A stand-alone deployment is useful when you want a deployment for product evaluation or demonstration, a small-scale deployment with minimal administrative overhead, or a development environment or test environment. A stand-alone deployment is an installation of FAST Search Server 2010 for SharePoint on a single server. All required components are installed on this server.

  • A multiple server deployment is recommended for most production environments, except small deployments without any fault-tolerance requirements. A multiple server deployment is an installation of FAST Search Server 2010 for SharePoint services on multiple servers. During configuration, one server is defined as the administration server, which is the server where the administrative services are installed and run. The deployment is scaled out by adding one or more non-administration servers. A non-administration server does not have the administrative services set up, but connects to a running administration server in a multiple server deployment. The non-administration server runs services such as query matching, indexing and document/item processing.

Configure a stand-alone deployment (single server deployment)

This section describes how to configure a stand-alone deployment (single server deployment) by running the configuration wizard.

In this section:

  1. Configure a stand-alone deployment

  2. Verify that all modules are running

Configure a stand-alone deployment

  1. From the Start menu, run Microsoft FAST Search Server 2010 for SharePoint Configuration Wizard as a local administrator.

  2. On the Welcome page, click Next.

  3. On the Select the deployment type page, select Single server (stand-alone) and then click Next.

  4. On the Enter the user name and password page, enter the user name and password for the FAST Search Server 2010 for SharePoint user. This user must be a domain user. Refer to Required permissions for more information.

  5. On the Enter a certificate password page, enter a certificate password of your choice. This password will be used to protect the general purpose FAST Search certificate (see also Manage certificates (FAST Search Server 2010 for SharePoint)). You will need this password to install certificates on other servers. Retype the password and then click Next.

  6. On the Server settings page, enter the fully qualified domain name of the server and the base port. A deployment file will be generated automatically with all the settings configured.

    Note:

    If you require a specific setup, you can create and use a custom deployment file. If you want to do this, select Use an existing deployment file, and then enter the Deployment file location. Refer to deployment.xml reference for more information.

  7. On the Database settings page, enter connection information to the SQL database that you installed and created before you installed FAST Search Server 2010 for SharePoint. Enter the Database connection string in the format <ServerName>\<DBInstanceName> (Example: mysqlserver1.contoso.com\myinstance).

    To determine the database connection string: On the server where the SQL Server is installed, open SQL Server Configuration Manager from the Start menu. Under SQL Server Services, you see the SQL Server main service. The service name contains the connection string in parentheses.

    Enter the Database name and Database port. The database will be created when you complete the configuration wizard. Click Next.

  8. On the Click-through relevancy settings page, add the desired relevancy click-through settings. Click-through relevancy enables automatic relevancy tuning based on how search users click on results. Select the relevant Microsoft SharePoint Server 2010 installation type.

    • If you have installed SharePoint Server 2010 using the Standalone option, specify the Fully Qualified Domain Name (FQDN) of the SharePoint Server 2010.

    • If you have installed SharePoint Server 2010 using the Server Farm option, specify the user who is running the Microsoft SharePoint 2010 Timer Service. This user is specified in the SharePoint Server 2010 post-setup configuration.

    • If you do not want click-through relevancy, or if you do not have the necessary information, select Do not enable click-through relevancy.

    Click Next.

  9. On the Configuration settings summary page, review the settings and then click Configure.

    Wait while the configuration finishes.

  10. On the Post-setup configuration was successful page, click Finish.

    Note:

    If the configuration fails, you will see an error message with a link to the relevant log file if possible. For information about how to troubleshoot a configuration failure, see Installation: Configuration wizard fails (FAST Search Server 2010 for SharePoint).

  11. Before you restart the server, make sure that administrative permissions are defined. Update the local FASTSearchAdministrators group (which was created by the configuration wizard) to include either the user who will administer the FAST Search Server 2010 for SharePoint farm, or the domain group of your FAST Search Administrators.

  12. Restart the server.

Verify that all modules are running

  1. Log in as a user who is a member of the local FASTSearchAdministrators group.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Right-click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

  5. At the Windows PowerShell command prompt, type the following command:

    nctrl status

  6. Make sure that all modules have the status Running.

Next step

You have now completed the initial configuration of a stand-alone deployment. To be able to crawl content by using the connector framework provided by SharePoint Server, follow the steps in Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint) to create and set up the FAST Search Content Search Service Application (Content SSA).

Configure a multiple server deployment

This section describes how to configure a multiple server deployment by running the configuration wizard. For more information about how to plan your farm topology, see Plan search topology (FAST Search Server 2010 for SharePoint).

In this section:

  1. Prepare the deployment file

  2. Configure the administration server in a multiple server deployment

  3. Verify that all modules are running on the administration server

  4. Configure the non-administration server(s) in a multiple server deployment

  5. Verify that all modules are running on the non-administration server

Prepare the deployment file

The deployment file, deployment.xml, specifies the distribution of components and services across servers in a FAST Search Server 2010 for SharePoint farm. Each FAST Search Server 2010 for SharePoint farm has one deployment file. Each server in the farm can read the deployment file and determine the services that run locally on that server. Each server also determines the location of other services that it communicates with.

The deployment file is a required input parameter when you run the configuration wizard (or configuration script) on the administration server. Non-administration servers will download the deployment file from the administration server, determine which services should run on the local server, and start these services.

An example deployment.xml template is shown later in this document: Deployment.xml template. You can also find example deployment.xml templates directly under <FASTSearchFolder>\etc\, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

For information about how to prepare the deployment file, see deployment.xml reference.

The completed deployment file to be used for the installation must be named deployment.xml.

The FAST Search Server 2010 for SharePoint administrator can reconfigure the deployment by adding or removing servers or services in the deployment file. For more information about reconfiguring, see Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint).

Deployment.xml template

The following template shows a multiple server deployment file, with three servers.

Copy Code
<?xml version="1.0" encoding="utf-8" ?>
<deployment comment="FAST Search Server three node deployment example" xmlns="http://www.microsoft.com/enterprisesearch">
  <instanceid>FASTSearchMultiNodeDemo</instanceid>
  <!--
 NOTE: Host names "fastserver1.contoso.com", "fastserver2.contoso.com" and "fastserver3.contoso.com"
       must be changed to the actual server names in use
  -->
  <connector-databaseconnectionstring />
  <host name="fastserver1.contoso.com">
    <admin />
    <indexing-dispatcher />
    <content-distributor />
    <webanalyzer server="true" link-processing="true" lookup-db="true" />
    <document-processor processes="2" />
  </host>
  <host name="fastserver2.contoso.com">
    <searchengine row="0" column="0" />
  </host>
  <host name="fastserver3.contoso.com">
    <searchengine row="1" column="0" />
    <query />
  </host>
  <searchcluster>
    <row id="0" index="primary" search="true" />
    <row id="1" index="none" search="true" />
  </searchcluster>
</deployment>

Configure the administration server in a multiple server deployment

This section describes how to configure a server as the administration server in a multiple server deployment by running the configuration wizard. The administration server must be configured and running before you can add one or more non-administration servers to the deployment.

Make sure to follow these steps as a user who is a member of the local Administrators group.

  1. Prepare a deployment file as described in the deployment.xml reference document.

  2. From the Start menu, run the Microsoft FAST Search Server 2010 for SharePoint Configuration Wizard.

  3. On the Welcome page, click Next.

  4. On the Select the deployment type page, select Admin server and then click Next.

  5. On the Enter the user name and password page, enter the user name and password for the FAST Search Server 2010 for SharePoint user. This user must be a domain user.

  6. On the Enter a certificate password page, enter a certificate password of your choice. This password will be used to protect the general purpose FAST Search certificate (see also Manage certificates (FAST Search Server 2010 for SharePoint)). You will need this password to install certificates on other servers. Retype the password and then click Next.

  7. On the Server settings page, enter the fully qualified domain name of the server and the base port. Make sure that the server name is the same as the administration server that is defined in deployment.xml.

    In the Deployment file location box, enter the path of the deployment.xml file that you created. If there is an error in your deployment.xml file, you will get a warning. If no warnings or errors appear, your deployment.xml file syntax is correct. Click Next.

  8. On the Database settings page, enter the connection information to the SQL database that you installed and created before you installed FAST Search Server 2010 for SharePoint. Enter the Database connection string in the format <ServerName>\<DBInstanceName> (Example: mysqlserver1.contoso.com\myinstance). Enter the Database name and Database port. The database will be created when you complete this configuration wizard. Click Next.

  9. On the Click-through relevancy settings page, add the desired relevancy click-through settings. Click-through relevancy enables automatic relevancy tuning based on how search users click on results. Select the relevant SharePoint Server 2010 installation type.

    • If you have installed SharePoint Server 2010 using the Standalone option, specify the Fully Qualified Domain Name (FQDN) of the SharePoint Server 2010.

    • If you have installed SharePoint Server 2010 using the Server Farm option, specify the user who is running the timer service in SharePoint Server 2010. This user is specified in the SharePoint Server 2010 post-setup configuration.

    • If you do not want click-through relevancy, or if you do not have the necessary information, select Do not enable click-through relevancy.

    Click Next.

  10. On the Configuration settings summary page, review the settings and then click Configure.

    Wait while the configuration finishes.

  11. On the Post-setup configuration was successful page, click Finish.

    Note:

    If the configuration fails, you will see an error message with a link to the relevant log file if possible. For information about how to troubleshoot a configuration failure, see Installation: Configuration wizard fails (FAST Search Server 2010 for SharePoint).

  12. Before you restart the server, make sure that administrative permissions are defined. Update the local FASTSearchAdministrators group (which was created by the configuration wizard) to include either the user who will administer the FAST Search Server 2010 for SharePoint farm, or the domain group of your FAST Search Administrators.

  13. Restart the server.

Verify that all modules are running on the administration server

  1. Log in as a user who is a member of the local FASTSearchAdministrators group.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Right-click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

  5. At the Windows PowerShell command prompt, type the following command:

    nctrl status

  6. Make sure that all modules have the status Running.

Configure the non-administration server(s) in a multiple server deployment

By default, all servers in a multiple server deployment, except the administration server, are non-administration servers. In this section, you will configure each non-administration server that is specified in your deployment file as a non-administration server by running the configuration wizard. As part of this configuration process, the server will connect to the administration server to join the FAST Search Server 2010 for SharePoint farm.

First, verify that the newly configured administration server is running, by following the steps in Verify that all modules are running on the administration server.

On each non-administration server, do the following:

  1. Log in as a user who is a member of the local Administrators group.

  2. On the Start menu, right-click Microsoft FAST Search Server 2010 for SharePoint Configuration Wizard, and select Run as administrator.

  3. On the Welcome page, click Next.

  4. On the Select the deployment type page, select Non-admin server and then click Next.

  5. On the Enter the user name and password page, enter the user name and password for the FAST Search Server 2010 for SharePoint user. This must be the same user as you specified for the administration server during post-setup configuration.

  6. On the Enter a certificate password page, enter the certificate password. Use the password that was used for the certificate in the administration server post-setup configuration. Click Next.

  7. On the Server settings page, enter the fully qualified domain name of the server. Make sure that the server name is the same as the corresponding server name that is defined in deployment.xml.

    Enter the fully qualified domain name and the base port of the administration server and then click Next.

  8. On the Configuration settings summary page, review the settings and then click Configure.

    Wait while the configuration finishes.

  9. On the Post-setup configuration was successful page, click Finish.

    Note:

    If the configuration fails, you will see an error message with a link to the relevant log file if possible. For information about how to troubleshoot a configuration failure, see Installation: Configuration wizard fails (FAST Search Server 2010 for SharePoint).

  10. Before you restart the server, make sure that administrative permissions are defined. Update the local FASTSearchAdministrators group (which was created by the configuration wizard) to include either the user who will administer the FAST Search Server 2010 for SharePoint farm, or the domain group of your FAST Search Administrators.

  11. Restart the server.

    Important:

    Skip this step if you perform this procedure during recovery after an indexer server failure (the procedures described in Replace a failed primary indexer server and Replace a failed backup indexer server).

Verify that all modules are running on the non-administration server

  1. Log in as a user who is a member of the local FASTSearchAdministrators group.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Right-click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

  5. At the Windows PowerShell command prompt, type the following command:

    nctrl status

  6. Make sure that all modules have the status Running.

Next step

You have now completed the initial configuration of a multiple server deployment. To be able to crawl content by using the connector framework provided by SharePoint Server, follow the steps in Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint) to create and set up the FAST Search Content Search Service Application (Content SSA).

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint
Install FAST Search Server 2010 for SharePoint
Plan search topology (FAST Search Server 2010 for SharePoint)
Run scripted installation (FAST Search Server 2010 for SharePoint)
Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)
Manage certificates (FAST Search Server 2010 for SharePoint)
Enable click-through relevancy after initial configuration (FAST Search Server 2010 for SharePoint)


Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint)
See Also

To be able to crawl content with the indexing connectors integrated in Microsoft SharePoint Server 2010, you must prepare the FAST Search Content Search Service Application (Content SSA). The Content SSA will crawl content and feed it into the FAST Search Server 2010 for SharePoint back-end.

After you have created the Content SSA and before you add a content source and configure crawling with the Content SSA, you must enable SSL communication between FAST Search Server 2010 for SharePoint and Microsoft SharePoint Server 2010. This is mandatory; you will not be able to feed content to FAST Search Server 2010 for SharePoint without enabling SSL communication.

In this article:

  • Create and set up the Content SSA

  • Configure SSL enabled communication

  • Add a content source

Create and set up the Content SSA

Important:

Make sure that the Windows Service called SharePoint 2010 Administration (SPAdminV4) is running on Microsoft SharePoint Server. If it is stopped, it must be started.

The install_info.txt file contains information to help you configure the Content SSA. The file is located directly under <FASTSearchFolder> (the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch) on the administration server or single server.

  1. On SharePoint Server 2010, on the Central Administration Home page, click Application Management and then click Manage Service Applications.

  2. On the Manage Service Applications page, click New and then click Search Service Application.

  3. On the Create New Search Service Application page, in the Name section, provide a unique name for the Service Application.

    Example: FASTContent

  4. In the FAST Service Application section, select FAST Search Connector.

  5. Set the Search Service Account.

    Note that the Search Service Account is the same across all Search Services. If you set this account to something other than the default value, it will affect all Search Services.

    If you want to change the Search Service Account later, go to the Central Administration Home page. Click Security and then click Configure Service Accounts. Change the user who runs Windows Service - SharePoint Server Search.

    By default, the user specified in the Search Service Account is used to access the content sources that will be crawled. If a certain content source should be crawled by a different user, this can be specified with a crawl rule for that content source.

  6. In the Application Pool for Search Admin Web Service section, select either an existing application pool under Use existing application pool, or select Create new application pool and give it a unique name. Example: FAST_Content_App_Pool.

    If you create a new application pool, select a security account.

    This security account is used to run the application pool for the administrative Web service. Through this service, an administrator will communicate all search administration operations such as starting crawls and creating crawl rules.

  7. In the Content Distributors and Content Collection Name sections, enter the connection information to the FAST Search Server 2010 for SharePoint installation. Use the install_info.txt file as a reference for the location of the content distributor.

    Important:

    The default FAST Search Server 2010 for SharePoint content collection is named sp.

    For example:

    • Content Distributors: fastsearch1.contoso.com:13391

      If you have more than one, use a semicolon as separator.

    • Content Collection Name: sp

  8. Click OK. It may take a while to finish.

    When finished, you will receive a confirmation that your new Content SSA is created. The Content SSA is added to the list of Service Applications.

    Note:

    If the Content SSA creation fails, you must enter new unique names for the Content SSA and the application pool. Otherwise, it will fail again and you must re-enter all the data.

You have now created the Content SSA. To be able to feed content to FAST Search Server 2010 for SharePoint, you must configure SSL enabled communication between Microsoft SharePoint Server 2010 and FAST Search Server 2010 for SharePoint. Continue with Configure SSL enabled communication.

Configure SSL enabled communication

Warning:

Configuring SSL enabled communication is mandatory.

Secure Sockets Layer (SSL) communication is required to feed content from SharePoint Server 2010 to FAST Search Server 2010 for SharePoint. To enable secure communications, configure the Content SSA to use an SSL certificate for FAST Search Server 2010 for SharePoint. FAST Search Server 2010 for SharePoint generates a self-signed certificate during post-setup configuration. This section describes how to use this default self-signed certificate to enable SSL communication. The self-signed certificate is mainly intended to be used in a pre-production or test environment to set up and verify the initial deployment. It is recommended to use certificates issued by a certification authority in production environments, refer to Manage certificates (FAST Search Server 2010 for SharePoint) for advanced instructions.

The following procedure has to be followed at least one time, to enable SSL communication between the SharePoint Server 2010 server that runs the Content SSA crawl component and the FAST Search Server 2010 for SharePoint back-end. If your environment contains additional SharePoint Server 2010 servers running a Content SSA crawl component, additional steps are required to enable SSL communication for those servers. Refer to Multiple server deployment of the Content SSA (FAST Search Server 2010 for SharePoint) for more information.

Important:

This procedure includes copying sensitive information. Ensure that the certificate information is kept private.

  1. Copy the script securefastsearchconnector.ps1 from the FAST Search Server 2010 for SharePoint administration server to the SharePoint Server 2010 server. The securefastsearchconnector.ps1 script is in the <FASTSearchFolder>, under installer\scripts\.

  2. Copy the certificate file FASTSearchCert.pfx from the FAST Search Server 2010 for SharePoint administration server to the SharePoint Server 2010 server. The certificate file is in the <FASTSearchFolder>, under data\data_security\cert\.

  3. Open a Microsoft SharePoint 2010 Management Shell with the Run as administrator option on the SharePoint Server 2010 server.

    Navigate to the directory where you copied the securefastsearchconnector.ps1 script and run it, replacing the necessary parameters with the values for your environment. The domain and user name should reflect the details of the user running the SharePoint Server Search 14 (OSearch14) service.

    .\SecureFASTSearchConnector.ps1 -certPath "path of the certificate\certificatename.pfx" -ssaName "name of your content SSA" -username "domain\username"

    When prompted to enter the certificate password, enter the certificate password that you supplied when you ran the post-setup configuration of FAST Search Server 2010 for SharePoint.

    Important:

    The script verifies the connection to FAST Search Server 2010 for SharePoint. Therefore, make sure that all servers in the FAST Search Server 2010 for SharePoint farm are running before you run this script.

You have now configured the Content SSA to use SSL communication. You should now add at least one content source to the Content SSA to be able to verify the installation. Continue with Add a content source.

Add a content source

This section describes the steps to add a content source to the FAST Search Content SSA.

A content source is a set of options that you can use to specify what type of content is crawled, what URLs to crawl, and how deep and when to crawl. You must create at least one content source before a crawl can occur.

In order to feed content to the FAST Search Server 2010 for SharePoint content collection using the integrated indexing connectors, you should add at least one content source to the FAST Search Content SSA. The default content collection name is sp.

The FAST Search Content SSA can have up to 500 content sources in it, and each content source can have a different configuration.

Note:

It is not necessary to configure the FAST Search Content SSA and set up a content source when you choose to deploy one or more FAST Search Server 2010 for SharePoint specific indexing connectors: the FAST Search Web crawler, the FAST Search database connector and/or the FAST Search Lotus Notes connector. These connectors require completing different configuration procedures.

To verify a successful FAST Search Server 2010 for SharePoint installation, we recommend that you add a small content source to the Content SSA and start a full crawl. This can be one folder in a file share for example, or one Web site.

Navigate to the Manage Content Sources page

  1. Verify that the user account that is performing this procedure is a service application administrator for the FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, click your FAST Search Content Search Service Application.

  4. On the Search Administration Page, in the Crawling section, click Content Sources.

To create a content source

  1. On the Manage Content Sources page, click New Content Source.

  2. On the Add Content Source page, in the Name section, in the Name box, type a name for the new content source.

  3. In the Content Source Type section, select the type of content that you want to crawl. We recommend that you pick a small content source to verify the installation. You can add additional content sources, or edit or delete this content source after you have completed the installation.

    Warning:

    You can only crawl one type of content per content source. That is, you can create a content source that contains URLs for SharePoint sites and another that contains URLs for file shares. However, you cannot create a single content source that contains URLs to both SharePoint sites and file shares.

  4. In the Start Addresses section, in the Type start addresses below (one per line) box, type the URLs from which the crawl should start.

  5. In the Crawl Settings section, select the crawling behavior that you want.

  6. In the Crawl Schedules section, to specify a schedule for full crawls, select a defined schedule from the Full Crawl list. A full crawl crawls all content that is specified by the content source, regardless of whether the content has changed. To define a full crawl schedule, click Create schedule.

    To specify a schedule for incremental crawls, select a defined schedule from the Incremental Crawl list. An incremental crawl crawls content that is specified by the content source that has changed since the last crawl. To define a schedule, click Create schedule.You can change a defined schedule by clicking Edit schedule.

    Tip:

    It is recommended to complete a manually started full crawl cycle before scheduling more crawls. This ensures that the time that is scheduled between incremental crawls is not shorter than the time that is required to run a full crawl.

  7. To prioritize this content source, in the Content Source Priority section, on the Priority list, select Normal or High.

  8. To immediately begin a full crawl, in the Start Full Crawl section, select the Start full crawl of this content source check box, and then click OK.

Next step

You have now set up the Content SSA, enabled SSL communication and added a content source to the Content SSA.

To be able to query crawled content, create a FAST Search Query SSA and follow the instructions to enable queries from SharePoint Server. Continue with Create and set up the Query Search Service Application (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Manage certificates (FAST Search Server 2010 for SharePoint)
Plan search topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Multiple server deployment of the Content SSA (FAST Search Server 2010 for SharePoint)
Plan for crawling and federation (FAST Search Server 2010 for SharePoint)
Add, edit, or delete a content source (FAST Search Server 2010 for SharePoint)


Create and set up the Query Search Service Application (FAST Search Server 2010 for SharePoint)
See Also

This article describes how to create a FAST Search Query Search Service Application (Query SSA) in SharePoint Server Central Administration. The FAST Search Query SSA provides query results from all content sources.

Note:

The Query SSA may also be used to provide people search results. You must then add a content source for people to the Query SSA, see Crawl the user profile store (FAST Search Server 2010 for SharePoint) for more information. The Query SSA will feed people data to the Search Server index and route people queries to Search Server.

Before you create the FAST Search Query SSA, follow the steps in Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint).

In this article:

  • Create and set up the FAST Search Query SSA

  • Verify user access to FAST Search Server 2010 for SharePoint administrative services

  • Connect the FAST Search Query SSA to your web application(s)

Create and set up the FAST Search Query SSA

Important:

Make sure that the Windows Service called SharePoint 2010 Administration (SPAdminV4) is running on Microsoft SharePoint Server 2010. If it is stopped, it must be started.

The install_info.txt file, located directly under the FASTSearch installation folder on the FAST Search Server 2010 for SharePoint administration server or single server (default c:\FASTSearch\install_info.txt), contains the URLs for configuring the SSA.

  1. On SharePoint Server 2010, on the Central Administration Home page, click Application Management and then Manage Service Applications.

  2. On the Manage Service Applications page, click New and then Search Service Application.

  3. On the Create New Search Service Application page, in the Name section, provide a unique name for the Service Application.

    Example: FAST Query SSA

    Warning: If the SSA creation fails, you must enter a new unique name for this field. Otherwise, it will fail again and you must re-enter all data.

  4. In the FAST Service Application section, select FAST Search Query.

  5. In the Application Pool for Search Admin Web Service section, select either an existing application pool under Use existing application pool, or select Create new application pool and give it a unique name. Example: FAST_Query_Admin_SSA_App_Pool.

    If you create a new application pool, select a security account. The security account is used to run the SSA Admin Web service.

    Select Configurable, and enter the user account. The default is the managed service account.

  6. In the Application Pool for Search Query and Site Settings Web Service section, select either an existing application pool under Use existing application pool, or select Create new application pool, and give it a unique name. Example: FAST_Query_SSA_App_Pool.

    If you create a new application pool, select a security account. The security account is used to run the Web service for Search Query and Site Settings.

    Select Configurable, and enter the user account.

  7. In the Query Service Location, Administration Service Location and Resource Store Location sections, enter the connection information to the FAST Search Server 2010 for SharePoint installation. The location (URLs/port numbers) of the services is included in the install_info.txt file.

    The examples here assume a base port of 13000 for FAST Search Server 2010 for SharePoint.

    • Query Service Location:

      http://fastsearch1.contoso.com:13287 or

      https://fastsearch1.contoso.com:13286 (If you have enabled HTTPS for queries)

    • Administration Service Location:

      http://fastsearch1.contoso.com:13257 or

      https://fastsearch1.contoso.com:13258 (If you have enabled HTTPS for queries.)

    • Resource Store Location: http://fastsearch1.contoso.com:13255

    • Account for Administration Service: domain\user.

  8. Click OK. It may take a while to finish.

    When finished, a new SSA is added to the list of Service Applications.

  9. Review the redundancy and availability requirements for the Query SSA. For planning considerations, see Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint). For deployment details, see Multiple server deployment of the Query SSA (FAST Search Server 2010 for SharePoint).

Verify user access to FAST Search Server 2010 for SharePoint administrative services

The user who runs the Application Pool for the web application in which you will configure FAST Search Server 2010 for SharePoint, is the user who will communicate with the FAST Search Server 2010 for SharePoint administration server. Therefore, make sure that the user who runs this Application Pool is a member of the FASTSearchAdministrators group on the FAST Search Server 2010 for SharePoint administration server:

  1. On SharePoint Server 2010, on the Central Administration Home page, under Security, select Configure Service Accounts.

  2. Each web application has a separate Web Application Pool. Select the relevant component, typically Web Application Pool - SharePoint - 80.

  3. Make sure that the user account for this component is a member of the FASTSearchAdministrators group on the FAST Search Server 2010 for SharePoint administration server.

Limit access to keyword management only

In a parent and child farm scenario, the child farm web application account must be authorized to access the FAST Search Server 2010 for SharePoint (back-end) keyword service. The account can be authorized by adding it to the FASTSearchKeywordAdministrators group on the FAST Search Server 2010 for SharePoint administration server. The FASTSearchKeywordAdministrators group is not automatically created during installation, but can be created manually if you want to use this level of authorization.

You can also select to give the account full access to the FAST Search Server 2010 for SharePoint administrative services by adding it to the FASTSearchAdministrators group.

Connect the FAST Search Query SSA to your web application(s)

Important:

The following steps require that at least one web application was created. For information about how to create web applications in SharePoint Server 2010, see Create a Web application.

  1. On SharePoint Server 2010, on the Central Administration Home page, select Application Management.

  2. Under Service Applications, click Configure service application associations.

  3. Click the Application Proxy Group that you want to change, typically default.

  4. Make sure that the Query SSA (for example FAST Query SSA) is the default and only selected Search Service Application Proxy, and then click OK.

  5. Verify that the correct Query SSA is associated with the web application.

All the default search service applications should now point to FAST Search Server 2010 for SharePoint.

Next step

When you have completed the steps in this article, continue with Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint
Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint)
Enable Administration Service over HTTPS (FAST Search Server 2010 for SharePoint)
Plan search topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Multiple server deployment of the Query SSA (FAST Search Server 2010 for SharePoint)
Manage people search (FAST Search Server 2010 for SharePoint)


Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint)
See Also

This article describes how to enable queries from Microsoft SharePoint Server 2010 on the FAST Search Server 2010 for SharePoint. You must configure claims authentication, which means that you export a certificate from SharePoint Server 2010 and import it on all query servers in FAST Search Server 2010 for SharePoint. If you want to help secure query traffic by enabling HTTPS (optional), you must perform some additional steps.

Before you enable queries as described in this article, follow the steps in Create and set up the Query Search Service Application (FAST Search Server 2010 for SharePoint).

In this article:

  • Configure claims authentication

  • Enable HTTPS (optional)

Configure claims authentication

This section describes how to transfer a Microsoft SharePoint Server 2010 STS certificate (MOSS_STS) from SharePoint Server 2010 to all FAST Search Server 2010 for SharePoint query servers to provide security trimmed queries.

Security trimming, or item level security, limits user access to items in search results that the user who submitted the query is authorized to see.

On SharePoint Server 2010, export the SharePoint STS certificate:

  1. On the Start menu, click All Programs.

  2. Click Microsoft SharePoint 2010 Products.

  3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.

  4. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $stsCert = (Get-SPSecurityTokenServiceConfig).LocalLoginProvider.SigningCertificate
$stsCert.Export("cert") | Set-Content  -encoding byte MOSS_STS.cer

    The file MOSS_STS.cer is created in the directory from which you run the commands.

  5. Copy MOSS_STS.cer to a location that is available to the FAST Search Server 2010 for SharePoint query servers.

On each FAST Search Server 2010 for SharePoint query server, import the SharePoint STS certificate:

  1. On the Start menu, click All Programs.

  2. Click Microsoft FAST Search Server 2010 for SharePoint.

  3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

  4. At the Windows PowerShell command prompt, browse to <FASTSearchFolder>\installer\scripts\, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch. Type the following command:

    Copy Code
    .\InstallSTSCertificateForClaims.ps1 -certPath <"full path of MOSS STS certificate">

    Where:

    • <"full path of MOSS STS certificate"> is the path of the location where you copied the MOSS_STS.cer file.

After finishing these steps, you have completed the configuration of the FAST Search Server 2010 for SharePoint query server. Repeat for all query servers if you have more than one.

Enable HTTPS (optional)

If you want to help secure query traffic by using HTTPS, you must perform some additional optional steps. With FAST Search Server 2010 for SharePoint, you can choose between HTTP and HTTPS for query traffic. HTTP is used as a default.

  • HTTP: All query traffic from the SharePoint Server 2010 search application to FAST Search Server 2010 for SharePoint will be sent via HTTP requests. The information is not encrypted. However, this communication channel is faster than HTTPS. Use HTTP if your documents do not contain sensitive information, or if you want maximum query performance. This is the default method.

  • HTTPS: All query traffic from the SharePoint Server 2010 search application to FAST Search Server 2010 for SharePoint will be sent via secured requests. The information is encrypted so that no one can access sensitive information that is contained in the query traffic. Use HTTPS if your documents contain sensitive information.

Note:

This configuration step is optional. Follow this procedure only if you decide to use HTTPS for queries between FAST Search Server 2010 for SharePoint and SharePoint Server 2010.

HTTPS for query traffic require server-specific SSL certificates. Each query server in FAST Search Server 2010 for SharePoint must have a certificate where one of the following must be true:

  • The certificate subject property is set to the FQDN of the query server.

  • The subject alternate name (SAN) property contains the FQDN of the query server.

We recommend that you use your company's existing public key infrastructure (PKI) to generate these SSL certificates.

On each query server in FAST Search Server 2010 for SharePoint, follow these steps:

  1. Import the server-specific SSL certificate into the certificate store. The certificate must be saved under Certificates(Local Computer)\Personal.

    Grant the FASTSearchAdministrators group full access to the certificate, by using winhttpcertcfg or the Microsoft Management Console (MMC) Certificates snap-in.

    To grant access by using the MMC snap-in:

    1. Click Start, type mmc in the Search programs and files box, and then press ENTER.

    2. Right-click the certificate, click All tasks, and then click Manage Private Keys.

    3. Click Add, add the FastSearchAdministrators group and select Full control.

  2. Configure the query server to use the certificate on base port + 286.

    1. On the Start menu, click All Programs.

    2. Click Microsoft FAST Search Server 2010 for SharePoint.

    3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

    4. At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      netsh http add sslcert ipport=0.0.0.0:<baseport+286> appid={a5455c78-6489-4e13-b395-47fbdee0e7e6} certhash=<Cert_Thumbprint>

      Where:

      • <baseport+286> is the actual port number. Example: If the base port is 13000, the actual port number will be 13286 (ipport=0.0.0.0:13286).

      • <Cert_Thumbprint> is the thumbprint of the certificate.

On the SharePoint Server 2010 server, follow these steps:

  1. Enable a trust relationship in SharePoint Server 2010 for the SSL certificate(s) that you created for each query server. Do this by importing the public certificate of the signing authority of the SSL Certificate(s) into SharePoint Server 2010:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.

    4. At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      $trustCert = Get-PfxCertificate '<SSL_CA_Public_Cert>.cer'
New-SPTrustedRootAuthority "FASTSearchHostQuerySSLCert" -Certificate $trustCert

      Where:

      • <SSL_CA_Public_Cert> is the name of the certificate from the signing authority of the SSL certificate(s).

      You will typically export this certificate from the Trusted Root Store on the query server.

  2. Change the Query Service Location configuration setting for the Query Search Service Application (SSA) to use the HTTPS endpoint. See Create and set up the Query Search Service Application (FAST Search Server 2010 for SharePoint) for detailed information.

Next step

When you have configured the claims authentication, and, optionally, enabled HTTPS, follow the steps in Create a FAST Search Center site (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint
FAST Search Authorization (FSA) overview
Manage certificates (FAST Search Server 2010 for SharePoint)

Other Resources

Add the Certificates Snap-in to an MMC


Create a FAST Search Center site (FAST Search Server 2010 for SharePoint)
See Also

This article describes how you can create a FAST Search Center site, which means that you set up a SharePoint Server 2010 site to use the FAST Search Query SSA.

Before you create the FAST Search Center site, follow the steps in Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint).

  1. Open the SharePoint Server 2010 Web application where you want to create a new site.

  2. Enable features unless this has already been done:

    1. Click Site Actions, then click Site Settings, and then click Site collection features.

    2. Make sure that the following are enabled:

      • SharePoint Server Publishing Infrastructure

      • SharePoint Server Enterprise Site Collection features

  3. Create a new site:

    1. Click Site Actions and then click New Site.

    2. Select the FAST Search Center template.

    3. Click Create. A FAST Search Center is created at the specified URL.

Next step

When you have created the FAST Search Center site, follow the steps in Verify successful installation (FAST Search Server 2010 for SharePoint).

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint


Verify successful installation (FAST Search Server 2010 for SharePoint)
See Also

This article describes some procedures that you can use to verify that the system is set up correctly.

Note:

All procedures are optional.

In this article:

  • Verify that all services are running

  • Feed and query test documents

  • Check the Event Log

Verify that all services are running

On each server in the FAST Search Server 2010 for SharePoint farm:

  1. On the Start menu, click All Programs.

  2. Click Microsoft FAST Search Server 2010 for SharePoint.

  3. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  4. At the Windows PowerShell command prompt, type the following command:

    Copy Code
    nctrl status

  5. Verify that all services have the status running.

Feed and query test documents

Verify that you can feed a document and do a search that returns that document. This task can be divided in two parts:

  • Test integration between FAST Search Server 2010 for SharePoint and SharePoint Server by testing feeding and searching from SharePoint Server 2010 using FAST Search Server 2010 for SharePoint as the back-end. See Test integration between FAST Search Server 2010 for SharePoint and SharePoint Server.

  • Test feeding and searching directly on FAST Search Server 2010 for SharePoint. See Test feeding and searching on FAST Search Server 2010 for SharePoint.

By testing feeding and searching in this two-step process, you can narrow down any potential problems. If the first step fails, but the second step succeeds, there is a problem with the integration between SharePoint Server 2010 and FAST Search Server 2010 for SharePoint. If the second step fails, the installation or configuration of FAST Search Server 2010 for SharePoint is incorrect.

Test integration between FAST Search Server 2010 for SharePoint and SharePoint Server

After you completed the steps in Create and set up the Content SSA, a Content SSA with a content source should have been created and some items should have been crawled.

  1. Verify that documents have been fed to FAST Search Server 2010 for SharePoint:

    1. On the Start menu, click All Programs.

    2. Click Microsoft FAST Search Server 2010 for SharePoint.

    3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

    4. At the Windows PowerShell command prompt, type the following command:

      Copy Code
      Get-FASTSearchContentCollection -Name "sp"

  2. Verify that one or more documents are displayed in the search result from the FAST Search Center site:

    1. Open the FAST Search Center site that you created, and do a search for the documents that you fed to FAST Search Server 2010 for SharePoint.

    2. Verify that one or more of the documents that you fed to FAST Search Server 2010 for SharePoint are returned in the search results.

Test feeding and searching on FAST Search Server 2010 for SharePoint

Follow these steps on a FAST Search Server 2010 for SharePoint server that has the query processing component (query role in the deployment.xml file). By default, single server deployments have this component. Refer to deployment.xml reference for more information.

  1. On the Start menu, click All Programs.

  2. Click Microsoft FAST Search Server 2010 for SharePoint.

  3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

  4. At the Windows PowerShell command prompt, type the following command to feed a single document:

    Copy Code
    Docpush -c sp <filename>

    Where <filename> is the full path to a test document that can be indexed by FAST Search Server 2010 for SharePoint. Use any text document, Office document or HTML document.

  5. Verify that the document or documents were indexed correctly, by running the following command:

    Copy Code
    Get-FASTSearchContentCollection -name "sp"

  6. Open a browser on the FAST Search Server 2010 for SharePoint server that has the query processing component and go to the page: http://localhost:<baseport + 280>.

    The baseport is the base TCP/IP port for the FAST Search Server 2010 for SharePoint farm. In a default deployment, the base port is 13000. Replace <baseport + 280> with the actual port number.

    Note:

    This is a test search page that is only accessible via localhost on the FAST Search Server 2010 for SharePoint server that has the query processing component.

  7. In the FQL Query field, type a word that you know exists in the document(s) that you previously fed. Click Search.

  8. Verify that the XML test result page returns an item (document) that contains the content of your test document.

  9. To remove all the documents that you added, run the following command:

    Copy Code
    Clear-FASTSearchContentCollection -name "sp"
    Warning:

    This command will erase the content index. Be careful not to use this command in a production environment unless you are sure that the content collection should be deleted.

Check the Event Log

  1. On the FAST Search Server 2010 for SharePoint administration server, open the Event Log.

  2. Browse to Applications and Services Logs and FAST Search Farm.

  3. Verify that no error messages are displayed.

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint
Create a FAST Search Center site (FAST Search Server 2010 for SharePoint)
FSA: Queries are returning fewer documents than expected (FAST Search Server 2010 for SharePoint)


Advanced installation and configuration options (FAST Search Server 2010 for SharePoint)

The following sections contain information about advanced or optional installation and configuration of FAST Search Server 2010 for SharePoint.

In this section:

  • Manage certificates (FAST Search Server 2010 for SharePoint)

  • Enable Advanced Filter Pack (FAST Search Server 2010 for SharePoint)

  • Enable Administration Service over HTTPS (FAST Search Server 2010 for SharePoint)

  • Run the Prerequisite Installer in offline mode (FAST Search Server 2010 for SharePoint)

  • Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)

  • Run scripted installation (FAST Search Server 2010 for SharePoint)

  • Change user password (FAST Search Server 2010 for SharePoint)

  • Change FAST Search user (FAST Search Server 2010 for SharePoint)

  • Change database (FAST Search Server 2010 for SharePoint)

  • Enable click-through relevancy after initial configuration (FAST Search Server 2010 for SharePoint)


Manage certificates (FAST Search Server 2010 for SharePoint)

Published: May 16, 2012

FAST Search Server 2010 for SharePoint uses several certificates for authentication and encryption purposes. The certificates are used both for communication between servers in a multiple server FAST Search Server 2010 for SharePoint farm, and between FAST Search Server 2010 for SharePoint and Microsoft SharePoint Server 2010.

Each server in a FAST Search Server 2010 for SharePoint farm potentially has three kinds of certificates which serve different functions and which must be configured (and replaced) separately:

  • General purpose certificate

    A self-signed FAST Search certificate (for test environments) or a certificate signed by a certificate authority (for production environments). The self-signed certificate should be replaced with a farm-wide or server specific CA signed certificate that supports the existing certificate infrastructure of the organization. The general purpose certificate is used for internal communication, administration services and to enable secure content feeding from SharePoint Server to FAST Search Server 2010 for SharePoint.

  • Query HTTPS certificate

    A server-specific certificate to encrypt query traffic that uses HTTPS. Only used on query servers that have HTTPS query traffic enabled.

  • Claims certificate

    A claims certificate to enable item level security trimming on queries. Only used on query servers.

This article describes the steps needed to replace these certificates because of, for example, expiration or revocation.

We highly recommend that you replace the default self-signed general purpose certificate with a CA signed certificate when you move your deployment from test to production.

In this section:

  • Self-signed general purpose FAST Search certificate

  • Certificates signed by a certification authority (CA)

  • Replacing the default self-signed certificate

  • Replacing the Query HTTPS certificate

  • Replacing the claims certificate

Self-signed general purpose FAST Search certificate

During initial installation, FAST Search Server 2010 for SharePoint generates a self-signed certificate. The self-signed certificate is only meant to be used in test environments. There are several limitations to this default self-signed certificate:

  • It expires after one year from the time of configuration.

  • It provides limited security because it cannot be revoked. This could allow an attacker to spoof identities or insert data into connections if the private key was compromised.

  • It cannot be used to enable queries over HTTPS.

  • It cannot be used to enable administration services over HTTPS.

Certificates signed by a certification authority (CA)

To help achieve a high level of security in a production environment, we recommend that you use certificates signed by a common certification authority (CA) for FAST Search Server 2010 for SharePoint.

Your organization may have an existing public key infrastructure (PKI) that can issue these certificates. If your organization does not have an existing PKI, you can acquire certificates from a third-party certificate issuing authority. Your organization may also have its own business processes and tools to issue and manage CA signed certificates. There are no specific properties that the certificate must have, but there are some requirements that must be met. Authorization is done by matching the thumbprint of the CA signed certificate across servers and by checking that the certificate issuer is trusted.

The CA signed certificate must be installed on each server in a multiple server FAST Search Server 2010 for SharePoint farm.

The following requirements apply to each certificate:

  • The subject name or subject alternative name (SAN) field must contain the fully qualified domain name (FQDN) of the server that the certificate is issued to. This is required to support queries over HTTPS and administration services over HTTPS.

  • The certificate that is issued to SharePoint Server 2010 must have the same issuer as the certificates that are issued to servers in the FAST Search Server 2010 for SharePoint farm.

  • The FAST Search Server 2010 for SharePoint user must have access to the private key of the certificate.

  • The certificate must support private key exchange.

Replacing the default self-signed certificate

FAST Search Server 2010 for SharePoint includes a Windows PowerShell script that must be run on each server in the deployment to replace the default self-signed certificate. The script can perform two separate tasks:

  • Recommended: Configure FAST Search Server 2010 for SharePoint to use an existing certificate that is signed by a certification authority (CA) by supplying a thumbprint to an already installed certificate. See Replace the self-signed certificate with a certificate signed by a certification authority (CA)

  • Create a new self-signed certificate (with a one year expiration) and configure FAST Search Server 2010 for SharePoint to use the new certificate. See Replace the self-signed certificate with a new self-signed certificate.

Replace the self-signed certificate with a certificate signed by a certification authority (CA)

  1. Stop FAST Search Server 2010 for SharePoint on all servers in the farm, including the monitoring service.

  2. On each server in the FAST Search Server 2010 for SharePoint farm:

    1. Make sure that the new CA signed certificate is installed correctly:

      The CA signed certificate must be installed under Certificates(Local Computer)\Personal in the certificate store.

      The root CA signed certificate must be installed under Certificates(Local Computer)\Trusted Root Certification Authorities.

    2. Make sure that the FAST Search Server 2010 for SharePoint user has access to the private key of the certificate.

  3. On the FAST Search Server 2010 for SharePoint administration server, follow these steps:

    1. Find the thumbprint of the CA signed certificate.

      1. Open Microsoft Management Console. Click Start and then type MMC in the Search box. Click MMC under Programs.

      2. Expand the Certificates (Local computer) menu under Console Root.

        Optional: If the Certificates (Local computer) menu is not visible, the Certificates snap-in is not enabled. To enable the Certificates snap-in:

        1. Click File and then click Add/Remove Snap-in.

        2. Select Certificates from the list of Available snap-ins and then click Add.

        3. Select Computer account and then click Next.

        4. Select Local computer and then click Finish.

        5. Click OK in the Add or Remove Snap-ins menu.

      3. Expand the Personal folder and then click the Certificates folder. Double-click the CA signed certificate.

      4. Open the Details tab and then click Thumbprint.

    2. On the Start menu, click All Programs.

    3. Click Microsoft FAST Search Server 2010 for SharePoint and then Microsoft FAST Search Server 2010 for SharePoint shell.

    4. At the command prompt, browse to installer\scripts under the installation folder.

    5. Type one of the following commands:

      If you use one certificate for all the servers in the farm, type:

      Copy Code
      .\ReplaceDefaultCertificate.ps1 -thumbprint "certificate thumbprint"

      Where:

      • certificate thumbprint is the thumbprint of the CA signed certificate.

      If you use a server specific certificate, type:

      Copy Code
      .\ReplaceDefaultCertificate.ps1 -certificateValidationMode ChainTrust -thumbprint "certificate thumbprint"

      Where:

      • certificate thumbprint is the thumbprint of the CA signed certificate.

  4. Start FAST Search Server 2010 for SharePoint on the administration server.

  5. On each non-administration server, follow these steps:

    1. On the Start menu, click All Programs.

    2. Click Microsoft FAST Search Server 2010 for SharePoint and then Microsoft FAST Search Server 2010 for SharePoint shell.

    3. At the command prompt, browse to installer\scripts under the installation folder.

    4. Type one of the following commands:

      If you use one certificate for all the servers in the farm, type:

      Copy Code
      .\ReplaceDefaultCertificate.ps1 -thumbprint "certificate thumbprint"

      Where:

      • certificate thumbprint is the thumbprint of the CA signed certificate.

      If you use a server specific certificate, type:

      Copy Code
      .\ReplaceDefaultCertificate.ps1 -certificateValidationMode ChainTrust -thumbprint "certificate thumbprint"

      Where:

      • certificate thumbprint is the thumbprint of the CA signed certificate.

  6. Start FAST Search Server 2010 for SharePoint on all non-administration servers.

The SharePoint Server where the Content SSA is running also needs a certificate that is signed by the same CA to feed documents to FAST Search Server 2010 for SharePoint:

  1. Install the CA signed certificate on SharePoint Server 2010 under Certificates(Local Computer)\Personal in the certificate store.

  2. Install the root CA certificate under Certificates(Local Computer)\Trusted Root Certification Authorities.

  3. Copy the script SecureFASTSearchConnector.ps1 from the FAST Search Server 2010 for SharePoint administration server to the SharePoint Server 2010 server that is running the FAST Search connector. The SecureFASTSearchConnector.ps1 script can be found in the installation folder, under \installer\scripts\.

  4. On the SharePoint Server 2010 server that is running the FAST Search connector, follow these steps:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.

    4. Browse to the directory where you copied the SecureFASTSearchConnector.ps1 script and run it, replacing the necessary parameters with the values for your environment. The domain and user name should reflect the details of the user running the SharePoint Server Search 14 service (OSearch14):

      • If you know the thumbprint of your certificate, type the following command:

        Copy Code
        .\SecureFASTSearchConnector.ps1 -certThumbprint "certificate thumbprint" -ssaName "name of your content SSA" -username "domain\username"

      • If you do not know the thumbprint of your certificate, type the following command:

        Copy Code
        .\SecureFASTSearchConnector.ps1 -ssaName "name of your content SSA" -username "domain\username"

        This command will return the thumbprint of the available certificates and a prompt asking whether you want to use the suggested certificate.

        Enter y for yes, and then click Enter.

Multiple server deployments

If you have configured the FAST Search Content SSA to use more than one crawl component, you must install the same CA signed certificate on each SharePoint Server 2010 server that has a crawl component.

  1. Make sure that the server has a certificate installed that is issued and signed by the same Certificate Authority as the certificate on the host server of the FAST Search Content SSA. The certificate must be installed under Certificates(Local Computer)\Personal in the certificate store. The root CA signed certificate must be installed under Certificates(Local Computer)\Trusted Root Certification Authorities.

  2. Grant the Search Service Application account (the account under which the SharePoint Server Search 14 service (OSearch14) runs) access to the private key of the imported certificate.

Replace the self-signed certificate with a new self-signed certificate

  1. Stop FAST Search Server 2010 for SharePoint on all servers in the farm, including the monitoring service.

  2. On the administration server:

    1. On the Start menu, click All Programs.

    2. Click Microsoft FAST Search Server 2010 for SharePoint.

    3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.

    4. At the command prompt, browse to installer\scripts under the installation folder.

    5. Type the following command:

      Copy Code
      .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

    6. Enter a password for the certificate.

  3. Start FAST Search Server 2010 for SharePoint on the administration server.

  4. On each non-administration server:

    1. On the Start menu, click All Programs.

    2. Click Microsoft FAST Search Server 2010 for SharePoint

    3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.

    4. At the command prompt, browse to installer\scripts under the installation folder.

    5. Type the following command:

      Copy Code
      .\ReplaceDefaultCertificate.ps1 -generateNewCertificate $true

    6. Enter the password that you defined for the certificate on the administration server.

  5. Start FAST Search Server 2010 for SharePoint on all non-administration servers.

The new self-signed certificate will expire after one year.

If FAST Search Server 2010 for SharePoint is already added as a back-end for SharePoint Server 2010, you must also redo the steps in Configure SSL enabled communication under Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint).

Multiple server deployments

If you have configured the FAST Search Content SSA to use more than one crawl component, you must import the new self-signed certificate to each SharePoint Server 2010 server that has a crawl component.

  1. Import the new FASTSearchCert.pfx certificate in the certificate store under Certificates(Local Computer)\Personal.

  2. Grant the Search Service Application account (the account under which the SharePoint Server Search 14 service (OSearch14) runs) access to the private key of the imported certificate.

Replacing the Query HTTPS certificate

This server-specific certificate is used to encrypt query traffic that uses HTTPS. For initial setup, see Enable HTTPS (optional).

Replace the query HTTPS certificate

To replace the query HTTPS certificate, follow these steps on each FAST Search Server 2010 for SharePoint query server:

  1. Import the new server-specific CA signed SSL certificate into the certificate store. The certificate must be saved under Certificates(Local Computer)\Personal.

    Grant the FASTSearchAdministrators group full access to the certificate, by using winhttpcertcfg or the Microsoft Management Console (MMC) Certificates snap-in.

    To grant access by using the MMC snap-in:

    1. Click Start, type mmc in the Search programs and files box, and then press ENTER.

    2. Right-click the certificate, click All tasks, and then click Manage Private Keys.

    3. Click Add, add the FastSearchAdministrators group and select Full control.

  2. Delete the previous certificate binding from baseport+286:

    1. On the Start menu, click All Programs.

    2. Click Microsoft FAST Search Server 2010 for SharePoint.

    3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

    4. At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      netsh http delete sslcert ipport=0.0.0.0:<baseport+286>

      Where:

      • <baseport+286> is the actual port number.

  3. Configure the query server to use the new certificate on baseport+286:

    1. At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      netsh http add sslcert ipport=0.0.0.0:<baseport+286>  appid={a5455c78-6489-4e13-b395-47fbdee0e7e6} certhash=<Cert_Thumprint>

      Where:

      • <Cert_Thumbprint> is the thumbprint of the new certificate.

      • <baseport+286> is the actual port number.

In addition, if the new certificate was not signed by the same certification authority (CA) as the previous certificate, you must add the CA certificate to the SharePoint Server:

On SharePoint Server 2010:

  1. Enable a trust relationship in SharePoint Server for the SSL certificate(s) that you created for each FAST Search Server 2010 for SharePoint query server. Do this by importing the public certificate of the signing authority of the SSL Certificate(s) into SharePoint Server 2010:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.

    4. At the command prompt, type the following command(s):

      Copy Code
      $trustCert = Get-PfxCertificate '<SSL_CA_Public_Cert>.cert'
New-SPTrustedRootAuthority "FASTSearchHostQuerySSLCert" -Certificate $trustCert

      Where:

      • <SSL_CA_Public_Cert> is the name of the certificate from the signing authority of the SSL certificate(s)

Replacing the claims certificate

The claims certificate enables item level security trimming on queries. To replace this certificate, repeat the steps listed under Configure claims authentication.


Enable Advanced Filter Pack (FAST Search Server 2010 for SharePoint)
See Also

Advanced Filter Pack is a FAST Search Server 2010 for SharePoint feature that enables text and metadata extraction from several hundred file formats, complementing the document formats that are supported by the Microsoft Filter Pack.

By default, the Advanced Filter Pack is disabled.

To enable the Advanced Filter Pack

On the FAST Search Server 2010 for SharePoint administration server (or single server), follow these steps:

  1. On the Start menu, click All Programs.

  2. Click Microsoft FAST Search Server 2010 for SharePoint.

  3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

  4. At the command prompt, browse to installer\scripts under the installation folder.

  5. Type the following command:

    Copy Code
    .\AdvancedFilterPack.ps1 -enable

To disable the Advanced Filter Pack

On the FAST Search Server 2010 for SharePoint administration server (or single server), follow these steps:

  1. On the Start menu, click All Programs.

  2. Click Microsoft FAST Search Server 2010 for SharePoint.

  3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

  4. At the command prompt, browse to installer\scripts under the installation folder.

  5. Type the following command:

    Copy Code
    .\AdvancedFilterPack.ps1 -disable

See Also

Concepts

IFilter and file type reference (FAST Search Server 2010 for SharePoint)
Include a file type in the content index (FAST Search Server 2010 for SharePoint)


Enable Administration Service over HTTPS (FAST Search Server 2010 for SharePoint)

Published: May 16, 2012

The Administration Service administers and configures FAST Search Server 2010 for SharePoint. Clients of the Administration Service include the following:

  • Administrators on Microsoft SharePoint Server

  • Internal FAST Search Server 2010 for SharePoint components which retrieve configuration information

By default, this traffic is using HTTP. You can help secure the traffic by using HTTPS through these configuration steps:

  1. Create Certificates:

    The default self-signed certificate which is generated during installation cannot be used if you want to enable Administration Service over HTTPS. Follow the steps in Replace the self-signed certificate with a certificate signed by a certification authority (CA) to replace the general purpose self-signed certificate with certificates which fulfill the requirements listed in Certificates signed by a certification authority (CA).

  2. Enable a trust relationship in Microsoft SharePoint Server for the certificate(s) you created in the previous step:

    Do this by importing the public certificate from the signing authority of the certificate(s) into Microsoft SharePoint Server.

    On Microsoft SharePoint Server, follow these steps:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Right-click SharePoint 2010 Management Shell, and select Run as administrator.

    4. At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      $trustCert = Get-PfxCertificate '<SSL_CA_Public_Cert>.cert'
New-SPTrustedRootAuthority "FASTSearchHostAdminCert" -Certificate $trustCert

      Where:

      • <SSL_CA_Public_Cert> is the name of the certificate of the signing authority of the certificate(s).

  3. Configure Microsoft SharePoint Server to use HTTPS for Administration Services:

    Change the Administration Service Location configuration setting for the Query Search Service Application (Query SSA) to use the HTTPS endpoint. The HTTPS endpoint is listed in the install_info.txt file. Refer to Create and set up the Query Search Service Application (FAST Search Server 2010 for SharePoint).

  4. Configure FAST Search Server 2010 for SharePoint to use HTTPS for Administration Services:

    To enable the FAST Search services to use HTTPS for the Administration Services, a reconfiguration is required. Edit the deployment.xml file and set the attribute useadminservicehttps to true on each host element. Follow the steps in Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint) to complete the reconfiguration. Refer to deployment.xml reference for detailed information.


Run the Prerequisite Installer in offline mode (FAST Search Server 2010 for SharePoint)

The Prerequisite Installer, by default, downloads components from the Microsoft Download Center to the server on which you are installing FAST Search Server 2010 for SharePoint. This requires Internet access and Windows Firewall open on port 80.

If the server where you are running the Prerequisite Installer does not have Internet access, you can use the Prerequisite Installer in offline mode.

To install prerequisites using the offline mode, follow these steps:

  1. Manually download each prerequisite applicable to your operating system.

  2. Copy the prerequisites to the server where you will run the Prerequisite Installer.

  3. From a command prompt, run the Prerequisite Installer, and specify each prerequisite as a command-line option.

The following table specifies the relevant command-line options:

Command-line option | Description | Prerequisite downloads | Operating system
/IDFX:file | Identity foundation (KB 974405) | http://go.microsoft.com/fwlink/p/?LinkID=160381 | Windows 2008 SP2
/IDFXR2:file | Identity foundation (KB 974405) | http://go.microsoft.com/fwlink/p/?LinkID=166363 | Windows 2008 R2
/VCCRedistPack:file | Microsoft Visual C++ 2008 SP1 Redistributable Package (x64) | http://go.microsoft.com/fwlink/p/?LinkId=188658 | Windows 2008 SP2 / Windows 2008 R2
/NETFX35SP1:file | Microsoft .NET Framework 3.5 SP1 | http://go.microsoft.com/fwlink/p/?LinkId=188659 | Windows 2008 SP2 / Windows 2008 R2
/PowerShell:file | Windows PowerShell 2.0 | http://go.microsoft.com/fwlink/p/?LinkID=161023 | Windows 2008 SP2
/KB976394:file | Hotfix for Microsoft Windows (KB976394) | http://go.microsoft.com/fwlink/p/?LinkID=166368 | Windows 2008 SP2
/KB976462:file | Hotfix for Microsoft Windows (KB976462) | http://go.microsoft.com/fwlink/p/?LinkID=166369 | Windows 2008 R2

To see all Prerequisite Installer command-line options, run the following command from a command prompt:

Copy Code
PrerequisiteInstaller.exe /?

The following example installs the prerequisites on Windows 2008 R2:

Copy Code
PrerequisiteInstaller.exe /IDFXR2:"C:\Windows6.1-KB974405-x64.msu" /VCCRedistPack:"C:\vcredist_x64.exe" /NETFX35SP1:"C:\dotnetfx35setup.exe"  /KB976462:"C:\Windows6.1-KB976462-x64.msu"


Run scripted installation (FAST Search Server 2010 for SharePoint)

Installation of FAST Search Server 2010 for SharePoint can be automated through scripting. This section outlines the steps that you must follow to run a scripted installation.

Install and do initial configuration using scripts

Note:

If you are installing a multiple server FAST Search Server 2010 for SharePoint farm, install and start the administration server first.

  1. Copy the installation files (see Overview of installation files) to a directory on the server where you want to install.

  2. Open a command prompt (cmd) and browse to the directory where you saved the installation files.

  3. Run the Prerequisite Installer by typing the following command:

    Copy Code
    PrerequisiteInstaller.exe /unattended

  4. Set the Windows PowerShell Execution policy by typing the following command:

    Copy Code
    Set-ExecutionPolicy RemoteSigned

  5. Restart the server.

  6. Install MSI by typing the following command:

    Copy Code
    Msiexec /i fsserver.msi /q FASTSEARCHSERVERINSTALLLOCATION="<InstallDir>" /l msiLog.log

    Where <InstallDir> is the path of where you want to install, typically C:\FASTSearch.

  7. Run Windows Update. Download and install the latest FAST Search Server 2010 for SharePoint updates. Restart the server if it is required.

  8. Open Windows PowerShell as an administrator and browse to installer\scripts under the installation directory.

  9. Run the post-setup configuration script by typing the following command:

    Copy Code
    .\Psconfig.ps1 [options]

    Where [options] are replaced with relevant configuration script options. See Configuration script options (psconfig.ps1) for more information.

  10. Restart the server.

Configuration script options (psconfig.ps1)

This section lists options that are available when you run the post-setup configuration script (psconfig.ps1).

Option | Value | Required | Description
-action | i|u | Yes | Specifies the installation action; i: Install u: Uninstall
-roleName | single|admin|nonadmin | Yes | Specifies the kind of node to install: single: Install as a single server/stand-alone server admin: Install as the administration server in a multiple server deployment nonadmin: Install as a non-administration server in a multiple server deployment
-userName | <domain\user> | Yes | The user name of the FAST Search Server 2010 for SharePoint user
-password | <user password as a secure string> | No | The user password of the FAST Search Server 2010 for SharePoint user. If you do not specify a password in the script, you are prompted for one.
-certificatePassword | <Password for Fast Search Server certificate> | No | If you do not specify a certificate password in the script, you are prompted for one. Enter a password of your choice. (You will need this password to move certificates between servers.)
-localMachineName | <fully qualified local server name> Server name cannot be longer than 15 characters. | Yes | Name of the local server name. Example: fastserver.contoso.com
-baseport | <port number> | No | Identifies the base port, which is the first port number in a range of TCP/UDP port numbers that are used by an application. A port number is specified as the base port plus an integer. Default:13000
-databaseConnectionString | <server and instance name> | Yes | Example with instance name set: localhost\sqlexpress Example without any instance name set in SQL Server: localhost
-databaseName | <database name> | Yes | A unique name assigned to the database
-logFileName | <name of the log file> | No | Specifies the name of the log file. .log is automatically appended to end of the name. Default: psconfig_TIMESTAMP.log
-logLevelAsString | Debug|Info|Warning|Error | No | Specifies the amount and kind of information to log during installation. Debug Info Warning Error Default: Debug
-adminMachineName | <fully qualified administration server name> | Required for non-administration server installation | Identifies the administration server that a non-administration server connects to.
-deploymentFile | <deployment file name> | No | Specifies the deployment file to use. Optional parameter for administration server installation action.
-adminBasePort | <base port of administration server> | Not required for non-administration server installation | Identifies the base port of the administration server that the non-administration server connects to. Default:13000
-SharePointInstalledMode | <basic or advanced> | No | Specifies the SharePoint Server 2010 installation mode: basic (corresponds to the Standalone option) or advanced (corresponds to the Server Farm option).
-SharePointServerName | <server name> | No | Specifies the server name where SharePoint Server 2010 is installed. Only needed if SharePointInstalledMode is basic (corresponds to the Standalone option).
-SharePointUserIdentity | <user name> | No | Specifies the user who is running the timer service in Microsoft SharePoint Server 2010. This user is specified in the SharePoint Server post-setup configuration. Only needed if SharePointInstalledMode is advanced (corresponds to the Server Farm installation option).
-sqlUserName | <SQL user account> | Yes, if SQL authentication was used when the database was created. | The user name that is used for SQL authentication.
-sqlPassword | <SQL user password as a secure string> | Yes, if SQL authentication was used when the database was created. | The password of the user that is used for SQL authentication.


Change user password (FAST Search Server 2010 for SharePoint)

The FAST Search Server 2010 for SharePoint user is a domain user. When the user password is changed in Active Directory Domain Services, the password of the FAST Search Server 2010 for SharePoint user must be changed on all servers in the FAST Search Server 2010 for SharePoint farm.

All content input sources that modify the index should be paused before you change the password. This includes pausing crawls, link analysis and other connectors that feed content into the index.

To change the user password

  1. Pause (suspend) crawling.

    • If you are using the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about pausing a crawl.

    • If you are using the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about suspending a crawl.

  2. Pause the Web Analyzer components. These components analyze links between items and search click-through logs. The components must be stopped to avoid partial updates for indexed items during the procedure.

    1. On the administration server, open a FAST Search Server 2010 for SharePoint shell:

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

    2. Verify the Web Analyzer schedule status:

      At the Windows PowerShell command prompt, type the following command:

      Copy Code
      waadmin ShowStatus
      Note:

      Refer to waadmin.exe reference if you are using multiple views for the Web Analyzer.

      In the Views section of the command output, for the default view, inspect the Schedule status.

      1. If the Schedule status is set to paused, type the following command:

        Copy Code
        waadmin enqueueview

      2. Wait until the default Web Analyzer view is running. Type the command waadmin ShowStatus and inspect the Schedule status. The Schedule status should be set to running.

    3. Stop Web Analyzer processing:

      At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      waadmin AbortProcessing
spreladmin AbortProcessing

  3. Update the passwords in Search Server 2010. For more information, see Change passwords used for administration accounts (SharePoint Server 2010).

  4. On every FAST Search Server 2010 for SharePoint server, update the password for all FAST Search Server 2010 for SharePoint Windows Services:

    • FAST Search for SharePoint

    • FAST Search for SharePoint Browser Engine

    • FAST Search for SharePoint Monitoring

    • FAST Search for SharePoint QRProxy

    • FAST Search for SharePoint Sam Admin

    • FAST Search for SharePoint SAM Worker

    Note:

    In a multiple server deployment, not all services will run on all servers.

    Perform the following steps for all services:

    1. On the Start menu, click Control Panel, click System and Security, and then click Administrative Tools.

    2. Double-click Services.

    3. Right-click the service name, and then click Properties.

    4. On the Log On tab, type the new password in the Password and Confirm password boxes, and then click OK.

  5. On the FAST Search Server 2010 for SharePoint administration server, in Internet Information Services (IIS), update the password for the FAST Search application pool and then restart the application pool, by performing the following steps:

    1. On the Start menu, click Control Panel, click System and Security and then click Administrative Tools.

    2. Double-click Internet Information Services (IIS) Manager.

    3. In the Connections dialog box, click Application Pools.

    4. Right-click FASTSearchAdminAppPool, and then click Advanced Settings.

    5. In Process Model, expand Identity.

    6. Click Set and re-type the user credentials.

    7. Right-click FASTSearchAdminAppPool, and then click Stop.

    8. Right-click FASTSearchAdminAppPool again, and then click Start.

  6. Restart all servers in the FAST Search Server 2010 for SharePoint farm. Verify that all services are running.

  7. On the administration server, at the Windows PowerShell command prompt, run the following command(s) to resume processing for the Web Analyzer components:

    Copy Code
    waadmin EnqueueView
spreladmin Enqueue

  8. Resume crawling:

    • If you are using the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about resuming a crawl.

    • If you are using the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about resuming a crawl.


Change FAST Search user (FAST Search Server 2010 for SharePoint)

If you want to change the user that FAST Search Server 2010 for SharePoint is running under, follow these steps:

  1. Stop FAST Search Server 2010 for SharePoint on all servers in the farm.

  2. On each server in the FAST Search Server 2010 for SharePoint farm, starting with the administration server:

    Note:

    First, follow these steps on the administration server. Then, restart the administration server before repeating the steps for the other servers in the deployment.

    1. Add the new FAST Search user to the local FASTSearchAdministrators group. Remove the old FAST Search user.

    2. Update the user credentials for all FAST Search Server 2010 for SharePoint Windows Services:

      • FASTSearchService

      • FASTSearchBrowserEngine

      • FASTSearchMonitoring

      • QRProxyService

      • FASTSearchSAMAdmin

      • FASTSearchSAMWorker

    3. Open a Microsoft FAST Search Server 2010 for SharePoint shell as an administrator:

      1. On the Start menu, click All Programs.

      2. Click Microsoft FAST Search Server 2010 for SharePoint.

      3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

    4. Set the Local Security Authority (LSA) policy for the new FAST Search user:

      At the command prompt, type the following command(s):

      Copy Code
      Set-LsaPolicy -AccountName <domain\user> -Privilege "SeServiceLogonRight"
Set-LsaPolicy -AccountName <domain\user> -Privilege "SeAssignPrimaryTokenPrivilege"
Set-LsaPolicy -AccountName <domain\user> -Privilege "SeIncreaseQuotaPrivilege"

      Where:

      • <domain\user> is the new FAST Search user.

    5. Set a new discretionary access control list (DACL) on ports:

      At the command prompt, type the following command(s):

      Copy Code
      Set-Services -UserName <domain\user> -BasePort <baseport>

      Where:

      • <domain\user> is the new FAST Search user.

      • <baseport> is the base port value in etc\hostconfiguration.xml.

    6. Grant permission to the new FAST Search user to stop and start services:

      At the command prompt, navigate to the bin directory under the installation folder and type the following command(s):

      Copy Code
      .\ServiceAccessControl.exe FASTSearchMonitoring <domain\user>
.\ServiceAccessControl.exe FASTSearchSAMWorker <domain\user>
.\ServiceAccessControl.exe FASTSearchSAMAdmin <domain\user>
.\ServiceAccessControl.exe QRProxyService <domain\user>
.\ServiceAccessControl.exe FASTSearchBrowserEngine <domain\user>

      Where:

      • <domain\user> is the new FAST Search user.

    7. Update the reference to the user name in etc\hostconfiguration.xml.

    8. Recreate the configuration files:

      At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      Set-FASTSearchConfiguration

    9. Make sure that the new user has access to the certificates and the corresponding private keys that are used by FAST Search Server 2010 for SharePoint. Refer to Manage certificates (FAST Search Server 2010 for SharePoint) for more information.

    10. Grant the new user full access to the FAST Search Server 2010 for SharePoint installation folder. Remove the old FAST Search user from the security description for the installation folder.

  3. On the administration server in the FAST Search Server 2010 for SharePoint farm, perform the following additional steps:

    1. In Internet Information Services (IIS), change the user who runs the application pool FASTSearchApplicationPool.

    2. Restart the application pool.

    3. In the params.xml file, under \etc\config_data\deployment\, update the user name value to reflect the new user.

  4. Update the Query SSA on SharePoint Server.

    1. From the SharePoint Server Central Administration home page, select Application Management > Manage Service Applications. In the list of Service Applications, highlight the name of your Query SSA. Click Properties.

    2. Update the Account for Administration Service field.

    3. Make sure that the security account selected for running the application pool for Search Admin Web Service is a member of the FASTSearchAdministrators group on the FAST Search Server 2010 for SharePoint administration server.

  5. Make sure that the new FAST Search user has read and write access to the SQL Server.

  6. Restart all servers in the FAST Search Server 2010 for SharePoint farm.


Change database (FAST Search Server 2010 for SharePoint)

In some cases, you may want to change the database that FAST Search Server 2010 for SharePoint uses, after the initial installation and configuration. An example may be if you have to reinstall the administration server.

Note:

The post-setup configuration wizard creates a new database when you configure a single server or administration server. However, you can configure FAST Search Server 2010 for SharePoint to use another existing database after the configuration wizard has finished.

You can find the current settings for the database in <FASTSearchFolder>\etc\hostconfiguration.xml, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

To change the database, follow these steps:

  1. Make sure that the FAST Search user has access to the database server.

  2. If you want to create a new empty database:

    1. On the Start menu, click All Programs.

    2. Click Microsoft FAST Search Server 2010 for SharePoint.

    3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

    4. At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      Install-FASTSearchAdminDatabase -DbName <DatabaseName> -dbserver <ConnectionString>

      Where:

      • <DatabaseName> and <ConnectionString> are replaced with the appropriate values. See Configure the administration server in a multiple server deployment for more information.

  3. On the FAST Search Server 2010 for SharePoint administration server, do the following:

    1. Stop FAST Search Server 2010 for SharePoint.

    2. Update the database server and database name in <FASTSearchFolder>\etc\hostconfiguration.xml.

      Note:

      In case you use the FAST Search Lotus Notes connector or the FAST Search database connector, also update the database server and database name in <FASTSearchFolder>\etc\dbconfig.xml.

    3. On the Start menu, click All Programs.

    4. Click Microsoft FAST Search Server 2010 for SharePoint and then Microsoft FAST Search Server 2010 for SharePoint shell.

    5. At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      Set-FASTSearchConfiguration

    6. Start FAST Search Server 2010 for SharePoint.


Enable click-through relevancy after initial configuration (FAST Search Server 2010 for SharePoint)

You may decide not to enable click-through relevancy when you run the configuration wizard. If you want to enable click-through relevancy later, follow these steps:

  1. On the Start menu, click All Programs.

  2. Click Microsoft FAST Search Server 2010 for SharePoint.

  3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

  4. At the Windows PowerShell command prompt, browse to installer\scripts under the installation folder.

  5. Type the following command:

    Copy Code
    .\configuresharepointauthorization.ps1 -installedMode <mode> [-sharepointserver <servername>] [-sharepointUserIdentity <userIdentity>]

    • Replace <mode> with "basic" if Microsoft SharePoint Server was installed using the Standalone option, or with "advanced" if it was installed by using the Server Farm option.

    • If Microsoft SharePoint Server was installed using the Standalone option, specify -sharepointserver and replace <servername> with the fully qualified name of the Microsoft SharePoint Server.

    • If Microsoft SharePoint Server was installed using the Server Farm option, specify -sharepointUserIdentity and replace <userIdentity> with the user who is running the Microsoft SharePoint 2010 Timer Service. This user is specified in the Microsoft SharePoint Server post-setup configuration.

In the following example, Microsoft SharePoint Server was installed using the Standalone option ("basic" installation mode):

Copy Code
.\configuresharepointauthorization.ps1 -installedMode "basic" -sharepointserver "moss1.contoso.com"


Overview of installation files

The FAST Search Server 2010 for SharePoint installation package contains the following files:

File | Description
prerequisiteinstaller.exe | The Prerequisite Installer, an executable program that installs the prerequisites
PrerequisiteInstallerFiles\ (folder) | The installation folder for prerequisites
fsserver.msi | The FAST Search Server 2010 for SharePoint MSI
fs1.cab | Compressed installation files
fs2.cab | Compressed installation files
Catalog (folder | Used by start page (splash)
Setup (folder) | Images used in start page (splash)
autorun.ico | Used by autorun
autorun.inf | Starts start page (splash)
default.hta | Used by start page (splash)
FAST_Search_Server_2010_for_SharePoint_Attributions.xps | List of open source (external) components in FAST Search Server 2010 for SharePoint
setup.cmd | Used by autorun
splash.hta | Start page (splash)


Deploy System Center Operations Manager (SCOM) 2007 (FAST Search Server 2010 for SharePoint)

This section describes the procedures to deploy System Center Operations Manager (SCOM) 2007.

Consult the topic Plan to monitor the health of the search system (FAST Search Server 2010 for SharePoint) for considerations before you start the installation.

In this section:

SCOM 2007 prerequisites (FAST Search Server 2010 for SharePoint)

Install SCOM 2007 (FAST Search Server 2010 for SharePoint)

Install the SCOM 2007 agent on managed servers (FAST Search Server 2010 for SharePoint)

Obtain and import the management pack


SCOM 2007 prerequisites (FAST Search Server 2010 for SharePoint)

This topic describes the prerequisites for deploying System Center Operations Manager (SCOM) 2007 for use with Microsoft FAST Search Server 2010 for SharePoint.

Software prerequisites

Install server roles and features

In Windows Server 2008, use the Server Manager to install these required roles and features:

Features:

  • .NET Framework 3.x Features

  • Windows PowerShell

Role:

Web Server (IIS 7.0), with the following role services:

  • ASP.NET

  • Windows Authentication

  • IIS 6 Metabase Compatibility

  • IIS 6 WMI Compatibility

The following IIS role services are also required, and are enabled by default:

  • Static Content

  • Default Document

  • Directory Browsing

  • HTTP Errors

  • NET Extensibility

Other prerequisites

  • Microsoft SQL Server 2008 (or SQL Server 2005, for use with SCOM 2007 SP1)

  • ASP.NET 2.0 AJAX Extensions v1.0, downloadable from http://go.microsoft.com/fwlink/p/?LinkId=188656

    If the SCOM 2007 prerequisite checker cannot verify the installation of ASP.NET, start a command shell as administrator, and run the commands aspnet_regiis.exe -i -enable and IISReset from the directory where you installed ASP.NET. If the commands are successful, the output will look as follows:

    Copy Code
    C:\Windows\Microsoft.NET\Framework64\v2.0.50727>aspnet_regiis.exe -i -enable
Finished installing ASP.NET (2.0.50727).

C:\Windows\Microsoft.NET\Framework64\v2.0.50727>IISReset
Attempting stop...
Internet services successfully stopped
Attempting start...
Internet services successfully restarted

  • For Windows Server 2003, the Windows Server IIS 6.0 component must be present.


Install SCOM 2007 (FAST Search Server 2010 for SharePoint)
See Also

After successfully running the prerequisite checker, start the SCOM 2007 installer and follow directions from the installer dialog boxes.

Recommended settings

These are the recommended settings:

Management Group Configuration: Enter a unique name for your management group. Make a note of the exact string. You will need it when you install agents.

SQL Server Database Instance: Select your SQL Server database instance. By default, TCP connectivity for your SQL Server instance is disabled and not required. However, if you are running multiple instances of SQL Server on the same server, make sure that the selected TCP port is not used by a different SQL instance.

Database and Log File Options: The default database name is OperationsManager. There is no requirement to edit this name unless the selected SQL Server instance already has a database that has the same name.

Note:

The recommended database size for the FAST Search Server 2010 for SharePoint management pack is about 10 GB (10000 MB). A larger database size could handle more historical data, but can result in reduced SCOM 2007 performance.

Post installation configuration steps for SCOM 2007 SP1

Note:

The following steps are not required for SCOM 2007 R2. The following procedure is only relevant when you deploy SCOM 2007 SP1.

  1. Open the Authoring view in the SCOM Operations Console.

  2. Select Rules from the Management Pack Objects folder.

  3. Enter Performance Data Source Module in the Look for: box and then click Find Now.

  4. Right-click the Performance Data Source Module could not find a performance counter entry in the type: Health Service (2) group and select Overrides > Disable the Rule > For all objects of type: Health Service.

Once SCOM 2007 is successfully installed, continue by installing the SCOM 2007 agent on all search nodes. Make them Agent Managed servers, and obtain and import the System Center Operations Manager Management Pack for FAST Search Server 2010 for SharePoint.

See Also

Concepts

Install the SCOM 2007 agent on managed servers (FAST Search Server 2010 for SharePoint)
Obtain and import the management pack


Install the SCOM 2007 agent on managed servers (FAST Search Server 2010 for SharePoint)

A simple and secure way to install the SCOM 2007 agent is to install the agent manually on each managed server.

Adding a search node as an Agent Managed server

SCOM 2007 includes a Discovery Wizard that automatically installs the SCOM 2007 agent on managed servers. However, there are requirements associated with this kind of installation that you can avoid when you install manually instead.

The Discovery Wizard uses a process that is known as agent push installation. In a secure, firewalled environment, agent push installations do not work. Many firewall ports must be opened and various Windows services such as file and printer sharing must be enabled. The SQL Server Broker must be enabled too, and the SQL Server service must run using an account that has permission to perform Active Directory lookups. The Local System account does not have the required permission to do this.

You can avoid the requirements related to using the agent push installation if you install the agent manually on each managed server. The process can also be automated and performed without user interaction because the agent MSI installer accepts command-line parameters for unattended installations.

Agents can also be installed by using Active Directory integration. See Operations Manager 2007 Deployment Guide for more information.

Manually install the SCOM 2007 agent

Perform the following pre-configuration steps when you plan to add managed servers to SCOM 2007 using manual agent installation.

  1. Open the Administration view in the SCOM Operations Console.

  2. Click Settings in the Administration tree view.

  3. Double-click Security in the Settings pane.

  4. Select the setting Review new manual agent installations in pending management view.

The agent software is available from both the SCOM 2007 installation media and the installed Program Files folder:

  • <Installation media>\agent\amd64

  • <Program Files folder>\System Center Operations Manager 2007\AgentManagement\amd64

Important:

You must install the 64-bit (amd64) version of the agent. The 32-bit (x86) version does not give access to the correct set of registry keys used by 64-bit applications such as FAST Search Server 2010 for SharePoint.

After you have performed the pre-configuration steps, do the following:

  1. Run the MOMAgent.msi installer on each monitored server.

  2. Enter the name of the management group. This should be the same management group name as specified when you install SCOM 2007. The name can be found in the main window caption of the Operations Console.

  3. Enter the host name of the SCOM 2007 server in the Management Server field.

  4. Enter the management server port number. Use the supplied default. An exception for this port is automatically inserted in the Windows Server 2008 firewall by the SCOM 2007 R2 installer. No additional firewall configuration is needed unless you have other physical firewalls in the network. If that is the case, you should create exception rules that enable traffic using TCP connections from each monitored server (random port) to the SCOM 2007 server on port 5723.

  5. Enter the Agent Action Account. Enter Local System, or another account that has read access to the FAST Search Server 2010 for SharePoint installation folder and registry. The required permissions are available for any user account that is a member of the FASTSearchAdministrators local NT security group.

The following is an example of an automated agent installation:

Copy Code
%windir%\system32\msiexec.exe /i "\\fileserver\....amd64\MOMAgent.msi" /qb
 USE_SETTINGS_FROM_AD=0 MANAGEMENT_GROUP=VMDSCOM MANAGEMENT_SERVER_DNS=vm3.vmd.local
 SECURE_PORT=5723 ACTIONS_USE_COMPUTER_ACCOUNT=1

See the article Use the Command Line to Deploy Agents in the System Center Operations Manager 2007 documentation for more information.

Important:

Hotfix KB954049 must be applied on all SCOM 2007 SP1 agent installations. (This is not necessary for SCOM 2007 R2 agents).

Important:

Use the R2 agent with SCOM 2007 R2 servers, and the SP1 agent with SCOM 2007 SP1 servers.

Tip:

If an anti-virus service is running, exclude the folder <Program Files>\System Center Operations Manager 2007\Health Services State\Health Service Store\ from the anti-virus scan.

Approve agents in the Operations Console

  1. From the SCOM Operations Console, open Administration, then Device Management and then Pending Management. Managed nodes appear in this list several minutes after you install the agent.

  2. Right-click on each node and then click Approve. The nodes appear in the Agent Managed folder in a minute.


Obtain and import the management pack

This topic explains how to obtain the management pack for FAST Search Server 2010 for SharePoint and how to import it into SCOM 2007.

Obtain the management pack

The main distribution channel for the Microsoft FAST Search Server 2010 for SharePoint management pack is the System Center Management Pack Catalog Web site. Direct link to the FAST Search Server 2010 for SharePoint management pack: http://go.microsoft.com/fwlink/p/?LinkID=182110.

Import the management pack into SCOM 2007

The default installation folder for management pack files is <Program Files [(x86)]>\System Center Management Packs.

In the SCOM Operations Console, right-click the Management Packs entry in the Administration view and select Import Management Packs .

In SCOM 2007 R2, this import dialog supports automatic download and installation of management packs from the Management Pack Catalog using the Add > Add from catalog option.

You should also import the following management packs to make sure that all aspects of the FAST Search Server 2010 for SharePoint installation are monitored:

  • Windows Server 2008 (Core OS)

  • Internet Information Services (IIS) 7.0

  • Microsoft SQL Server 2008

  • Microsoft SharePoint Foundation 2010

  • Microsoft SharePoint Server 2010

Before you start to work with SCOM 2007, make sure that object discovery has finished. Object discovery can take anywhere from ten minutes to several hours. Use Refresh (F5).

Note:

If there is no discovery after you load a new management pack, restart the agent service on the respective node.

The name of the service is HealthService. In SCOM 2007 SP1, it is displayed as OpsMgr Health Service; in SCOM 2007 R2, it is displayed as System Center Management.


Install a software update (FAST Search Server 2010 for SharePoint)

Published: June 28, 2011

Microsoft periodically releases software updates for Microsoft FAST Search Server 2010 for SharePoint, such as cumulative updates and service packs. This article describes how to install a FAST Search Server 2010 for SharePoint software update in a single server deployment or in a multiple server deployment.

In this article:

  • Prepare to deploy a software update

  • Install a software update

  • Verify successful update

  • Troubleshooting

For information about how to update Microsoft SharePoint Server 2010, see Deploy software updates for SharePoint Server 2010.

Prepare to deploy a software update

Caution:

When you have installed an update, you cannot uninstall it from the system. There is no roll-back process available.

Before you install the update:

  • Document your current deployment to determine what is unique in your farm. Document the deployment.xml file and any customizations that you have applied.

    You can use different techniques to gather information about your deployment, such as manual inspection, comparisons by using tools such as Windiff.exe, or by using Windows PowerShell commands.

  • Back up your deployment. See Backup and restore (FAST Search Server 2010 for SharePoint).

    Recommended:

    • We recommend that you have a full backup of the current system before you apply the update.

    • If possible, create a similar pre-production system with the same configuration as your current system and test the update on that system before you apply the update to your production system.

Install a software update

The software update is delivered as an executable that updates the binaries and registry keys. After you have run the executable, you must run the post-setup configuration script in patch mode (update mode).

Important:

In a multiple server deployment, you must apply the update to all servers in the FAST Search Server 2010 for SharePoint deployment. Always update the administration server first, and ensure that the FAST Search Server 2010 for SharePoint service is running on the administration server before you continue with installing the update on the non-administration servers.

To install a software update in a single server deployment

  1. Double click the executable that contains the FAST Search Server 2010 for SharePoint update.

  2. Run the post-setup configuration script:

    1. Open a FAST Search Server 2010 for SharePoint shell as an administrator:

      1. On the Start menu, click All Programs

      2. Click Microsoft FAST Search Server 2010 for SharePoint.

      3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

    2. Browse to <FASTSearchFolder>\installer\scripts, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    3. Type the following command to run the post-setup configuration script in patch mode:

      .\psconfig.ps1 -action p

  3. Verify that the FAST Search for SharePoint service has started.

To install a software update in a multiple server deployment

  1. First update the administration server by following these steps:

    1. Double-click the executable that contains the FAST Search Server 2010 for SharePoint update.

    2. Run the post-setup configuration script:

      1. Open a FAST Search Server 2010 for SharePoint shell as an administrator:

        1. On the Start menu, click All Programs

        2. Click Microsoft FAST Search Server 2010 for SharePoint.

        3. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

      2. Browse to <FASTSearchFolder>\installer\scripts, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

      3. Type the following command to run the post-setup configuration script in patch mode:

        .\psconfig.ps1 -action p

    3. Verify that the FAST Search for SharePoint service has started.

  2. Follow the steps above on all non-administration servers in the deployment.

Verify successful update

After you have installed a FAST Search Server 2010 for SharePoint software update, verify that the update completed successfully.

  • Check that the update is shown in Installed Updates. In Control Panel, under Programs and Features, click View installed updates.

  • Check the installation log, here: <FASTSearchFolder>\var\log\installer\Psconfig-patch_Server_Name_Random_Number

Troubleshooting

Issue: When you run the post-setup configuration script, you receive the following error message:

Unable to regenerate schema. Please check that the user running this script is a member of the FASTSearchAdministrators group and that the admin services and configuration server are running. Unable to apply schema updates. Please check that the user running this script is a member of the FASTSearchAdministrators group and that the admin services and configuration server are running.

Cause: The World Wide Web Publishing Service has stopped.

Resolution: Start the World Wide Web Publishing Service and re-run the post-setup configuration script: .\psconfig.ps1 -action p


Uninstall FAST Search Server 2010 for SharePoint (FAST Search Server 2010 for SharePoint)
See Also

You can uninstall an existing FAST Search Server 2010 for SharePoint installation from Control Panel. Files, registered snap-ins, and registry keys that were installed by the Installer (the MSI) are removed from the system.

Prerequisites, logs, data (including the index), the database, and other temporary or generated files are not removed by the uninstall option.

If you only want to uninstall one server in a multiple server FAST Search Server 2010 for SharePoint farm, you must first make a deployment reconfiguration to remove the server in question from the farm. Refer to Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint) for more information.

To uninstall FAST Search Server 2010 for SharePoint

  1. On the Start menu, click Control Panel.

  2. In Control Panel, under Programs, click Uninstall a program.

  3. Click FAST Search for SharePoint.

  4. When the uninstaller has finished, restart the server.

  5. Delete the files in the <FASTSearchFolder> (<FASTSearchFolder> is the folder where you installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch).

  6. Optional: If you no longer want to use FAST Search Server 2010 for SharePoint, you can delete the FAST Search Content Search Service Application (Content SSA) and the FAST Search Query Search Service Application (Query SSA) from Central Administration.

  7. Optional: If you no longer need the data in the FAST Search Server 2010 for SharePoint database that was created when you installed FAST Search Server 2010 for SharePoint, you can delete the database. The database is stored in SQL Server.

You do not have to uninstall any parts of Microsoft SharePoint Server or delete any SharePoint Server databases when uninstalling FAST Search Server 2010 for SharePoint only.

See Also

Concepts

Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)
Change database (FAST Search Server 2010 for SharePoint)

Other Resources

To delete a service application by using Central Administration


Operations (FAST Search Server 2010 for SharePoint)

This section includes articles that describe how to do the following tasks:

  • Manage search topology (FAST Search Server 2010 for SharePoint)

  • Backup and restore (FAST Search Server 2010 for SharePoint)

  • Optimize search relevance (FAST Search Server 2010 for SharePoint)

  • Manage the Search Center (FAST Search Server 2010 for SharePoint)

  • Manage crawling (FAST Search Server 2010 for SharePoint)

  • Manage federated locations (FAST Search Server 2010 for SharePoint)

  • Manage people search (FAST Search Server 2010 for SharePoint)

  • Monitor FAST Search Server 2010 for SharePoint

  • Manage property extraction (FAST Search Server 2010 for SharePoint)

  • Manage content collections (FAST Search Server 2010 for SharePoint)

  • Manage index schema (FAST Search Server 2010 for SharePoint)

  • Manage search scopes (FAST Search Server 2010 for SharePoint)


Manage search topology (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

This section describes how to change the search topology configuration in a multiple server FAST Search Server 2010 for SharePoint deployment.

The search topology consists of the Search Service Applications in the SharePoint Server 2010 farm and the FAST Search Server 2010 for SharePoint farm.

In this section:

  • Manage FAST Search Server farm topology

  • Multiple server deployment of the Query SSA

  • Multiple server deployment of the Content SSA

The following figure shows the high-level topology of a FAST Search Server 2010 for SharePoint deployment.

FAST Search Server in an overall SharePoint Server farm topology

FAST Search Server in an overall SharePoint Server

The FAST Search Server 2010 for SharePoint farm (back-end) provides the processing, indexing, and query matching capabilities.

The parent SharePoint Server 2010 farm (front-end) communicates with the FAST Search Server 2010 for SharePoint farm through the FAST Search Query SSA and the FAST Search Content SSA.

SQL Server is used for configuration data, crawl database and property store for the people search index.

Note:

The FAST Search Server 2010 for SharePoint farm does not use the SQL Server database to store indexed properties. This differs from SharePoint Server 2010 search, which uses SQL Server as the property store.

See Also

Concepts

Plan search topology (FAST Search Server 2010 for SharePoint)
Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Plan search service applications (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)


Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

This section describes how to change the FAST Search Server 2010 for SharePoint farm topology.

In this section:

  • Manage performance and availability

  • Reconfigure the farm deployment

  • Increase indexing capacity

  • Increase query capacity

  • Manage high availability of the content index

Manage performance and availability

The following table lists typical FAST Search Server 2010 for SharePoint performance and availability related tasks with the associated operational procedures:

Task | Required steps
Increase indexing capacity to make more content searchable | Increase indexing capacity
Increase query capacity to handle more queries per second or reduce query latency | Increase query capacity
Configure query matching high availability | Increase query capacity
Configure indexing high availability | Manage high availability of the content index

The following figure shows the main topology components for the FAST Search Server 2010 for SharePoint farm.

FAST Search Server for SharePoint farm topology

FAST Search Server Farm Topology

The search cluster provides the topology for the indexing and query matching components. The following figure shows an example search cluster configuration with two index columns and three rows.

FAST search cluster topology

FAST search cluster architecture

See Also

Concepts

Plan FAST Search Server farm topology
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Performance and capacity planning (FAST Search Server 2010 for SharePoint)
Plan search service applications
Manage search topology


Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)
See Also

You can change the deployment of an existing Microsoft FAST Search Server 2010 for SharePoint farm by reconfiguring the deployment.xml file. Deployment changes include adding or removing servers and assigning components to servers.

Note:

This article contains generic procedures that are used when you reconfigure the farm topology. For information about topology operations, see Manage FAST Search Server farm topology.

In this article:

  • Manage configuration files

  • Reconfigure the farm deployment without adding new servers

  • Reconfigure the farm deployment including adding new servers

  • Remove servers from the farm deployment

  • Make the necessary changes on Microsoft SharePoint Server

  • About search rows and index columns in deployment.xml

Manage configuration files

FAST Search Server 2010 for SharePoint includes some configuration files that can be modified by developers or IT Professionals. For information about which configuration files that can be modified, see About configuration files (FAST Search Server 2010 for SharePoint). Changing other configuration files is not supported and could lead to system inconsistencies during re-configuration (Set-FASTSearchConfiguration cmdlet) and software updates.

Reconfigure the farm deployment without adding new servers

Follow these steps to make deployment changes for existing servers:

  1. On the administration server, open the deployment configuration file, <FASTSearchFolder>\etc\config_data\deployment\deployment.xml.

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    Make the necessary changes to the deployment configuration. For more information about the deployment configuration file syntax, see deployment.xml reference. For more information about index column and search row deployment configuration, see About search rows and index columns in deployment.xml.

  2. If you change the configuration of an indexer row or a search row, delete the following file on all servers in the row:

    <FASTSearchFolder>\etc\rtsplatformcache.xml

  3. On the administration server, follow these steps to reconfigure the servers in the deployment:

    1. Stop the FAST Search for SharePoint service.

    2. Stop the FAST Search for SharePoint Monitoring service.

    3. Open a Microsoft FAST Search Server 2010 for SharePoint shell:

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

      5. To re-create configuration files, at the Windows PowerShell command prompt, type the following command:

        Copy Code
        Set-FASTSearchConfiguration

      6. Start the FAST Search for SharePoint service.

        Note:

        This will also start the FAST Search for SharePoint Monitoring service.

      7. To verify that all processes are running, at the Windows PowerShell command prompt, type the following command:

        Copy Code
        nctrl status

  4. When the administration server is started and running, repeat step 3 on all other servers in the FAST Search Server 2010 for SharePoint deployment.

Reconfigure the farm deployment including adding new servers

Follow these steps to make deployment changes that include adding or removing servers:

  1. On the administration server, open <FASTSearchFolder>\etc\config_data\deployment\deployment.xml and make the necessary changes.

    <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    For more information about the deployment configuration file (deployment.xml) syntax, see deployment.xml reference. For more information about index column and search row deployment configuration, see About search rows and index columns in deployment.xml.

  2. On the administration server, follow these steps to reconfigure the deployment:

    1. Stop the FAST Search for SharePoint service.

    2. Stop the FAST Search for SharePoint Monitoring service.

    3. Open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator:

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

      5. To configure Internet Protocol security (IPsec), at the Windows PowerShell command prompt, type the following command:

        Copy Code
        Set-FASTSearchIPSec -create

        Use the same user name and password that you used during installation.

      6. To re-create configuration files, at the Windows PowerShell command prompt, type the following command:

        Copy Code
        Set-FASTSearchConfiguration

      7. Start the FAST Search for SharePoint service.

        Note:

        This will also start the FAST Search for SharePoint Monitoring service.

      8. To verify that all processes are running, at the Windows PowerShell command prompt, type the following command:

        Copy Code
        nctrl status

  3. When the administration server is started and running, repeat step 2 on all other servers in the FAST Search Server 2010 for SharePoint farm.

  4. Install and configure FAST Search Server 2010 for SharePoint on the new server(s). For each server, perform the following steps:

    1. Install FAST Search Server 2010 for SharePoint

    2. Configure the non-administration server(s) in a multiple server deployment.

Remove servers from the farm deployment

Follow these steps to make deployment changes that include removing servers:

  1. On the administration server, open <FASTSearchFolder>\etc\config_data\deployment\deployment.xml and make the necessary changes.

    <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    For more information about the deployment configuration file (deployment.xml) syntax, see deployment.xml reference. For more information about index column and search row deployment configuration, see About search rows and index columns in deployment.xml.

  2. On the administration server, follow these steps to reconfigure the deployment:

    1. Stop the FAST Search for SharePoint service.

    2. Stop the FAST Search for SharePoint Monitoring service.

    3. Open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator:

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Right click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

      5. To configure Internet Protocol security (IPsec), at the Windows PowerShell command prompt, type the following command:

        Copy Code
        Set-FASTSearchIPSec -create

        Use the same user name and password that you used during installation.

      6. To re-create configuration files, at the Windows PowerShell command prompt, type the following command:

        Copy Code
        Set-FASTSearchConfiguration

      7. Start the FAST Search for SharePoint service.

        Note:

        This will also start the FAST Search for SharePoint Monitoring service.

      8. To verify that all processes are running, at the Windows PowerShell command prompt, type the following command:

        Copy Code
        nctrl status

  3. When the administration server is started and running, repeat step 2 on all other servers in the FAST Search Server 2010 for SharePoint farm (not including the servers you shall remove).

  4. Uninstall FAST Search Server 2010 for SharePoint on the server(s) you shall remove. For each server, perform the steps in Uninstall FAST Search Server 2010 for SharePoint (FAST Search Server 2010 for SharePoint).

Make the necessary changes on Microsoft SharePoint Server

The FAST Search Content Search Service Application (Content SSA) and the FAST Search Query Search Service Application (Query SSA) on the Microsoft SharePoint Server 2010 are configured to have information about the FAST Search Server 2010 for SharePoint farm. Therefore, when you change the deployment of FAST Search Server 2010 for SharePoint, you may also have to update the configuration of the Content SSA and the Query SSA.

Update Content SSA connection configuration

For the FAST Search Content SSA, you must update the Content Distributors field in the Content SSA configuration if you have added, removed, or moved content distributors (content-distributor element) in the deployment.xml file:

To update Content SSA connection configuration

  1. Verify that the user account that is performing this procedure is a service application administrator for the FAST Search Content Search Service Application.

  2. On SharePoint Server 2010, in Central Administration, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, highlight your FAST Search Content SSA and then click Properties in the ribbon.

  4. On the Edit Search Service page, update the Content Distributors field. Add each server that has the content distributor component to this field. Copy the information from the install_info.txt file, which is found directly under the installation directory on any FAST Search Server 2010 for SharePoint server. (The file is updated as part of reconfiguration and will therefore contain updated information after reconfiguration.)

Update Query SSA connection configuration

For the FAST Search Query SSA, you must update the Query Service Location field if you have added, removed, or moved query processing servers (query element) in the deployment configuration file deployment.xml.

To update the Query Service Location

  1. Verify that the user account that is performing this procedure is a service application administrator for the FAST Search Content Search Service Application.

  2. On SharePoint Server 2010, in Central Administration, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, highlight your FAST Search Query SSA and then click Properties in the ribbon.

  4. On the Edit Search Service page, update the Query Service Location field. Add each server that has the query role in deployment.xml to this field. Copy the information from the install_info.txt file, which is found directly under the installation directory on any FAST Search Server 2010 for SharePoint server. (The file is updated as part of reconfiguration and will therefore contain updated information after reconfiguration.)

  5. Follow the steps in Configure claims authentication to configure the connection between the query processing server and the Query SSA.

    Note:

    This step is not required if you remove a query processing server.

About search rows and index columns in deployment.xml

You configure the search row and index column roles of the FAST Search Server 2010 for SharePoint topology in the deployment.xml configuration file.

The searchengine sub-element of the host element specifies the index column and the search/indexer row identity. The row attribute value refers to the corresponding row id in the searchcluster specification. The index attribute specifies the indexing role of the particular row:

  • primary: Primary indexer

  • secondary: Backup indexer

  • none: No indexer running on the servers in this row. This row acts as a stand-alone search row

The search attribute specifies the query matching role of the particular row:

  • true: Query matching is performed on this row. This row acts as a separate search row.

  • false: No query matching is running on the servers in this row. This row acts as a dedicated indexer row.

Example configuration without backup indexer

The following example shows an extract of a deployment.xml file that has the host definition for three indexer servers.

  • The first host element represents the administration server, which is also configured for query processing and the Web Analyzer.

  • The three following host elements represent the indexer server for index column 0, 1 and 2.

  • There are no backup indexer defined, and there are only one search row.

Copy Code
  <host name="fs4sp1.contoso.com">
    <admin />
    <query />
    <webanalyzer server="true" link-processing="true"
                 lookup-db="true" max-targets="4"/>
    <document-processor processes="12" />
  </host>
  <host name="fs4sp2.contoso.com">
    <content-distributor />
    <searchengine row="0" column="0" />
    <document-processor processes="12" />
  </host>
  <host name="fs4sp3.contoso.com">
    <content-distributor />
    <indexing-dispatcher />
    <searchengine row="0" column="1" />
    <document-processor processes="12" />
  </host>
  <host name="fs4sp4.contoso.com">
    <indexing-dispatcher />
    <searchengine row="0" column="2" />
    <document-processor processes="12" />
  </host>

  <searchcluster>
      <row id="0" index="primary" search="true" />
  </searchcluster>

Example configuration with backup indexer and two search rows

The following example shows an extract of a deployment.xml file that has the host definition for three indexer servers that have a backup indexer.

  • The first host element represents the administration server, which is also configured for query processing and the Web Analyzer.

  • The three following host elements represent the primary indexer server for index column 0, 1 and 2.

  • The three last host elements represent the backup indexer server for index column 0, 1 and 2.

  • Both rows are configured as search rows.

Copy Code
  <host name="fs4sp1.contoso.com">
    <admin />
    <query />
    <webanalyzer server="true" link-processing="true"
                          lookup-db="true" max-targets="4"/>
    <document-processor processes="12" />
  </host>
  <host name="fs4sp2.contoso.com">
    <content-distributor />
    <searchengine row="0" column="0" />
    <document-processor processes="12" />
  </host>
  <host name="fs4sp3.contoso.com">
    <content-distributor />
    <indexing-dispatcher />
    <searchengine row="0" column="1" />
    <document-processor processes="12" />
  </host>
  <host name="fs4sp4.contoso.com">
    <indexing-dispatcher />
    <searchengine row="0" column="2" />
    <document-processor processes="12" />
  </host>
  <host name="fs4sp5.contoso.com">
    <query />
    <searchengine row="1" column="0" />
  </host>
  <host name="fs4sp6.contoso.com">
    <query />
    <searchengine row="1" column="1" />
  </host>
  <host name="fs4sp7.contoso.com">
    <query />
    <searchengine row="1" column="2" />
  </host>

  <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="secondary" search="true" />
  </searchcluster>

See Also

Concepts

deployment.xml syntax reference
Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Plan search service applications (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)


Increase indexing capacity (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

A default FAST Search Server 2010 for SharePoint configuration can index up to 15 million items per index column. Indexing more than 15 million items per column will adversely affect indexing and query performance.

In this section:

  • Add an index column.

    You can scale out by adding an index column on a new set of servers.

  • Optimize for extended content capacity.

    For scenarios with low content update rates and moderate query volume, you can configure the index to handle up to 40 million items per index column.

  • Remove an index column.

    If you remove content sources from the index, you can remove an index column to save hardware cost.

Remove an index column

You remove an index column in the same way as you add an index column, but with the following modification to the steps in Add an index column:

  • Remove the host element in the deployment configuration file for each server in the index column you shall remove.

  • Remove and uninstall the indexer server(s).

See Also

Concepts

Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Performance and capacity planning (FAST Search Server 2010 for SharePoint)
Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)
deployment.xml reference


Add an index column (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

If the total index is too large to reside on one server, you can scale out your FAST Search Server 2010 for SharePoint farm by adding an index column.

Warning:

Before you add an index column, you must first remove all indexed content from the FAST Search Server 2010 for SharePoint farm. Depending on the size of the deployment, it may take days to re-index the content.

To add an index column

  1. Stop crawling:

    • If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about stopping a crawl.

    • If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about stopping a crawl.

    • If you use the FAST Search database connector to crawl content, see Test, start or stop a crawl (FAST Search database connector) for information about stopping a crawl.

    • If you use the FAST Search Lotus Notes connector to crawl content, see Test, start or stop a crawl (FAST Search Lotus Notes content connector) for information about stopping a crawl.

  2. Pause the Web Analyzer components. These components analyze links between items and search clickthrough logs. The components must be stopped to avoid partial updates for indexed items during the procedure.

    1. On the administration server, open a FAST Search Server 2010 for SharePoint shell.

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

    2. Inspect the Web Analyzer schedule status:

      Copy Code
      waadmin ShowStatus
      Note:

      Refer to waadmin.exe reference if you use multiple views for the Web Analyzer.

      In the Views section of the command output, for the default view, inspect the Schedule status. If the Schedule status is set to paused, follow these steps:

      1. Start the Web Analyzer view:

        Copy Code
        waadmin enqueueview

      2. Wait until the default Web Analyzer view is running. Type the following command to inspect the Schedule status:

        Copy Code
        waadmin ShowStatus

        Repeat the command until the Schedule status is set to running.

    3. Stop Web Analyzer processing:

      Copy Code
      waadmin AbortProcessing
spreladmin AbortProcessing

  3. Wait until all item processing is finished:

    Copy Code
    psctrl status

    Repeat the command until all processor servers report "idle".

  4. Remove all items from the content index:

    • If you use the FAST Search Content SSA to crawl content, follow the steps in Reset the content index (FAST Search Server 2010 for SharePoint) to remove all items from the default content collection named "sp".

    • If you use the FAST Search Web crawler to crawl content, follow the steps in Reset the content index (FAST Search Web crawler) to remove all items from the content collection or collections that are associated with this connector.

    • If you use the FAST Search database connector to crawl content, follow the steps in Reset the content index (FAST Search database connector) to remove all items from the content collection or collections that are associated with this connector.

    • If you use the FAST Search Lotus Notes connector to crawl content, follow the steps in Reset the content index (FAST Search Lotus Notes connector) to remove all items from the content collection or collections that are associated with this connector.

  5. Make sure that no items are indexed on any of the indexers.

    On the administration server, run the following command:

    Copy Code
    indexerinfo -a doccount
    Note:

    All indexers must report 0 items before you proceed to the next step.

  6. Reconfigure the deployment to add an index column:

    1. Determine the changes that you must apply to the deployment configuration file (deployment.xml):

      • Add a host element for each server in the new index column.

        Note:

        All index columns must have the same configuration.

    2. Follow the steps in Reconfigure the farm deployment including adding new servers.

  7. On the administration server, run the following commands to resume processing for the Web Analyzer components:

    Copy Code
    waadmin EnqueueView
spreladmin Enqueue

  8. Start a full crawl of all content:

    • If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about starting a crawl.

    • If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about starting a crawl.

    • If you use the FAST Search database connector to crawl content, see Test, start or stop a crawl (FAST Search database connector) for information about starting a crawl.

    • If you use the FAST Search Lotus Notes connector to crawl content, see Test, start or stop a crawl (FAST Search Lotus Notes content connector) for information about starting a crawl.

See Also

Reference

waadmin.exe reference
spreladmin.exe reference

Concepts

Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Manage high availability of the content index (FAST Search Server 2010 for SharePoint)
Increase query capacity (FAST Search Server 2010 for SharePoint)
deployment.xml reference
Extend content capacity for the indexer (FAST Search Server 2010 for SharePoint)


Extend content capacity for the indexer (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

The default indexer configuration does not allow for more than 30 million items per index column, and indexing more than 15 million items per index column will adversely affect indexing and query performance.

For scenarios with low content update rates and moderate query volume it may be desired to index more items per column in order to save hardware cost. This section describes how to configure the indexer to handle up to 40 million items per column.

In this section:

  • Configure a new deployment for extended content capacity

  • Configure an existing deployment for extended content capacity

  • Handle software updates when configured for extended content capacity

See Also

Concepts

Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint)
Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Add an index column (FAST Search Server 2010 for SharePoint)

Other Resources

FAST Search Server 2010 for SharePoint Capacity Planning (white paper)


Extend content capacity for a new deployment (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

A default Microsoft FAST Search Server 2010 for SharePoint indexer configuration is optimized for content volumes up to 15 million items per index column. You can change the indexer configuration to handle up to 40 million items per column.

Important:

Before you extend the content capacity, read the recommendations in Performance and capacity recommendations (FAST Search Server 2010 for SharePoint).

To extend content capacity for a new deployment

  1. Install FAST Search Server 2010 for SharePoint by following the steps in Install FAST Search Server 2010 for SharePoint.

  2. Change the index partition configuration:

    1. On the administration server, locate the indexer template configuration file:

      <FASTSearchFolder>\META\config\profiles\default\templates\installer\etc\config_data\RTSearch\clusterfiles\rtsearchrc.xml.win32.template

      Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    2. Save a backup of the file. You may need the backup at a later stage if the configuration file is modified in a software update.

    3. In a text editor, open the file and change the following configuration parameters:

      • numberPartitions:

        Default value: 5

        New value: 10

      • docsDistributionMax:

        Default value:

        6000000,6000000,6000000,6000000,6000000

        New value:

        6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000

  3. Configure FAST Search Server 2010 for SharePoint by following the steps in Configure a multiple server deployment.

See Also

Concepts

Test results: Extra-small scenario (FAST Search Server 2010 for SharePoint)
Extend content capacity for an existing deployment (FAST Search Server 2010 for SharePoint)
Handle software updates when you have extended the content capacity (FAST Search Server 2010 for SharePoint)
Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Add an index column (FAST Search Server 2010 for SharePoint)

Other Resources

FAST Search Server 2010 for SharePoint Capacity Planning (white paper)


Extend content capacity for an existing deployment (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

A default Microsoft FAST Search Server 2010 for SharePoint indexer configuration is optimized for content volumes up to 15 million items per index column. You can change the indexer configuration to handle up to 40 million items per column.

Warning:

Before you change the configuration, you must first remove all indexed content from your FAST Search Server 2010 for SharePoint farm. Depending on the size of your deployment, it may take days or even weeks to re-index the content.

Before you extend the content capacity, read the Index capacity tuning recommendations.

To extend content capacity for a deployment with indexed content

  1. Stop crawling:

    1. If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about stopping a crawl.

    2. If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about stopping a crawl.

    3. If you use the FAST Search database connector to crawl content, see Test, start or stop a crawl (FAST Search database connector) for information about stopping a crawl.

    4. If you use the FAST Search Lotus Notes connector to crawl content, see Test, start or stop a crawl (FAST Search Lotus Notes content connector) for information about stopping a crawl.

  2. Pause the Web Analyzer components. These components analyze links between items and search clickthrough logs. The components must be stopped to avoid partial updates for indexed items during the procedure.

    1. On the administration server, open a FAST Search Server 2010 for SharePoint shell.

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

    2. Inspect the Web Analyzer schedule status:

      Copy Code
      waadmin ShowStatus
      Note:

      Refer to waadmin.exe reference if you use multiple views for the Web Analyzer.

      In the Views section of the command output, for the default view, inspect the Schedule status.

      1. If the Schedule status is set to paused, type the following command:

        Copy Code
        waadmin enqueueview

      2. Wait until the default Web Analyzer view is running. Type the command waadmin ShowStatus and inspect the Schedule status. The Schedule status should be set to running.

    3. Stop Web Analyzer processing:

      Copy Code
      waadmin AbortProcessing
spreladmin AbortProcessing

  3. Wait until all item processing is finished:

    Copy Code
    psctrl status

    Repeat the command until all processor servers report "idle".

  4. Remove all items from the content index:

    1. If you use the FAST Search Content SSA to crawl content, follow the steps in Reset the content index (FAST Search Server 2010 for SharePoint) to remove all items from the default content collection named "sp".

    2. If you use the FAST Search Web crawler to crawl content, follow the steps in Reset the content index (FAST Search Web crawler) to remove all items from the content collection or collections that are associated with this connector.

    3. If you use the FAST Search database connector to crawl content, follow the steps in Reset the content index (FAST Search database connector) to remove all items from the content collection or collections that are associated with this connector.

    4. If you use the FAST Search Lotus Notes connector to crawl content, follow the steps in Reset the content index (FAST Search Lotus Notes connector) to remove all items from the content collection or collections that are associated with this connector.

  5. Make sure that no items are indexed on any of the indexers:

    On the administration server, type the following command:

    Copy Code
    indexerinfo -a doccount
    Note:

    All indexers must report 0 items before you proceed to the next step.

  6. Change the index partition configuration:

    1. On the administration server, locate the indexer template configuration file:

      <FASTSearchFolder>\META\config\profiles\default\templates\installer\etc\config_data\RTSearch\clusterfiles\rtsearchrc.xml.win32.template

      Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    2. Save a backup of the file. You may need the backup at a later stage if the configuration file is modified in a software update.

    3. In a text editor, open the file and change the following configuration parameters:

      • numberPartitions:

        Default value: 5

        New value: 10

      • docsDistributionMax:

        Default value:

        6000000,6000000,6000000,6000000,6000000

        New value:

        6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000

  7. Re-deploy the farm configuration to all servers:

    Important:

    In this step, you must change the deployment configuration file (deployment.xml) without changing the actual content. You can do that by adding a space and then save the file.

    Follow the steps in Reconfigure the farm deployment including adding new servers.

  8. On the administration server, run the following commands to resume processing for the Web Analyzer components:

    Copy Code
    waadmin EnqueueView
spreladmin Enqueue

  9. Start a full crawl of all content:

    1. If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about starting a crawl.

    2. If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about starting a crawl.

    3. If you use the FAST Search database connector to crawl content, see Test, start or stop a crawl (FAST Search database connector) for information about starting a crawl.

    4. If you use the FAST Search Lotus Notes connector to crawl content, see Test, start or stop a crawl (FAST Search Lotus Notes content connector) for information about starting a crawl.

See Also

Concepts

Extend content capacity for a new deployment (FAST Search Server 2010 for SharePoint)
Handle software updates when you have extended the content capacity (FAST Search Server 2010 for SharePoint)
Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Add an index column (FAST Search Server 2010 for SharePoint)

Other Resources

FAST Search Server 2010 for SharePoint Capacity Planning (white paper)


Handle software updates when you have extended the content capacity (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

Follow this procedure if you install a software update or a service pack, and have extended the content capacity for your indexer servers.

To change the configuration file in association with a software update

  1. Review the readme file to determine if the following configuration file is changed as part of a software update:

    <FASTSearchFolder>\META\config\profiles\default\templates\installer\etc\config_data\RTSearch\clusterfiles\rtsearchrc.xml.win32.template

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    If the file is affected by the update, follow the next steps.

  2. Replace the configuration file with the backup file that you saved when you modified the index partition configuration.

  3. Perform the software update.

    Important:

    Do not start the FAST Search Server 2010 for SharePoint service after the software update. You start the service later in this procedure.

  4. Change the index partition configuration:

    1. On the administration server, locate the indexer template configuration file:

      <FASTSearchFolder>\META\config\profiles\default\templates\installer\etc\config_data\RTSearch\clusterfiles\rtsearchrc.xml.win32.template

    2. Save a backup of the file. You may need the backup at a later stage if the configuration file is modified in a software update.

    3. In a text editor, open the file and change the following configuration parameters:

      • numberPartitions:

        Default value: 5

        New value: 10

      • docsDistributionMax:

        Default value:

        6000000,6000000,6000000,6000000,6000000

        New value:

        6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000,6000000

  5. Re-deploy the farm configuration to all servers:

    Important:

    In this step, you must change the deployment configuration file (deployment.xml) without changing the actual content. You can do that by adding a space and then save the file.

    To re-deploy the farm configuration, follow the steps in Reconfigure the farm deployment including adding new servers.

See Also

Concepts

Performance and capacity recommendations (FAST Search Server 2010 for SharePoint)
Extend content capacity for the indexer (FAST Search Server 2010 for SharePoint)
Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)

Other Resources

FAST Search Server 2010 for SharePoint Capacity Planning (white paper)


Increase query capacity (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

To improve query performance and availability, you can scale out with an additional search row.

Note:

When you add a second search row you should normally have two query processing servers in your deployment. For more information about topology options and planning, see Plan search topology (FAST Search Server 2010 for SharePoint).

In this article:

  • Add a search row

  • Add the search row role to a backup indexer row

  • Configure a new combined backup indexer and search row

  • Remove a search row

  • Add a query processing server

Add a search row

To add a search row, you install and deploy one query matching server for each index column in the farm.

To add a search row to a new set of servers

  1. Determine the changes you must apply to the deployment configuration file (deployment.xml):

    • For the searchcluster element, add a row sub-element with the search attribute set to true.

    • Add a host element for each server in the new search row. Add a searchengine sub-element with the row number of the new search row.

  2. Follow the steps in Reconfigure the farm deployment including adding new servers.

Add the search row role to a backup indexer row

If your farm deployment includes a backup indexer row, you can add the search row role to the servers of the backup indexer row.

To add the search row role to the servers in a backup indexer row

  1. Determine the changes you must apply to the deployment configuration file (deployment.xml):

    • searchcluster: For the row element that represents the backup indexer, you change the search attribute value from false to true.

  2. Follow the steps in Reconfigure the farm deployment without adding new servers.

Configure a new combined backup indexer and search row

For many deployments it is convenient to deploy backup indexer and query matching on the same row.

Follow the steps in Add a backup indexer row. Make sure that you set the search attribute value to true in the deployment configuration file (deployment.xml).

Remove a search row

Follow the steps in Remove servers from the farm deployment, where you change the following elements in the deployment configuration file (deployment.xml):

  • searchcluster: Remove the row element for the search row.

  • host: Remove the host element for each server in the search row.

Add a query processing server

You can add the query processing role to a new or an existing server in the deployment.

To add a query processing server

  1. Determine the changes you must apply to the deployment configuration file (deployment.xml):

    • If you add a new server, add a host element.

    • Add a query element to the host element for the server that shall have the query processing role.

  2. Reconfigure the deployment:

    • If you add the query processing role to a new server, follow the steps in Reconfigure the farm deployment including adding new servers.

    • If you add the query processing role to an existing server, follow the steps in Reconfigure the farm deployment without adding new servers.

  3. Update the connection configuration for the interface between the Query SSA and the query processing server. Follow the steps in Update Query SSA connection configuration.

See Also

Concepts

Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Increase indexing capacity (FAST Search Server 2010 for SharePoint)
Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)
deployment.xml reference

Other Resources

FAST Search Server 2010 for SharePoint Capacity Planning (white paper)


Manage high availability of the content index (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

This section describes how to manage a FAST Search Server 2010 for SharePoint backup indexer configuration.

Note:

You can also perform regular full data backup to ensure high availability for the content index.

For more information and planning considerations, see Indexer high availability (Plan for redundancy and availability).

In this section:

  • Add a backup indexer row (FAST Search Server 2010 for SharePoint)

  • Verify the status of the backup indexer (FAST Search Server 2010 for SharePoint)

  • Synchronize the primary and the backup indexer servers (FAST Search Server 2010 for SharePoint)

  • Replace a failed primary indexer server (FAST Search Server 2010 for SharePoint)

  • Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)

  • Remove a backup indexer row

Remove a backup indexer row

Follow the steps in Remove servers from the farm deployment, where you modify the following element in the deployment configuration file (deployment.xml):

  • searchcluster: Remove the row element for the search row.

  • Remove the host element for each server in the search row.

See Also

Reference

Components within the FAST Search Server 2010 for SharePoint farm

Concepts

Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Manage search topology (FAST Search Server 2010 for SharePoint)


Add a backup indexer row (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

You can add a backup indexer row on a new set of servers, or you can add the backup indexer role to an existing search row in your farm deployment.

To add a backup indexer row

  1. Determine the changes you need to apply to the deployment configuration file (deployment.xml):

    • If you add a backup indexer row on a new set of servers:

      • searchcluster: Add a row element for the backup indexer row, with the index attribute set to "secondary".

      • host: Add a host element for each server in the new backup indexer row. Add a searchengine sub-element with correct row and column attribute values.

    • If you add the backup indexer role to an existing search row:

      • searchcluster: For the existing search row, set the index attribute to "secondary".

  2. Reconfigure the deployment:

    • If you add a backup indexer row on a new set of servers, follow the steps in Reconfigure the farm deployment including adding new servers.

    • If you add the backup indexer role to an existing search row, follow the steps in Reconfigure the farm deployment without adding new servers.

  3. Copy the index data from the primary indexer to the backup indexer:

    • For all index columns in your deployment, follow the steps in Synchronize the primary and the backup indexer server.

See Also

Concepts

Manage high availability of the content index (FAST Search Server 2010 for SharePoint)
Verify the status of the backup indexer (FAST Search Server 2010 for SharePoint)
Synchronize the primary and the backup indexer servers (FAST Search Server 2010 for SharePoint)
Replace a failed primary indexer server (FAST Search Server 2010 for SharePoint)
Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)
deployment.xml reference


Verify the status of the backup indexer (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

This article describes how to verify if the Microsoft FAST Search Server 2010 for SharePoint backup indexer is synchronized with the primary indexer.

Verify this status whenever there has been any downtime on one of the indexer servers or you have observed issues with indexing or search availability.

In this article:

  • Verify the status of the backup indexer by using SCOM.

    You may use this procedure if you already use SCOM for monitoring your FAST Search Server 2010 for SharePoint farm.

  • Verify the status of the backup indexer by using RC.

    RC is a FAST Search Server 2010 for SharePoint tool that can be used to collect monitoring data from different components of the farm.

  • About FiXML files.

Verify the status by using SCOM

To verify the status of the backup indexer by using SCOM

  1. Ensure that you have installed Microsoft System Operations Manager (SCOM) 2007 SP1 or R2 and the FAST Search Server 2010 for SharePoint Management Pack for SCOM.

  2. Open System Center Operations Manager (SCOM) 2007 Performance Monitor.

  3. For each backup indexer server, view the performance counters for FAST Search Indexer. View the counter ManualFTSyncNeeded.

    If this counter has the value "1.000", you must manually synchronize the indexer servers. For more information, see Synchronize the primary and the backup indexer servers (FAST Search Server 2010 for SharePoint).

Verify the status by using RC

To verify the status of the backup indexer by using RC

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    Copy Code
    rc -r | Select-String "manual FT sync needed","indexer column id"

    All indexers must report "0" for manual ft sync needed. If any of them report "1", you must manually synchronize the indexers. For more information, see Synchronize the primary and the backup indexer servers (FAST Search Server 2010 for SharePoint).

    In the following example command output for a deployment with three index columns we can see that one indexer server is out of sync in index column 2.

    PS C:\FASTSearch> rc -r | Select-String "manual FT sync needed","indexer column id"
    indexer column id                         : 0
    manual ft sync needed                     : 0
    indexer column id                                : 0
    manual ft sync needed                            : 0
    indexer column id                         : 1
    manual ft sync needed                     : 0
    indexer column id                                : 1
    manual ft sync needed                            : 0
    indexer column id                         : 2
    manual ft sync needed                     : 0
    indexer column id                                : 2
    manual ft sync needed                            : 1
About FiXML files
After item processing, the crawled items are distributed to individual index columns based on a statistical routing algorithm. The routing algorithm ensures that items are evenly distributed across the columns, and also ensures that updates to an item are routed to the correct column.
The items are stored in an XML-based pre-index format (FiXML files). The indexer uses the FiXML files as input to the indexing process.
In a fault-tolerant indexer setup, the primary indexer copies the FiXML files to the backup indexer. Manual recovery is required if a backup indexer gets out of sync with the primary index and is unable to recover automatically.
  See Also
Reference
rc.exe reference
Concepts
Manage high availability of the content index (FAST Search Server 2010 for SharePoint)
Synchronize the primary and the backup indexer servers (FAST Search Server 2010 for SharePoint)
Replace a failed primary indexer server (FAST Search Server 2010 for SharePoint)
Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)
Monitor FAST Search Server 2010 for SharePoint with SCOM


Synchronize the primary and the backup indexer servers (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

There are three situations where you must manually synchronize the backup indexer with the primary indexer to ensure that the primary indexer server and the backup indexer server have the same set of indexed items:

  • You have added a backup indexer to an existing FAST Search Server 2010 for SharePoint farm that contains indexed content.

  • You have detected that the backup indexer is out of sync with the primary indexer.

  • You have replaced an indexer server after a non-recoverable error.

To synchronize the primary indexer server and the backup indexer server in an index column

  1. Stop crawling:

    • If you are using the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about stopping a crawl.

    • If you are using the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about stopping a crawl.

  2. Pause the Web Analyzer components. These components analyze links between items and search clickthrough logs. The components must be stopped to avoid partial updates for indexed items during the procedure.

    1. On the administration server, open a FAST Search Server 2010 for SharePoint shell.

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

    2. Inspect the Web Analyzer schedule status. At the Windows PowerShell command prompt, type the following command:

      Copy Code
      waadmin ShowStatus
      Note:

      Refer to waadmin.exe reference if you are using multiple views for the Web Analyzer.

      In the Views section of the command output, for the default view, inspect the Schedule status. If the Schedule status is set to paused, perform the following steps:

      1. At the Windows PowerShell command prompt, type the following command:

        Copy Code
        waadmin enqueueview

      2. Wait until the default Web Analyzer view is running. Type the command waadmin ShowStatus and inspect the Schedule status. The Schedule status shall be set to running.

    3. At the Windows PowerShell command prompt, type the following commands:

      Copy Code
      waadmin AbortProcessing
spreladmin AbortProcessing

  3. Wait until all item processing is finished:

    At the Windows PowerShell command prompt, type the following command:

    Copy Code
    psctrl status
    Note:

    All processor servers must report "idle" before you proceed to the next step.

  4. Stop indexing. Perform the following steps on all indexer servers in the index column:

    1. Open a FAST Search Server 2010 for SharePoint shell.

    2. At the Windows PowerShell command prompt, type the following command:

      Copy Code
      nctrl stop indexer

  5. Copy all FiXML files between the indexer servers in the index column.

    • If you have replaced the primary indexer server, copy the FiXML files from the backup indexer server (source) to the primary indexer server (destination).

    • For all other cases, copy files from the primary indexer server (source) to the backup indexer server (destination).

    The FiXML files are stored in the following directory on the indexer servers:

    <FASTSearchFolder>\data\data_fixml\

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    1. On the destination indexer server, create two temporary folders for the index data you shall copy. At the Windows command prompt, type the following commands:

      Copy Code
      mkdir <FASTSearchFolder>\data\data_fixml_new
mkdir <FASTSearchFolder>\data\ftStorage_new

    2. On the destination indexer server, at the Windows command prompt, copy the data_fixml folder from the source server:

      Copy Code
      robocopy /E /MT:100 /NFL /COPYALL /LOG:\incoming\robocopy_fixml.log <source_path>\data\data_fixml <FASTSearchFolder>\data\data_fixml_new
copy <source_path>\data\ftStorage\processed_checkpoint.txt <FASTSearchFolder>\data\ftStorage_new

      Where <source_path> is the network path of the FAST Search Server 2010 for SharePoint installation folder on the source server.

      Note:

      The /LOG: folder (\incoming\) does not exist by default and you need to have write access to it to run the command.

    3. Swap the copied folders (<FASTSearchFolder>\data_fixml_new and <FASTSearchFolder>\ftStorage_new) with the old folders (<FASTSearchFolder>\data\data_fixml and <FASTSearchFolder>\data\ftStorage) on the destination server.

  6. On all indexer servers in the index column, at the Windows PowerShell command prompt, type the following command:

    Copy Code
    nctrl start indexer

  7. On the administration server, at the Windows PowerShell command prompt, type the following commands to resume processing for the Web Analyzer components:

    Copy Code
    waadmin EnqueueView
spreladmin Enqueue

  8. Start crawling:

    • If you are using the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about starting a crawl.

    • If you are using the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about starting a crawl.

See Also

Reference

About FiXML files

Concepts

Manage high availability of the content index (FAST Search Server 2010 for SharePoint)
Verify the status of the backup indexer (FAST Search Server 2010 for SharePoint)
Add a backup indexer row (FAST Search Server 2010 for SharePoint)
Replace a failed primary indexer server (FAST Search Server 2010 for SharePoint)
Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)


Replace a failed primary indexer server (FAST Search Server 2010 for SharePoint)
See Also

Published: December 16, 2010

If the farm deployment includes a backup indexer row, you must manually change the deployment configuration if a non-recoverable server error occurs on the primary indexer server. You must also synchronize the indexer servers after you have replaced the failed server.

Follow these steps to replace a primary indexer server if a non-recoverable hardware failure has occurred:

  • Step 1: Pause item processing and crawling

  • Step 2: Reconfigure the backup indexer server to become the new primary indexer server

  • Step 3: Install and configure a new backup indexer server

  • Step 4: Resume item processing and crawling

Step 1: Pause item processing and crawling

You must pause item processing and crawling before you reconfigure the indexer servers.

To pause item processing and crawling

  1. Pause crawling:

    • If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about pausing a crawl.

    • If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about suspending a crawl.

    • If you use another FAST Search specific connector, make sure that all crawling is paused or stopped.

  2. Pause the Web Analyzer components. These components analyze links between items and search clickthrough logs. The components must be stopped to avoid partial updates for indexed items during the procedure.

    1. On the administration server, open a FAST Search Server 2010 for SharePoint shell.

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

    2. Inspect the Web Analyzer schedule status:

      Copy Code
      waadmin ShowStatus
      Note:

      Refer to waadmin.exe reference if you use multiple views for the Web Analyzer.

      In the Views section of the command output, for the default view, inspect the Schedule status. If the Schedule status is set to paused, perform the following steps:

      1. Set the view to scheduled state:

        Copy Code
        waadmin enqueueview

      2. Wait until the default Web Analyzer view is running. Type the command waadmin ShowStatus and verify that the Schedule status is set to running.

    3. Stop all Web Analyzer processing:

      Copy Code
      waadmin AbortProcessing
spreladmin AbortProcessing

  3. Wait until all item processing is finished. On the administration server, type the following command:

    Copy Code
    psctrl status
    Note:

    All processor servers must report "idle" before you proceed to the next step.

Step 2: Reconfigure the backup indexer server to become the new primary indexer server

Follow these steps to ensure that the new primary indexer server contains the same set of indexed items as the failed primary indexer server.

To reconfigure the backup indexer server to become the new primary indexer server

  1. Ensure that the failed primary indexer server is shut down or disconnected.

  2. Modify the deployment configuration (deployment.xml).

    1. On the administration server, open the deployment configuration file: <FASTSearchFolder>\etc\config_data\deployment\deployment.xml.

      Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

      Locate the host element that specifies the failed primary indexer server. The column attribute of the searchengine element indicates the index column for this indexer.

      The following example shows an extract of a deployment.xml file with the host definition for the primary and backup indexer for index column 0. The first host element (fs4sp2.contoso.com) specifies the primary indexer server. This is the server that is not working and must be replaced. The second host element (fs4sp5.contoso.com) specifies the backup indexer server. The searchcluster element specifies which server that is the primary indexer and which server that is the backup indexer, by using the row element. Typically, row 0 corresponds to the primary indexer server.

      Copy Code
      <host name="fs4sp2.contoso.com">
    <content-distributor />
    <searchengine row="0" column="0" />
    <document-processor processes="12" />
  </host>
  ...
  <host name="fs4sp5.contoso.com">
    <query />
    <searchengine row="1" column="0" />
  </host>
  ...
  <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="secondary" search="true" />
  </searchcluster>

    2. Change the host element for the two indexer servers. The current backup indexer server will now take the role as primary indexer server. The following example shows the new configuration, where the new primary indexer server is the host fs4sp5.contoso.com:

      Copy Code
      <host name="fs4sp5.contoso.com">
    <content-distributor />
    <searchengine row="0" column="0" />
    <document-processor processes="12" />
  </host>
  ...
  <host name="fs4sp2.contoso.com">
    <query />
    <searchengine row="1" column="0" />
  </host>
  ...
  <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="secondary" search="true" />
  </searchcluster>
      Important:

      In the deployment example, we have switched the host names for the two host elements, and that way changed the content-distributor and query roles for the two servers. This is recommended in order to keep the same distribution of server roles after the switch of primary indexer server. This means that you must update the connection information for the Query SSA and Content SSA, which is covered later in this procedure.

      It is also possible to switch the row numbers for the two servers, and keep the other server roles assigned to the same servers. This ensures that there will be no change to the SSA connection information, but the performance characteristics of the servers may change.

  3. Reconfigure the deployment to switch the primary and backup indexer roles for this index column:

    1. Apply the new configuration to the administration server. Perform the following steps on the administration server:

      1. Stop the FAST Search for SharePoint service.

      2. Stop the FAST Search for SharePoint Monitoring service.

      3. To recreate configuration files, in a Microsoft FAST Search Server 2010 for SharePoint shell, type the following command:

        Copy Code
        Set-FASTSearchConfiguration

      4. Start the FAST Search for SharePoint service.

        Note:

        This will also start the FAST Search for SharePoint Monitoring service.

      5. To verify that all processes are running, type the following command:

        Copy Code
        nctrl status

        When all processes are running OK, proceed to the next step.

    2. Apply the new configuration to the other servers in the FAST Search Server 2010 for SharePoint farm, except the failed server. Perform the following steps on one server at a time:

      1. If the server has the query matching or indexer role (searchengine element in deployment.xml), delete the following file:

        <FASTSearchFolder>\etc\rtsplatformcache.xml

      2. Stop the FAST Search for SharePoint service.

      3. Stop the FAST Search for SharePoint Monitoring service.

      4. To recreate configuration files, in a Microsoft FAST Search Server 2010 for SharePoint shell, type the following command:

        Copy Code
        Set-FASTSearchConfiguration

      5. Start the FAST Search for SharePoint service.

      6. To verify that all processes are running, type the following command:

        Copy Code
        nctrl status

  4. If you have moved content-distributor or query roles to new servers in deployment.xml, you must also perform the steps in Make the necessary changes on Microsoft SharePoint Server.

    Important:

    For the deployment example earlier in this article, you must perform this step. The content-distributor and query roles are changed for the two servers. This is done to keep the same distribution of components across the servers in the farm.

  5. Verify that the new primary indexer server is properly deployed. At the administration server, type the following command:

    Copy Code
    indexerinfo --row=<rowNumber> --column=<columnNumber> status

    <rowNumber> is the row number for the primary indexer row, typically 0.

    <columnNumber> is the index column number.

    Verify that the following status information is correct:

    • indexer element: status="running ok"

    • indexer/column_role element: state="master" backups="0"

      state="master" indicates that this is the new primary indexer server for this index column.

      backups="0" indicates that there is not yet any working backup indexer for this index column.

  6. Determine if you want to replace the failed server at this stage or later.

    • If you want to configure a new backup indexer server at this stage, proceed with Step 3: Install and configure a new backup indexer server.

    • If you do not want to configure a new backup indexer server at this stage, proceed with Step 4: Resume item processing and crawling. You must then follow the steps in Replace a failed backup indexer server at a later stage.

      Important:

      Until you have configured a new backup indexer server, your deployment will not be fault-tolerant for all components. In the deployment example earlier in this article, index column 0 will not have a backup indexer. The query processing role (query role in deployment.xml) is now associated with the backup indexer server that is not working. Queries will be served correctly as long as you have at least one other server in your farm with the query processing role.

Step 3: Install and configure a new backup indexer server

Follow these steps to create a new backup indexer server. You have already configured the backup indexer server to become the new primary indexer server.

To install and configure a new backup indexer server

  1. Verify that the host name for the new server is correct in the deployment configuration file (deployment.xml).

  2. Stop the new primary indexer server you configured in Step 2: Reconfigure the backup indexer server to become the new primary indexer server. On the primary indexer server, type the following command:

    Copy Code
    nctrl stop indexer

  3. Install the new server and reconfigure the deployment to add a backup indexer:

    Important:

    When you perform this step, do not restart the new server after you have installed and configured it. You restart the server later in this procedure.

    Follow the steps in Reconfigure the farm deployment including adding new servers, but do not restart the new server.

  4. If you have changed the host name of the new server, update the connection information in the SharePoint Server 2010 farm. Perform the steps in Make the necessary changes on Microsoft SharePoint Server.

  5. Copy all FIXML files from the primary indexer server to the new backup indexer server.

    The indexer uses the FIXML files as input to the indexing process. The FIXML files are stored in the following directory on the indexer servers:

    <FASTSearchFolder>\data\data_fixml\

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    1. On the new backup indexer server, create two folders for the index data that you will copy. At the Windows command prompt, type the following commands:

      Copy Code
      mkdir <FASTSearchFolder>\data
mkdir <FASTSearchFolder>\data\data_fixml
mkdir <FASTSearchFolder>\data\ftStorage

    2. On the new backup indexer server, at the Windows command prompt, copy the data_fixml folder from the source server:

      Copy Code
      robocopy /E /MT:100 /NFL /COPYALL /LOG:\incoming\robocopy_fixml.log <source_path>\data\data_fixml <FASTSearchFolder>\data\data_fixml
copy <source_path>\data\ftStorage\processed_checkpoint.txt <FASTSearchFolder>\data\ftStorage

      Where <source_path> is the network path of the FAST Search Server 2010 for SharePoint installation folder on the primary indexer server.

      Note:

      The /LOG: folder (\incoming\) does not exist by default and you must have write access to the folder to run the command.

  6. Restart the new backup indexer server.

  7. Start the primary indexer server. On the primary indexer server in the affected index column, type the following command:

    Copy Code
    nctrl start indexer

  8. Verify that the new backup indexer server is registered properly with the primary indexer server. At the administration server, type the following command:

    Copy Code
    indexerinfo --row=<rowNumber> --column=<columnNumber> status

    <rowNumber> is the row number for the primary indexer row, typically 0.

    <columnNumber> is the index column number.

    Verify that the following status information is correct:

    • indexer element: status="running ok"

    • indexer/column_role element: state="master" backups="1"

      backups="1" indicates that the primary indexer server has identified the new backup indexer server.

Step 4: Resume item processing and crawling

Follow these steps to resume item processing and start a full crawl.

To resume item processing and crawling

  1. Resume processing for the Web Analyzer components. On the administration server, type the following commands:

    Copy Code
    waadmin EnqueueView
spreladmin Enqueue

  2. Start a full crawl:

    • If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about starting a crawl.

    • If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about starting a crawl.

    • If you use another FAST Search specific connector, make sure that a full new crawl is initiated.

    Note:

    We recommend to start a new full crawl, as there is a risk that items may be lost at the time of the primary indexer server failure. This will include items that are received correctly by the old primary indexer server just before the failure, but that have not yet been copied to the backup indexer server.

See Also

Reference

About FiXML files

Concepts

Manage high availability of the content index (FAST Search Server 2010 for SharePoint)
Verify the status of the backup indexer (FAST Search Server 2010 for SharePoint)
Add a backup indexer row (FAST Search Server 2010 for SharePoint)
Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)


Replace a failed backup indexer server (FAST Search Server 2010 for SharePoint)
See Also

Published: July 16, 2012

If the farm deployment includes a backup indexer row, you must manually change the deployment configuration if a non-recoverable server error occurs on the backup indexer server. You must also synchronize the indexer servers after you have replaced the failed server.

Follow these steps to replace a backup indexer server if a non-recoverable hardware failure has occurred:

  • Step 1: Pause item processing and crawling

  • Step 2: Install and configure a new backup indexer server

  • Step 3: Resume item processing and crawling

Step 1: Pause item processing and crawling

You must pause item processing and crawling before you reconfigure the indexer servers.

To pause item processing and crawling

  1. Pause crawling:

    • If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about pausing a crawl.

    • If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about suspending a crawl.

    • If you use another FAST Search specific connector, make sure that all crawling is paused or stopped.

  2. Pause the Web Analyzer components. These components analyze links between items and search clickthrough logs. The components must be stopped to avoid partial updates for indexed items during the procedure.

    1. On the administration server, open a FAST Search Server 2010 for SharePoint shell.

      1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the server where FAST Search Server 2010 for SharePoint is installed.

      2. On the Start menu, click All Programs.

      3. Click Microsoft FAST Search Server 2010 for SharePoint.

      4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

    2. Inspect the Web Analyzer schedule status:

      Copy Code
      waadmin ShowStatus
      Note:

      Refer to waadmin.exe reference if you use multiple views for the Web Analyzer.

      In the Views section of the command output, for the default view, inspect the Schedule status. If the Schedule status is set to paused, perform the following steps:

      1. Set the view to scheduled state:

        Copy Code
        waadmin enqueueview

      2. Wait until the default Web Analyzer view is running. Type the command waadmin ShowStatus and verify that the Schedule status is set to running.

    3. Stop all Web Analyzer processing:

      Copy Code
      waadmin AbortProcessing
spreladmin AbortProcessing

  3. Wait until all item processing is finished. On the administration server, type the following command:

    Copy Code
    psctrl status
    Note:

    All processor servers must report "idle" before you proceed to the next step.

Step 2: Install and configure a new backup indexer server

Follow these steps to create a new backup indexer server to replace the failed server.

To install and configure a new backup indexer server

  1. Verify that all indexing is finished. On the administration server, type the following command:

    Copy Code
    indexerinfo status -a

    Verify that all index partitions have partition status="idle". Repeat this step until all index partitions are idle.

  2. On the administration server, open the deployment configuration file: <FASTSearchFolder>\etc\config_data\deployment\deployment.xml.

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    Locate the host element that specifies the failed indexer. The column attribute of the searchengine element indicates the index column for this indexer.

    The following example shows an extract of a deployment.xml file that has the host definition for the primary and backup indexer for index column 0. The first host element specifies the primary indexer server. The second host element specifies the failed backup indexer server. The searchcluster element specifies which server that is the primary indexer and which server that is the backup indexer, by using the row element. Typically, row 0 corresponds to the primary indexer server.

    Copy Code
    <host name="fs4sp2.contoso.com">
    <content-distributor />
    <searchengine row="0" column="0" />
    <document-processor processes="12" />
  </host>
  ...
  <host name="fs4sp5.contoso.com">
    <query />
    <searchengine row="1" column="0" />
  </host>
  ...
  <searchcluster>
      <row id="0" index="primary" search="true" />
      <row id="1" index="secondary" search="true" />
  </searchcluster>

  3. Stop the primary indexer server. On the primary indexer server in the index column with the failed backup indexer server, type the following command:

    Copy Code
    nctrl stop indexer

  4. Install the new server and reconfigure the deployment to change the indexer server configuration:

    1. Change the host element that specifies the failed backup indexer server for this index column. Change the name attribute of the host element to specify the host name of the new server that will replace the failed backup indexer server.

      Note:

      If you use the same host name for the new server, you do not have to modify the content of the file. However, you must modify the file for the Windows PowerShell cmdlet Set-FASTSearchConfiguration to redeploy the configuration. You can modify the file by opening it in Notepad, add a space and save the file.

    2. Install the new server and reconfigure the deployment to add a backup indexer:

      Important:

      When you perform this step, do not restart the new server after you have installed and configured it. You restart the server later in this procedure.

      Follow the steps in Reconfigure the farm deployment including adding new servers, but do not restart the new server.

    3. If you have changed the host name of the new server, you must update the connection information in the SharePoint Server 2010 farm. Perform the steps in Make the necessary changes on Microsoft SharePoint Server.

  5. Copy all FIXML files from the primary indexer server to the new backup indexer server.

    The indexer uses the FIXML files as input to the indexing process. The FIXML files are stored in the following directory on the indexer servers:

    <FASTSearchFolder>\data\data_fixml\

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    1. On the new backup indexer server, create two folders for the index data that you will copy. At the Windows command prompt, type the following commands:

      Copy Code
      mkdir <FASTSearchFolder>\data
mkdir <FASTSearchFolder>\data\data_fixml
mkdir <FASTSearchFolder>\data\ftStorage

    2. On the new backup indexer server, at the Windows command prompt, copy the data_fixml folder from the source server:

      Copy Code
      robocopy /E /MT:100 /NFL /COPYALL /LOG:\incoming\robocopy_fixml.log <source_path>\data\data_fixml <FASTSearchFolder>\data\data_fixml
copy <source_path>\data\ftStorage\processed_checkpoint.txt <FASTSearchFolder>\data\ftStorage

      Where <source_path> is the network path of the FAST Search Server 2010 for SharePoint installation folder on the primary indexer server.

      Note:

      The /LOG: folder (\incoming\) does not exist by default and you must have write access to the folder to run the command.

  6. Restart the new backup indexer server.

  7. Start the primary indexer server. On the primary indexer server in the affected index column, type the following command:

    Copy Code
    nctrl start indexer

  8. Verify that the new backup indexer server is registered properly with the primary indexer server. At the administration server, type the following command:

    Copy Code
    indexerinfo --row=<rowNumber> --column=<columnNumber> status

    <rowNumber> is the row number for the primary indexer row, typically 0.

    <columnNumber> is the index column number.

    Verify that the following status information is correct:

    • indexer element: status="running ok"

    • indexer/column_role element: state="master" backups="1"

      backups="1" indicates that the primary indexer server has identified the new backup indexer server.

Step 3: Resume item processing and crawling

Follow these steps to resume item processing and start a full crawl.

To resume item processing and crawling

  1. Resume processing for the Web Analyzer components. On the administration server, type the following commands:

    Copy Code
    waadmin EnqueueView
spreladmin Enqueue

  2. Start a full crawl:

    • If you use the FAST Search Content SSA to crawl content, see Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint) for information about starting a crawl.

    • If you use the FAST Search Web crawler to crawl content, see Start, suspend, resume, or stop a crawl process (FAST Search Web crawler) for information about starting a crawl.

    • If you use another FAST Search specific connector, make sure that a full new crawl is initiated.

    Note:

    We recommend starting a new full crawl, as there is a risk that items may be lost at the time of the primary indexer server failure. This will include items that are received correctly by the old primary indexer server just before the failure, but that have not yet been copied to the backup indexer server.

See Also

Reference

About FiXML files

Concepts

Manage high availability of the content index (FAST Search Server 2010 for SharePoint)
Verify the status of the backup indexer (FAST Search Server 2010 for SharePoint)
Add a backup indexer row (FAST Search Server 2010 for SharePoint)
Replace a failed primary indexer server (FAST Search Server 2010 for SharePoint)


Multiple server deployment of the Content SSA (FAST Search Server 2010 for SharePoint)
See Also

When you have created and configured your FAST Search Content SSA and enabled SSL communication, it is possible to distribute the crawling task over additional servers within your Microsoft SharePoint Server 2010 farm.

Follow the procedures outlined in this section to scale out crawling over multiple servers.

Note:

You must only deploy one Content SSA associated with your Microsoft FAST Search Server 2010 for SharePoint farm. You can scale out the FAST Search Content SSA across multiple servers for performance and failover.

In this section:

  • Add or remove a crawl component (FAST Search Server 2010 for SharePoint)

  • Add or remove a crawl database (FAST Search Server 2010 for SharePoint)

  • Add or remove a host distribution rule (FAST Search Server 2010 for SharePoint)

See Also

Concepts

Plan search topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Manage search topology (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Multiple server deployment of the Query SSA (FAST Search Server 2010 for SharePoint)


Add or remove a crawl component (FAST Search Server 2010 for SharePoint)
See Also

Crawl components process crawls of content sources, add information about the location and crawl schedule of content sources to their associated crawl databases, and pass the crawled item to the FAST Search Server 2010 for SharePoint farm. Crawl components are associated with a single FAST Search Content Search Service Application. You can distribute the crawl load by adding crawl components to different farm servers.

You assign a farm server to participate in crawling by creating a crawl component on that server. If you want to balance the load of servicing crawls across multiple farm servers, add crawl components to the farm and associate them with the servers you want to crawl content.

To add a crawl component to the FAST Search Content Search Service Application

  1. On the Home page of the SharePoint Central Administration, in the Application Management section, click Manage service applications.

  2. On the Manage Service Applications page, click the name of the FAST Search Content SSA to which you want to add a crawl component.

  3. On the Search Administration page, in the Search Application Topology section, click the Modify button.

  4. On the Manage Search Topology page, click New, and then click Crawl Component.

  5. In the Add Crawl Component dialog box, in the Server list, click the farm server to which you want to add the crawl component.

  6. In the Associated Crawl Database list, click the crawl database you want to associate with the new crawl component.

  7. The Temporary Location of Index field is not applicable to FAST Search Server 2010 for SharePoint, and can be left unchanged.

  8. Click OK to add the new crawl component to the job queue.

  9. On the Manage Search Topology page, click the Apply Topology Changes button to start the SharePoint timer job that will add the new crawl component to the farm on the specified server.

    Note:

    You can make other changes to the Search topology and then apply them all at the same time by clicking the Apply Topology Changes button.

Configure SSL enabled communication on each of the servers that hosts a crawl component

Secure Sockets Layer (SSL) communication is required for feeding from SharePoint Server 2010 (through the FAST Search Content SSA) to FAST Search Server 2010 for SharePoint. To enable secure communications, you must configure each server that hosts a crawl component to use an SSL certificate for FAST Search Server 2010 for SharePoint. You can use the default FAST Search Server 2010 for SharePoint self-signed certificate that is generated during post-setup configuration. However, in a production environment we highly recommend that you use a certificate that is issued by a certification authority (CA). See Manage certificates (FAST Search Server 2010 for SharePoint) for more information.

Use the same certificate type (self-signed or signed by a CA) on the additional host(s) for crawl components as you used to enable SSL communication for the original FAST Search Content SSA.

To configure SSL communication using the self-signed certificate created by the FAST Search Server 2010 for SharePoint post-setup configuration procedure

  1. Import the FASTSearchCert.pfx certificate in the certificate store under Certificates(Local Computer)\Personal.

  2. Grant the Search Service Application account (the account under which the SharePoint Server Search 14 (OSearch14) service runs) access to the private key of the imported certificate.

To configure SSL communication using a certificate signed by a certification authority

  1. Make sure that the server has a certificate installed that is issued and signed by the same Certificate Authority as the certificate on the host server of the FAST Search Content SSA. The certificate must be installed under Certificates(Local Computer)\Personal in the certificate store. The root CA signed certificate must be installed under Certificates(Local Computer)\Trusted Root Certification Authorities.

  2. Grant the Search Service Application account (the account under which the SharePoint Server Search 14 service (OSearch14) runs) access to the private key of the imported certificate.

To remove a crawl component from the FAST Search Content Search Service Application

  1. On the Home page of the SharePoint Central Administration, in the Application Management section, click Manage service applications.

  2. On the Manage Service Applications page, click the name of the FAST Search Content SSA from which you want to remove a crawl component.

  3. On the Search Administration page, in the Search Application Topology section, click the Modify button.

  4. On the Manage Search Topology page, click the crawl component you want to remove, and then click Delete.

  5. In the message box that appears, click OK to add the removal of the crawl component to the job queue.

  6. On the Manage Search Topology page, click the Apply Topology Changes button to start the SharePoint timer job that will delete the crawl component from the farm.

    Note:

    You can make other changes to the Search topology and then apply them all at the same time by clicking the Apply Topology Changes button.

See Also

Concepts

Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint)


Add or remove a crawl database (FAST Search Server 2010 for SharePoint)
See Also

Crawl databases contain data related to the location of content sources, crawl schedules, and other information specific to crawl operations for a specific FAST Search Content Search Service Application. You can distribute the database load by adding crawl databases to different computers that are running SQL Server. Crawl databases are associated with crawl components, and can be dedicated to specific hosts by creating host distribution rules. For more information about crawl components, see Add or remove a crawl component (FAST Search Server 2010 for SharePoint). For more information about host distribution rules, see Add or remove a host distribution rule (FAST Search Server 2010 for SharePoint).

To add a crawl database to the FAST Search Content Search Service Application

  1. In Central Administration, in the Application Management section, click Manage service applications.

  2. On the Service Applications page, click the name of the FAST Search Content SSA to which you want to add a crawl database.

  3. On the Search Administration page, in the Search Application Topology section, click the Modify button.

  4. On the Manage Search Topology page, click New, and then click Crawl Database.

  5. In the Add Crawl Database dialog box, in the Add Crawl Database section, specify a database server to which you want to add the crawl database, the database name, and database authentication information. By default, the Database Server field contains the host name and instance that is used to store the farm's configuration database, and the Database Name field is pre-populated with a suggested name.

  6. In the Failover Database Server field, you can optionally specify a failover database server that is used in conjunction with SQL Server database mirroring.

    Important:

    Do not enter a server name into this field unless SQL Server database mirroring is currently configured and operational.

  7. In the Dedicated Database section, you can select the Dedicate this crawl store to hosts as specified in Host Distribution Rules check box if appropriate.

    Note:

    It is not possible to set a crawl store (that is, a crawl database) as dedicated after it has been created.

  8. Click OK to add the new crawl database to the job queue.

  9. On the Manage Search Topology page, click the Apply Topology Changes button to start the SharePoint timer job that will add the new crawl database to the specified SQL Server computer and make it available for association with crawl components.

    You can make other changes to the Search topology and then apply them all at the same time by clicking the Apply Topology Changes button.

To remove a crawl database from a FAST Search Content Search Service Application

Important:

Before you can delete a crawl database, you must remove associations with any crawl components, either by removing the crawl components or by assigning them to a different crawl database. If any crawl components are currently associated with the crawl database, the Delete option will not appear when you click the crawl database.

  1. In Central Administration, in the Application Management section, click Manage service applications.

  2. On the Service Applications page, click the name of the FAST Search Content Search Service Application from which you want to remove a crawl database.

  3. On the Search Administration page, in the Search Application Topology section, click the Modify button.

  4. On the Manage Search Topology page, click the crawl database you want to remove, and then click Delete.

    Note:

    If any crawl components are currently associated with the crawl database, the Delete option will not appear when you click the crawl database.

  5. In the message box, click OK to add the removal of the crawl database to the job queue.

  6. On the Manage Search Topology page, click the Apply Topology Changes button to start the SharePoint timer job that will delete the crawl database from the specified SQL Server computer and remove it from the farm.

    Note:

    You can make other changes to the Search topology and then apply them all at the same time by clicking the Apply Topology Changes button.

See Also

Concepts

Add or remove a crawl component (FAST Search Server 2010 for SharePoint)
Add or remove a host distribution rule (FAST Search Server 2010 for SharePoint)


Add or remove a host distribution rule (FAST Search Server 2010 for SharePoint)
See Also

Host distribution rules are used to associate a host with a specific crawl database. By default, hosts are load balanced across crawl databases based on space availability. However, you may want to assign a host to a specific crawl database for availability and performance optimization. For more information about crawl databases, see Add or remove a crawl database (FAST Search Server 2010 for SharePoint).

To add a host distribution rule

  1. In Central Administration, in the Application Management section, click Manage service applications.

  2. On the Service Applications page, click the name of the FAST Search Content SSA to which you want to add a host distribution rule.

  3. On the Search Administration page, under Crawling, click Host Distribution Rules.

  4. On the Host Distribution Rules page, click Add Distribution Rule.

    Note:

    If there is only one crawl database in the FAST Search Content SSA, you cannot add a host distribution rule, and this option will not appear.

  5. On the Add Host Rule page, in the Hostname field, enter the name of the host to which this rule applies.

  6. In the Distribution Configuration list, select the crawl database that you want to participate in the crawl of the host you specified in the Hostname field.

  7. Click OK to add the new mirror query component to the job queue.

  8. On the Host Distribution Rules page, click Apply Changes.

  9. Click OK to apply the rule.

To remove a host distribution rule

  1. In Central Administration, in the Application Management section, click Manage service applications.

  2. On the Service Applications page, click the name of the FAST Search Content SSA from which you want to remove a host distribution rule.

  3. On the Search Administration page, under Crawling, click Host Distribution Rules.

  4. On the Host Distribution Rules page, point to the host distribution rule that you want to delete, click the arrow that appears, and then click Delete.

  5. In the message box that appears, click OK to add the removal of the host distribution rule to the job queue.

  6. On the Host Distribution Rules page, click Apply Changes.

  7. Click OK to delete the rule.

See Also

Concepts

Add or remove a crawl database (FAST Search Server 2010 for SharePoint)


Multiple server deployment of the Query SSA (FAST Search Server 2010 for SharePoint)
See Also

The FAST Search Query Search Service Application (SSA) provides the query-side integration between the FAST Search Server 2010 for SharePoint farm and the parent Microsoft SharePoint Server 2010 farm. It also includes a crawl component and query component that handles the People Search index.

It is possible to scale out the Query SSA to run on multiple servers. In most cases this is only needed to provide failover for the search solution, as most of the query handling (and associated load) takes place within the FAST Search Server 2010 for SharePoint farm.

Note:

You must only deploy one Query SSA associated with your Microsoft FAST Search Server 2010 for SharePoint farm. You can scale out the FAST Search Query SSA across multiple servers for performance and failover.

To add a query component

  1. Verify that the user account that is performing this procedure is a member of the Farm Administrators group.

  2. On the Home page of the SharePoint Server 2010 Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, click the name of the Query Search Service Application (SSA) to which you want to add a query component.

  4. On the Search Administration page, in the Search Application Topology section, click the Modify button.

  5. On the Manage Search Topology page, click a query component in the index partition that you want to modify, and then click Add Mirror.

  6. In the Add mirror query component dialog box, in the Server field, select the farm server to which you want to add the query component.

  7. Leave the content of the Location of Index field unchanged.

  8. In the Failover-only Query Component section, you can select the Set this query component as failover-only check box if you want the query component to receive queries only in the event of a failure of the primary query component in the same index partition.

  9. Click OK to add the new mirror query component to the job queue.

  10. On the Manage Search Topology page, click the Apply Topology Changes button to start the SharePoint timer job that will add the new mirror query component to the selected index partition.

See Also

Concepts

Plan search topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
Manage search topology (FAST Search Server 2010 for SharePoint)
Manage FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Multiple server deployment of the Query SSA (FAST Search Server 2010 for SharePoint)


Backup and restore (FAST Search Server 2010 for SharePoint)
See Also

Before you put a FAST Search Server 2010 for SharePoint farm into production, you should plan your backup strategy. Make sure that you back up the farm on a regular basis, and that you also make a backup before you introduce major changes to the system. You should combine the backup of FAST Search Server 2010 for SharePoint with a backup of Microsoft SharePoint Server. You can also combine the FAST Search Server 2010 for SharePoint backup with third-party backup systems.

There are two kinds of backups for FAST Search Server 2010 for SharePoint: configuration backup and full backup.

In this article:

  • About configuration backup

  • About full backup

  • What parts of the system are included in the backup

About configuration backup

A configuration backup backs up configuration data, including the schema data, the relevance configuration, and other configuration files. The searchable index and the indexed documents themselves are not included in the backup. The configuration backup is faster, uses fewer system resources, and requires less storage space than a full backup.

If you unsuccessfully modify the FAST Search Server 2010 for SharePoint system, you can use the configuration backup to restore a safe set of system configuration data. You can also use a configuration backup to migrate a configuration to a different set of servers.

When you run a configuration backup, you do not have to shut down any services. The backup is non-intrusive and operates without interrupting services.

You can restore a configuration backup on a running system. After the configuration has been restored, you must shut down and restart the system to restart the services.

About full backup

A full backup backs up the complete FAST Search Server 2010 for SharePoint farm. You can decide whether you want to include or exclude the searchable index in the full backup.

Depending on the size of the system and the data in the index, a full backup can require a large amount of hard disk space.

Before you run a full backup, you must stop the feeding and indexing parts of the system. You also have to shut down the system to restore a full backup.

What parts of the system are included in the backup

The following table shows what components that are covered by the different backup and restore options of FAST Search Server 2010 for SharePoint.

Component | Configuration backup | Full backup | Full backup + SharePoint Server backup
Files generated by the installer | No | Yes | Yes
Configserver files | Yes | Yes | Yes
FSA admin configuration | Yes | Yes | Yes
SPRel configuration | Yes | Yes | Yes
Web Analyzer configuration | Yes | Yes | Yes
Web Analyzer lookup/link database | No | Yes | Yes
FAST Crawler configuration | Yes | Yes | Yes
Binaries and DLLs | No | No | No
Searchable index | No | Yes* | Yes*
FIXML files | No | Yes | Yes
PeopleSearch | No | No | Yes**
Content SSA state | No | No | Yes**

* Can be disabled with the -excludeindex parameter in the backup.ps1 script.

** The states of People Search and Content SSA are not covered by the FAST Search Server 2010 for SharePoint backup, but by the SharePoint Server backup. See Backup and recovery (SharePoint Server 2010).

See Also

Concepts

Plan the backup and restore strategy (FAST Search Server 2010 for SharePoint)
Backup and restore prerequisites (FAST Search Server 2010 for SharePoint)

Other Resources

Backup and recovery overview (SharePoint Server 2010)


Plan the backup and restore strategy (FAST Search Server 2010 for SharePoint)
See Also

Use this article to select the backup type that best suits your requirements, and to plan the backup and restore strategy for FAST Search Server 2010 for SharePoint.

Important:

Always test and verify your backup and restore procedures before you implement a new backup and restore procedure in a production system, and before you put a system into production.

For an overview of the different FAST Search Server 2010 for SharePoint backup and restore options, see Backup and restore (FAST Search Server 2010 for SharePoint).

You should combine the FAST Search Server 2010 for SharePoint backup with a Microsoft SharePoint Server backup. For information about SharePoint Server 2010 backup and recovery, see Backup and recovery (SharePoint Server 2010).

In this section:

  • Plan where you want to store the backup data

  • General considerations when planning the backup and restore strategy

  • Considerations for configuration backup and restore

  • Considerations for full backup and restore

  • Downtime considerations when planning the restore strategy

  • Run scripts with a saved password

Plan where you want to store the backup data

All backup scenarios require a backup store. The backup store is a location where the backup data is stored, and it should be an external data resource separate from the FAST Search Server 2010 for SharePoint farm. The backup store must be available as a UNC path, such as \\contoso\FASTSearchBackup, so that all servers can reach it. The user who runs the backup and restore scripts must have read and write access to this path.

Note:

Access control lists (ACLs) are not preserved on the files that are stored in the backup stores. To prevent unauthorized users from reading indexed data, protect the backup store with an appropriate ACL.

For a full backup you can use multiple backup stores. The FAST Search Server 2010 for SharePoint backup script will try to balance how much data is stored in the different backup stores by assigning an even number of servers to each backup store.

To save bandwidth and time, you can re-use the backup store when you run a new backup. The full backup and restore runs the Robocopy command in incremental mode to transfer data. When you overwrite an existing backup, Robocopy only replaces the files that have been modified since the previous backup.

Important:

Always make sure that you have a safe copy of the backup store before you run another backup to the same backup store. If the backup procedure fails while you run the new backup, the backup store might be corrupted.

When there are multiple servers in the FAST Search Server 2010 for SharePoint deployment, and multiple backup stores are specified in the -backuppath parameter, the system searches the content of all the backup stores to determine whether there is a backup for all servers in the backup stores. If all servers have an existing backup, and you have specified the -force parameter, the scripts will update the existing backups.

General considerations when planning the backup and restore strategy

You cannot use the backup and restore scripts to back up or restore individual servers in a multiple server deployment.

The certificates that authenticate FAST Search Server 2010 for SharePoint to Microsoft SharePoint Server are not included by the FAST Search Server 2010 for SharePoint backup and restore scripts. If the certificates have changed, for example if you restore on a reinstalled system, you must restore the certificates separately. See Restore certificates.

The administration database backup file is generated locally on the SQL Server, and is not stored with the backup of FAST Search Server 2010 for SharePoint. To maintain a complete backup file set, you should copy the administration database backup to the backup store after you have run the backup script.

The FAST Search Server 2010 for SharePoint backup is sensitive to version differences between the backed up SQL Server and the restored SQL Server. The versions must be identical. If the SQL version changes between backup and restore, export the FAST Search Server 2010 for SharePoint administration database to a canonical format and import it.

Note:

Backup procedures may interfere with the spell tuning process. To avoid this problem, schedule regular backups and spell tuning runs so that they do not interfere with one another. By default, the spell tuner is scheduled to run at 2:05 AM.

Considerations for configuration backup and restore

The configuration backup is not deployment-aware, and you can restore a configuration backup to a different set of servers from the servers where you made the backup.

Only configuration files that are not generated by the FAST Search Server 2010 for SharePoint installer are backed up. If you migrate the configuration to a different system, the content of the directory <FASTSearchFolder>\etc\config_data\deployment on the restored system is not modified, and the server-specific configuration from the installation is not overwritten.

The user inclusion and exclusion lists for company, location and person name property extraction, and the dictionaries of the custom properties extractors, are stored in the resource store and are not included in a configuration backup.

Considerations for full backup and restore

A full backup can only be restored to exactly the same set of servers as it was backed up from, and the farm topology cannot be changed between the backup and the restore operation. The number of servers must be the same, the server names must be the same, and the roles that the installer assigns to the servers must be the same.

For each server, the backup script stores a copy of the file system, except for the executable binaries and DLLs.

A full backup may require a large amount of storage space. The backup data volume will be equal to the sum of all installation folders on all servers. To help reduce disk space requirements, you can exclude the searchable index from the backup. If you exclude the searchable index from the backup, you can reconstruct it after a restore by using the backed up FIXML. This method saves time and space during backup because less data is transferred. The disadvantage is that it takes longer to get a restored system up and running, because the indexer must rebuild the searchable index from the FIXML files.

During a full backup, feeding and indexing will stop. The system will serve queries but no new items will become available in the index during the backup process. During a full restore, the complete FAST Search Server 2010 for SharePoint farm must be shut down.

Downtime considerations when planning the restore strategy

The downtime required to complete a system restore depends on the complexity of the deployment, how much data you have in the system, and whether the searchable index was included in the backup, or not. The targeted Recovery Time Objective (RTO), which is described in Plan for backup and recovery (SharePoint Server 2010), should influence your backup and restore strategy for FAST Search Server 2010 for SharePoint.

Use the following table to find the restore procedure that best matches your downtime requirements.

Acceptable downtime | Backup and restore strategy
No downtime accepted | Use a fully or partly redundant setup of FAST Search Server 2010 for SharePoint to always have service available while you restore the farm. For redundancy recommendations, see Search redundancy and availability.
One to several days of downtime accepted | Use a full backup or a third-party backup solution to restore both configuration and data to the existing farm.
Days to weeks of downtime accepted | Install FAST Search Server 2010 for SharePoint on the same set of servers, restore the configuration from a valid configuration backup, and refeed the content.

Run scripts with a saved password

The backup and restore scripts forward their credentials to the remote servers so that the scripts can authenticate themselves to remote resources. The scripts prompt for credentials to perform the authentication, and an administrator must be present to enter the credentials.

To run the scripts without the administrator having to enter passwords at a prompt, you can specify a saved password file. This password must be DPAPI (Windows Data Protection API) protected.

To generate a password file

  1. Open a Windows PowerShell prompt as an administrator and run the following command:

    Copy Code
    $mypw = Read-Host -AsSecureString

  2. Enter your password when you are prompted.

  3. Enter the command:

    Copy Code
    Set-Content passwordfile (ConvertFrom-SecureString $mypw)

You can now run the backup and restore scripts with the -passwordfile parameter.

Note:

The file that contains the password is encrypted by DPAPI, and you should protect it with a strong ACL.

Warning:

If the password of the user who runs the backup and restore scripts is changed, you must update and regenerate the password file.

See Also

Concepts

Backup and restore prerequisites (FAST Search Server 2010 for SharePoint)
Configuration backup and restore (FAST Search Server 2010 for SharePoint)
Full backup and restore (FAST Search Server 2010 for SharePoint)
About backup and restore scripts (FAST Search Server 2010 for SharePoint)

Other Resources

Plan for backup and recovery (SharePoint Server 2010)


Backup and restore prerequisites (FAST Search Server 2010 for SharePoint)
See Also

Make sure that you meet the following prerequisites before you run a FAST Search Server 2010 for SharePoint backup or restore script.

Note:

Always run the backup and restore scripts from the FAST Search Server 2010 for SharePoint administration server. Scripts that are started on the administration server, in turn starts backup or restore tasks on remote servers by using the Windows PowerShell remoting feature.

In this article:

  • Prerequisites on all FAST Search Server 2010 for SharePoint servers

  • Prerequisites on the FAST Search Server 2010 for SharePoint administration server

Prerequisites on all FAST Search Server 2010 for SharePoint servers

On all servers in the FAST Search Server 2010 for SharePoint farm, make sure that you meet the following prerequisites:

  • Ensure that the user who runs the backup and restore scripts, is a member of the local FASTSearchAdministrators group, and of the local administrators group.

  • Enable Windows PowerShell remoting. Open a Windows PowerShell prompt as an administrator and run this command:

    Copy Code
    Enable-PSRemoting -Force

  • Ensure that Windows PowerShell remoting is set up as a Credential Security Support Provider (CredSSP) server. Open a Windows PowerShell prompt as an administrator and run this command:

    Copy Code
    Enable-WSManCredSSP -Role Server -Force

  • Increase the maximum amount of memory that is allocated to a Windows PowerShell instance on a remote server. Open a Windows PowerShell prompt as an administrator and run this command:

    Copy Code
    Set-Item WSMan:\localhost\Shell\MaxMemoryPerShellMB 1000

Prerequisites on the FAST Search Server 2010 for SharePoint administration server

On the administration server in the FAST Search Server 2010 for SharePoint farm, make sure that you meet the following prerequisites:

  • Ensure that Windows PowerShell remoting is set up as a Credential Security Support Provider (CredSSP) client. Open a Windows PowerShell prompt as an administrator and run the following command:

    Copy Code
    Enable-WSManCredSSP -Role Client -DelegateComputer <server1,server2,…,server> -Force

    Where <server1,server2,…,server> lists the server names of all servers in the FAST Search Server 2010 for SharePoint farm.

    Important:

    The server names must be identical to the server names as they appear in the <FASTSearchFolder>\etc\config_data\deployment\deployment.xml file.

  • Ensure that the SQL Server Management Objects (SMO) .Net assemblies are installed. If SQL Server is installed on the FAST Search Server 2010 for SharePoint administration server, you already have the SMO assemblies. Otherwise, you can install the SMO assemblies by installing the Microsoft SQL Server System CLR Types and Microsoft SQL Server 2008 Management Objects packages that are available from http://go.microsoft.com/fwlink/p/?LinkId=188653.

  • Ensure that the user who runs the backup and restore scripts, has sysadmin permissions on the FAST Search Server 2010 for SharePoint administration database (SQL database).

See Also

Concepts

Backup and restore (FAST Search Server 2010 for SharePoint)
Plan the backup and restore strategy (FAST Search Server 2010 for SharePoint)
Configuration backup and restore (FAST Search Server 2010 for SharePoint)
Full backup and restore (FAST Search Server 2010 for SharePoint)
About backup and restore scripts (FAST Search Server 2010 for SharePoint)


Configuration backup and restore (FAST Search Server 2010 for SharePoint)
See Also

The configuration backup makes a snapshot of the FAST Search Server 2010 for SharePoint configuration setup and the FAST Search Server 2010 for SharePoint administration database. You must restore the backup to an existing FAST Search Server 2010 for SharePoint deployment, either to the same deployment as the backup was made from, or to another deployment with other servers or with a different deployment.xml file.

Before you run a configuration backup or restore, see Backup and restore prerequisites (FAST Search Server 2010 for SharePoint).

In this article:

  • Make a configuration backup

  • Copy the backup of the administration database to the backup store

  • Restore a configuration backup to the same deployment

  • Restore a configuration backup to a different deployment

  • Troubleshooting configuration backup and restore

Make a configuration backup

  1. Open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator.

  2. At the command prompt, browse to <FASTSearchFolder>\bin.

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

  3. Run the following command:

    Copy Code
    .\backup.ps1 -action config -backuppath <UNC path of backup store>

The script will prompt you for a password to authenticate itself to remote resources. To run the script without having to enter a password, use the -passwordfile option to specify a file that contains a saved password. See Run scripts with a saved password for more information.

Before the backup starts, the script checks if the backup store already exists, or not. If you want to overwrite an existing backup set, use the -force option when you run the backup script.

Copy the backup of the administration database to the backup store

The administration database backup file is generated locally on the SQL Server, and is not stored in the FAST Search Server 2010 for SharePoint backup store. To maintain a complete backup file set, you should copy the database backup to the backup store after you run the backup script.

There can be several SQL Server backup files in the SQL Server folder. You can find the exact name of your SQL Server backup file in the dbbackup.txt file in the FAST Search Server 2010 for SharePoint backup store. You can also find more information about the SQL Server backup process in the SQLServerMetaData.txt file in the backup store. The lines that contain BackupDirectory and NetName display the path and host name for the SQL Server.

  1. Locate the administration database backup file in the SQL Server folder.

  2. Copy the file to the backup store.

Restore a configuration backup to the same deployment

To restore a configuration backup to the same system as it was backed up from, with the same server name and the same deployment file:

  1. Make sure that the FAST Search Server 2010 for SharePoint administration database backup file is present on the SQL Server.

  2. Open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator.

  3. At the command prompt, browse to <FASTSearchFolder>\bin.

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

  4. Run the following command:

    Copy Code
    .\restore.ps1 -action config -backuppath <UNC path of backup store>

    The script will prompt you for a password to authenticate itself to remote resources. To run the script without having to enter a password, use the -passwordfile option to specify a file that contains a saved password. See Run scripts with a saved password for more information.

  5. When the configuration restore script is complete, restart the FAST Search Server 2010 for SharePoint service on all servers:

    Copy Code
    Stop-Service FASTSearchService
Start-Service FASTSearchService

Restore a configuration backup to a different deployment

To restore the configuration backup to different servers or to a system with a different deployment file:

  1. Make sure that the FAST Search Server 2010 for SharePoint administration database backup file is present on the SQL Server.

  2. Open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator.

  3. At the command prompt, browse to <FASTSearchFolder>\bin.

    Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

  4. Run the following command:

    Copy Code
    .\restore.ps1 -action configmigrate -backuppath <UNC path of backup store>

    The script will prompt you for a password to authenticate itself to remote resources. To run the script without having to enter a password, use the -passwordfile option to specify a file that contains a saved password. See Run scripts with a saved password for more information.

  5. When the configuration restore script is complete, restart the FAST Search Server 2010 for SharePoint service on all servers:

    Copy Code
    Stop-Service FASTSearchService
Start-Service FASTSearchService

Troubleshooting configuration backup and restore

Issue: The content is not searchable after a configuration restore

Symptom: The content is not searchable after you have restored a configuration backup.

Cause: The backed up system and the restored system have different sets of managed properties.

The managed properties are part of the index schema, which specifies which content parts are indexed. If the managed properties have been changed after the configuration backup was taken, the restored configuration includes a set of managed properties that does not match the indexed content.

Resolution: Re-feed all content.

To prevent this problem, make sure that you make a configuration backup when you change the set of managed properties changes, and that you only restore the latest configuration backup.

Issue: The SPREL configuration restore fails

Symptom: A typical indication of this problem is this error message: "Can't change FDM config when FDM is running".

Cause: Relevancy processing is active.

In rare cases, restoring the SharePoint Relevancy (SPREL) configuration can fail because the relevancy processing is running at the same time as the restore process.

Resolution: Try running the restore script later.

See Also

Concepts

Backup and restore (FAST Search Server 2010 for SharePoint)
Full backup and restore (FAST Search Server 2010 for SharePoint)
About backup and restore scripts (FAST Search Server 2010 for SharePoint)

Other Resources

Back up Search (SharePoint Server 2010)
Restore Search (SharePoint Server 2010)


Full backup and restore (FAST Search Server 2010 for SharePoint)
See Also

A full backup includes the FAST Search Server 2010 for SharePoint installation folder and the FAST Search Server 2010 for SharePoint administration database, and you can choose to include or exclude the searchable index in the full backup.

A full restore sets the FAST Search Server 2010 for SharePoint farm back to the state it had at the time of backup. A full restore can also be used to recover and re-deploy the farm after a disaster.

You should combine the backup of FAST Search Server 2010 for SharePoint with a Microsoft SharePoint Server backup to ensure that both systems work as expected after a restore. The searchable index and the state of the Content Search Service Application (Content SSA) must be synchronized to avoid duplicates and search problems.

Before you start a full backup or restore procedure, see Plan the backup and restore strategy (FAST Search Server 2010 for SharePoint) and Backup and restore prerequisites (FAST Search Server 2010 for SharePoint).

In this article:

  • Make a full backup

  • Copy the backup of the administration database to the backup store

  • Restore a full backup

  • Restore certificates

Make a full backup

  1. On the main SharePoint server, run the following commands to pause the SharePoint Server Content SSA:

    Copy Code
    $ssa = Get-SPenterpriseSearchServiceApplication "<FAST Search Content SSA>"
$ssa.Pause()

  2. Back up the SharePoint Server farm. See Back up a farm (SharePoint Server 2010) for more information.

  3. On the FAST Search Server 2010 for SharePoint administration server, follow these steps (you can do this in parallel with step 2):

    1. Open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator.

    2. At the command prompt, browse to <FASTSearchFolder>\bin.

      Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    3. Suspend the services:

      Copy Code
      .\suspend.ps1
      Note:

      The suspend.ps1 script suspends the FAST Search Server 2010 for SharePoint feeding and indexing chain, but does not suspend the query processing and query matching components.

    4. Run the backup script:

      Copy Code
      .\backup.ps1 -action full -backuppath <UNC path of backup store(s)> (-excludeindex)

      The script will prompt you for a password to authenticate itself to remote resources. To run the script without having to enter a password, use the -passwordfile option to specify a file that contains a saved password. See Run scripts with a saved password for more information.

      Note:

      If you want to exclude the searchable index from the backup, use the -excludeindex option.

    5. When the script has completed, resume the services:

      Copy Code
      .\resume.ps1

  4. On the SharePoint server, resume the SSA by running the following command:

    Copy Code
    $ssa.Resume()

Copy the backup of the administration database to the backup store

The administration database backup file is generated locally on the SQL Server, and is not stored in the FAST Search Server 2010 for SharePoint backup store. To maintain a complete backup file set, you should copy the database backup to the backup store after you run the backup script.

There can be several SQL Server backup files in the SQL Server folder. You can find the exact name of your SQL Server backup file in the dbbackup.txt file in the FAST Search Server 2010 for SharePoint backup store. You can also find more information about the SQL Server backup process in the SQLServerMetaData.txt file in the backup store. The lines that contain BackupDirectory and NetName display the path and host name for the SQL Server.

  1. Locate the database backup file in the SQL Server folder.

  2. Copy the file to the backup store.

Restore a full backup

  1. On the main SharePoint server, to disable updates of managed properties in the Content SSA, run these commands:

    Copy Code
    $extractorJob = Get-SPTimerJob | where {$_.Name.StartsWith("FAST Search Server 2010 for SharePoint Alternate Access Mapping Extractor Job")}
Disable-SPTimerJob $extractorJob

  2. Restore the SharePoint Server farm. See Restore a farm (SharePoint Server 2010) for more information.

  3. On the main SharePoint server, run these commands to pause the Content SSA:

    Copy Code
    $extractorJob = Get-SPTimerJob | where {$_.Name.StartsWith("FAST Search Server 2010 for SharePoint Alternate Access Mapping Extractor Job")}
Enable-SPTimerJob $extractorJob
$ssa = Get-SPenterpriseSearchServiceApplication "<FAST Search Content SSA>"
$ssa.Pause()

  4. On all FAST Search Server 2010 for SharePoint servers, stop the FAST Search Server 2010 for SharePoint services:

    Copy Code
    Stop-Service FASTSearchService
Stop-Service FASTSearchMonitoring

  5. When all services have stopped, on the FAST Search Server 2010 for SharePoint administration server, run the restore script:

    1. Open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator.

    2. At the command prompt, browse to <FASTSearchFolder>\bin.

      Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    3. Run the following command:

      Copy Code
      .\restore.ps1 -action full -backuppath <UNC path of backup store(s)>

      The script will prompt you for a password to authenticate itself to remote resources. To run the script without having to enter a password, use the -passwordfile option to specify a file that contains a saved password. See Run scripts with a saved password for more information.

  6. When the script has completed, on all FAST Search Server 2010 for SharePoint servers, start FAST Search Server 2010 for SharePoint services by running these commands:

    Copy Code
    Start-Service FASTSearchService
Start-Service FASTSearchMonitoring

  7. On the FAST Search Server 2010 for SharePoint administration server, resume the services that were suspended during the backup:

    Copy Code
    .\resume.ps1

  8. On the SharePoint Server, resume the Content SSA:

    Copy Code
    $ssa.ForceResume($ssa.IsPaused())

After restoring the SharePoint Server and FAST Search Server 2010 for SharePoint farms, consider updating the FAST Search Server 2010 for SharePoint back-end with any backlog of click-through data from the SharePoint Server front-end. See Resend backlog clicks for more information.

If you have restored the backup to a reinstalled FAST Search Server 2010 for SharePoint or SharePoint Server system, you must restore the certificates for SharePoint Server and FAST Search Server 2010 for SharePoint as described in Restore certificates.

Restore certificates

The FAST Search Server 2010 for SharePoint full backup and restore process does not handle certificates. You must manually replace the certificates after reinstalling and restoring a FAST Search Server 2010 for SharePoint farm.

Depending on whether the backed up FAST Search Server 2010 for SharePoint installation used the default FASTSearchCert or a certificate signed by a certification authority, follow the steps in one of these sections:

  • Restore certificate configuration when the backed up installation used the self-signed FASTSearchCert certificate

  • Restore certificate configuration when the backed up installation used certificates signed by certification authority

Restore certificate configuration when the backed up installation used the self-signed FASTSearchCert certificate

  1. On all FAST Search Server 2010 for SharePoint servers, stop the FAST Search Server 2010 for SharePoint services:

    Copy Code
    Stop-Service FASTSearchService
Stop-Service FASTSearchMonitoring

  2. On the SharePoint server, verify that there are no FAST Search Server 2010 for SharePoint certificates from the previous installation:

    1. Run the Microsoft Management Console (mmc.exe).

    2. Use Add or Remove Snap-ins to add the Certificates snap-in for the account on the Local Computer.

    3. Use Find certificates to search for a certificate that contains the word FAST. Remove any such certificate on all SharePoint servers by right-clicking and selecting Delete.

  3. On all FAST Search Server 2010 for SharePoint servers, verify that there are no existing SharePoint certificates. Repeat step 2 on all FAST Search Server 2010 for SharePoint servers, and delete any certificates that contains the word sharepoint.

  4. You can now set up the new certificates on all servers. First, you must set up the certificate on the FAST Search Server 2010 for SharePoint administration server:

    1. Open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator.

    2. At the command prompt, browse to <FASTSearchFolder>\installer\scripts.

      Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    3. Run the following command:

      Copy Code
      .\replacedefaultcertificate.ps1 -generateNewCertificate $true

    4. Start the FAST Search for SharePoint service by running the following command:

      Copy Code
      Start-Service FASTSearchService

  5. Repeat step 4 on the remaining FAST Search Server 2010 for SharePoint servers. Use the same password for the certificate that you used on the FAST Search Server 2010 for SharePoint administration server.

  6. Copy the SecureFASTSearchConnector.ps1 script from <FASTSearchFolder>\installer\scripts to the SharePoint Server.

  7. Copy the recently created FAST Search Server 2010 for SharePoint certificate from <FASTSearchFolder>\data\data_security\cert\FASTSearchCert.pfx to the SharePoint Server.

  8. On the SharePoint Server, run the SecureFASTSearchConnector.ps1 script that you copied in step 6:

    Copy Code
    SecureFASTSearchConnector.ps1 -certPath <path of FASTSearchCert.pfx> -ssaName "<FAST Search Content SSA>" -username "<domain\username>"

    Make sure that the ssaName matches the SSA that you set up in the SharePoint Server installation.

  9. Export the SharePoint Server certificate (not to be confused with the previous FAST Search Server 2010 for SharePoint certificate):

    1. In a SharePoint Server shell, run the following commands:

      Copy Code
      $stsCert = (GetSPSecurityTokenServiceConfig).LocalLoginProvider.SigningCertificate
$stsCert.Export("cert") | Set-content -Encoding byte MOSS_STS.cer

    2. Copy the resulting MOSS_STS.cer file to all FAST Search Server 2010 for SharePoint servers.

  10. On all FAST Search Server 2010 for SharePoint servers, open a Microsoft FAST Search Server 2010 for SharePoint shell as administrator, and run the following command:

    Copy Code
    <FASTSearchFolder>\installer\scripts\installSTSCertificateForClaims.ps1 -certPath '<path of MOSS certificate...\MOSS_STS.cer>'

  11. Restart FAST Search Server 2010 for SharePoint and SharePoint servers to make sure that the new certificates are being used.

Restore certificate configuration when the backed up installation used certificates signed by certification authority

  1. To replace the self-signed FASTSearchCert certificate, see Replace the self-signed certificate with a certificate signed by a certification authority (CA).

  2. To reinstall the SharePoint claims certificate on each FAST Search Server 2010 for SharePoint query server, see Configure claims authentication.

See Also

Concepts

Backup and restore (FAST Search Server 2010 for SharePoint)
Configuration backup and restore (FAST Search Server 2010 for SharePoint)

Other Resources

Back up Search (SharePoint Server 2010)
Restore Search (SharePoint Server 2010)


Optimize search relevance (FAST Search Server 2010 for SharePoint)

One of the key features in a search solution is the ability to return relevant information to the user. Relevance describes how well a given search satisfies a user's information needs. Relevance includes which documents are returned in the results (recall) and the order of those documents in the results (ranking). FAST Search Server 2010 for SharePoint provides good search relevance out of the box in addition to many possibilities for custom tuning. Optimizing search for relevance includes refining content for searching, tuning sort order and ranking. It is a task that should be done by advanced search administrators and developers.

In this section:

  • Manage index schema for relevance (FAST Search Server 2010 for SharePoint)

  • Manage rank profiles (FAST Search Server 2010 for SharePoint)

  • Tune relevance factors (FAST Search Server 2010 for SharePoint)

  • Manage content feeding and freshness (FAST Search Server for SharePoint)

See also

Plan the relevance of search results (FAST Search Server 2010 for SharePoint)


Manage index schema for relevance (FAST Search Server 2010 for SharePoint)

In this section:

  • Identify unmapped crawled properties by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Exclude crawled properties from searching by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Map a managed property to a full-text index at a specific importance level by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

See also

The index schema and its features (FAST Search Server 2010 for SharePoint)


Identify unmapped crawled properties by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Not all new crawled properties are automatically assigned a managed property mapping and made available for searching. If the MapToContents property on the category is set to $false, all new crawled properties discovered for that category will not be mapped into the default full-text index and made available for searching. Use the following procedure to look up unmapped and potentially new crawled properties.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command to display all categories:

    Copy Code
    Get-FASTSearchMetadataCategory

>> CrawledPropertyCount      DiscoverNewProperties      MapToContents   Name
>>                  133      True                       True            SharePoint
>>                   15      True                       True            Mail
>>                   25      True                       True            Office
>>                   54      True                       True            Basic
>>                   52      True                       True            Web
>>                   15      True                       True            OutsideIn
>>                   10      False                      False           MESG Lingustics
>>                   10      True                       False           Enterprise Crawler
>>                    3      False                      False           Format Information
>>                    7      True                       False           Managed Metadata
>>                    0      True                       True            JDBC
>>                   19      True                       True            Lotus Notes
>>                    0      True                       False           People
>>                    3      True                       False           Notes
>>                    0      True                       True            Business Data
>>                    1      False                      False           FAST Search Connector

  6. Get the category:

    Copy Code
    $category = Get-FASTSearchMetadataCategory -Name "<CategoryName>"

    Where:

    • <CategoryName> is the name of the category from which you want to display unmapped crawled properties, for example Notes.

  7. Get the list of unmapped crawled properties for the category:

    Copy Code
    $unmappedCPs = $category.GetUnmappedCrawledProperties()

  8. Display the list:

    Copy Code
    $unmappedCPs

See also

Category.GetUnmappedCrawledProperties Method

Get-FASTSearchMetadataCategory


Exclude crawled properties from searching by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

One way to exclude a crawled property from searching is to set the MapToContents flag on the category to $false and all new crawled properties in that category will no longer get mapped to the full-text index. An alternative approach is to exclude individual crawled properties by setting the IsMappedToContents property on the crawled property object. This will override whatever value its category has for MapToContents.

In this article:

  • Exclude a category from searching

  • Exclude a crawled property from searching

Exclude a category from searching

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command to display all categories:

    Copy Code
    Get-FASTSearchMetadataCategory

>> CrawledPropertyCount      DiscoverNewProperties      MapToContents   Name
>>                  133      True                       True            SharePoint
>>                   15      True                       True            Mail
>>                   25      True                       True            Office
>>                   54      True                       True            Basic
>>                   52      True                       True            Web
>>                   15      True                       True            OutsideIn
>>                   10      False                      False           MESG Lingustics
>>                   10      True                       False           Enterprise Crawler
>>                    3      False                      False           Format Information
>>                    7      True                       False           Managed Metadata
>>                    0      True                       True            JDBC
>>                   19      True                       True            Lotus Notes
>>                    0      True                       False           People
>>                    3      True                       False           Notes
>>                    0      True                       True            Business Data
>>                    1      False                      False           FAST Search Connector

  6. Exclude category from searching:

    Copy Code
    Set-FASTSearchMetadataCategory -Name "<CategoryName>" -MapToContents 0

>> CrawledPropertyCount      DiscoverNewProperties      MapToContents   Name
>>                   0       True                       False           JDBC

    Where:

    • <CategoryName> is the name of the category that you want to exclude from searching, for example JDBC.

  7. Verify that the category has been excluded from searching:

    Copy Code
    Get-FASTSearchMetadataCategory

>> CrawledPropertyCount      DiscoverNewProperties      MapToContents   Name
>>                  133      True                       True            SharePoint
>>                   15      True                       True            Mail
>>                   25      True                       True            Office
>>                   54      True                       True            Basic
>>                   52      True                       True            Web
>>                   15      True                       True            OutsideIn
>>                   10      False                      False           MESG Lingustics
>>                   10      True                       False           Enterprise Crawler
>>                    3      False                      False           Format Information
>>                    7      True                       False           Managed Metadata
>>                    0      True                       False           JDBC
>>                   19      True                       True            Lotus Notes
>>                    0      True                       False           People
>>                    3      True                       False           Notes
>>                    0      True                       True            Business Data
>>                    1      False                      False           FAST Search Connector

See also

Category.MapToContents Property

Get-FASTSearchMetadataCategory

Exclude a crawled property from searching

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command (s):

    Copy Code
    $CrawledProperty = Get-FASTSearchMetadataCrawledProperty -Name "<CrawledPropertyName>"

    Where:

    • <CrawledPropertyName> is the name of the crawled property that you want to exclude from searching, for example ows_WorkPhone.

  6. Exclude the crawled property from searching:

    Copy Code
    Set-FASTSearchMetadataCrawledProperty -CrawledProperty $CrawledProperty -IsMappedToContents $false

>>CategoryName         :   SharePoint
>>IsMappedtoContents   :   False
>>IsNameEnum           :   False
>>IsMultiValued        :   False
>>Name                 :   ows_WorkPhone
>>Propset              :   00130329-0000-0130-c000-000000131346
>>VariantType          :   31

See also

CrawledProperty.IsMappedToContents Property

Get-FASTSearchMetadataCrawledProperty


Map a managed property to a full-text index at a specific importance level by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $mp = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>

    Where:

    • <ManagedProperty> is the name of the managed property that you want to map to a full-text index, for example body.

  6. View the importance level for the managed property:

    Copy Code
    $mp.GetFullTextIndexMappings()

>> ImportanceLevel: 1
>> ManagedProperty: body
>> FullTextIndex: content

  7. Get the full text index:

    Copy Code
    $fuu = Get-FASTSearchMetadataFullTextIndexMapping|Where-Object {$_.ManagedProperty.Name -eq "<ManagedProperty>"}

    Where:

    • <ManagedProperty> is the name of the managed property that you want to map to a full-text index, for example body.

  8. Remove the current importance level mapping:

    Copy Code
    Remove-FASTSearchMetadataFullTextIndexMapping -Mapping $fuu

  9. Set the new importance level:

    Copy Code
    New-FASTSearchMetadataFullTextIndexMapping -ManagedProperty (Get-FASTSearchMetadataManagedProperty Body) -FullTextIndex (Get-FASTSearchMetadataFullTextIndex content) -Level <ImportanceLevel>

    Where:

    • <ImportanceLevel> is the importance level you want to give this managed property, for example 2 (valid values are 1 through 7, with 7 indicating the highest importance).

Note:

Do not use a higher importance level value than necessary. Using too many levels within a full-text index will lead to an unnecessary large index file structure. You should always have at least one managed property with importance level 1, and in most cases you do not need to use more than the three first levels. This should be the managed property containing the body of your items.

Note:

You have to re-feed the content for the importance level change to take effect.

See also

FullTextIndexMapping Interface

Set-FASTSearchMetadataFullTextIndexMapping


Manage rank profiles (FAST Search Server 2010 for SharePoint)

The rank profile is configured to calculate the rank score with input from quality rank components (also known as static rank components) and dynamic rank components. The default rank profile is named default, which is also used as the default sort criteria (Sort by Relevance) on the search result page. Be aware that you tune rank components independently of one another. The value of two rank components with the same weight is not necessarily the same so that changing a weight from 50 to 100 on one component could lead to more effect on the final rank score than changing a weight from 50 to 100 on another component.

The rank profile

Optimize search relevance in FAST Search Server

Rank Factor Description

Quality (static) rank components

Quality

Quality rank points, also known as static rank points, are derived from the following managed properties:

  • Urldepthrank: Rank points given to boost shorter URLs.

  • Docrank: Rank points given based on the number of and relative importance of links pointing to an item.

  • Siterank: Rank points given based on the number of and relative importance of links pointing to the items on a site.

  • Hwboost: FAST Search Server 2010 for SharePoint placeholder for generic usage of static / quality rank points.

In addition to these, you can add any custom integer managed property to the list of static/quality rank components.

Dynamic rank components

Freshness

Rank points given based on the age (last modified) of the item compared to the time of the search. The source for the last modified time is the managed property named "Write".

The resolutions used (second/minutes/hour/day/year) are important for the ranking, because a lower resolution implies that smaller time differences will affect the ranking.

Context

Rank points given based on where in the item the search words hit. Context boost is assigned as an overall weight and as a weight for an individual managed property. The different managed properties such as title and body are mapped into different importance levels. This makes it possible to give more rank points when the search word hit in a title instead of in the body text of the item.

Proximity

Rank points given depending on the distance between the search words and where these words are located in the search item. For example, if two search words appear in the same managed property within the full-text index, they will get more rank points.

Managed property field boost

Rank point given based on a specific item type of a managed property. For example, if you want to give additional rank points to indexed items of a specific type such as Word documents, you would give search items with the file extension .docx additional rank points.

Authority (anchor text)

Rank points given based on when a search word retrieves hits in the link text (anchor text). This rank boost has two components:

  • Partial match - the link text is Microsoft Word, and the search is Microsoft.

  • Complete match - additional score if there is a match between the search word and link text.

Query authority (click-through)

Rank points given based on when a search retrieves hits in items associated with previously clicked on search results (click-through). In the search front-end, all of a user's searches and the items that the user clicks on that are associated with those searches are stored and processed. For example, if a user searched for vacation and clicked on the vacation request template in the search result (e.g., http://myserver/template/vacation.aspx), this URL is associated with the search word and receives additional rank points when any new search with the same search word is performed.

Processing occurs one time each day. The last 30 days of clicks are kept for processing.

This rank score also has two components:

  • Partial match - the initial search words that resulted in a click to this item was Microsoft Word, and the search word now is Microsoft.

  • Complete match - an additional score if there is a match between the search word and link text.

In this section:

  • Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Change the overall weight of the static rank (quality rank) components by using Windows PowerShell (FAST Search Server 2010 for SharePoint))

See also

The index schema and its features (FAST Search Server 2010 for SharePoint)


Change the overall weight of the static rank (quality rank) components by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Published: July 16, 2012

If you want to give more rank point to search words entered by the user versus rank points given to documents independent of search word used, you should increase the overall weight of the dynamic rank components. You do this by reducing the overall weight of the quality, or static rank components.

  1. On the Start menu, click All Programs.

  2. Click Microsoft FAST Search Server 2010 for SharePoint.

  3. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  4. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to change the overall static rank components weight for. The default rank profile is named default.

  5. Look at the current value:

    Copy Code
    $RankProfile

>> Name                                   :         Time
>> isDefault                              :         False
>> RankModelName                          :         Default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         50
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         0
>> FreshnessResolution                    :         Second
>> FreshnessManagedPropertyReference      :         Write

  6. Change the quality weight:

    Copy Code
    $RankProfile.QualityWeight = <QualityWeight>

    Where:

    • <QualityWeight> is the new overall weight you want to give the static rank components, for example 30.

  7. Update the rank profile with the new value:

    Copy Code
    $RankProfile.Update()

  8. Look at the rank profile to confirm that the new value is being used:

    Copy Code
    $RankProfile

>> Name                                   :         Time
>> isDefault                              :         False
>> RankModelName                          :         Default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         30
>> AuthorityWeight                        :         50
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         0
>> FreshnessResolution                    :         Second
>> FreshnessManagedPropertyReference      :         Write

See also

RankProfile.QualityWeight Property

Get-FASTSearchMetadataRankProfile


Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Published: July 16, 2012

In this article the following procedures are described:

  • Create a new rank profile by using Windows PowerShell

  • Create a new rank profile by using the default rank profile as a template by using Windows PowerShell

  • Change the default rank profile by using the graphical interface

Create a new rank profile by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    New-FASTSearchMetadataRankProfile -name <NewRankProfile>

>> Name                                 :  NewRankProfile
>> isDefault                            :  False
>> RankModelName                        :  default
>> StopWordThreshold                    :  2000000
>> PositionStopWordThreshold            :  20000000
>> QualityWeight                        :  50
>> AuthorityWeight                      :  50
>> QueryAuthorityWeight                 :  100
>> FreshnessWeight                      :  1000
>> FreshnessResolution                  :  Second
>> FreshnessManagedPropertyReference    :  Write

    Where:

    • <NewRankProfile> is the name of the new rank profile that you want to create, for example NewRankProfile.

See also

RankProfileCollection.Create Method (String)

Get-FASTSearchMetadataRankProfile

Create a new rank profile by using the default rank profile as a template by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $rp = Get-FASTSearchMetadataRankProfile -name <TemplateRankProfile>
$np = New-FASTSearchMetadataRankProfile -name <NewRankProfile> -template $rp

    Where:

    • <TemplateRankProfile> is the name of the rank profile that you want to use as template, for example default.

    • <NewRankProfile> is the name of the new rank profile that you want to create, for example NewRankProfile.

  6. View the newly created rank profile:

    Copy Code
    $np

>> Name                                 :  NewRankProfile
>> isDefault                            :  False
>> RankModelName                        :  default
>> StopWordThreshold                    :  2000000
>> PositionStopWordThreshold            :  20000000
>> QualityWeight                        :  50
>> AuthorityWeight                      :  50
>> QueryAuthorityWeight                 :  100
>> FreshnessWeight                      :  1000
>> FreshnessResolution                  :  Second
>> FreshnessManagedPropertyReference    :  Write

See also

RankProfileCollection.Create Method (String, RankProfile)

Get-FASTSearchMetadataRankProfile

Change the default rank profile by using the graphical interface

When a user types one or more search words into the Search Box Web Part on the search page (default.aspx) in the Search Center site, the search service returns the results displayed in various Web Parts on the search results page (results.aspx). If the search words did not contain any keywords or associated synonyms, the order of the search results are based on the rank score calculated by the default rank profile. The label of the default rank profile is displayed in the Sort by pull down menu. The default search results page uses the default rank profile, which is labeled Relevance in the Sort by pull down menu. Use the following procedure to change the default rank profile via the graphical interface:

  1. On the Search Center site, enter a search word and then click the Search icon.

  2. On the Search results page, click the Site Actions menu, and then click Edit Page.

  3. From the Search Action Links Web Part Menu select Edit Web Part.

  4. In the web part tool pane, expand Display Properties to display all existing rank profiles.

  5. Select the rank profile that you want to set as default by clicking the Default radio button and checking the Enabled check box next to it.

  6. Click OK.

  7. Click Save & Close.

The new default rank profile will now appear in the Sort by pull down menu.

Note:

The name that you gave the rank profile when you created it, is displayed in the ID column under Display Properties in the Search Action Link Web Part tool pane. You can change the name of the rank profile as displayed in the Sort by pull down menu on the Search results page by editing the name in the Label column.


Tune relevance factors (FAST Search Server 2010 for SharePoint)

FAST Search Server 2010 for SharePoint lets you tune relevance in several ways. Predefined relevance (delivered out of the box) was tested and tuned against large sets of different data. However, relevance is not absolute, and because perceived relevance differs greatly among users, organizations, and locations, it is important to tune relevancy according to your specific needs.

The relevance score is quantified in a rank score for each searchable item. This section introduces different tuning possibilities, discusses tuning details, and provides usage examples. Since some tuning is performed through programming and Windows PowerShell cmdlets, this section also describes some technical aspects related to relevance.

The following are involved when tuning relevance:

  • Static (quality) rank Rank points added to a search item before it is indexed. The number of points given is based on boost values that are stored in four predefined static/quality rank components. In addition, any custom integer managed property can also be used to tune static rank. You can decide which of these components to use at search time so that you only get the rank points from the components that you specify. You can control the effect of the static rank points by modifying the weight of the different components. Static rank points assigned to an indexed item are independent of the search words used, so static rank only indicates something about the general importance of an item.

  • Dynamic rank: Rank points added to search items at search time. The number of points given is based on a combination of the search words used and boost values that are stored in dynamic rank components. You can control the effect of the dynamic rank points by modifying the weight of the different components.

  • Keyword rank: Rank points added to an item when a search word matches a specified keyword. Recall is improved by searching for the associated synonyms. Also, the search experience is improved by displaying additional relevant information as Best Bets and Visual Best Bets and incorporating document- and site promotions to move relevant search results higher up in the search results list.

  • Linguistic features: like stemming, spell checking, stop words and tokenization effect both relevance and recall.

  • Fast Query Language (FQL): a query language intended for creating queries programmatically. FQL can be used only for solutions developed for FAST Search Server 2010 for SharePoint.

In this section:

  • Tune static rank (quality rank) (FAST Search Server 2010 for SharePoint)

  • Tune dynamic rank (FAST Search Server 2010 for SharePoint)

  • Tune keyword rank (FAST Search Server 2010 for SharePoint)

  • Tune linguistic relevance (FAST Search Server 2010 for SharePoint)

See also

Which relevance factors can you tune and what is the impact (FAST Search Server 2010 for SharePoint)

FAST Query Language (FQL) (FAST Search Server 2010 for SharePoint)


Tune static rank (quality rank) (FAST Search Server 2010 for SharePoint)

In this section:

  • Change the weight, remove or add a static rank component by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

See also

For more information about the different components involved in tuning static rank and their effect, see Static rank (quality rank) (FAST Search Server 2010 for SharePoint).


Change the weight, remove or add a static rank component by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

In this article the following Windows PowerShell procedures are described:

  • Change the weight of a static rank component

  • Remove a static rank component

  • Add a static rank component

Note:

When tuning static rank, we recommend that you create a new rank profile, on which you can test the new component settings. For more on how to create new rank profiles, see Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

Change the weight of a static rank component

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that contains the component that you want to configure. The default rank profile is named default.

  6. Get all the static rank components:

    Copy Code
    $QualityComponents = $RankProfile.GetQualityComponents()

  7. Look at the current values:

    Copy Code
    $QualityComponents

>> ManagedPropertyReference : hwboost
>> Weight                   : 0
>> ManagedPropertyReference : docrank
>> Weight                   : 70
>> ManagedPropertyReference : siterank
>> Weight                   : 100
>> ManagedPropertyReference : urldepthrank
>> Weight                   : 100

  8. Set the new weight value:

    Copy Code
    $QualityComponents | where-object -filterscript { if ( $_.ManagedProperty
Reference.Name -eq "<QualityRankComponent>" ) { $_.Weight=<RelativeWeight>; $_.Update() } }

    Where:

    • <QualityRankComponent> is the name of the static rank component to which you want to change the weight, for example urldepthrank.

    • <RelativeWeight> is the new weight you want to give the static rank component, for example 200.

  9. Check that the weight was changed:

    Copy Code
    $QualityComponents

>> ManagedPropertyReference : hwboost
>> Weight                   : 0
>> ManagedPropertyReference : docrank
>> Weight                   : 70
>> ManagedPropertyReference : siterank
>> Weight                   : 100
>> ManagedPropertyReference : urldepthrank
>> Weight                   : 200

See also

QualityComponent.Weight Property

Get-FASTSearchMetadataRankProfile

Remove a static rank component

To remove a static rank component, either set the static component weight = 0, or remove the component from the rank profile. Since the previous example can also be used to set the weight = 0, the following example shows how to remove a static rank component.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell..

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that contains the component that you want to remove. The default rank profile is named default.

  6. Get all the static rank components:

    Copy Code
    $QualityComponents = $RankProfile.GetQualityComponents()

  7. Look at the current components:

    Copy Code
    $QualityComponents

>> ManagedPropertyReference : hwboost
>> Weight                   : 0
>> ManagedPropertyReference : docrank
>> Weight                   : 70
>> ManagedPropertyReference : siterank
>> Weight                   : 100
>> ManagedPropertyReference : urldepthrank
>> Weight                   : 100

  8. Loop over the components and remove the component that you want to exclude from static rank when found:

    Copy Code
    $QualityComponents | where-object -filterscript { if ( $_.ManagedProperty
Reference.Name -eq "<QualityRankComponent>" ) { $_.Delete() } }

    Where:

    • <QualityRankComponent> is the name of the static rank component that you want to remove, for example urldepthrank.

  9. Verify that the static rank component was removed:

    Copy Code
    $QualityComponents

>> ManagedPropertyReference : hwboost
>> Weight                   : 0
>> ManagedPropertyReference : docrank
>> Weight                   : 70
>> ManagedPropertyReference : siterank
>> Weight                   : 100

See also

QualityComponent.Delete Method

Get-FASTSearchMetadataRankProfile

Add a static rank component

Any managed property that is defined as an integer can be used as input to the static rank. The weight of the new static rank component should be set depending on the scale used in that managed property. For example, using a static component weight of 200 will give rank points equal to double of the integer value of the managed property. Using a static component weight of 1000 will give rank points equal to 10 times the integer value of the managed property.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to add the component to. The default rank profile is named default.

  6. Get all the static rank components:

    Copy Code
    $QualityComponents = $RankProfile.GetQualityComponents()

  7. Look at the current components:

    Copy Code
    $QualityComponents

>> ManagedPropertyReference : hwboost
>> Weight                   : 0
>> ManagedPropertyReference : docrank
>> Weight                   : 70
>> ManagedPropertyReference : siterank
>> Weight                   : 100
>> ManagedPropertyReference : urldepthrank
>> Weight                   : 100

  8. Get the managed property that you want to add as a static rank component:

    Copy Code
    $ManagedProperty = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>

    Where:

    • <ManagedProperty> is the name of the managed property that you want to add as a static rank component, for example rating.

  9. Add the managed property to the list of static rank components:

    Copy Code
    $QualityComponents.Create($ManagedProperty, <RelativeWeight>)

    Where:

    • <RelativeWeight> is the relative weight you want to give the new component, for example 200.

  10. Verify that the managed property was added:

    Copy Code
    $QualityComponents

>> ManagedPropertyReference : rating
>> Weight                   : 200
>> ManagedPropertyReference : hwboost
>> Weight                   : 0
>> ManagedPropertyReference : docrank
>> Weight                   : 70
>> ManagedPropertyReference : siterank
>> Weight                   : 100
>> ManagedPropertyReference : urldepthrank
>> Weight                   : 100

See also

QualityComponentCollection.Create Method

Get-FASTSearchMetadataRankProfile


Tune dynamic rank (FAST Search Server 2010 for SharePoint)

In this section:

  • Change the freshness resolution and weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Change the overall- and managed property context weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Change the proximity weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Change the managed property field boost weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Change the authority (anchor text) weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Change the query authority (click-through) weight and resend backlog clicks by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Change the Stop Word Threshold by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

See also

For more information about the different components involved in tuning dynamic rank and their effect, see Dynamic rank (FAST Search Server 2010 for SharePoint).


Change the freshness resolution and weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Note:

When tuning dynamic rank, we recommend that you create a new rank profile, on which you can test the new component settings. For more on how to create new rank profiles, see Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to change the freshness resolution and weight of. The default rank profile is named default.

  6. Look at the current value:

    Copy Code
    $RankProfile

>> Name                                   :         default
>> isDefault                              :         True
>> RankModelName                          :         default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         80
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         100
>> FreshnessResolution                    :         Day
>> FreshnessManagedPropertyReference      :         Write

  7. Change the freshness resolution and weight:

    Copy Code
    $RankProfile.FreshnessResolution = "<FreshnessResolution>"
$RankProfile.FreshnessWeight = "<FreshnessWeight>"

    Where:

    • <FreshnessResolution> is the resolution that you want to use, for example Minute. Valid resolutions are as follows: Second, Minute, Hour, Day and Year.

    • <FreshnessWeight> is the weight you want the Freshness component to have relative to the other dynamic rank components, for example 1000.

  8. Update the rank profile with the new values:

    Copy Code
    $RankProfile.Update()

  9. Look at the rank profile to verify that the new values are being used:

    Copy Code
    $RankProfile

>> Name                                   :         default
>> isDefault                              :         True
>> RankModelName                          :         default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         80
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         1000
>> FreshnessResolution                    :         Minute
>> FreshnessManagedPropertyReference      :         Write

See also

RankProfile.FreshnessResolution Property

RankProfile.FreshnessWeight Property

Set-FASTSearchMetadataRankProfile


Change the overall- and managed property context weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

In this article, the following Windows PowerShell procedures are described:

  • Change the context weight of a managed property

  • Change the overall context component weight

Note:

When tuning dynamic rank, we recommend that you create a new rank profile, on which you can test the new component settings. For more on how to create new rank profiles, see Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

Change the context weight of a managed property

The context component lets you increase rank score for certain managed properties. By for example increasing the context weight for the managed property title, a search word hit in a title will receive a higher rank score than a hit in the body. This causes the item with a search word hit in the title to appear higher up on the search results list. The different managed properties, such as title and body, are mapped into different important levels. The context weight of a managed property is changed in association with the importance level to which it is mapped.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile which is mapped to the full-text index containing the managed property you want to change the context weight of. The default rank profile is named default.

  6. Get the full text index:

    Copy Code
    $content = $RankProfile.GetFullTextIndexRanks()|where-Object -filterscript {$_.FullTextIndexReference.Name -eq "content"}

  7. Get the managed property:

    Copy Code
    $mp = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>

    Where

    • <ManagedProperty> is the name of the managed property that you want to change the context boost for, for example Title.

  8. Get the importance level for the managed property.

    Copy Code
    $mp.GetFullTextIndexMappings()

>> ImportanceLevel: 7
>> ManagedProperty: Title
>> FullTextIndex: content

  9. View the weight for that importance level:

    Copy Code
    $content.GetImportanceLevelWeight(<ImportanceLevel>)

>> 60

    Where:

    • <ImportanceLevel> is the importance level of the managed property that you want to change the context boost of, for example 7.

  10. Set the new context weight:

    Copy Code
    $content.SetImportanceLevelWeight(<ImportanceLevel>, <ContextWeight>)

    Where:

    • <ImportanceLevel> is the importance level for the managed property, for example 7.

    • <ContextWeight> is the context weight you want to assign this managed property, for example 200.

  11. Update the weight:

    Copy Code
    $content.Update()

  12. Verify that the context weight for the managed property has changed:

    Copy Code
    $content.GetImportanceLevelWeight(<ImportanceLevel>)

>> 200

    Where:

    • <ImportanceLevel> is the importance level for the managed property that you changed the weight for, for example 7.

See also

Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

Get-FASTSearchMetadataRankProfile

Get-FASTSearchMetadataFullTextIndex

Get-FASTSearchMetadataManagedProperty

Change the overall context component weight

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to change the overall context component weight of. The default rank profile is named default.

  6. Get the full text index:

    Copy Code
    $content = $RankProfile.GetFullTextIndexRanks()|where-Object -filterscript {$_.FullTextIndexReference.Name -eq "content"}

  7. Look at the current value:

    Copy Code
    $content

>> FullTextIndexReference : content
>> ProximityWeight: 50
>> ContextWeight: 50

  8. Change the weight:

    Copy Code
    $content.ContextWeight = <ContextWeight>

    Where:

    • <ContextWeight> is the weight you want to give the overall context component, for example 200.

  9. Update the full text index with the new value:

    Copy Code
    $content.Update()

  10. Look at the full text index to verify that the new value is being used:

    Copy Code
    $content

>> FullTextIndexReference : content
>> ProximityWeight: 50
>> ContextWeight: 200

See also

FullTextIndexRankComponent.ContextWeight Property

Get-FASTSearchMetadataRankProfile

Get-FASTSearchMetadataFullTextIndex


Change the proximity weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Note:

When tuning dynamic rank, we recommend that you create a new rank profile, on which you can test the new component settings. For more on how to create new rank profiles, see Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to change the proximity weight of. The default rank profile is named default.

  6. Get the full text index:

    Copy Code
    $content = $RankProfile.GetFullTextIndexRanks()|where-Object -filterscript {$_.FullTextIndexReference.Name -eq "content"}

  7. Look at the current value:

    Copy Code
    $content

>> FullTextIndexReference : content
>> ProximityWeight: 50
>> ContextWeight: 50

  8. Change the proximity weight:

    Copy Code
    $content.ProximityWeight = <ProximityWeight>

    Where:

    • <ProximityWeight> is the weight you want to give the proximity component, for example 200.

  9. Update the full text index with the new value:

    Copy Code
    $content.Update()

  10. Look at the full text index to confirm that the new value is being used:

    Copy Code
    $content

>> FullTextIndexReference : content
>> ProximityWeight: 200
>> ContextWeight: 50

See also

FullTextIndexRankComponent.ProximityWeight Property

Get-FASTSearchMetadataRankProfile

Get-FASTSearchMetadataFullTextIndex


Change the managed property field boost weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

To be able to give rank points to items that have a particular value within a managed property, for example add rank points when the value of the managed property language is English or the value of the managed property fileextension is .docx, you first have to create a Managed Property Boost Component. The weight for the managed property field boost is then set on this component.

Note:

When you tune dynamic rank, we recommend that you create a new rank profile, on which you can test the new component settings. For more information about how to create new rank profiles, see Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to change the managed property field boost of. The default rank profile is named default.

  6. Get the managed property that you want change the field boost of:

    Copy Code
    $Property = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>

    Where:

    • <ManagedProperty> is the managed property that you want change the field boost of, for example fileextension.

  7. Create the Managed Property Boost Component and set the weight:

    Copy Code
    $RankProfile.CreateManagedPropertyBoostComponent($Property, "<ManagedPropertyValue>,<ManagedPropertyBoostWeight>")

    Where:

    • <ManagedPropertyValue> is the value of the managed property that you want to add rank points to, for example docx.

    • <ManagedPropertyBoostWeight> is the weight you want to add to the Managed Property Boost Component, for example 1000.

  8. Update the rank profile with the new values:

    Copy Code
    $RankProfile.Update()

  9. Look at the managed property to confirm that the new value is being used:

    Copy Code
    $RankProfile.GetManagedPropertyBoosts()|where-object { $_.ManagedPropertyReference.Name -eq "<ManagedProperty>"}

>> ManagedPropertyReference : fileextension
>> BoostValue               : csv,-4000,txt,-4000,msg,-4000,oft,-4000,zip,-4000,vsd,-4000,rtf,-4000

>> ManagedPropertyReference : fileextension
>> BoostValue               : docx,1000

    Where:

    • <ManagedProperty> is the managed property changed the field boost of, for example fileextension.

See also

ManagedPropertyBoostComponent.BoostValue Property

Get-FASTSearchMetadataRankProfile

Get-FASTSearchMetadataManagedProperty


Change the authority (anchor text) weight by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Note:

When tuning dynamic rank, we recommend that you create a new rank profile, on which you can test the new component settings. For more on how to create new rank profiles, see Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to change the authority (anchor text) weight of. The default rank profile is named default.

  6. Look at the current value:

    Copy Code
    $RankProfile

>> Name                                   :         default
>> isDefault                              :         False
>> RankModelName                          :         default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         80
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         100
>> FreshnessResolution                    :         Day
>> FreshnessManagedPropertyReference      :         Write

  7. Change the authority (anchor text) weight:

    Copy Code
    $RankProfile.AuthorityWeight = <AuthorityWeight>

    Where:

    • <AuthorityWeight> is the weight you want the parameter Authority to have relative to the other dynamic rank parameters, for example 1000.

  8. Update the rank profile with the new values:

    Copy Code
    $RankProfile.Update()

  9. Look at the rank profile to confirm that the new value is being used:

    Copy Code
    $RankProfile

>> Name                                   :         default
>> isDefault                              :         False
>> RankModelName                          :         default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         1000
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         100
>> FreshnessResolution                    :         Day
>> FreshnessManagedPropertyReference      :         Write

See also

RankProfile.AuthorityWeight Property

Set-FASTSearchMetadataRankProfile


Change the query authority (click-through) weight and resend backlog clicks by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

In this article, the following Windows PowerShell procedures are described:

  • Change the query authority (click-through) weight

  • Resend backlog clicks

Note:

When tuning dynamic rank, we recommend that you create a new rank profile, on which you can test the new component settings. For more on how to create new rank profiles, see Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

Change the query authority (click-through) weight

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to change the query authority (click-through) weight of. The default rank profile is named default.

  6. Look at the current value:

    Copy Code
    $RankProfile

>> Name                                   :         default
>> isDefault                              :         False
>> RankModelName                          :         default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         80
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         100
>> FreshnessResolution                    :         Day
>> FreshnessManagedPropertyReference      :         Write

  7. Change the query authority (click-through) weight:

    Copy Code
    $RankProfile.QueryAuthorityWeight = <QueryAuthorityWeight>

    Where:

    • <QueryAuthorityWeight> is the weight you want the parameter Query authority (click-through) to have relative to the other dynamic rank parameters, for example 1000.

  8. Update the rank profile with the new values:

    Copy Code
    $RankProfile.Update()

  9. Look at the rank profile to confirm that the new value is being used:

    Copy Code
    $RankProfile
>> Name                                   :         default
>> isDefault                              :         False
>> RankModelName                          :         default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         80
>> QueryAuthorityWeight                   :         1000
>> FreshnessWeight                        :         100
>> FreshnessResolution                    :         Day
>> FreshnessManagedPropertyReference      :         Write

See also

RankProfile.QueryAuthorityWeight Property

Set-FASTSearchMetadataRankProfile

Resend backlog clicks

The first time that you match a SharePoint Server front-end with a FAST Search Server 2010 for SharePoint back-end, 30 days of clicks are sent to the FAST Search Server 2010 for SharePoint back-end (even if no clicks have been done). This starts the search click-through functionality and keeps the system from running 30 days before click-through boosting becomes effective. However, if you re-install the FAST Search Server 2010 for SharePoint back-end, make sure that the 30 days of clicks are resent to the back-end by following these steps:

  1. Verify that you have Farm Administrator permissions.

  2. On the Start menu, click All Programs.

  3. Click Microsoft SharePoint 2010 Products.

  4. Click SharePoint 2010 Management shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $ctejd = Get-SPEnterpriseSearchExtendedClickThroughExtractorJobDefinition -SearchApplication "<SearchApplication>"
$ctejd.ResetLastExtractionDates()
$ctejd.Update()
$ctejd.RunNow()

    Where:

    • <SearchApplication> is the name of your FAST Search Query Search Service Application (SSA).


Change the Stop Word Threshold by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

The number of matching items used in the rank computation is determined by the StopWordThreshold attribute in the rank profile. If the number of matching items for a search term is larger than the StopWordThreshold for the rank profile being used, the Microsoft FAST Search Server 2010 for SharePoint system automatically retries the search by limiting the managed properties searched. This process is known as drilling. Managed properties are ranked by importance level. Importance level 1, the first level that is used in a search, includes managed properties from levels 1 and up. Drilling to importance level 2 means that only managed properties from levels 2 and up will be used in the search. The maximum importance level is 7.

The value of the StopWordThreshold is relative to a full-text index size of 10,000,000 items. In the default rank profile the StopWordThreshold value is 2,000,000, which means that drilling is performed when a search word is found in more than 20% of items in the full-text index (2,000,000 divided by 10,000,000).

Tip:

If you have a small full-text index with a uniform set of items, you might want to increase the StopWordThreshold in order to retrieve rank score from the managed properties in the lower levels (such as body). The performance effect of having a high StopWordThreshold when you have a small full-text index is limited.
To make sure that you get rank from everything, change the StopWordThreshold to 10,000,000. The new StopWordThreshold and the predefined reference number are then equal, which gives a 100% rank calculation at importance level 1.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $RankProfile = Get-FASTSearchMetadataRankProfile -Name <RankProfile>

    Where:

    • <RankProfile> is the name of the rank profile that you want to change the StopWordThreshold of. The default rank profile is named default.

  6. Look at the current value:

    Copy Code
    $RankProfile

>> Name                                   :         default
>> isDefault                              :         False
>> RankModelName                          :         default
>> StopWordThreshold                      :         2000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         80
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         100
>> FreshnessResolution                    :         Day
>> FreshnessManagedPropertyReference      :         Write

  7. Change the StopWordThreshold:

    Copy Code
    $RankProfile.StopWordThreshold = <StopWordThresholdValue>

    Where:

    • <StopWordThresholdValue> is the value that you want to set the StopWordThreshold to, for example 5000000.

  8. Update the rank profile with the new value:

    Copy Code
    $RankProfile.Update()

  9. Look at the rank profile to confirm that the new value is being used:

    Copy Code
    $RankProfile

>> Name                                   :         default
>> isDefault                              :         False
>> RankModelName                          :         default
>> StopWordThreshold                      :         5000000
>> PositionStopWordThreshold              :         20000000
>> QualityWeight                          :         50
>> AuthorityWeight                        :         80
>> QueryAuthorityWeight                   :         50
>> FreshnessWeight                        :         100
>> FreshnessResolution                    :         Day
>> FreshnessManagedPropertyReference      :         Write

See also

RankProfile.StopWordThreshold Property

Importance Level and Drilling

Map a managed property to a full-text index at a specific importance level by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Set-FASTSearchMetadataRankProfile


Tune keyword rank (FAST Search Server 2010 for SharePoint)

In this section:

  • Add, edit, remove and display keywords by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Add, remove and display synonyms for keywords by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Add, remove and display Best Bets for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Add, remove and display Visual Best Bets for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Add, remove and display document promotions for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Add, remove and display site promotions for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Add, remove and display user contexts by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

See also

For more information about the different components involved in tuning keyword rank and their effect, see Keyword rank (FAST Search Server 2010 for SharePoint).


Add, edit, remove and display keywords by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Published: July 16, 2012

Note:

This article describes how to manage keywords by using Windows PowerShell. For information about how to manage keywords by using the graphical user interface (GUI), see Manage FAST Search keywords (Office.com).

In this article the following Windows PowerShell procedures are described:

  • Add a keyword

  • Add multiple keywords

  • Edit a keyword definition

  • Remove a keyword

  • Remove all keywords

  • Display all keywords and their search settings

Important:

When tuning keyword rank by using Windows PowerShell, you associate the different search settings to a search setting group. If you have already added a keyword by using the GUI, the SharePoint front end has automatically created a search setting group. If you have not added a keyword via the GUI, you will have to create a search setting group before you can start doing keyword management.

Add a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a keyword to the search setting group:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Keywords.AddKeyword("<KeywordTerm>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword that you want to add.

See also

Keyword.Term Property

Get-FASTSearchSearchSettingGroup

New-FASTSearchSearchSettingGroup

Add multiple keywords

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of the keywords that you want to add:

    Copy Code
    $keywordsToBeAdded = @("<Keyword1>", "<Keyword2>", "<Keyword3>", "<Keyword4>")

    Where:

    • <Keyword1-4> are the keywords that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of keywords to the search setting group:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
foreach($keywordToBeAdded in "$keywordsToBeAdded") { $searchSettingGroup.Keywords.AddKeyword($keywordToBeAdded)}

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

KeywordCollection Interface

Keyword.Term Property

Get-FASTSearchSearchSettingGroup

New-FASTSearchSearchSettingGroup

Edit a keyword definition

Note:

A keyword definition is a text that will be displayed under the keyword on the search result page when users search for the keyword.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Edit a keyword definition in the search setting group:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$keyword.Definition = "<NewKeywordDefinition>"

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term to which the definition should be added.

    • <NewKeywordDefinition> is the new keyword definition string.

See also

Keyword.Definition Property

Get-FASTSearchSearchSettingGroup

Remove a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a keyword from the search setting group:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Keywords.RemoveKeyword("<KeywordTerm>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term of an existing keyword that is to be removed.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove all keywords

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all keywords from the search setting group:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Keywords.Clear()

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Display all keywords and their search settings

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Display all keywords and their search settings in the search setting group:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Keywords

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Keyword.Term Property

Get-FASTSearchSearchSettingGroup


Add, remove and display synonyms for keywords by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Published: July 16, 2012

Note:

This article describes how to manage synonyms by using Windows PowerShell. For information about how to manage synonyms by using the graphical user interface, see Manage FAST Search keywords (Office.com).

Use synonyms only to improve recall for small result sets. If you add synonyms to keywords that already have a large result set, the additional results will be added to the end of the result set, and the search performance will be reduced.

Do not use synonyms for multiple versions of a word, for example car and cars. The stemming functionality in FAST Search Server 2010 for SharePoint will handle different versions of the same word. For more information about stemming, see Tune linguistic relevance (FAST Search Server 2010 for SharePoint).

In this article the following Windows PowerShell procedures are described:

  • Add a synonym to a keyword

  • Add multiple synonyms to a keyword

  • Remove a synonym from a keyword

  • Remove all synonyms from a keyword

  • Display all synonyms

  • Turn off recall for synonyms

Add a synonym to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a synonym to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$keyword.AddSynonym("<SynonymTerm>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <SynonymTerm> is the synonym term to be added.

See also

Keyword.AddSynonym Method

Get-FASTSearchSearchSettingGroup

Add multiple synonyms to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of the synonyms that you want to add:

    Copy Code
    $synonymsToBeAdded = @("<Synonym1>", "<Synonym2>", "<Synonym3>", "<Synonym4>")

    Where:

    • <Synonym1-4> are the synonyms that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of synonyms to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
foreach($synonymToBeAdded in "<$synonymsToBeAdded>") {$keyword.AddSynonym($synonymToBeAdded)}

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

See also

SynonymCollection Interface

Keyword.AddSynonym Method

Get-FASTSearchSearchSettingGroup

Remove a synonym from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a synonym from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$keyword.RemoveSynonym("<SynonymTerm>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <SynonymTerm> is the synonym term to be removed.

See also

Keyword.RemoveSynonym Method

Get-FASTSearchSearchSettingGroup

Remove all synonyms from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all synonyms from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$keyword.Synonyms.Clear()

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Display all synonyms

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Display all synonyms:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
foreach ($keyword in $searchSettingGroup.Keywords) { $keyword.Synonyms }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Keyword.Synonyms Property

Get-FASTSearchSearchSettingGroup

Turn off recall for synonyms

When a search is rewritten because of keyword synonyms, search performance is affected. The degree of search latency increase depends on:

  • The number of synonyms added to the search.

  • The number of documents in the corpus.

  • The number of terms in each synonym, if it is a phrase.

If you do not want the increased recall from synonyms, or if the search latency increase is unacceptable, you can turn off recall of keyword synonyms by changing a configuration option. On every FAST Search Server 2010 for SharePoint server that runs the query service, follow these steps:

  1. You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the FAST Search Server 2010 for SharePoint administration server, open <FASTSearchInstallationFolder>\META\config\profiles\default\templates\installer\etc\admin.config.template in a text editor (<FASTSearchInstallationFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch).

  3. Replace this line:

    Copy Code
    <add key="SynonymQueryRecall" value="true" />

    with the following:

    Copy Code
    <add key="SynonymQueryRecall" value="false" />

  4. Save the file.

  5. Stop the FAST Search for SharePoint service.

  6. On the Start menu, click All Programs.

  7. Click Microsoft FAST Search Server 2010 for SharePoint.

  8. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  9. At the Windows PowerShell command prompt, type the following command:

    Copy Code
    Set-FASTSearchConfiguration

  10. Start the FAST Search for SharePoint service.

See also

Set-FASTSearchConfiguration


Add, remove and display Best Bets for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Note:

This article describes how to manage Best Bets by using Windows PowerShell. For information about how to manage document promotions by using the graphical user interface, see Manage FAST Search keywords (Office.com).

In this article the following PowerShell procedures are described:

  • Add a Best Bet to a keyword

  • Add multiple Best Bets to a keyword

  • Remove a Best Bets from a keyword

  • Remove all Best Bets from a keyword

  • Add a global Best Bet

  • Add multiple global Best Bets

  • Remove a global Best Bet

  • Remove all global Best Bets

  • Display all Best Bets

Add a Best Bet to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a Best Bet to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$uri = New-Object -TypeName System.Uri -ArgumentList "<URI>"
$keyword.AddBestBet("<BestBetName>", "<HTMLTeaser>", $uri)

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <URI> is the URI to the content of the Best Bet.

    • <BestBetName> is the title of the Best Bet to be added.

    • <HTMLTeaser> is the teaser in HTML format to be displayed with the link to the Best Bet. Please note that the teaser will not be displayed by the standard web part.

See also

Keyword.AddBestBet Method

Get-FASTSearchSearchSettingGroup

Add multiple Best Bets to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of existing Best Bets that you want to add:

    Copy Code
    $bestBetsToBeAdded = @("<BestBet1>", "<BestBet2>", "<BestBet3>", "<BestBet4>")

    Where:

    • <BestBet1-4> are the Best Bets that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of Best Bets to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
foreach($bestBetsToBeAdded in "<$bestBetsToBeAdded>") {$keyword.AddBestBet($bestBetToBeAdded)}

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <$bestBetsToBeAdded> is a variable that contains an existing array or collection of strings specifying the names of the existing Best Bets to be added.

See also

BestBetCollection Interface

Keyword.AddBestBet Method

Get-FASTSearchSearchSettingGroup

Remove a Best Bets from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a Best Bet from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$keyword.RemoveBestBet("<BestBetName>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <BestBetName> is the title of the Best Bet to be removed.

See also

Keyword.RemoveBestBet Method

Get-FASTSearchSearchSettingGroup

Remove all Best Bets from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all Best Bets from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$keyword.BestBets.Clear()

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add a global Best Bet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a global Best Bet:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.BestBetsWithoutKeyword.AddBestBet("<GlobalBestBetName>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <GlobalBestBetName> is the title of the global Best Bet to be added.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add multiple global Best Bets

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of existing global Best Bets that you want to add:

    Copy Code
    $globalBestBetsToBeAdded = @("<GlobalBestBet1>", "<GlobalBestBet2>", "<GlobalBestBet3>", "<GlobalBestBet4>")

    Where:

    • <GlobalBestBet1-4> are the global Best Bets that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of global Best Bets:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
foreach ($globalBestBetToBeAdded in <$globalBestBetsToBeAdd>) { $searchSettingGroup.BestBetsWithoutKeyword.AddBestBet( $globalBestBetToBeAdded ) }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <$globalBestBetsToBeAdded> is a variable that contains an existing array or collection of strings specifying the names of the existing global Best Bets to be added.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a global Best Bet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove the global Best Bet:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.BestBetsWithoutKeyword.RemoveBestBet("<GlobalBestBetName>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <GlobalBestBetName> is the title of the global Best Bet to be removed.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove all global Best Bets

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all global Best Bets:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.BestBetsWithoutKeyword.Clear()

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Display all Best Bets

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Display all Best Bets:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.BestBets

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Keyword.BestBets Property

Get-FASTSearchSearchSettingGroup


Add, remove and display Visual Best Bets for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Note:

This article describes how to manage Visual Best Bets by using Windows PowerShell. For information about how to manage Visual Best Bets by using the graphical user interface, see Manage FAST Search keywords (Office.com).

In this article the following PowerShell procedures are described:

  • Add a Visual Best Bet to a keyword

  • Add multiple Visual Best Bets to a keyword

  • Remove a Visual Best Bet from a keyword

  • Remove all Visual Best Bets from a keyword

  • Add a global Visual Best Bet

  • Add multiple global Visual Best Bets

  • Remove a global Visual Best Bet

  • Remove all global Visual Best Bets

  • Display all Visual Best Bets

Add a Visual Best Bet to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a Visual Best Bet to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$uri = New-Object -TypeName System.Uri -ArgumentList "<URI>"
$visualBestBet = $keyword.AddFeaturedContent("<VisualBestBetName>")
$visualBestBet.Uri = $uri

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <URI> is the URI to the content of the Visual Best Bet.

    • <VisualBestBetName> is the title of the Visual Best Bet to be added.

See also

Keyword.AddFeaturedContent Method

Get-FASTSearchSearchSettingGroup

Add multiple Visual Best Bets to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of existing Visual Best Bets that you want to add:

    Copy Code
    $visualBestBetsToBeAdded = @("<VisualBestBet1>", "<VisualBestBet2>", "<VisualBestBet3>", "<VisualBestBet4>")

    Where:

    • <VisualBestBet 1-4> are the Visual Best Bets that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of Visual Best Bets to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
foreach($visualBestBetToBeAdded in <$visualBestBetsToBeAdded>) {$keyword.AddFeaturedContent($visualBestBetToBeAdded)}

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <$visualBestBetsToBeAdded> is an existing variable that contains an existing array or collection of strings specifying the names of the existing Visual Best Bets to be added.

See also

FeaturedContentCollection Interface

Keyword.AddFeaturedContent Method

Get-FASTSearchSearchSettingGroup

Remove a Visual Best Bet from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a Visual Best Bets from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$keyword.RemoveFeaturedContent("<VisualBestBetName>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <VisualBestBetName> is the name of the Visual Best Bet to be removed.

See also

Keyword.RemoveFeaturedContent Method

Get-FASTSearchSearchSettingGroup

Remove all Visual Best Bets from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all Visual Best Bets from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$keyword.FeaturedContent.Clear()

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add a global Visual Best Bet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a global Visual Best Bet:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.FeaturedContentWithoutKeyword.AddFeaturedContent("<GlobalVisualBestBetName>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <GlobalVisualBestBetName> is the title of the global Visual Best Bet to be added.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add multiple global Visual Best Bets

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of existing global Visual Best Bets that you want to add:

    Copy Code
    $globalVisualBestBetsToBeAdded = @("<GlobalVisualBestBet1>", "<GlobalVisualBestBet2>", "<GlobalVisualBestBet3>", "<GlobalVisualBestBet4>")

    Where:

    • <GlobalVisualBestBet 1-4> are the global Visual Best Bets that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of global Visual Best Bets:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
foreach ($globalVisualBestBetToBeAdded in <$globalVisualBestBetsToBeAdd>) { $searchSettingGroup.FeaturedContentWithoutKeyword.AddFeaturedContent( $globalVisualBestBetToBeAdded ) }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <$globalVisualBestBetsToBeAdded> is an existing variable that contains an existing array or collection of strings specifying the names of the existing global Visual Best Bets to be added.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a global Visual Best Bet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a global Visual Best Bets:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.FeaturedContentWithoutKeyword.RemoveFeaturedContent("<GlobalVisualBestBetName>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <GlobalVisualBestBetName> is the name of the global Visual Best Bet to be removed.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove all global Visual Best Bets

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all global Visual Best Bets:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.FeaturedContentWithoutKeyword.Clear()

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Display all Visual Best Bets

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Display all Visual Best Bets:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.FeaturedContent

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Keyword.FeaturedContent Property

Get-FASTSearchSearchSettingGroup


Add, remove and display document promotions for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Note:

This article describes how to manage document promotions by using Windows PowerShell. For information about how to manage document promotions by using the graphical user interface, see Manage FAST Search keywords (Office.com).

In this article the following Windows PowerShell procedures are described:

  • Add a document promotion to a keyword

  • Add multiple document promotions to a keyword

  • Remove a document promotion from a keyword

  • Remove all document promotions from a keyword

  • Add a global document promotion

  • Add multiple global document promotions

  • Remove a global document promotion

  • Remove all global document promotions

  • Display all promotions

Add a document promotion to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a document promotion to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$promotion = $keyword.AddPromotion("<PromotionName>")
$promotion.BoostValue = "<BoostValue>"
$promotion.PromotedItems.AddPromotedDocument("<DocumentId>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <PromotionName> is the title of the document promotion to be added.

    • <BoostValue> is the boost value to associate with the document promotion, for example 1000.

    • <DocumentId> is the document ID (URL) for the document to be promoted.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add multiple document promotions to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of existing document promotions that you want to add:

    Copy Code
    $promotedDocumentIds = @("<DocPromotionId1>", "<DocPromotionId2>", "<DocPromotionId3>", "<DocPromotionId4>")

    Where:

    • < DocPromotionId1-4> are the document promotions that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of document promotions to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$promotion = $keyword.AddPromotion("<PromotionName>")
$promotion.BoostValue = "<BoostValue>"
foreach ($promotedDocumentId in <$promotedDocumentIds>) { $promotion.PromotedItems.AddPromotedDocument($promotedDocumentId) }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <PromotionName> is the title of the promotion to be added.

    • <BoostValue> is the boost value to associate with the promotion, for example 1000.

    • <$promotedDocumentIds> is an existing variable that contains an array or collection of strings specifying the documents to be promoted with document ids (URIs.)

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a document promotion from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a document promotion from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$promotedItems = $keyword.Promotions["<PromotionName>"].PromotedItems
foreach ($promotedDocument in @($promotedItems.GetPromotedDocumentEnumerator())) { if ($promotedDocument.DocumentId -eq "<DocumentId>") {
$promotedItems.Remove($promotedDocument)}}

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <PromotionName> is the title of the document promotion to be removed.

    • <DocumentId> is the document ID (URL) of the document promotion to be removed.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove all document promotions from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all document promotions from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$promotedItems = $keyword.Promotions["<PromotionName>"].PromotedItems
foreach ($promotedDocument in @($promotedItems.GetPromotedDocumentEnumerator())) { $promotedItems.Remove($promotedDocument) }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <PromotionName> is the title of an existing promotion from which to remove all document promotions.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add a global document promotion

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a global document promotion:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
$globalPromotion = $globalPromotions.AddPromotion("<GlobalPromotionName>")
$globalPromotion.BoostValue = "<BoostValue>"
$globalPromotion.PromotedItems.AddPromotedDocument("<DocumentId>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <GlobalPromotionName> is the name of the global document promotion to be added.

    • <BoostValue> is the boost value to associate with the global document promotion, for example 1000.

    • <DocumentId> is the document ID (URL) of the document to be promoted.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add multiple global document promotions

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of existing global document promotions that you want to add:

    Copy Code
    $globalDocPromotionsToAdd = @("<GlobalDocPromotion1>", "<GlobalDocPromotion2>", "<GlobalDocPromotion3>", "<GlobalDocPromotion4>")

    Where:

    • <GlobalDocPromotion1-4> are the global document promotions that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of global document promotions:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
foreach ($promotionToAdd in <$globalDocPromotionsToAdd>) { $newPromotion = $globalPromotions.AddPromotion($promotionToAdd) ; $newPromotion.PromotedItems.AddPromotedDocument($promotionToAdd) }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <$globalDocPromotionsToAdd > is an existing variable that contains an existing array or collection of strings specifying the names of the existing global document promotions to be added.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a global document promotion

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a global document promotion:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
$globalPromotion = $globalPromotions.GetPromotion("<GlobalPromotionName>")
$promotedItems = $globalPromotion.PromotedItems
foreach ($promotedDocument in @($promotedItems.GetPromotedDocumentEnumerator())) { if ($promotedDocument.DocumentId -eq "<DocumentId>") { $promotedItems.Remove($promotedDocument) }}

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <GlobalPromotionName> is the name of an existing promotion from which to remove a global document promotion.

    • <DocumentId> is the document ID (URL) of the global document promotion to be removed.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove all global document promotions

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all global document promotions:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
foreach ($globalPromotion in $globalPromotions) { $globalPromotedItems = $globalPromotion.PromotedItems;  foreach ($promotedDoc in @($globalPromotedItems.GetPromotedDocumentEnumerator())) { $globalPromotedItems.Remove($promotedDoc) } }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Display all promotions

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Display all promotions:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.promotions

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup


Add, remove and display site promotions for a keyword by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Note:

This article describes how to manage site promotions by using Windows PowerShell. For information about how to manage site promotions by using the graphical user interface, see Promoting or demoting sites (Office.com).

In this article the following Windows PowerShell procedures are described:

  • Add a site promotion to a keyword

  • Add multiple site promotions to a keyword

  • Remove a site promotion from a keyword

  • Remove all site promotions from a keyword

  • Add a global site promotion

  • Add multiple global site promotions

  • Remove a global site promotion

  • Remove all global site promotions

  • Display all promotions

Add a site promotion to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a site promotion to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$promotion = $keyword.AddPromotion("<PromotionName>")
$promotion.BoostValue = "<BoostValue>"
$uri = New-Object -TypeName System.Uri -ArgumentList "<URI>"
$promotion.PromotedItems.AddPromotedLocation($uri)

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <PromotionName> is the title of the site promotion to be added.

    • <BoostValue> is the boost value to associate with the site promotion, for example 1000.

    • <URI> is the URI of the site that is to be promoted.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add multiple site promotions to a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of existing site promotions that you want to add:

    Copy Code
    $sitePromotionsToBeAdded = @("<SitePromotion1>", "<SitePromotion2>", "<SitePromotion3>", "<SitePromotion4>")

    Where:

    • <SitePromotion1-4> are the site promotions that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of site promotions to a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
foreach ($uriToAdd in <$sitePromotionsToBeAdded>) { $sitePromotion = $keyword.AddPromotion( $uriToAdd ) ;
$uri = New-Object -TypeName System.Uri -ArgumentList $uriToAdd ; $sitePromotion.PromotedItems.AddPromotedLocation( $uri ) }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <$sitePromotionsToBeAdded > is an existing variable that contains a collection of strings of the existing site promotions you want to add.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a site promotion from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a site promotion from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$promotedItems = $keyword.Promotions["<PromotionName>"].PromotedItems
$uri = New-Object -TypeName System.Uri -ArgumentList "<URI>"
foreach ($promotedLocation in @($promotedItems.GetPromotedLocationEnumerator())) { if ($promotedLocation.Uri -eq $uri) { $promotedItems.Remove($promotedLocation) } }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <PromotionName> is the title of the site promotion to be removed.

    • <URI> is the URI of the site promotion that is to be removed.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove all site promotions from a keyword

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all site promotions from a keyword:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
foreach ($promotion in $keyword.Promotions) { $promotedItems = $promotion.PromotedItems; foreach
($promotedLocation in @($promotedItems.GetPromotedLocationEnumerator() ) )
{ $promotedItems.Remove( $promotedLocation ) } }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add a global site promotion

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a global site promotion:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
$globalPromotion = $globalPromotions.AddPromotion("<GlobalPromotionName>")
$globalPromotion.BoostValue = "<BoostValue>"
$uri = New-Object -TypeName System.Uri -ArgumentList <URI>
$globalPromotion.PromotedItems.AddPromotedLocation($uri)

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <GlobalPromotionName> is the name of the global site promotion to be added.

    • <BoostValue> is the boost value to associate with the global site promotion, for example 1000.

    • <URI> is the URI of the site that is to be promoted.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add multiple global site promotions

Note:

In order to add multiple global site promotions you have to create a collection of the global site promotions you want to add.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, create a collection of existing global site promotions that you want to add:

    Copy Code
    $globalSitePromotionsToBeAdded = @("<GlobalSitePromotion1>", "<GlobalSitePromotion2>", "<GlobalSitePromotion3>", "<GlobalSitePromotion4>")

    Where:

    • <GlobalSitePromotion1-4> are the site promotions that you want to add.

  6. Get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  7. Add the collection of global site promotions:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
foreach ($uriToAdd in <$globalSitePromotionsToBeAdded>) { $sitePromotion = $globalPromotions.AddPromotion( $uriToAdd ) ; $uri = New-Object -TypeName System.Uri -ArgumentList $uriToAdd ; $sitePromotion.PromotedItems.AddPromotedLocation( $uri ) }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <$globalSitePromotionsToBeAdded> is an existing variable that contains a collection of strings of the existing global site promotions you want to add.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a global site promotion

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a global site promotion:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
$globalPromotion = $globalPromotions.GetPromotion("<GlobalPromotionName>")
$promotedItems = $globalPromotion.PromotedItems
$uri = New-Object -TypeName System.Uri -ArgumentList <URI>
foreach ($promotedLocation in @($promotedItems.GetPromotedLocationEnumerator())) { if ($promotedLocation.Uri -eq $uri) { $promotedItems.Remove($promotedLocation) } }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <GlobalPromotionName> is the name of an existing global promotion from which to remove a global site promotion.

    • <URI> is the URI of the global site promotion to be removed.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove all global site promotions

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all global site promotions:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$globalPromotions = $searchSettingGroup.PromotionsWithoutKeyword
foreach ($globalPromotion in $globalPromotions) { $globalPromotedItems = $globalPromotion.PromotedItems;  foreach ($promotedLoc in @($globalPromotedItems.GetPromotedLocationEnumerator())) {  $globalPromotedItems.Remove($promotedLoc) } }

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Display all promotions

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Display all promotions:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.promotions

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup


Add, remove and display user contexts by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

This article describes how to manage user contexts by using Windows PowerShell. For information about how to manage user contexts by using the graphical user interface, see Manage FAST Search user contexts (Office.com).

In this article the following Windows PowerShell procedures are described:

  • Create a user context

  • Display all user contexts

  • Remove a user context

  • Remove all user contexts

  • Add a user context to a synonym

  • Remove a user context from a synonym

  • Add a user context to a Best Bet

  • Remove a user context from a Best Bet

  • Add a user context to a Visual Best Bet

  • Remove a user context from a Visual Best Bet

  • Add a user context to a document or site promotion

  • Remove a user context from a document or site promotion

Create a user context

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Create a user context:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$context = $searchSettingGroup.Contexts.AddContext("<ContextName>")
$andExpression = $context.AddAndExpression()
$orExpression = $andExpression.AddOrExpression()
$orExpression.AddMatchExpression("<PropertyName0>","<PropertyValue0>")
$orExpression.AddMatchExpression("<PropertyName1>","<PropertyValue1>")
$notExpression = $andExpression.AddNotExpression()
$notExpression.AddMatchExpression("<PropertyName2>","<PropertyValue2>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <ContextName> is the name of the user context you want to create.

    • <PropertyName0> is the name of a user context property to be added to the context.

    • <PropertyValue0> is the value of the corresponding user context property (<PropertyName0>) to be used in the context expression.

    • <PropertyName1> is the name of a user context property to be added to context.

    • <PropertyValue1> is the value of the corresponding user context property (<PropertyName1>) to be used in the context expression.

    • <PropertyName2> is the name of a user context property to be added to context.

    • <PropertyValue2> is the value of the corresponding user context property (<PropertyName2>) to be used in the context expression.

Note:

A user context forms a logical match expression which is evaluated when executing a search. You can make the expression arbitrarily complex by adding AND, OR, and NOT expressions. The procedure above creates an expression with the form: (<PropertyName0>:<PropertyValue0>OR<PropertyName1>:<PropertyValue1>) AND (NOT (<PropertyName2>:<PropertyName2>))

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Display all user contexts

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Display all user contexts:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Contexts

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a user context

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a user context:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Contexts.RemoveContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <ContextName> is the name of the user context you want to remove.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove all user contexts

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove all user contexts:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Contexts.Clear()

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add a user context to a synonym

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a user context to a synonym:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$synonym = $keyword.Synonyms.GetSynonym("<SynonymTerm>")
$synonym.Contexts.AddContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <SynonymTerm> is the synonym term that you want to add a user context to.

    • <ContextName> is the name of an existing user context to be added to the synonym.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a user context from a synonym

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a user context from a synonym:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$synonym = $keyword.Synonyms.GetSynonym("<SynonymTerm>")
$synonym.Contexts.AddContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <SynonymTerm> is the synonym term that you want to remove a user context from.

    • <ContextName> is the name of an existing user context to be removed from the synonym.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add a user context to a Best Bet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a user context to a Best Bet:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$bestBet = $keyword.BestBets.GetBestBet("<BestBetName>")
$bestBet.Contexts.AddContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <BestBetName> is the name of the Best Bet you want to add a user context to.

    • <ContextName> is the name of an existing user context to be added to the Best Bet.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a user context from a Best Bet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a user context from a Best Bet:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$bestBet = $keyword.BestBets.GetBestBet("<BestBetName>")
$bestBet.Contexts.RemoveContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <BestBetName> is the name of the Best Bet you want to remove a user context from.

    • <ContextName> is the name of an existing user context to be removed from the Best Bet.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add a user context to a Visual Best Bet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a user context to a Visual Best Bet:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$visualBestBet = $keyword.FeaturedContent.GetFeaturedContent ("<VisualBestBetName>")
$visualBestBet.Contexts.AddContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <VisualBestBetName> is the name of the Visual Best Bet you want to add a user context to.

    • <ContextName> is the name of an existing user context to be added to the Visual Best Bet.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a user context from a Visual Best Bet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a user context from a Visual Best Bet:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$visualBestBet = $keyword.FeaturedContent.GetFeaturedContent("<VisualBestBetName>")
$visualBestBet.Contexts.RemoveContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <VisualBestBetName> is the name of the Visual Best Bet you want to remove a user context from.

    • <ContextName> is the name of an existing user context to be removed from the Visual Best Bet.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Add a user context to a document or site promotion

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Add a user context to a document or site promotion:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$promotion = $keyword.Promotions["<PromotionName>"]
$promotion.Contexts.AddContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <PromotionName> is the name of the document or site promotion you want to add a user context to.

    • <ContextName> is the name of an existing user context to be added to the document or site promotion.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup

Remove a user context from a document or site promotion

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, get all search setting groups:

    Copy Code
    Get-FASTSearchSearchSettingGroup

>>Name        : 9c65a55f-eafc-451d-b0f5-56fbed465141
>>Description :
>>Id          : 1
>>LastChanged : 10/19/2010 9:04:46 AM

  6. Remove a user context from a document from site promotion:

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$keyword = $searchSettingGroup.Keywords.GetKeyword("<KeywordTerm>")
$promotion = $keyword.Promotions["<PromotionName>"]
$promotion.Contexts.RemoveContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of the search setting group.

    • <KeywordTerm> is the keyword term string name of an existing keyword.

    • <PromotionName> is the name of the document or site promotion you want to remove a user context from.

    • <ContextName> is the name of an existing user context to be removed from a document or site promotion.

See also

Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

Get-FASTSearchSearchSettingGroup


Tune linguistic relevance (FAST Search Server 2010 for SharePoint)

In this section:

  • Enable stemming for a managed property by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Add, remove and display spell checking exceptions by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • East Asian language specific tokenization tasks by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

See also

Linguistic features (FAST Search Server 2010 for SharePoint)

Add, remove and display synonyms for keywords by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Linguistic features per language (FAST Search Server 2010 for SharePoint)


Enable stemming for a managed property by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Published: September 16, 2010

Tip:

As a best practice, you should only enable stemming for a managed property if the property contains free text with inflected noun/adjective forms or user searches this property using inflectional variants.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $mp = Get-FASTSearchMetadataManagedProperty -name <ManagedProperty>

    Where:

    • <ManagedProperty> is the name of the managed property for which you want to enable stemming, for example DocComments.

  6. Enable stemming:

    Copy Code
    $mp.StemmingEnabled=1

  7. Update the managed property:

    Copy Code
    $mp.update()

  8. Verify that stemming was enabled for the managed property:

    Copy Code
    $mp

>> Name                                   :         DocComments
>> Description                            :
>> Type                                   :         Text
>> Queryable                              :         True
>> StemmingEnabled                        :         True
>> RefinementEnabled                      :         False
>> MergeCrawledPfopertiesAuthorityWeight  :         False
>> SubstringEnabled                       :         False
>> DeleteDisallowed                       :         False
>> MappingDisallowed                      :         False
>> MaxIndexSize                           :         1024
>> MaxResultSize                          :         64
>> DecimalPlaces                          :         3
>> SortableType                           :         SortableDisabled
>> SummaryType                            :         Disabled

See also

Query.EnableStemming Property


Add, remove and display spell checking exceptions by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Published: September 16, 2010

Note:

This article describes how to manage spell checking by using Windows PowerShell. For information about how to manage spell checking by using the graphical user interface, see Manage Spell Checking (Office.com).

In this article the following Windows PowerShell procedures are described:

  • Add a spell checking exception

  • Add multiple spell checking exceptions

  • Remove a spell checking exception

  • Remove all spell checking exceptions

  • Display all spell checking exceptions

Add a spell checking exception

Important:

Every entry in the spell checking exception list must be a single word. Multi-word entries or phrases are not supported.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $spellCheckerContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.SpellCheckerContext
$spellCheckers = $spellCheckerContext.SpellCheckers
$spellChecker
foreach ($checker in $spellCheckers) { if ($checker.Name -eq "spell check exclusion") { $spellChecker = $checker } }
$spellChecker.Exclusions.Add("<ExclusionTerm>")

    Where:

    • <ExclusionTerm> is the term that you want to excluded from spell checking.

See also

SpellCheckerContext Class

Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)

Add multiple spell checking exceptions

Note:

In order to add multiple spell checking exceptions, you have to create a collection of the exceptions that you want to add.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $spellCheckerContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.SpellCheckerContext
$spellCheckers = $spellCheckerContext.SpellCheckers
$spellChecker
foreach ($checker in $spellCheckers) { if ($checker.Name -eq "spell check exclusion") { $spellChecker = $checker } }
foreach ($exclusionTermToAdd in <$exclusionTermsToAdd>) { $spellChecker.Exclusions.Add($exclusionTermToAdd) }

    Where:

    • <$exclusionTermsToAdd> is a variable that contains an existing array or collection of strings specifying existing spell checking exceptions to be excluded from spell checking.

See also

SpellCheckerContext Class

Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)

Remove a spell checking exception

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $spellCheckerContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.SpellCheckerContext
$spellCheckers = $spellCheckerContext.SpellCheckers
$spellChecker
foreach ($checker in $spellCheckers) { if ($checker.Name -eq "spell check exclusion") { $spellChecker = $checker } }
$spellChecker.Exclusions.Remove("<ExclusionTerm>")

    Where:

    • <ExclusionTerm > is the term that you want to remove from the spell checking exception list.

See also

SpellCheckerContext Class

Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)

Remove all spell checking exceptions

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $spellCheckerContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.SpellCheckerContext
$spellCheckers = $spellCheckerContext.SpellCheckers
foreach ($checker in $spellCheckers) { if ($checker.Name -eq "spell check exclusion") { $checker.Exclusions.Clear() } }

See also

SpellCheckerContext Class

Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)

Display all spell checking exceptions

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $spellCheckerContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.SpellCheckerContext
$spellCheckers = $spellCheckerContext.SpellCheckers
$spellChecker
foreach ($checker in $spellCheckers) { if ($checker.Name -eq "spell check exclusion") { $spellChecker = $checker } }

See also

SpellCheckerContext Class

Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)


East Asian language specific tokenization tasks by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Published: September 16, 2010

In FAST Search Server 2010 for SharePoint you can influence the default tokenization using two methods: linguistic tokenization and substring tokenization.

Linguistic tokenization

Linguistic tokenization means that a string of text is split into individual tokens based on language-specific rules. For East Asian languages, you can influence tokenization by creating custom dictionaries. If words are missing from the system dictionary provided by FAST Search Server 2010 for SharePoint, for example technical terms, person names or company names, or if the default tokenization is incorrect, you can add words to the custom dictionary to ensure that they are tokenized as required.

Substring tokenization

Substring tokenization is especially useful for applications where recall is very important. Substring tokenization removes all spaces from the text and then splits it into bigrams (overlapping two character long tokens). For example, "アメリカ" (America) is split up into: ア,アメ,メリ,リカ (a, ame, meri, ca).

Substring tokenization increases recall, but reduces precision and greatly increases the size of the index. You should not use substring tokenization if it is more important to keep the index size down than to increase recall. To minimize the drop in precision, you can use a combination of substring tokenization and linguistic tokenization.

Tip:

If you use substring tokenization only without any linguistic tokenization, the precision will be very low. Typically hits in the linguistic tokenized index are ranked higher than hits in the substring tokenized index, so a combination of the two tokenization types will give the best precision and recall.

In this article:

  • Create a custom dictionary for East Asian word breakers (linguistic tokenization)

  • Configure substring tokenization

  • Configure a combination of linguistic tokenization and substring tokenization

Create a custom dictionary for East Asian word breakers (linguistic tokenization)

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. Open a new file in a text editor.

  3. Type the words that you want in the custom dictionary according to the following rules:

    • The first line must be #CUSTOMER_WB

    • Do not use the # character as the first character of an entry.

    • Do not use spaces in entries.

    • Use a forward slash (/) to mark the token boundaries in a compound word.

      Examples:

      • English: book/shelf/cover

      • Japanese/Chinese: 朝鲜/民主/主義/人民/共和國

      • Thai: กระทรวง/มหาด/ไทย

    • Use the following file names:

      Language File name

      Japanese

      custom0011.lex

      Thai

      custom001e.lex

      Chinese Simplified

      custom0804.lex

      Chinese Traditional

      custom0404.lex

  4. On the File menu, click Save As.

  5. In the Save as type list, select All Files.

  6. In the Encoding list, select Unicode.

  7. Save the custom dictionaries in the <FASTSearchInstallationFolder>\lib\nlg\ folder on each server where FAST Search Server 2010 for SharePoint search or item processing is performed (<FASTSearchInstallationFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch).

  8. On the Start menu, click All Programs.

  9. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  10. At the Windows PowerShell command prompt restart item processing and query processing by typing the following commands:

    Copy Code
    psctrl reset
nctrl restart qrserver

  11. Re-crawl all content to enable the customization for the indexed content.

Note:

Words in the custom dictionaries are given absolute priority during tokenization. If a word is found in the custom dictionary, the same word in a document is guaranteed to be tokenized as defined in the custom dictionary.

Configure substring tokenization

Use this procedure to disable linguistic tokenization and enable substring tokenization:

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the FAST Search Server 2010 for SharePoint administration server, open <FASTSearchInstallationFolder> components\admin-services\web.config in a text editor (<FASTSearchInstallationFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch). Enable automatic re-indexing of all items by setting the value of AllowIndexPurgeOnSchemaUpdate to yes.

  3. Save the file.

  4. On the Start menu, click All Programs.

  5. Click Microsoft FAST Search Server 2010 for SharePoint.

  6. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  7. At the Windows PowerShell command promt, type the following command(s):

    Copy Code
    $mp= Get-FASTSearchMetadataManagedProperty -name <ManagedPropertyName>

    Where:

    • <ManagedPropertyName> is the name of the managed property that you want to configure substring tokenization for, for example title.

  8. View the properties of the managed property:

    Copy Code
    $mp

>> Name                                   :         Title
>> Description                            :         The title of the document
>> Type                                   :         Text
>> Queryable                              :         True
>> StemmingEnabled                        :         True
>> RefinementEnabled                      :         False
>> MergeCrawledPfopertiesAuthorityWeight  :         False
>> SubstringEnabled                       :         False
>> DeleteDisallowed                       :         True
>> MappingDisallowed                      :         False
>> MaxIndexSize                           :         1024
>> MaxResultSize                          :         64
>> DecimalPlaces                          :         3
>> SortableType                           :         SortableDisabled
>> SummaryType                            :         Dynamic

  9. Disable stemming:

    Copy Code
    $mp.StemmingEnabled=0

  10. Enable substring:

    Copy Code
    $mp.SubstringEnabled=1

  11. Update the managed property:

    Copy Code
    $mp.Update()

  12. Verify that the changes were made:

    Copy Code
    $mp

>> Name                                   :         Title
>> Description                            :         The title of the document
>> Type                                   :         Text
>> Queryable                              :         True
>> StemmingEnabled                        :         False
>> RefinementEnabled                      :         False
>> MergeCrawledPfopertiesAuthorityWeight  :         False
>> SubstringEnabled                       :         True
>> DeleteDisallowed                       :         True
>> MappingDisallowed                      :         False
>> MaxIndexSize                           :         1024
>> MaxResultSize                          :         64
>> DecimalPlaces                          :         3
>> SortableType                           :         SortableDisabled
>> SummaryType                            :         Dynamic

    It can take several minutes before substring tokenization is enabled after you have run the Update() command. Wait several minutes before you feed any documents.

  13. Re-crawl all content.

Configure a combination of linguistic tokenization and substring tokenization

To combine the two tokenization modes, you must create two full-text indexes and two sets of managed properties. One index/property pair will have substring tokenization enabled (SubstringEnabled=1) and one pair will have linguistic tokenization enabled (StemmingEnabled=1).

If you want the two managed properties to be assigned a different rank in the search results, the two managed properties must be mapped to different full-text indexes.

The following example procedure describes how to apply both substring tokenization and linguistic tokenization to a managed property:

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the FAST Search Server 2010 for SharePoint administration server, open <FASTSearchInstallationFolder>\components\admin-services\web.config in a text editor (<FASTSearchInstallationFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch). Enable automatic re-indexing of all items by setting the value of AllowIndexPurgeOnSchemaUpdate to yes.

  3. Save the file.

  4. On the Start menu, click All Programs.

  5. Click Microsoft FAST Search Server 2010 for SharePoint.

  6. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  7. At the Windows PowerShell command prompt, type the flowing commands(s).

    Copy Code
    Get-FASTSearchMetadataFullTextIndexMapping -ManagedProperty (Get-FASTSearchMetadataManagedProperty -name <ManagedProperty>)

>> ImportanceLevel:     :  7
>> ManagedProperty:     :  Title
>> FullTextIndex:       :  content

    Where:

    • <ManagedProperty> is the name of the managed property that you want to configure linguistic- and substring tokenization for, for example title.

  8. Create a new managed property that has the same content as the managed property above:

    Copy Code
    $mpSubstring=New-FASTSearchMetadataManagedProperty -name <ManagedProperty>Substring -type 1

    Where:

    • <Managed_Property> is the managed property that you want to use the content of to create a new managed property, for example title.

  9. View the properties of the new managed property:

    Copy Code
    $titleSubstring

>> Name                                   :         titleSubstring
>> Description                            :
>> Type                                   :         Text
>> Queryable                              :         True
>> StemmingEnabled                        :         False
>> RefinementEnabled                      :         False
>> MergeCrawledPfopertiesAuthorityWeight  :         False
>> SubstringEnabled                       :         False
>> DeleteDisallowed                       :         False
>> MappingDisallowed                      :         False
>> MaxIndexSize                           :         1024
>> MaxResultSize                          :         64
>> DecimalPlaces                          :         3
>> SortableType                           :         SortableDisabled
>> SummaryType                            :         Static

  10. Disable linguistic tokenization for the new managed property:

    Copy Code
    $mpSubstring.StemmingEnabled=0

  11. Enable substring tokenization for the new managed property:

    Copy Code
    $mpSubstring.SubstringEnabled=1

  12. Update the new managed property:

    Copy Code
    $mpSubstring.update()

  13. To ensure that the content of the two managed properties are the same, map all the crawled properties of the first managed property to the newly created managed property:

    Copy Code
    Get-FASTSearchMetadataCrawledPropertyMapping -name <ManagedProperty>| ForEach-Object { New-FASTSearchMetadataCrawledPropertyMapping -managedproperty $mpSubstring -crawledproperty $_}

    Where:

    • <ManagedProperty> is the managed property that you used to create the new managed property, for example title.

  14. Create a new full-text index with lowercase characters only for the name of the full-text index:

    Copy Code
    $mpSubstringIndex= New-FASTSearchMetadataFullTextIndex -Name <NewFullTextIndex> -Description "Title Substring"

    Where:

    • <NewFullTextIndex> is the name of the new full text index, for example titlesubstringindex.

  15. View the new full text index:

    Copy Code
    $mpSubstringIndex

>> Name                :   titlesubstringindex
>> Description         :   Title Substring
>> StemmingEnabled:    :   True
>> isDefault           :   False
>> DeleteDisallowed:   :   False

  16. Disable linguistic tokenization for the new full-text index:

    Copy Code
    $mpSubstringIndex.StemmingEnabled=0

  17. Update the new full text index:

    Copy Code
    $mpSubstringIndex.Update()

  18. Verify that the new full text index was updated:

    Copy Code
    $mpSubstringIndex

>> Name                :   titlesubstringindex
>> Description         :   Title Substring
>> StemmingEnabled:    :   False
>> isDefault           :   False
>> DeleteDisallowed:   :   False

  19. Map the newly created substring-enabled managed property to the newly created substring-enabled full-text index:

    Copy Code
    New-FASTSearchMetadataFullTextIndexMapping -ManagedProperty $<NewManagedProperty> -FullTextIndex $<NewFullTextIndex> -ImportanceLevel 1

    Where:

    • <NewManagedProperty> is the name of the newly created managed property, for example titleSubstring.

    • <NewFullTextIndex> is the name of the newly created full text index, for example titlesubstringindex.

  20. Create a new rank profile to contain both indexes:

    Copy Code
    $NewRankProfile=New-FASTSearchMetadataRankProfile -Name <NewRankProfile>

    Where

    • <NewRankProfile> is the name of the new rank profile, for example DualRankProfile.

  21. Determine if your full-text index has a pre-defined rank component in the newly created rank profile:

    Copy Code
    $DualRankProfile.GetFullTextIndexRanks()

>> FullTextIndexReference    :  content
>> ProximityWeight           :  50
>> ContextWeight:            :  50

    In a default installation there is one pre-defined rank component only, for the full-text index named content.

  22. Create index rank components for the newly created full-text index that do not already have an index rank:

    Copy Code
    $mpSubstringIndex= get-FASTSearchMetadataFullTextIndex -Name <NewFullTextIndex>
$DualRankProfile.CreateFullTextIndexRankComponent($<NewFullTextIndex>)

    Where:

    • <NewFullTextIndex> is the name of the full text index you just created, for example titlesubstringindex.

  23. Verify that both full-text indexes have a full-text index rank component in the rank profile:

    Copy Code
    $DualRankProfile.GetFullTextIndexRanks()

>> FullTextIndexReference    :  content
>> ProximityWeight           :  50
>> ContextWeight:            :  50

>> FullTextIndexReference    :  titlesubstringindex
>> ProximityWeight           :  140
>> ContextWeight:            :  50

  24. The full-text index that contains the linguistically tokenized words (content) should be ranked higher than the full-text index with substring-tokenized words (titlesubstringindex).

    To rank the full index that contains the linguistically tokenized words higher than the full-text index with substring-tokenized words, set the context weight of the substring-enabled full-text index lower than that of the linguistic-tokenized full-text index:

    Copy Code
    $ranks = $DualRankProfile.GetFullTextIndexRanks()
$ranks|Where-Object -filterscript {$_.FullTextIndexReference.Name -eq "titlesubstringindex"}|ForEach-Object {$_.ContextWeight=<ContextWeight>; $_.Update()}

    Where:

    • <ContextWeight> is the context weight you want the new full text index (titlesubstringindex) to have relative to the default full text index (content), for example 30.

  25. Verify that the context weight was updated:

    Copy Code
    $DualRankProfile.GetFullTextIndexRanks()

>> FullTextIndexReference    :  content
>> ProximityWeight           :  50
>> ContextWeight:            :  50

>> FullTextIndexReference    :  titlesubstringindex
>> ProximityWeight           :  140
>> ContextWeight:            :  30


Manage content feeding and freshness (FAST Search Server for SharePoint)
See Also

There are two important aspects of search relevance that are related to the document feeding process:

  • Get the relevant data, and

  • Get it with the required freshness

You can configure relevance of items crawled by the FAST Search connector Content Search Service Application by configuring relevance settings on each content source.

Tuning the FAST Search connector Content Search Service Application

To define a content source and set crawl rules for the FAST Search connector Content SSA, from the SharePoint Server Central Admin home page, select Application Management > Manage Service Applications. In the list of Service Applications, click the name of your Content SSA. Then, from the Crawling menu, select Content sources > New Content Source. Or select Content sources from the Crawling menu and then select the content source that you want to change the relevance settings for if you have already configured a content source but want to improve relevance.

There are several possibilities for configuration:

  • Only include the start addresses that are relevant for search when you define the content source.

  • Differentiate priorities among your crawled sources, run as either normal or high priority. The most relevant content should receive a high priority.

  • Configure the crawl schedule for the content source to run in sync with the freshness requirements that you have defined.

  • Make sure that each crawl cycle is not too large to complete within the defined crawl cycles that are scheduled.

  • Use the Crawl Rules option from the Crawling menu to use regular expressions to include and exclude content from search.

See Also

Concepts

Manage crawl rules (FAST Search Server 2010 for SharePoint)
Add, edit, or delete a content source (FAST Search Server 2010 for SharePoint)


Manage the Search Center (FAST Search Server 2010 for SharePoint)
See Also

Published: May 5, 2011

The FAST Search Center is the user interface where end-users enter their searches and view the corresponding search results. The Search Center includes several components that you can customize to improve the end-user search experience.

In this section:

  • Manage the Search Center Home page (FAST Search Server 2010 for SharePoint)

  • Manage the Search Results page (FAST Search Server 2010 for SharePoint)

See Also

Concepts

Create a FAST Search Center site (FAST Search Server 2010 for SharePoint)

Other Resources

Customizing the Enterprise Search UI


Manage the Search Center Home page (FAST Search Server 2010 for SharePoint)
See Also

Published: May 5, 2011

In a default installation, the Search Center home page is called http://<ServerName>/default.aspx. The field where the end-users enter the search term is called the Search Box Web Part.

Change properties for the Search Box Web Part

By changing properties for the Search Box Web Part, you can for example:

  • Turn query suggestions on or off

  • Change the appearance of the search box

  • Specify whether to append keyword terms or property filters to all queries

For more information, see Change properties for the Search Box Web Part (SharePoint Server 2010)

Use search scopes to narrow a search to particular locations or content

By using search scopes, end-users can search across a narrower set of content. A search scope defines a subset of information in the searchable index, and can for example direct a query to particular sites, or to items marked with particular property values.

End-users can apply the search scope by selecting a pre-defined scope from a drop-down list next to the search box (part of the Search Box Web Part), or they can specify the search scope directly in the Keyword Syntax query string by using the special property filter scope:scopename, where scope is a pre-defined property name and scopename is the name of the defined search scope.

For more information about how to use search scopes to narrow a search, see Manage search scopes (FAST Search Server 2010 for SharePoint)

Display query suggestions as end-users type their query

Query suggestions, also known as search suggestions, help end-users see other, similar queries that have successfully returned results. A query term becomes a query suggestion when the term has returned a result that was clicked on at least six times by other users. For example, the query term FAST Search Server 2010 for SharePoint has returned a result that was clicked on over six times. When a user starts to type FAST in the Search Box Web Part, FAST Search for SharePoint will be displayed in a drop-down list below the Search Box.

Because query suggestions are based on end-users first searching for information and then clicking the results, it might take some time before query suggestions are displayed in your Search Center. However, you can manually create query suggestions by using Windows PowerShell.

For more information about how to use query suggestions, see Manage query suggestions (SharePoint Server 2010).

See Also

Other Resources

Customizing the Search Center


Manage the Search Results page (FAST Search Server 2010 for SharePoint)
See Also

Published: May 16, 2012

In a default installation, the Search Results page is called http://<ServerName>/results.aspx. The Search Results page includes several components that you can configure to improve the end-user experience.

Improve how search results are displayed

  • Change how search results appear in the Search Core Results Web Part

    The relevant results of an end-user's query are displayed in the Search Core Results Web Part. By configuring the properties in this Web Part, you can control how search results are displayed. You can for example:

    • Change the number of results that appear on a page

    • Change the number of characters in the summary description

    • Change the number of characters of the URL in search results

    For more information, see Change how search results appear in the Search Core Results Web Part (SharePoint Server 2010)

  • Enable document previews and the Did you mean feature

    The Search Core Results Web Part in FAST Search Server 2010 for SharePoint has some additional features that enable you to enhance the search results:

    • Show thumbnail preview for Word documents

    • Show document preview for PowerPoint documents

    • Enable a Did you mean feature

    For more information, see Configure additional features for the Search Core Results Web Part (FAST Search Server 2010 for SharePoint)

    Note:

    For thumbnail preview and document preview to work, Windows Integrated Mode authentication (NTLM or Kerberos) must be configured on the web application of the search site.

  • Get search results from items containing a synonym of the search term

    By adding Keywords and associating Synonyms to each keyword, you can increase recall, which is the number of search items returned related to a search. If a search includes a keyword, search items that contain the synonym are also returned, regardless of whether they contain the keyword. If a search includes a synonym for a keyword, search items that contain the related keyword are also returned. For example, if you have defined bike as keyword and associated the words bicycle and cycle as synonyms to this keyword, search items that contain the words bike and cycle are also returned when an end-user searches for bicycle.

    For more information about how to add keywords and synonyms, see Tune keyword rank (FAST Search Server 2010 for SharePoint).

  • Define relevant results to be displayed towards the top of the search results list

    By associating Best Bets, Visual Best Bets and Document Promotions to keywords, you can influence the search results so that results that you have identified as particularly relevant to a search term, are displayed towards the top of the search results list. When a search includes a keyword or one of its synonyms associated with a Best Bet, Visual Best Bet or Document Promotion, links to these associations are displayed in a prominent position on the Search Results page.

    For information on how to add Best Bets, Visual Bet Bets or Document Promotions, see Tune keyword rank (FAST Search Server 2010 for SharePoint).

  • Display search results specific to a user group

    User contexts enable targeted communication and targeted search improvements by displaying Best Bets, Visual Best Bets and Document Promotions to a specific user group only. User contexts match the properties that are defined on the user's SharePoint User Profile page. For example, a Visual Best Bet banner can be configured to be displayed only for employees who belong to the sales organization and who are located in Germany. The different search settings - Best Bets, Visual Best Bets, Document Promotions and Site Promotions - can have one or more user contexts associated with them.

    For information on how to manage user contexts, see Tune keyword rank (FAST Search Server 2010 for SharePoint).

  • Highlight search words entered by the end-user in the summary description

    In the search results list, for each matching search item, a text called summary description is displayed under the title of the search item. You can enable hit highlighted summary for selected managed properties. This will dynamically generate the summary description and highlight the search words that the end-user enters. For example, if an end-user searches for system requirement and a match is found in a managed property where the hit highlighted summary is enabled, the summary description displayed for this match would be "…another important point is the system requirement specified in the manual…".

    For information about how to enable this feature, see Manage hit highlighted summary for a managed property by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Change the number of search results returned per page

    If more than one page of search results is returned for a query, the Search Paging Web Part displays page numbers and forward and back arrows between the search result pages. You can edit different properties of the Search Paging Web Part, such as how many page links should be displayed, and how they are displayed.

    For more information, see Change properties for the Search Paging Web Part (SharePoint Server 2010).

Add sort options to search results

End-users can sort their search results by using the Sort by menu on the Search Results page. The default sort order for search results is Relevance, which is based on the default rank profile named default. You can give end-users additional options for sorting the search results by enabling additional rank profiles or managed properties as sort options.

For information about how to add sort options to search results, see Set a rank profile as sort option on the Search Results page via the graphical user interface (FAST Search Server 2010 for SharePoint) and Set a managed property as sort option on the Search Results page (FAST Search Server 2010 for SharePoint).

Use refiners to drill down in search results

Refiners enable end-users to drill down into their search results based on managed properties that are associated with the indexed search items, such as creation date, author, and company names. By using refiners, end-users can narrow their search results to only present search items like for example certain document types, created in a certain time period, authored by a given person. Refiners are displayed in the Refinement Panel Web Part.

For information about how to add refiners, see Manage refiners and refiner settings (FAST Search Server 2010 for SharePoint).

Display information from external sources by adding federations

By adding federated locations, you can expand the search results to also include information that exists outside your internal network. When adding federated locations, you enable queries to be sent to remote search engines and feeds, such as www.bing.com or wikipedia.com.

For information about how to add federations, see Manage federated locations (FAST Search Server 2010 for SharePoint).

Make it easy to find people

FAST Search Server 2010 for SharePoint provides an address book-style name lookup experience that makes it easier to find people by name, title and organizational structure. This includes phonetic name matching that will return names that sound similar to what the end-user has typed in a query. It will also return all variations of common names, such as nicknames (for supported languages).

For information about how to enable and configure people search, see Manage people search (FAST Search Server 2010 for SharePoint).

Display related queries

By displaying query suggestions in the Related Queries Web Part, you can help end-users see similar queries that have successfully returned results. A query term becomes a query suggestion when the query term has returned a result that was clicked on at least six times by other users. For example, the query term FAST Search Server for SharePoint has returned a result that was clicked on over six times. When a user searches for FAST, the query suggestion FAST Search Server for SharePoint will be displayed in the Related Queries Web Part on the Search Results page. Clicking on the query suggestion will run that query.

Because query suggestions are based on end-users first searching for information and then clicking the results, it might take some time before query suggestions are displayed in your Search Center. However, you can manually create query suggestions by using Windows PowerShell.

For more information about how to use related queries and query suggestions, see Manage query suggestions (SharePoint Server 2010).

See Also

Other Resources

Customizing Search Results


Configure additional features for the Search Core Results Web Part (FAST Search Server 2010 for SharePoint)

Published: May 5, 2011

The Search Core Results Web Part in FAST Search Server 2010 for SharePoint has some additional features that enable you to enhance the search results.

  1. Verify that the user account that is performing this procedure is a member of the Designers group.

  2. On the Search Center site, enter a search word and then click Search.

  3. On the Search Results page, click the Site Actions menu, and then click Edit Page.

  4. In the Search Core Results Web Part, click the edit arrow to display the web part menu and then click Edit Web Part.

  5. Use the information in the following table:

To do this | Perform these actions
Show thumbnail preview for Word documents | Under Display Properties, select Enable Thumbnail Preview for Word.
Show document preview for PowerPoint documents | Under Display Properties, select Enable Document Preview for PowerPoint.
Enable the Did you mean feature | Under Results Query Options in Display Properties, select Suggest from the Enable Spellchecking menu.
Enable options when a search returns zero results | Under Results Query Options, select one or more of the following options Resubmit With Spellcheck, Resubmit With Stemming and Suggest Spelling.
Note:

NTLM authentication must be configured for thumbnail preview and document preview to work.


Manage hit highlighted summary for a managed property by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Published: May 5, 2011

You can enable hit highlighted summary for selected managed properties. This will dynamically generate the summary description in the search results list and highlight the search words that are entered by the end-user. By default, hit highlighted summary is enabled for the managed properties body and title.

Caution:

Enabling hit highlighted summary for other managed properties than body and title will have an adverse performance effect on the query result creation, in particular if the managed property in average contains lots of text.

A key performance parameter that affects hit highlighted summary creation is the managed property parameter MaxResultSize. This parameter specifies the maximum number of kilobytes used to store the result of the managed property in the index. For managed properties that are not configured for hit highlighted summary, this parameter affects how much content is returned in the query results, with direct effect on query performance. In particular,r this applies for disk accesses and network I/O. For managed properties that are configured for hit highlighted summary, the MaxResultSize parameter affects the processing load of creating the hit highlighted summary for each hit in the query match list.

In this article:

  • Enable hit highlighted summary for a managed property

  • Set hit highlighted summary fallback for a managed property

Enable hit highlighted summary for a managed property

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    Set-FASTSearchMetadataManagedProperty -Name <ManagedProperty> -SummaryType <SummaryType>

    Where:

    • <ManagedProperty> is the name of the managed property that you want to enable hit highlighted summary for.

    • <SummaryType> is the value for the summary type that you want to use for the managed property, for example 2. Valid values:

      • 0 - Disabled. The managed property will not be returned as part of the result.

      • 1 - Static. The entire content of the managed property is returned (up until the value for the MaxResultSize property.)

      • 2 - Dynamic. A dynamic summary is generated based on the managed property content and the query and returned as part of the result.

See also

Set-FASTSearchMetadataManagedProperty

Set hit highlighted summary fallback for a managed property

If you have set the summary type of a managed property to 2 (dynamic), you can specify the static content of a managed property as a fallback. The fallback managed property will be used as the result summary if the system does not generate a dynamic summary. The fallback managed property can be the static summary of the same managed property, or the summary of another managed property.

By default, the managed property Title has a fallback reference to itself, and if it does not create a dynamic summary from the query, it will use its static summary instead. The following example sets the fallback reference for Title to the managed property DocSubject.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $title = Get-FASTSearchMetadataManagedProperty -Name Title
$docsubject = Get-FASTSearchMetadataManagedProperty DocSubject
$title.SetResultFallBack($docsubject)

See also

Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

Get-FASTSearchMetadataManagedProperty


Set a rank profile as sort option on the Search Results page via the graphical user interface (FAST Search Server 2010 for SharePoint)

Published: May 5, 2011

You can give end-users additional options for sorting the search results by changing the default sort option rank profile or by enabling additional rank profiles as sort options.

In this article:

  • Change the default sort option rank profile by using the graphical interface

  • Enable multiple rank profiles as sort options by using the graphical user interface

Change the default sort option rank profile by using the graphical interface

When a user types one or more search words into the search box on the search page (default.aspx) on the Search Center site, the search service returns the results displayed in various Web Parts on the Search Results page (results.aspx). If the search words did not include any keywords or associated synonyms, the order of the search results is based on the rank score calculated by the default rank profile. The label of the default rank profile is displayed in the Sort by pull-down menu. The default Search Results page uses the default rank profile, which is labeled Relevance. Use the following procedure to change the default rank profile via the graphical interface:

  1. On the Search Center site, enter a search word and then click Search.

  2. On the Search Results page, click Site Actions, and then click Edit Page.

  3. In the Search Action Links Web Part click the edit arrow to display the Web Part menu, and then click Edit Web Part.

  4. In the Web Part tool pane, expand Display Properties to display all existing rank profiles.

  5. Select the rank profile that you want to set as default by clicking the Default radio button and selecting the Enabled check box next to it.

  6. Click OK.

  7. Click Save & Close.

The new default rank profile will now appear in the Sort by menu.

Note:

The name that you gave the rank profile when you created it, is displayed in the ID column under Display Properties in the Search Action Link Web Part tool pane. You can change the name of the rank profile as displayed in the Sort by pull-down menu on the Search Results page by editing the name in the Label column.

See also

Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

Enable multiple rank profiles as sort options by using the graphical user interface

You can enable multiple rank profiles as sort options and display them in the Sort by pull-down menu on the Search Results page. This gives the end-user more options for filtering the search result. To enable multiple rank profiles as sort options:

  1. On the Search Center site, enter a search word and then click Search.

  2. On the Search Results page, click Site Actions menu, and then click Edit Page.

  3. From the Search Action Links Web Part click the edit arrow to display the Web Part menu, and then click Edit Web Part.

  4. In the Web Part tool pane, expand Display Properties to display all existing rank profiles.

  5. Select the Enabled check box next to the rank profiles that you want to make available as sort options.

  6. Click OK.

  7. Click Save & Close.

The enabled rank profiles will now appear in the Sort by menu.

Note:

The name that you gave the rank profile when you created it is displayed in the ID column under Display Properties in the Search Action Link Web Part tool pane. You can change the name of the rank profile as displayed in the Sort by pull-down menu on the Search Results page by editing the name in the Label column.

See also

Create and manage rank profiles by using Windows PowerShell (FAST Search Server 2010 for SharePoint)


Set a managed property as sort option on the Search Results page (FAST Search Server 2010 for SharePoint)
See Also

Published: May 5, 2011

You can set a managed property as sort option on the Search Result page. Enabling a managed property as a sort option gives the end-user the opportunity to sort the search results alphabetically. For example, by enabling the managed property Title as sort option, end-users can sort their search results in alphabetical order according to the title of the search items. Only managed properties of type string can be enabled as a sort option.

Caution:

Enabling many managed properties as sort options will have a significant adverse effect on memory usage in the query matching component.

Setting a managed property as a sort option is a two-step process: First, enable the managed property as a sort option, and then make it available in the web front-end.

In this article:

  • Enable a managed property as sort option by using Windows PowerShell

    You can also enable the managed property as a sort option by using the graphical user interface (GUI). For more information, see Property Management (office.com).

  • Make a managed property sort option available in the web front-end by using the graphical user interface

Enable a managed property as sort option by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $sort = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>

    Where:

    • <ManagedProperty> is the name of the managed property that you want to set as sort option, for example Author.

  6. Set sorting value:

    Copy Code
    Set-FASTSearchMetadataManagedProperty -ManagedProperty $sort -SortableType <ConfigurationValue>

    Where:

    • <ConfigurationValue> is one of the following values:

      • 0 - "Sorting disabled". The managed property is not sortable.

      • 1 - "Sorting enabled". The managed property is enabled for sorting.

      • 2 - "Latent sorting". The managed property is prepared for sorting, but must be changed to "Sorting enabled" for sorting to be supported.

Note:

In contrast to switching from "sorting disabled" to "sorting enabled", this does not require reprocessing of data.

Consider setting the configuration value to latent sorting if you want to prepare the index data structures for result-sorting but do not yet want to enable the feature for query evaluation. When using this option, the required data structures for result sorting are not loaded into main memory, and it will therefore have no performance effect. The setting can later be changed to enabled, and the change will have immediate effect (no requirement to re-index search items.)

Make a managed property sort option available in the web front-end by using the graphical user interface

  1. On the Search Center site, enter a search word and then click Search.

  2. On the Search Results page, click Site Actions, and then click Edit Page.

  3. In the Search Action Links Web Part click the edit arrow to display the Web Part menu, and then click Edit Web Part.

  4. In the Web Part tool pane, expand Display Properties.

  5. Select the Enabled check box next to the managed property that you want to set as sort option.

  6. Click OK.

  7. Click Save & Close.

    The enabled managed property will appear in the Sort by pull-down menu on the Search Results page. Both Ascending and Descending sort order will be enabled.

See Also

Reference

Set-FASTSearchMetadataManagedProperty


Manage refiners and refiner settings (FAST Search Server 2010 for SharePoint)

Published: May 5, 2011

Refiners enable end-users to drill down into their search results based on managed properties that are associated with the indexed search items, such as creation date, author, and company names.

FAST Search Server 2010 for SharePoint supports two kinds of query refiners: deep refiners and shallow refiners.

Deep refiners

Deep refinement is based on the aggregation of managed property statistics for all of the search results. The indexer creates aggregation data that is used in the query matching process. The advantage of using deep refiners is that the refinement options will reflect all the search items matching a query. The number of matching search items is displayed in parentheses behind each refiner. This is usually the recommended mode, but defining many deep refiners may have a significant adverse effect on memory usage in the query matching component.

Important:

Deep refiners that have many unique values will have significant adverse performance effect on internal I/O communication between the query matching node and the query processing node (if on different servers). If your installation has many index columns, this interface may become a bottleneck. In this case, consider using the configuration parameter CutoffMaxBuckets to limit the number of refinement bins to be evaluated on each index column.

Shallow refiners

Shallow refinement is based on the aggregation of managed property statistics for the top 100 results for a search query. The refinement result data is created during result processing. Because the refinement is limited to the top matching results, you may be unable to find results hidden deeper in the query results. However, this refinement option does not affect the indexing process and can therefore be applied immediately after it is enabled.

Shallow refiners will have significant adverse performance effect on the query processing node and will reduce the query performance. Consider using deep refiners instead.

Note:

When you create a managed property, it is not refinement-enabled by default.

In this article:

  • Set a managed property as a refiner on the Search Results page

  • Update refiner settings for a managed property by using Windows PowerShell

Set a managed property as a refiner on the Search Results page

Setting a managed property as refiner on the Search Results page is a two-step process. First, you must enable the managed property as a refiner (deep or shallow), and then you must make it available in the web-front end.

You can also set a managed property as a refiner by using the graphical user interface (GUI). For more information, see Property Management (office.com).

To enable a managed property as deep refiner by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    Set-FASTSearchMetadataManagedProperty -Name <Name> -RefinementEnabled <Flag>

    Where:

    • <Name> is the name of the managed property to be set as a refiner enabled.

    • <Flag> is a Boolean indicating whether the managed property should be enabled as a refiner. Use $true if it should be enabled, $false if it should not be enabled.

Note:

For managed properties of type Integer, you should not enable query refinement if the managed property may contain negative values. If a refinement bin (value range) contains negative values, it is not possible to drill down into this value range.

To enable a managed property as shallow refiner by using Windows PowerShell

  1. Set the managed property as refiner-enabled by following the steps under To enable a managed property as deep refiner by using Windows PowerShell

  2. Get the managed property that you want to set as shallow refiner:

    Copy Code
    $mp = Get-FASTSearchMetadataManagedProperty -Name <ManagedProperty>

    Where:

    • <ManagedProperty> is the name of the managed property that you want to set as shallow refiner.

  3. Get the refiner configuration:

    Copy Code
    $refiner = $mp.GetRefinerConfiguration()

  4. Set the managed property as shallow refiner:

    Copy Code
    $refiner.RefinementType = "DeepRefinementDisabled"

  5. Update the configuration:

    Copy Code
    $mp.SetRefinerConfiguration($refiner)

  6. Verify that the configuration was changed:

    Copy Code
    $refiner

>>  RefinementType       : DeepRefinementDisabled
>>  Divisor              :
>>  Intervals            :
>>  Resolution           :
>>  Algorithm            :
>>  DefaultValue         :
>>  CutoffMaxBuckets     : 1000
>>  Anchoring            : Auto
Note:

Consider setting the RefinementType property to LatentRefinement if you want to prepare the index data structures for deep refinement, but you do not want to enable the feature yet for query evaluation. When using this option, the required data structures for deep refinement are not loaded into main memory, and it will therefore have no performance effect. The setting can later be changed to DeepRefinementEnabled to enable the feature. In that case, the change has an immediate effect (no requirement to re-index items).

To make a refiner available in the web front-end by using the graphical user interface

  1. On the Search Center site, enter a search word and then click Search.

  2. On the Search results page, click Site Actions, and then click Edit Page.

  3. In the Refinement Panel Web Part, click the edit arrow to display the web part menu, and then click Edit Web Part.

  4. In the Refinement section, clear the Use Default Configuration check box.

  5. In the Refinement section, click in the field Filter Category Definition and then click the ellipsis (…) button.

    This opens the Text Editor window for the Refinement Web Part's FilterCategoriesDefinition property. You can modify the XML directly in this window, but you may find it easier to copy the code to a file and then edit that file by using an application such as Visual Studio 2010.

  6. Add your refiner to the file before the end tag for the FilterCategories element. The following is an example of an XML element for a refinement enabled managed property named wineterms. The Category Title, in this example, Wine terminology, will be displayed on the Search Results page as the refiner title.

    Copy Code
    <Category Title="Wine terminology"
          Description="Use this filter to restrict results by items related to wine"
          Type="Microsoft.Office.Server.Search.WebControls.ManagedPropertyFilterGenerator"
          MetadataThreshold="1"
          NumberOfFiltersToDisplay="4"
          MaxNumberOfFilters="20"
          ShowMoreLink="True"
          MappedProperty="wineterms"
          MoreLinkText="show more"
          LessLinkText="show fewer"
          ShowCounts="Count"/>
    Note:

    The value of the MappedProperty attribute must be in lowercase, even if the property name includes uppercase letters.

    For more information about the refiner properties, see the XML schema definition of the Category element.

  7. After you finish editing the file, you can copy the modified code back into the Text Entry window and save your changes.

See also

  • Specifying Refiners in a Query

  • Set-FASTSearchMetadataManagedProperty

Update refiner settings for a managed property by using Windows PowerShell

Refiners have different settings based on the type of the managed property they are enabled for. Refiner settings for managed properties of type Text will differ from refiner settings for managed properties that use numeric types.

The managed property size exists by default, and is used for the size of the documents. It is of type Integer and is refinement-enabled by default. The indexed values for size are in bytes, and by default the refiner is configured to display the value as kilobytes (Refiner.divisor=1024.) In the following example, the refiner is changed to display values in megabytes instead (1024*1024=1048576).

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $size = Get-FASTSearchMetadataManagedProperty -Name size
$sizerefiner = $size.GetRefinerConfiguration()
$sizerefiner.Divisor = 1048576
$size.SetRefinerConfiguration($sizerefiner)

See also

  • Specifying Refiners in a Query

  • Get-FASTSearchMetadataManagedProperty


Manage crawling (FAST Search Server 2010 for SharePoint)
See Also

This section describes how to configure and manage crawling.

Plan for crawling and federation (FAST Search Server 2010 for SharePoint) provides an overview of all the available indexing connectors, and gives guidance on when to use which indexing connector.

Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint) provides a number of considerations that should be made before deploying any of the FAST Search Server 2010 for SharePoint specific indexing connectors.

In this section:

Manage crawling with the FAST Search Content SSA (FAST Search Server 2010 for SharePoint)

Manage crawling with the FAST Search specific connectors (FAST Search Server 2010 for SharePoint)

See Also

Concepts

Plan for crawling and federation (FAST Search Server 2010 for SharePoint)
Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint)


Manage crawling with the FAST Search Content SSA (FAST Search Server 2010 for SharePoint)
See Also

In most scenarios, you can use the indexing connectors that are integrated in SharePoint Server 2010 to crawl content.

Before you can start to set up a crawl, you have to set up a FAST Search Content Search Service Application (Content SSA), enable SSL communication and create at least one content source.

A content source is a set of options that you can use to specify what type of content is crawled, what URLs to crawl, and how deep and when to crawl. You must create at least one content source before a crawl can occur. Depending on the type of content source, additional configuration may be required.

In order to feed content to the FAST Search Server 2010 for SharePoint content collection using the integrated indexing connectors, you must add at least one content source to the FAST Search Content Search Service Application. The default content collection name is sp.

The FAST Search Content SSA can have up to 500 content sources in it, and each content source can have a different configuration.

Note:

It is not necessary to configure the FAST Search Content SSA and set up a content source when you choose to deploy one or more FAST Search Server 2010 for SharePoint specific indexing connectors: the FAST Search Web crawler, the FAST Search database connector and/or the FAST Search Lotus Notes connector. These connectors require completing different configuration procedures.

In this section:

  • Add, edit, or delete a content source (FAST Search Server 2010 for SharePoint)

  • Content sources that require additional configuration (FAST Search Server 2010 for SharePoint)

  • Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint)

  • Manage crawl rules (FAST Search Server 2010 for SharePoint)

  • Manage crawler impact rules (FAST Search Server 2010 for SharePoint)

  • Verify the crawl (FAST Search Server 2010 for SharePoint)

  • Verify feeding (FAST Search Server 2010 for SharePoint)

  • Remove URLs from search results (FAST Search Server 2010 for SharePoint)

  • Delete a file type from the content index (FAST Search Server 2010 for SharePoint)

  • Include a file type in the content index (FAST Search Server 2010 for SharePoint)

  • Reset the content index (FAST Search Server 2010 for SharePoint)

See Also

Concepts

Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint)
Plan for crawling and federation (FAST Search Server 2010 for SharePoint)


Add, edit, or delete a content source (FAST Search Server 2010 for SharePoint)
See Also

After having followed the instructions in Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint), you can add one or more content sources to the FAST Search Content SSA.

A content source is a set of options that you can use to specify what type of content is crawled, what URLs to crawl, and how deep and when to crawl.

You must create at least one content source before a crawl can occur. After you create a content source, you can edit or delete it at any time.

Add, edit, or delete a content source

To get to the Manage Content Sources page

  1. Verify that the user account that is performing this procedure is a service application administrator for the FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, click your FAST Search Content Search Service Application.

  4. On the Search Administration Page, in the Crawling section, click Content Sources.

To create a content source

  1. On the Manage Content Sources page, click New Content Source.

  2. On the Add Content Source page, in the Name section, in the Name box, type a name for the new content source.

  3. In the Content Source Type section, select the type of content that you want to crawl.

    Warning:

    You can only crawl one type of content per content source. That is, you can create a content source that contains URLs for SharePoint sites and another that contains URLs for file shares. However, you cannot create a single content source that contains URLs to both SharePoint sites and file shares.

  4. In the Start Addresses section, in the Type start addresses below (one per line) box, type the URLs from which the crawl should start.

  5. In the Crawl Settings section, select the crawling behavior that you want.

  6. In the Crawl Schedules section, to specify a schedule for full crawls, select a defined schedule from the Full Crawl list. A full crawl crawls all content that is specified by the content source, regardless of whether the content has changed. To define a full crawl schedule, click Create schedule.

    To specify a schedule for incremental crawls, select a defined schedule from the Incremental Crawl list. An incremental crawl crawls content that is specified by the content source that has changed since the last crawl. To define a schedule, click Create schedule.You can change a defined schedule by clicking Edit schedule.

    Tip:

    It is recommended to complete a manually started full crawl cycle before scheduling more crawls. This ensures that the time that is scheduled between incremental crawls is not shorter than the time that is required to run a full crawl.

  7. To prioritize this content source, in the Content Source Priority section, on the Priority list, select Normal or High.

  8. To immediately begin a full crawl, in the Start Full Crawl section, select the Start full crawl of this content source check box, and then click OK.

To edit a content source

  1. You can edit a content source to change the schedule on which the content is crawled, the crawl start addresses, the content source priority, or the name of the crawl. Crawl settings and content type cannot be changed when editing a content source.

  2. On the Manage Content Sources page, in the list of content sources, point to the name of the content source that you want to edit, click the arrow that appears, and then click Edit.

  3. After you have made the changes that you want, select the Start full crawl of this content source check box, and then click OK.

Delete a content source

  1. On the Manage Content Sources page, in the list of content sources, point to the name of the content source that you want to delete, click the arrow that appears, and then click Delete.

  2. Click OK to confirm that you want to delete this content source.

See Also

Concepts

Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint)


Content sources that require additional configuration (FAST Search Server 2010 for SharePoint)

Published: June 17, 2010

Through the FAST Search Content Search Service Application, the indexing connectors integrated in Microsoft SharePoint Server 2010 can crawl content by using the following indexing connector protocols:

  • bcs

  • bcs2

  • file

  • https

  • http

  • notes

  • rb

  • rbs

  • sps

  • sps3

  • sps3s

  • spss

  • sts

  • sts2

  • sts2s

  • sts3

  • sts3s

  • sts4

  • sts4s

For other types of content, a separate connector is required. SharePoint Server 2010 comes with two indexing connectors: Lotus Notes connector and Exchange connector. You can get other connectors from third-party vendors. If there is no connector for a protocol, you can create custom connectors by using Microsoft SharePoint Designer and Microsoft Visual Studio 2010.

Content sources that require additional configuration

The following sections provide an overview of the content sources that require additional configuration in order to be able to crawl that content type. The additional configuration procedures needed to crawl these content types are explained in the topics that are linked to at the end of each overview section.

Business Intelligence content

Important:

The Microsoft Business Intelligence Indexing Connector free download was removed from the download site. Customers who have downloaded the Business Intelligence Indexing Connector as of March 2011 will continue to be supported as per standard support procedures. However, Business Intelligence Indexing Connector will not be tested or supported for SharePoint Server 2010 Service Pack 1.

The Microsoft Business Intelligence Indexing Connector improves search capabilities for Microsoft Excel and Microsoft SQL Server Reporting Services (SSRS) reports and their underlying data sources. The searchable business intelligence assets include .xlsx and .xlsm documents, which are natively created by Microsoft Office Excel 2007 and Excel 2010. PowerPivot and SQL Server Reporting Services files are also searchable. For more information about how to plan for, install and configure the Business Intelligence Indexing Connector: Configure and use the Business Intelligence Indexing Connector (FAST Search Server 2010 for SharePoint).

Documentum content

The Indexing Connector for Documentum enables SharePoint Server 2010, through the FAST Search Content SSA, to crawl content that is stored in the EMC Documentum system. For more information about how to install and configure the Indexing Connector for Documentum, see Configure and use the Documentum connector (FAST Search Server 2010 for SharePoint).

Exchange content

The Exchange connector enables SharePoint Server 2010, through the FAST Search Content SSA, to crawl Microsoft Exchange Server data. The Exchange connector comes installed as part of the default SharePoint Server 2010 installation and requires only configuration to use. For more information about how to configure the Exchange connector, see Configure and use the Exchange connector (FAST Search Server 2010 for SharePoint).

Lotus Notes content

The Lotus Notes connector enables SharePoint Server 2010, through the FAST Search Content SSA, to crawl Lotus Notes Domino Server data. The Lotus Notes connector comes installed as part of the default SharePoint Server 2010 installation and requires only configuration to use. For more information about how to configure the Lotus Notes connector, see Configure and use the Lotus Notes connector (FAST Search Server 2010 for SharePoint).


Configure and use the Business Intelligence Indexing Connector (FAST Search Server 2010 for SharePoint)

Published: July 8, 2010

Important:

The Microsoft Business Intelligence Indexing Connector free download was removed from the download site. Customers who have downloaded the Business Intelligence Indexing Connector as of March 2011 will continue to be supported as per standard support procedures. However, Business Intelligence Indexing Connector will not be tested or supported for SharePoint Server 2010 Service Pack 1.

This article introduces the Microsoft Business Intelligence Indexing Connector and provides links to planning, installation and configuration instructions.

Introduction

The Business Intelligence Indexing Connector adds a search tab called Reports to the search front end of FAST Search Server 2010 for SharePoint. The Reports search tab displays improved results, descriptions, thumbnails, and previews for business intelligence assets. Users will be able to refine search results from the Reports tab.

Deploy the Business Intelligence Indexing Connector to improve search capabilities for Microsoft Excel and Microsoft SQL Server Reporting Services (SSRS) reports and their underlying data sources. The searchable business intelligence assets include .xlsx and .xlsm documents, which are natively created by Microsoft Office Excel 2007 and Excel 2010. PowerPivot and SQL Server Reporting Services files are also searchable.

By deploying the Business Intelligence Indexing Connector you can help users search and locate business intelligence assets that include the following characteristics:

  • Additional data that is not explicitly included in the document text of the report, but that is still searchable.

  • Additional business intelligence characteristics like dimensions and measures.

Documentation for the Business Intelligence Indexing Connector

The following links take you to articles that explain how to plan for and deploy the Business Intelligence Indexing Connector.

  • Plan for Business Intelligence Indexing Connector (SharePoint Server 2010)

  • Deployment for Business Intelligence Indexing Connector (SharePoint Server 2010)


Configure and use the Documentum connector (FAST Search Server 2010 for SharePoint)

Published: June 17, 2010

The Microsoft SharePoint 2010 Indexing Connector for Documentum enables Microsoft FAST Search Server 2010 for SharePoint (through Microsoft SharePoint Server 2010) to index content that is stored in the EMC Documentum system. This article describes how to install and configure the Indexing Connector for Documentum connector for use with FAST Search Server 2010 for SharePoint.

To download the Indexing Connector for Documentum from the Microsoft Download Center, use the following link: Microsoft SharePoint 2010 Indexing Connector for Documentum (http://go.microsoft.com/fwlink/p/?LinkId=191180&clcid=0x409).

The Indexing Connector for Documentum includes the following features:

  • Based on the SharePoint 2010 Search Connector Framework

  • 64-bit connector

  • One connector supports multiple versions of EMC Documentum Content Server

  • Indexes Documentum objects and object metadata

  • Supports Documentum security definitions and policies

  • Supports Windows PowerShell for automated configuration and administration

  • Configurable search results URL to support multiple Documentum client applications

  • Supports file and folder exclusion for crawling

The following lists describe supported and unsupported object types and properties for the Indexing Connector for Documentum.

Supported container objects and properties:

  • dm_cabinet and subtypes

  • dm_Folder and subtypes

  • r_object_type

  • object_name

  • title

  • subject

  • keywords

  • owner_name

  • r_creator_name

  • r_creation_date

  • r_modifier

  • r_modify_date

  • cabinetpath

  • folderpath

Supported document objects and properties:

  • dm_document and subtypes

  • authors

  • keywords

  • r_full_content_size

  • r_creation_date

  • object_name

  • r_modify_date

  • r_modifier

  • subject

  • title

  • r_object_type

  • a_content_type

  • owner_name

  • r_version_label

  • r_lock_date

  • r_lock_owner

  • r_policy_id

  • r_current_state

  • log_entry

  • r_creator_name

  • r_access_date

  • a_storage_type

  • i_retain_until

  • ContainerPath

  • All custom properties

Unsupported object types:

  • Temp cabinets

  • Temp folders

  • Temp files

Install and configure the prerequisites for the Indexing Connector for Documentum

Use the following procedure to install and configure the prerequisites for the Indexing Connector for Documentum. The steps are listed in the order that they must be performed.

The SharePoint 2010 Indexing Connector for Documentum has the following software prerequisites:

  • One of the following SharePoint Server 2010, Search Server 2010, or FAST Search Server 2010 for SharePoint products:

    • Microsoft SharePoint Server 2010

    • Microsoft Search Server 2010

    • Microsoft Search Server 2010 Express

    • Microsoft SharePoint 2010 for Internet Sites Enterprise

    • Microsoft SharePoint 2010 for Internet Sites Standard

    • Microsoft FAST Search Server 2010 for SharePoint

    • Microsoft FAST Search Server 2010 for SharePoint Internet Sites

  • DFS Server v6.5 with SP2 and DFS hotfix 1049. This server needs to be configured and connected to all repositories.

  • You must use DFS Productivity Layer v6.5 with SP2 and DFS hotfix 1049 .NET assemblies.

    The .NET assemblies are included in the DFS hotfix 1049 package. You can obtain the DFS hotfix 1049 package, which includes both a server side patch as well as a client side patch, by opening a service request on the EMC Powerlink Web site: http://powerlink.emc.com. Alternatively, you can contact your EMC customer representative.

    The Indexing Connector for Documentum uses EMC DFS (Documentum Foundation Services) as the connectivity application programming interface (API) to access Documentum repositories. Therefore, you must install and configure DFS Productivity Layer (client of DFS Server) .NET components on the SharePoint Server 2010 crawl server where the Indexing Connector for Documentum will be installed.

To install and configure the prerequisites for the Indexing Connector for Documentum

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content Search Service Application.

  2. Determine which Documentum content access account you will use for crawling. You have to specify this account and the password later in the configuration procedure when you set up crawl rules. The Indexing Connector for Documentum uses a Documentum content access account to retrieve content from the Documentum repository. This account must have the following credentials:

    • At least read permission to documents that you want to crawl.

    • At least browse permission to cabinets, folders, and records (documents with only metadata) that you want to crawl.

  3. On each SharePoint Server 2010 crawl server, deploy DFS Productivity Layer .NET assemblies to the global assembly cache %windir%\assembly. There are four DLLs that are used by the Indexing Connector for Documentum. Verify the DLL names and versions before you deploy them into the global assembly cache. The following files are included in the DFS1049 Hotfix. When extracted to the default path, these files are located in the directory %local%\emc-dfs-sdk-6.5\emc-dfs-sdk-6.5\lib\dotnet:

    • Emc.Documentum.FS.DataModel.Core.dll, version number 6.5.0.231

    • Emc.Documentum.FS.DataModel.Shared.dll, version number 6.5.0.231

    • Emc.Documentum.FS.runtime.dll, version number 6.5.0.231

    • Emc.Documentum.FS.Services.Core.dll, version number 6.5.0.231

    Note:

    You can drag and drop the four DLLs into the global assembly cache (%windir%\assembly) to deploy them, but you might have to turn off User Account Control to do this.

  4. In order for the DFS productivity layer .NET assemblies to function correctly, you must update the .NET machine.config file to include WCF settings for the DFS productivity layer. On each SharePoint Server 2010 crawl server, open the machine.config file located in the following directory: %windir%\Microsoft.NET\Framework64\V2.0.50727\CONFIG. The following WCF settings allow maximum 30 megabytes (MB) per Documentum content object (the document file plus its metadata) transferred. The administrator can increase "maxReceivedMessageSize" in "DfsDefaultService" binding for larger content. The default FAST Search Server 2010 for SharePoint search will handle files with a maximum size of 64 MB. To crawl files larger than 64 MB, follow step 4b.

    1. Go to %windir%\Microsoft.NET\Framework64\v2.0.50727\CONFIG, open the machine.config file, and then add the following XML snippet into the <configuration> element:

      Copy Code
      <system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="DfsAgentService" closeTimeout="00:01:00"
 openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
 allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
 maxBufferSize="10000000" maxBufferPoolSize="10000000" maxReceivedMessageSize="10000000"
 messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
 useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
  maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>

<binding name="DfsContextRegistryService" closeTimeout="00:01:00"
   openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
   allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
   maxBufferSize="10000000" maxBufferPoolSize="10000000" maxReceivedMessageSize="10000000"
   messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
   useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>
<binding name="DfsDefaultService" closeTimeout="00:01:00" openTimeout="00:10:00" receiveTimeout="00:20:00" sendTimeout="00:10:00" allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard" maxBufferSize="10000000" maxBufferPoolSize="10000000" maxReceivedMessageSize="30000000" messageEncoding="Text" textEncoding="utf-8" transferMode="StreamedResponse" useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="1048576" maxNameTableCharCount="16384"/>
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None" realm=""/>
<message clientCredentialType="UserName" algorithmSuite="Default"/>
</security>
</binding>
</basicHttpBinding>
</bindings>
</system.serviceModel>

    2. To crawl files larger than 64 MB, you have to get the FAST Search Content Search Service Application and store it in a variable, retrieve the current value of "MaxDownloadSize" and modify it to the size that you want to crawl.

      1. Run this procedure on the crawl server.

      2. On the Start menu, click All Programs.

      3. Click Microsoft SharePoint 2010 Products.

      4. Right-click SharePoint 2010 Management shell and select Run as administrator.

      5. At the Windows PowerShell command prompt, type the following command(s):

        Copy Code
        $ssa = Get-SPEnterpriseSearchServiceApplication -Identity <FAST Search Content Search Service Application>
$ssa.GetProperty("MaxDownloadSize")
$ssa.SetProperty("MaxDownloadSize", <File size larger than 64>)
$ssa.Update()

        Where:

        • <File size larger than 64> is the maximum file size (in MB) that you want to allow the Indexing Connector for Documentum to crawl.

        • <FAST Search Content Search Service Application> is the name of your FAST Search Content Search Service Application.

  5. The Indexing Connector for Documentum will crawl the Documentum document Access Control List (ACL) and map this list to the system ACLs. This allows users to search documents that they have permission to read in Documentum. The Indexing Connector for Documentum supports three kinds of ACL translations that you can configure in DCTMConfig.xml by using the Windows PowerShell cmdlet Set-SPEnterpriseSearchDCTMConnectorConfig.

    For more information about how to set these options and for examples, refer to Install and configure the Indexing Connector for Documentum.

    These are the options for setting up the system ACLs:

    • No Security

      The Indexing Connector for Documentum will ignore Documentum ACLs during crawl and every SharePoint user can search all crawled documents.

    • Assume the same account

      When Documentum and SharePoint 2010 Products are both using Active Directory Domain Services (AD DS) or Active Directory directory service, the Indexing Connector for Documentum assumes a user or group is using the same account in both systems.

    • Translate ACL according to a user mapping table

      If Documentum and SharePoint 2010 Products are not both using AD DS or Active Directory and you want to enable the security search, you have to set up a user mapping table to specify how to do the ACL translation.

  6. The user mapping table requires the following:

    • The user mapping table must be in a Microsoft SQL Server 2008 or later database.

    • The OSearch14 service account must have at least read permission on the user mapping table data.

    DCTMCredentialDomain

    Domain name of a Documentum account. Populate this column when the account comes from the local computer or an LDAP system. The User Source property of the Documentum account should equal None or LDAP, otherwise leave the column empty.

    DCTMCredentialRepository

    Repository name of a Documentum account. Populate this column when the account comes from a Documentum repository.

    DCTMCredentialLoginName

    Login name of the Documentum account

    NTCredential

    Windows domain user account that searches Documentum contents in SharePoint Server

    Example: A Documentum repository user Dan Park has a login that is linked to the Finance repository. Dan's Windows domain user account is Litwareinc\dpark. In this case, the user mapping table entry for Dan appears as the following:

    DCTMCredentialDomain

    ""

    DCTMCredentialRepository

    Finance

    DCTMCredentialLoginName

    dpark

    NTCredential

    Litwareinc\dpark

    Example: In Documentum there is a group called Marketing Department that is located in the M&S repository. The group is mapped to the AD group Marketing under the domain litware. In this case, the group mapping table entry appears as the following:

    DCTMCredentialDomain

    ""

    DCTMCredentialRepository

    M&S

    DCTMCredentialLoginName

    Marketing Department

    NTCredential

    Litware\Marketing

    Note:

    If any cells have no value assigned, they cannot be NULL or empty. You must assign the following empty string value: ''.

    For each Documentum group there must be an NT group in the user mapping table and they must both contain the same user information.

    The user mapping table contains mappings for both users and groups. The table must be kept up to date. If you add a group in Documentum, you need to add a corresponding group in the Active Directory and set up a mapping to it.

    Use the following script to create a user mapping table:

    Copy Code
    CREATE TABLE <replace with your user mapping table name>
(
DCTMCredentialDomain nvarchar (255) NOT NULL ,
DCTMCredentialRepository nvarchar (32) NOT NULL ,
DCTMCredentialLoginName nvarchar (80) NOT NULL ,
NTCredential nvarchar (255) NOT NULL ,
CONSTRAINT PK_CredentialMapping PRIMARY KEY CLUSTERED
( DCTMCredentialDomain, DCTMCredentialRepository, DCTMCredentialLoginName )
)

    Populate the new mapping table with Documentum/NT Credential pairs as seen in the above table. Grant the OSearch14 account read access to this table.

Install and configure the Indexing Connector for Documentum

Use the following procedure to install and configure the Indexing Connector for Documentum.

To install and configure the Indexing Connector for Documentum

  1. See Add-SPShellAdmin. This article contains information that helps you to verify the permissions that are required to perform this procedure.

  2. Open the Windows PowerShell command console.

  3. On each SharePoint Server 2010 server in the farm that is running a crawl component, run the Indexing Connector for Documentum DCTMIndexConn.exe. Follow the steps presented in the installation wizard.

  4. On the SharePoint Server 2010 crawl server, use the following Windows PowerShell cmdlet to register the indexing connector to the FAST Search Content search service application: New-SPEnterpriseSearchCrawlCustomConnector

  5. Use the following example for a single FAST Search Content search service application: New-SPEnterpriseSearchCrawlCustomConnector -SearchApplication "<name of your FAST Search Content search service application>" -Protocol "dctm" -ModelFilePath "<C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG\SearchConnectors\Documentum\MODEL.xml>" -Name "Microsoft SharePoint 2010 Indexing Connector for Documentum"

  6. Use the following example for all search service applications on the farm: Get-SPEnterpriseSearchServiceApplication | New-SPEnterpriseSearchCrawlCustomConnector -Protocol "dctm" -ModelFilePath "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG\SearchConnectors\Documentum\MODEL.xml" -Name "Microsoft SharePoint 2010 Indexing Connector for Documentum"

  7. On each SharePoint Server 2010 crawl server, set the configuration details using the following Windows PowerShell cmdlet: Set-SPEnterpriseSearchDCTMConnectorConfig. All the settings are stored in <system drive>\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG\SearchConnectors\Documentum\DCTMConfig.xml. If more than one SharePoint Server 2010 crawl server is used, all settings must be the same on each server.

    Use the following Windows PowerShell commands to display help and examples for the Indexing Connector for Documentum:

    • Get-help Set-SPEnterpriseSearchDCTMConnectorConfig -full shows full help.

    • Get-help Set-SPEnterpriseSearchDCTMConnectorConfig -examples shows only examples.

    The following tables describe the important parameters of the Set-SPEnterpriseSearchDCTMConnectorConfig cmdlet.

    ACLTranslation Directs the behavior of the ACL translation

    UserMappingTable

    Default value. The Indexing Connector for Documentum translates the Documentum ACL into a Windows ACL according to the user mapping table. UserMappingTableSQLServer, UserMappingTableSQLInstance, UserMappingTableDBName, UserMappingTableName and UnMappedAccount take effect only when ACLTranslation is set to "UserMappingTable".

    NoSecurity

    The Indexing Connector for Documentum ignores the Documentum ACL during crawl. For example, all documents from Documentum will be searchable by any SharePoint user. This option enables you to decline enforcement of security trimming or implement custom security trimming.

    SameAccountName

    The Indexing Connector for Documentum assumes Documentum and SharePoint users share the same account, such as a shared account in Active Directory. Once an invalid NT account is found, the Indexing Connector for Documentum discards the account permission.

    UnmappedAccount Defines how to process Documentum accounts which have no corresponding Windows account defined in the user mapping table.

    DiscardACE

    Default value of "UnmappedAccount". The Indexing Connector for Documentum discards a Documentum account when no mapped Windows account is found. If there is any other mapped account for the document, the document will be crawled. If none of the accounts for this document can be mapped, the document will be discarded and an error message will be entered in the crawl log.

    AssumeSameAccount

    Assumes the NT account is the same as the Documentum account.

    Other parameters Description

    UserMappingTableSQLServer

    Host name of the computer that is running SQL Server which contains the user mapping table.

    UserMappingTableSQLInstance

    Name of the SQL Server instance that contains the user mapping table.

    UserMappingTableDBName

    Name of the SQL Server database that contains the user mapping table.

    UserMappingTableName

    Name of the user mapping table.

    DisplayURLPatternForDocument

    DisplayURL pattern for documents. Any valid URL with part of the string replaced by placeholders such as {ObjectId}, {RepositoryName} or {Format}. You can find examples below.

    DisplayURLPatternForContainer

    DisplayURL pattern for folders and cabinets. Any valid URL with part of the string replaced by placeholders such as {ObjectId}, {RepositoryName} or {Format}. You can find examples below.

    DFSURL

    Specify the DFS Web Services URL for each repository that is to be crawled. More than one DFS Web Services URL can be specified for each repository. Use the following format: "RepositoryName1\DFSURL1.1\DFSURL1.2\...\DFSURL1.n\\RepositoryName2\DFSURL2.1\DFSURL2.2\...\DFSURL2.n\..."

    PersistDCTMACL

    Specify whether to store the Documentum ACL in a crawled property. If "PersistDCTMACL" is set to "True", the Indexing Connector for Documentum will store the Documentum ACL information as a crawled property. The default value is "False".

    C# Copy Code
    Example 1: Set to "UserMappingTable" mode.
Set-SPEnterpriseSearchDCTMConnectorConfig -ACLTranslation "UserMappingTable" -UnmappedAccount "DiscardACE" -UserMappingTableSQLServer "<YourDatabaseServerName>" -UserMappingTableSQLInstance "<YourDatabaseInstanceName>" -UserMappingTableDBName "<YourMappingDatabaseName>" -UserMappingTableName "<YourMappingTableName>" -DFSURL "RepositoryName1\http://MACHINENAME1:PORT1/services\\RepositoryName2\http://MACHINENAME2:PORT2/services\http://MACHINENAME3:PORT3/services" -DisplayURLPatternForDocument "http://MACHINENAME4:PORT4/webtop/component/drl?objectId={ObjectId}&format={Format}&RepositoryName={RepositoryName}" -DisplayURLPatternForContainer "http://MACHINENAME5:PORT5/webtop/component/drl?objectId={ObjectId}&RepositoryName={RepositoryName}"

Example 2: Set to "NoSecurity" mode.
Set-SPEnterpriseSearchDCTMConnectorConfig -ACLTranslation "NoSecurity" -DFSURL "RepositoryName1\http://MACHINENAME1:PORT1/services\\RepositoryName2\http://MACHINENAME2:PORT2/services\http://MACHINENAME3:PORT3/services" -DisplayURLPatternForDocument "http://MACHINENAME4:PORT4/webtop/component/drl?objectId={ObjectId}&format={Format}&RepositoryName={RepositoryName}" -DisplayURLPatternForContainer "http://MACHINENAME5:PORT5/webtop/component/drl?objectId={ObjectId}&RepositoryName={RepositoryName}"

Example 3: Set to "SameAccountName" mode.
Set-SPEnterpriseSearchDCTMConnectorConfig -ACLTranslation "SameAccountName" -DFSURL "RepositoryName1\http://MACHINENAME1:PORT1/services\\RepositoryName2\http://MACHINENAME2:PORT2/services\http://MACHINENAME3:PORT3/services" -DisplayURLPatternForDocument "http://MACHINENAME4:PORT4/webtop/component/drl?objectId={ObjectId}&format={Format}&RepositoryName={RepositoryName}" -DisplayURLPatternForContainer "http://MACHINENAME5:PORT5/webtop/component/drl?objectId={ObjectId}&RepositoryName={RepositoryName}"

  8. After setting the configuration details, restart the OSearch14 service on each SharePoint Server 2010 crawl server.

Create a crawl rule for the Indexing Connector for Documentum

Before a crawl, create crawl rules to include or exclude specific content in Documentum. Use the following procedure to create a crawl rule for the Indexing Connector for Documentum.

To create a crawl rule for the Indexing Connector for Documentum

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content search service application.

  2. Open SharePoint Central Administration, and then click Manage Service Applications.

  3. Click the name of your FAST Search Content search service application for which you want to create a crawl rule.

  4. Under Crawling, click Crawl Rules.

  5. On the Manage Crawl Rules page, click New Crawl Rule.

  6. On the Add Crawl Rule page, specify the following information to create at least one crawl rule.

    • In Path box, type the path of the content that you want to crawl.

      You can use wildcard "*" or regular expression syntax. Because Documentum uses case sensitive names for the content, select the Match case check box. Refer to the section Syntax to refer to a Documentum object for examples.

    • In Crawl Configuration section, select Include all items in this path, and then select Crawl complex URLs (URLs that contain a question mark - ?).

    • In the Specify Authentication section, select Specify a different content access account, and then type the Documentum content access account and password that you specified earlier in this article in the appropriate boxes.

    • Make sure that the Do not allow Basic Authentication check box is cleared.

  7. Click OK to finish configuration.

    Note:

    • You can create multiple crawl rules for Documentum to include or exclude Documentum content.

    • You can use different crawl rules to specify different content access accounts for different Documentum content. For example, you have two repositories and two content access accounts for each repository. The Documentum content access account specified in a crawl rule will only be applied to Documentum content covered by the path in that crawl rule.

Create a content source for the Indexing Connector for Documentum

Use the following procedure to create a content source.

To create a content source for the Indexing Connector for Documentum

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content search service application.

  2. Open SharePoint Central Administration, and then click Manage Service Applications.

  3. Click the name of your FAST Search Content search service application in which you want to create a content source.

  4. On the Search Administration page, under Crawling, click Content Sources.

  5. On the Manage Content Sources page, click New Content Source.

  6. On the Add Content Source page, do the following:

    1. In the Name box, type the name of the content source.

    2. In the Content Source Type section, select Custom Repository.

    3. In the Type of Repository section, select the SharePoint 2010 Indexing Connector for Documentum. Mark the radio button in front of the name that you specified for the Indexing Connector for Documentum when you registered it with the FAST Search Content search service application, for example dctm.

    4. In the Start Addresses section, type the start addresses. The start address format is the same as the format used to specify the path. You can type more than one start address for the content source, one per line.

      Refer to the section Syntax to refer to a Documentum object for examples.

    5. In the Crawl Schedules section, select schedules from the Full Crawl and Incremental Crawl drop-down lists, or create schedules for each kind of crawl.

    6. In the Content Source Priority section, assign a priority level to the content source according to your business requirements.

    7. Select Start full crawl of this content source to start a crawl immediately after the content source is created.

    8. Click OK to finish the configuration and accept all configured options.

    The Documentum content source is configured and the system can crawl Documentum content repositories that are specified in the content source.

SharePoint Server 2010 supports a scalable architecture for performance scale-out. You can deploy more than one crawl server and configure multiple SharePoint Server 2010 crawlers to crawl the EMC Documentum database simultaneously.

Syntax to refer to a Documentum object

The format to refer to a Documentum object that you use for the path (when you set up a crawl rule) and the start address (when you set up a content source) is defined in the following table:

Type of Documentum object | Syntax for the path or the start address
Repository | dctm://<clientapphostname>/<repository name>
Cabinet | dctm://<clientapphostname>/<repository name>/<cabinet name>
Folder | dctm://<clientapphostname>/<repository name>/<cabinet name>/<folder name>
Document | dctm://<clientapphostname>/<repository name>/<cabinet name>/<folder name>/…/<folder name>?DocSysID=<r_object_id> (where r_object_id is the object id of that document)

<clientapphostname> is the host name of your Documentum client application such as Webtop or DA. The <clientapphostname> configured here should be same as the same as the one used in content source. <repository name>, <cabinet name>, and <folder name> are case sensitive.


Configure TCS support for the Documentum connector (FAST Search Server 2010 for SharePoint)

Published: May 16, 2012

This article describes how to install and configure the Microsoft SharePoint 2010 Indexing Connector for Documentum to support EMC Documentum Trusted Content Services (TCS).

Tip:

You can use the procedures in this article to install and configure the Indexing Connector for Documentum and enable automatic user mapping between Windows Active Directory users and Documentum users. This can be useful whether or not you require support for TCS.

System prerequisites and requirements

  • Supported operating systems are either Windows Server 2008 R2 or Windows Server 2008 Service Pack 2.

  • An installed and configured installation of FAST Search Server 2010 for SharePoint with the February 2012 Cumulative Update.

  • An installed and configured installation of SharePoint Server 2010 with the December 2010 Cumulative Update.

  • A FAST Search Content Search Service Application.

  • Documentum Foundation Services (DFS) Server v6.5 SP2 patched with DFS hotfix 1049. You can ask for this hotfix from EMC customer support site http://powerlink.emc.com. This server must be configured and connected to all repositories.

  • DFS Productivity Layer v6.5 with SP2 and DFS hotfix 1049 .NET assemblies.

  • Documentum Content Server. The supported versions of Documentum Content Server are determined by DFS v6.5 SP2. You can find a detailed list in the DFS v6.5 SP2 release notes in the section "Backward compatibility for Content Server". You can download the release notes from the EMC customer support site http://powerlink.emc.com.

  • Documentum Trusted Content Services (TCS) is required if you have to support full EMC Documentum repository security. TCS is not required to enable automatic mapping between Windows Active Directory users and Documentum users.

  • A Windows Server 2008 Active Directory Lightweight Directory Services (AD LDS) instance.

Quick overview

  1. Before you begin:

    1. Ensure that your system meets the System prerequisites and requirements.

    2. Download the Indexing Connector for Documentum from the Microsoft Download Center.

    3. Determine which Documentum content access account to use for crawling.

    4. On each SharePoint 2010 crawl server:

      1. Set DFS Productivity Layer .NET assemblies.

      2. Edit the machine.config file to set the Documentum bindings.

  2. Install and configure the Indexing Connector for Documentum.

    1. Deploy the Indexing Connector for Documentum on each SharePoint 2010 crawl server in the farm.

    2. Register the Indexing Connector for Documentum to the FAST Search Content SSA using Windows PowerShell.

    3. Configure the Indexing Connector for Documentum on each SharePoint 2010 crawl server in the farm using the Indexing Connector for Documentum PowerShell commandlet.

  3. Set up AD LDS, run the custom security trimming script and configure the FAST Search Server for SharePoint Security Sync Service. This enables automatic user mapping and support for TCS.

    1. Stop IIS for the FASTSearchAdminAppPool on the FAST Search Server 2010 for SharePoint administration server.

    2. Install the AD LDS server role, create an AD LDS instance and create a partition for it.

    3. Set DFS Productivity Layer .NET assemblies for FAST Search Server 2010 for SharePoint in the bin directory of the installation directory.

    4. Enable custom security trimming with a Windows PowerShell script and provide the location of the AD LDS on all FAST Search Server 2010 for SharePoint servers in the farm.

    5. Restart IIS, if you have stopped it.

    6. Configure the FAST Search Server for SharePoint Security Sync Service.

    7. Verify the keys and the description field in the AD LDS partition.

  4. Configure a Documentum content source in the FAST Search Content SSA in the SharePoint 2010 Central Administration.

    1. Create a crawl rule for the Indexing Connector for Documentum.

    2. Create a content source for the Indexing Connector for Documentum.

Prepare the SharePoint 2010 crawl server(s).

Determine which Documentum content access account to use

  1. Determine which Documentum content access account you will use for crawling. You have to specify this account and the password later in the configuration procedure when you set up crawl rules. The Indexing Connector for Documentum uses a Documentum content access account to retrieve content from the Documentum repository. This account must have the following credentials:

    • At least read permission to documents that you want to crawl.

    • At least browse permission to cabinets, folders, and records (documents with only metadata) that you want to crawl.

Set DFS Productivity Layer .NET assemblies

  1. Locate and verify the version number of the following four DFS Productivity Layer .NET assemblies that are included in DFS hotfix 1049. When extracted to the default path, these files are located in the directory %local%\emc-dfs-sdk-6.5\emc-dfs-sdk-6.5\lib\dotnet:

    • Emc.Documentum.FS.DataModel.Core.dll (version number 6.5.0.231)

    • Emc.Documentum.FS.DataModel.Shared.dll (version number 6.5.0.231)

    • Emc.Documentum.FS.runtime.dll (version number 6.5.0.231)

    • Emc.Documentum.FS.Services.Core.dll (version number 6.5.0.231)

  2. On each SharePoint 2010 crawl server, log on with an account that has Administrator rights to the server and deploy the DFS Productivity Layer .NET assemblies to the global assembly cache %windir%\assembly.

    Note:

    You can drag and drop the four DLLs into the global assembly cache (%windir%\assembly) to deploy them, but you might have to turn off User Account Control to do this.

The next step is to edit the machine.config file on each SharePoint Server 2010 crawl server to include WCF settings for the DFS Productivity Layer. This is done to make sure that the DFS Productivity Layer .NET assemblies function correctly.

The WCF settings you are about to set allow a maximum of 30 megabytes (MB) per Documentum content object (the document file plus its metadata) transferred. The administrator can increase "maxReceivedMessageSize" in "DfsDefaultService" binding for larger content. However, the SharePoint Server 2010 crawler will handle files with a maximum size of 64 MB. To crawl files larger than 64 MB, follow step 4b.

Edit the machine.config file

  1. Go to %windir%\Microsoft.NET\Framework64\v2.0.50727\CONFIG, open the machine.config file, and then add the following XML snippet into the <configuration> element:

    Copy Code
    <system.serviceModel>
<bindings>
<basicHttpBinding>
<binding name="DfsAgentService" closeTimeout="00:01:00"
 openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
 allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
 maxBufferSize="10000000" maxBufferPoolSize="10000000" maxReceivedMessageSize="10000000"
 messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
 useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
  maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>

<binding name="DfsContextRegistryService" closeTimeout="00:01:00"
   openTimeout="00:01:00" receiveTimeout="00:10:00" sendTimeout="00:01:00"
   allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard"
   maxBufferSize="10000000" maxBufferPoolSize="10000000" maxReceivedMessageSize="10000000"
   messageEncoding="Text" textEncoding="utf-8" transferMode="Buffered"
   useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
maxBytesPerRead="4096" maxNameTableCharCount="16384" />
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None"
realm="" />
<message clientCredentialType="UserName" algorithmSuite="Default" />
</security>
</binding>
<binding name="DfsDefaultService" closeTimeout="00:01:00" openTimeout="00:10:00" receiveTimeout="00:20:00" sendTimeout="00:10:00" allowCookies="false" bypassProxyOnLocal="false" hostNameComparisonMode="StrongWildcard" maxBufferSize="10000000" maxBufferPoolSize="10000000" maxReceivedMessageSize="30000000" messageEncoding="Text" textEncoding="utf-8" transferMode="StreamedResponse" useDefaultWebProxy="true">
<readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="1048576" maxNameTableCharCount="16384"/>
<security mode="None">
<transport clientCredentialType="None" proxyCredentialType="None" realm=""/>
<message clientCredentialType="UserName" algorithmSuite="Default"/>
</security>
</binding>
</basicHttpBinding>
</bindings>
</system.serviceModel>

  2. To crawl files larger than 64 MB, you have to get the FAST Search Content Search Service Application and store it in a variable, retrieve the current value of "MaxDownloadSize" and modify it to the size that you want to crawl.

    1. Run this procedure on the crawl server.

    2. On the Start menu, click All Programs.

    3. Click Microsoft SharePoint 2010 Products.

    4. Right-click SharePoint 2010 Management shell and select Run as administrator.

    5. At the Windows PowerShell command prompt, type the following command(s):

      Copy Code
      $ssa = Get-SPEnterpriseSearchServiceApplication -Identity <FAST Search Content Search Service Application>
$ssa.GetProperty("MaxDownloadSize")
$ssa.SetProperty("MaxDownloadSize", <File size larger than 64>)
$ssa.Update()

      Where:

      • <File size larger than 64> is the maximum file size (in MB) that you want to allow the Indexing Connector for Documentum to crawl.

      • <FAST Search Content Search Service Application> is the name of your FAST Search Content Search Service Application.

Install and configure the Indexing Connector for Documentum

Deploy the Indexing Connector for Documentum

  1. Download the Indexing Connector for Documentum from the Microsoft Download Center.

  2. On each SharePoint Server 2010 server in the farm that is running a crawl component, run the Indexing Connector for Documentum by running the file DCTMIndexConn.exe. Follow the steps that are presented in the installation wizard.

Register the Indexing Connector for Documentum to the FAST Search Content SSA

  1. Run this procedure on the SharePoint Server 2010 crawl server to register the connector to the FAST Search Content search service application.

  2. On the Start menu, click All Programs.

  3. Click Microsoft SharePoint 2010 Products.

  4. Right-click SharePoint 2010 Management shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    New-SPEnterpriseSearchCrawlCustomConnector -SearchApplication "<name of your FAST Search Content search service application>" -Protocol "dctm" -ModelFilePath "<C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG\SearchConnectors\Documentum\MODEL.xml>" -Name "Microsoft SharePoint 2010 Indexing Connector for Documentum"

Configure the Indexing Connector for Documentum

  1. On each SharePoint Server 2010 crawl server, set the configuration details using the following Windows PowerShell cmdlet: Set-SPEnterpriseSearchDCTMConnectorConfig. All the settings are stored in <system drive>\Program Files\Common Files\Microsoft Shared\Web Server Extensions\14\CONFIG\SearchConnectors\Documentum\DCTMConfig.xml. If more than one SharePoint Server 2010 crawl server is used, all settings must be the same on each server.

    Use the following Windows PowerShell commands to display help and examples for the Indexing Connector for Documentum:

    • Get-help Set-SPEnterpriseSearchDCTMConnectorConfig -full shows full help.

    • Get-help Set-SPEnterpriseSearchDCTMConnectorConfig -examples shows only examples.

    The following table describes some of the parameters of the Set-SPEnterpriseSearchDCTMConnectorConfig cmdlet. It is important to set the parameter ACLTranslation to NoSecurity and the parameter PersistDCTMACL to True, as these settings enable automatic user mapping and TCS support.

    Command Option Description

    ACLTranslation

    NoSecurity

    The Indexing Connector for Documentum ignores the Documentum ACL during crawl. For example, all documents from Documentum will be searchable by any SharePoint user. This option enables you to decline enforcement of security trimming or implement custom security trimming.

    DisplayURLPatternForDocument

    DisplayURL pattern for documents. Any valid URL with part of the string replaced by placeholders such as {ObjectId}, {RepositoryName} or {Format}. You can find an example below.

    DisplayURLPatternForContainer

    DisplayURL pattern for folders and cabinets. Any valid URL with part of the string replaced by placeholders such as {ObjectId}, {RepositoryName} or {Format}. You can find an example below.

    DFSURL

    Specify the DFS Web Services URL for each repository that is to be crawled. More than 1 DFS Web Services URL can be specified for each repository. Use the following format: "RepositoryName1\DFSURL1.1\DFSURL1.2\...\DFSURL1.n\\RepositoryName2\DFSURL2.1\DFSURL2.2\...\DFSURL2.n\..."

    PersistDCTMACL

    True

    Specify whether to store the Documentum ACL in a crawled property. If "PersistDCTMACL" is set to "True", the Indexing Connector for Documentum will store the Documentum ACL information as a crawled property. The default value is "False".

    Example:

    Copy Code
    Set-SPEnterpriseSearchDCTMConnectorConfig -ACLTranslation "NoSecurity" -DFSURL "RepositoryName1\http://MACHINENAME1:PORT1/services\\RepositoryName2\http://MACHINENAME2:PORT2/services\http://MACHINENAME3:PORT3/services" -DisplayURLPatternForDocument "http://MACHINENAME4:PORT4/webtop/component/drl?objectId={ObjectId}&format={Format}&RepositoryName={RepositoryName}" -DisplayURLPatternForContainer "http://MACHINENAME5:PORT5/webtop/component/drl?objectId={ObjectId}&RepositoryName={RepositoryName}"

  2. After setting the configuration details, restart the OSearch14 service on each SharePoint Server 2010 crawl server.

Set up support for TCS and enable automatic user mapping

Before you set up support for TCS, verify that you have access to the Internet Information Services (IIS) Manager. To install the IIS Manager, follow these steps:

Install the Internet Information Services (IIS) Manager

  1. Run this procedure on the FAST Search Server 2010 for SharePoint administration server.

  2. Click Start, point to Administrative Tools, and then click Server Manager.

  3. Right-click on Server Manager (<server name>) and click Add Features.

  4. Expand the Remote Server Administration Tools menu, then expand the Role Administration Tools menu and then click Web Server (IIS) Tools.

  5. Click Install and follow the instructions in the Wizard.

Stop IIS for the FASTSearchAdminAppPool

  1. Run this procedure on the FAST Search Server 2010 for SharePoint administration server.

  2. Click Start, point to Administrative Tools, and then click Internet Information Services (IIS) Manager.

  3. In the Connections menu, expand the server menu and click Application Pools.

  4. Click FASTSearchAdminAppPool.

  5. If the IIS service is running, go to the Actions menu and click Stop.

Install the AD LDS server role, create an AD LDS instance and create a partition for it

  1. Run this procedure on the FAST Search Server 2010 for SharePoint administration server.

    You may install the AD LDS server role on a different FAST Search Server 2010 for SharePoint machine in your farm, but the AD LDS server role must be installed on a machine that is close to or contains the QRServer and there may only be one AD LDS instance per farm.

    Install the Active Directory Lightweight Directory Services (AD LDS) role.

    1. Click Start, point to Administrative Tools, and then click Server Manager.

    2. In the console tree, right-click Roles and then click Add Roles.

    3. Review the information on the Before You Begin page of the Add Roles Wizard, and then click Next.

    4. On the Select Server Roles page, in the Roles list, select the Active Directory Lightweight Directory Services check box, and then click Next.

    5. Finish adding the AD LDS server role by following the instructions in the wizard.

  2. Create an AD LDS instance and create an AD LDS partition on the instance.

    1. Click Start, point to Administrative Tools, and then click Active Directory Lightweight Directory Services Setup Wizard.

    2. On the Welcome to the Active Directory Lightweight Directory Services Setup Wizard page, click Next.

    3. On the Setup Options page, click A unique instance, and then click Next.

    4. On the Instance Name page, provide a name for the AD LDS instance that you are installing. This name is used on the local computer to uniquely identify the AD LDS instance. For example, type CustomSecurity. Click Next.

    5. On the Ports page, specify the communications ports that the AD LDS instance uses to communicate. AD LDS can communicate by using both LDAP and Secure Sockets Layer (SSL). Therefore, you must provide a value for each port.

      Note:

      The default value for the LDAP port is 389 and the default value for the SSL port is 636. If you install AD LDS on a computer where either of the default ports is in use, the Active Directory Lightweight Directory Services Setup Wizard automatically locates the first available port, starting at 50000. For example, Active Directory Domain Services (AD DS) uses ports 389 and 636, as well as ports 3268 and 3269 on global catalog servers. Therefore, if you install AD LDS on a domain controller, the Active Directory Lightweight Directory Services Setup Wizard provides a default value of 50000 for the LDAP port and 50001 for the SSL port.

      Click Next.

    6. On the Application Directory Partition page, create an application directory partition (or naming context) by clicking Yes, create an application directory partition. Then, type a name for the partition. We recommend to name the partition CN=CustomSecurity,DC=FS4SP. The full name of the AD LDS instance or farm follows the pattern LDAP://<FAST Search Server 2010 for SharePoint server host name>:<port number>/<AD LDS partition name>, for example LDAP://contoso.com:52000/CN=CustomSecurity,DC=FS4SP.

    7. On the File Locations page, you can view and change the installation directories for AD LDS data and recovery (log) files. By default, AD LDS data and recovery files are installed in %ProgramFiles%\Microsoft ADAM\instancename\data, where instancename represents the AD LDS instance name that you specified on the Instance Name page. Click Next.

    8. On the Service Account Selection page, make sure that the Network service account is selected. This should be the default selection. The account that you select determines the security context in which the AD LDS instance runs

    9. On the AD LDS Administrators page, select This account. This should be the account of the user who is running FAST Search Server 2010 for SharePoint. Provide the account name of the FAST Search Server 2010 for SharePoint user that runs Windows Services for FAST Search Server 2010 for SharePoint to become the default administrator for the AD LDS instance. The user or group that you select will have full administrative control of the AD LDS instance.

    10. On the Importing LDIF Files page, you can import schema LDAP Data Interchange Format (LDIF) files into the AD LDS instance. Mark the box in front of MS-InetOrgPerson.LDF and click Next.

    11. The Ready to Install page gives you an opportunity to review your installation selections. After you click Next, the Active Directory Lightweight Directory Services Setup Wizard copies files and sets up AD LDS on your computer.

    12. When the Active Directory Lightweight Directory Services Setup Wizard finishes installing AD LDS, it displays this message: "You have successfully completed the Active Directory Lightweight Directory Services Setup Wizard." When the Completing the Active Directory Lightweight Directory Services Setup Wizard page appears, click Finish to close the wizard.

Set DFS Productivity Layer .NET assemblies for FAST Search Server 2010 for SharePoint

  1. Locate and verify the version number of the following four DFS Productivity Layer .NET assemblies that are included in DFS hotfix 1049. When extracted to the default path, these files are located in the directory %local%\emc-dfs-sdk-6.5\emc-dfs-sdk-6.5\lib\dotnet:

    • Emc.Documentum.FS.DataModel.Core.dll (version number 6.5.0.231)

    • Emc.Documentum.FS.DataModel.Shared.dll (version number 6.5.0.231)

    • Emc.Documentum.FS.runtime.dll (version number 6.5.0.231)

    • Emc.Documentum.FS.Services.Core.dll (version number 6.5.0.231)

  2. On the FAST Search Server 2010 for SharePoint administration server, deploy the DFS Productivity Layer .NET assemblies to the bin directory of the installation directory.

    Note:

    You can drag and drop the four DLLs into the bin directory to deploy them, but you might have to turn off User Account Control to do this.

Enable custom security trimming

  1. Run this procedure on each server in the farm that is running FAST Search Server 2010 for SharePoint.

    Note:

    You must re-enable custom security trimming by repeating the steps in this procedure after you install a Service Pack (SP) or a Cumulative Update (CU) for FAST Search Server 2010 for SharePoint.

    Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. Stop the FAST Search for SharePoint service. You can use the nctrl.exe reference command-line tool.

  3. On the Start menu, click All Programs.

  4. Click Microsoft FAST Search Server 2010 for SharePoint.

  5. Right-click Microsoft FAST Search Server 2010 for SharePoint shell and select Run as administrator.

    Warning:

    If you do not run this script as an administrator, you will not be able to start the FAST Search Server 2010 for SharePoint Security Sync Service.

  6. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    customsecurity.ps1 -action enable

  7. Accept the default values, except when you are prompted to provide the full name of the AD LDS. Specify the full name of the AD LDS configuration location in the following pattern: LDAP://<FAST Search Server 2010 for SharePoint server host name>:<port number>/CN=Config,<AD LDS partition name>, for example LDAP://contoso.com:52000/CN=Config,CN=CustomSecurity,DC=FS4SP.

  8. Optional: You can choose to specify a claim type for the claim that will contain the group memberships from Documentum. To do this, define the claim type in a URI format, for example: http://custom.documentum.local/claims/group.

    Note:

    Although the format is a URI, it does not point to an object that can be physically reached.

  9. Start the FAST Search for SharePoint service and verify that all of the services are running. You can use the nctrl.exe reference command-line tool.

Restart IIS

  1. If you have stopped IIS on the FAST Search Server 2010 for SharePoint administration server, restart it.

  2. Click Start, point to Administrative Tools, and then click Internet Information Services (IIS) Manager.

  3. In the Connections menu, expand the server menu and click Application Pools.

  4. Click FASTSearchAdminAppPool.

  5. If the IIS service is stopped, go to the Actions menu and click Start.

Configure the FAST Search Server for SharePoint Security Sync Service

  1. Run this procedure on the FAST Search Server 2010 for SharePoint administration server.

  2. Go to the bin directory of your FAST Search Server 2010 for SharePoint installation directory and open the Microsoft.SharePoint.Search.Extended.Security.TrimmingSync.exe.config file.

  3. Using the same information you provided when you configured the Indexing Connector for Documentum, edit the <Datasource.Documentum> section as follows:

    1. Set the Documentum user and password for the login account. Use the account that you determined to be the Documentum content access account. This should be a user with elevated user permissions on the Documentum Content Server. For more information, see Determine which Documentum content access account to use.

    2. Set the name of the Documentum repository. This can be a repository that has Trusted Content Services (TCS) enabled.

    3. Set the location of the Documentum Foundation Services (DFS) by providing the URI for the DFS.

  4. Edit the <Emc.Documentum> section, using the same Documentum bindings you entered when you configured the machine.config file on the SharePoint Server 2010. See Edit the machine.config file.

  5. Optional: Modify the syncIntervalMinutes parameter to adjust the synchronization interval. The default value is 60 minutes.

  6. Save and close the Microsoft.SharePoint.Search.Extended.Security.TrimmingSync.exe.config file.

Set the log on account for the FAST Search Server for SharePoint Security Sync Service

  1. Run this procedure on the FAST Search Server 2010 for SharePoint administration server.

  2. Click Start, point to Administrative Tools and click Server Manager.

  3. Expand the Configuration menu and select Services.

  4. Locate and right-click the FAST Search Server for SharePoint Security Sync service and then click Properties.

  5. Click the Log On tab.

  6. Select This account and type the account name and the password of the user who is running FAST Search Server 2010 for SharePoint.

  7. Optional: On the General tab, change the startup type from Manual to Automatic.

  8. Click Apply and then click OK.

Verify the keys and the description field in the AD LDS partition

  1. Now that you have run the FAST Search Server for SharePoint Security Sync Service, verify that the following keys and descriptions exist in the AD LDS partition.

  2. Using ADSI Edit, or another tool that lets you view and edit the AD LDS partition, verify that the hierarchy looks like the following example. Also, make sure that the "description" field under CN=Setup contains a DataSource value of the CN=Setup (for example: DataSource:CN=Documentum,CN=DataSources,CN=CustomSecurity,DC=FS4SP):

    Copy Code
    CN=Config
   CN=Setup
      A "description" field containing a DataSource value of the CN=Setup.
CN=DataSources
   CN=Documentum
      CN=Config
   CN=GroupMemberships
      A "description" field containing semi-colon separated group membership information.

  3. If the CN=Config is missing, create a new object of the type "container" named Config.

  4. If the CN=Setup under CN=Config is missing, create a new object of the type "controlAccessRight" named Setup.

Optional: Troubleshoot the FAST Search Server for SharePoint Security Sync Service

  1. If the FAST Search Server for SharePoint Security Sync Service stops after it has started, investigate the Event Viewer for an event that mentions a host setup failure.

    Run this procedure on the FAST Search Server 2010 for SharePoint administration server.

    1. Open the Server Manager.

    2. On the Start menu, click Administrative Tools.

    3. Click Event Viewer.

    4. Investigate the events and look for host setup failure messages.

  2. If you find an event that indicates a host setup failure, open and edit the Microsoft.SharePoint.Search.Extended.Security.TrimmingSync.exe.config file located in the bin directory of your FAST Search Server 2010 for SharePoint installation directory.

    1. Search for 10100. You should retrieve the following line: <add baseAddress="http://localhost:10100/secTrimmingSync" />

    2. Edit the port number 10100 to a port number that the FAST Search Server 2010 for SharePoint Administrator user has access to. For example, you can use port number 13279. This port is opened by default on a FAST Search Server 2010 for SharePoint server.

    3. Save and close the Microsoft.SharePoint.Search.Extended.Security.TrimmingSync.exe.config file.

  3. Start the FAST Search Server for SharePoint Security Sync Service.

    1. On the Start menu, click Administrative Tools.

    2. Click Server Manager.

    3. Expand the Configuration menu and select Services.

    4. Locate and right click the FAST Search Server for SharePoint Security Sync service and then click Start.

Configure a Documentum content source in the FAST Search Content SSA in the SharePoint 2010 Central Administration.

Before a crawl, create crawl rules to include or exclude specific content in Documentum. Use the following procedure to create a crawl rule for the Indexing Connector for Documentum.

Run this procedure on the SharePoint Server 2010 server that contains the FAST Search Content search service application.

Create a crawl rule for the Indexing Connector for Documentum

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content search service application.

  2. Open SharePoint Central Administration, and then click Manage Service Applications.

  3. Click the name of your FAST Search Content search service application for which you want to create a crawl rule.

  4. Under Crawling, click Crawl Rules.

  5. On the Manage Crawl Rules page, click New Crawl Rule.

  6. On the Add Crawl Rule page, specify the following information to create at least one crawl rule.

    • In Path box, type the path of the content that you want to crawl.

      You can use wildcard "*" or regular expression syntax. Because Documentum uses case sensitive names for the content, select the Match case check box. Refer to the section Syntax to refer to a Documentum object for examples.

    • In Crawl Configuration section, select Include all items in this path, and then select Crawl complex URLs (URLs that contain a question mark - ?).

    • In the Specify Authentication section, select Specify a different content access account, and then type the Documentum content access account and password in the appropriate boxes. See the procedure Determine which Documentum content access account to use earlier in this article.

    • Make sure that the Do not allow Basic Authentication check box is cleared.

  7. Click OK to finish configuration.

    Note:

    • You can create multiple crawl rules for Documentum to include or exclude Documentum content.

    • You can use different crawl rules to specify different content access accounts for different Documentum content. For example, you have two repositories and two content access accounts for each repository. The Documentum content access account specified in a crawl rule will only be applied to Documentum content covered by the path in that crawl rule. In that case, you must set up a FAST Search Server for SharePoint Security Sync service for each repository.

Create a content source for the Indexing Connector for Documentum

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content search service application.

  2. Open SharePoint Central Administration, and then click Manage Service Applications.

  3. Click the name of your FAST Search Content search service application in which you want to create a content source.

  4. On the Search Administration page, under Crawling, click Content Sources.

  5. On the Manage Content Sources page, click New Content Source.

  6. On the Add Content Source page, do the following:

    1. In the Name box, type the name of the content source.

    2. In the Content Source Type section, select Custom Repository.

    3. In the Type of Repository section, select the SharePoint 2010 Indexing Connector for Documentum. Mark the radio button in front of the name that you specified for the Indexing Connector for Documentum when you registered it with the FAST Search Content search service application, for example dctm.

    4. In the Start Addresses section, type the start addresses. The start address format is the same as the format used to specify the path. You can type more than one start address for the content source, one per line.

      Refer to the section Syntax to refer to a Documentum object for examples.

    5. In the Crawl Schedules section, select schedules from the Full Crawl and Incremental Crawl drop-down lists, or create schedules for each kind of crawl.

    6. In the Content Source Priority section, assign a priority level to the content source according to your business requirements.

    7. Select Start full crawl of this content source to start a crawl immediately after the content source is created.

    8. Click OK to finish the configuration and accept all configured options.

    The Documentum content source is configured and the system can crawl Documentum content repositories that are specified in the content source.

    SharePoint Server 2010 supports a scalable architecture for performance scale-out. You can deploy more than one crawl server and configure multiple SharePoint Server 2010 crawlers to crawl the EMC Documentum database simultaneously.

Syntax to refer to a Documentum object

The format to refer to a Documentum object that you use for the path (when you set up a crawl rule) and the start address (when you set up a content source) is defined in the following table:

Type of Documentum object | Syntax for the path or the start address
Repository | dctm://<clientapphostname>/<repository name>
Cabinet | dctm://<clientapphostname>/<repository name>/<cabinet name>
Folder | dctm://<clientapphostname>/<repository name>/<cabinet name>/<folder name>
Document | dctm://<clientapphostname>/<repository name>/<cabinet name>/<folder name>/…/<folder name>?DocSysID=<r_object_id> (where r_object_id is the object id of that document)

<clientapphostname> is the host name of your Documentum client application such as Webtop or DA. The <clientapphostname> configured here should be same as the same as the one used in content source. <repository name>, <cabinet name>, and <folder name> are case sensitive.


Configure and use the Exchange connector (FAST Search Server 2010 for SharePoint)

This article describes how to create a crawl rule and add a content source for crawling Microsoft Exchange Server public folders.

Before you begin, ensure that the user running the FAST Search Content Search Service Application has at least Read permission to the public folder.

Create a crawl rule

This section describes the steps that are required to create a crawl rule. You must create a crawl rule if the default content access account does not have Read permission to the Exchange public folders that you want to crawl.

To create a crawl rule

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawl Rules.

  5. On the Manage Crawl Rules page, click New Crawl Rule.

  6. In the Path section, in the Path box, type a path to be crawled.

    When creating a crawl rule, the URL that you type in the Path box should be in the following form: <protocol>://hostname/*
    where <protocol> is the protocol that you want to use (typically http or https), and hostname is the NetBIOS or fully qualified domain name of the server that is running Exchange Server.

  7. In the Crawl Configuration section, select one of the following:

    • Exclude all items in this path (Default)

    • Include all items in this path

  8. In the Specify Authentication section, select the type of crawl authentication to use. This section is used only when Include all items in this path is selected.

  9. Click OK to add the crawl rule.

Add a content source for Microsoft Exchange public folders

Use the following procedures to create a content source for Exchange Server public folders. Note that there are two procedures listed that describe separate processes to add a content source based on the Exchange Server version. The first procedure describes how to add a content source for Exchange Server 2007 and earlier, the second describes how to add a content source from Exchange Server 2007 with Service Pack 2 (SP2) and later versions. This includes Microsoft Exchange Server 2010.

To add a content source for Exchange Server 2007 and earlier public folders

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Content Sources.

  5. On the Manage Content Sources page, click New Content Source.

  6. On the Add Content Source page, in the Name box, type the name that you want to use for the content source.

  7. In the Content Source Type section, select Exchange Public Folders.

  8. In the Start Addresses section, type the URLs for the Exchange Server public folders that you want to crawl. These URLs are typically in one of the following forms:

    • <protocol>://host name/public

      where <protocol> can be http or https, and host name is the NetBIOS or fully qualified domain name (FQDN) of the server that is running Exchange Server.

    • <protocol>://host name/public/subfolder

      where <protocol> can be http or https, host name is the NetBIOS or FQDN of the server that is running Exchange Server, and subfolder is the name of the specific subfolder that you want to crawl.

    For example, if you want to crawl all subfolders in the public folder on a server that is named exch-01 and that is in the Contoso domain, and that server does not use SSL, you could type either http://exch-01/public or http://exch-01.contoso.com. To crawl only a specific subfolder named Bob in the same public folder, type http://exch-01/public/bob or http://exch-01.contoso.com/bob.

    Note:

    For performance reasons, you cannot add the same start addresses to multiple content sources.

  9. In the Crawl Settings section, select the behavior for the kind of content that you selected.

  10. In the Crawl Schedules section, you can optionally specify when to start full and incremental crawls:

    • You can create a full crawl schedule by clicking the Create Schedule link below the Full Crawl list.

    • You can create an incremental crawl schedule by clicking the Create schedule link below the Incremental Crawl list.

  11. Click OK.

  12. Repeat steps 4 through 10 for any additional content sources that you want to create.

To add content sources for Microsoft Exchange 2007 SP2 and later versions public folders

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content Search Service Application.

  2. Open a Web browser and navigate to the Outlook Web Access Web page for the Exchange Server that has the public folders that you want to crawl.

  3. Log on to Outlook Web Access using any user account that has Read access to the public folders that you want to crawl.

  4. Navigate to the public folder that you want to crawl, right-click that folder, and then select Open in New Window.

  5. When the new window opens, navigate to the new window's address bar and copy the complete URL. This is the Outlook Web Access public folder address.

  6. In Central Administration, in the Application Management section, click Manage service applications.

  7. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  8. On the Search Administration page, under Crawling, click Content Sources.

  9. Click New Content Source.

  10. On the Add Content Source page, in the Name box, type the name that you want to use for the content source.

  11. In the Content Source Type section, select Exchange Public Folders.

  12. In the Start Addresses section, paste the Outlook Web Access public folder address that you copied in step 5.

  13. In the Crawl Settings section, select the behavior for the kind of content that you selected.

  14. In the Crawl Schedules section, you can optionally specify when to start full and incremental crawls:

    • You can create a full crawl schedule by clicking the Create Schedule link below the Full Crawl list.

    • You can create an incremental crawl schedule by clicking the Create schedule link below the Incremental Crawl list.

  15. To create the content source, click OK.


Configure and use the Lotus Notes connector (FAST Search Server 2010 for SharePoint)

Updated: April 2, 2013

This article describes the administrative roles, required software, user accounts, and processes that are required to install and operate the Lotus Notes Client and the Lotus Notes indexing connector integrated in SharePoint Server 2010 to work with FAST Search Server 2010 for SharePoint.

Required administrative roles

The following administrative roles are required to prepare the SharePoint Server 2010 crawl server to crawl Lotus Notes content that is hosted by one or more Lotus Domino databases:

  • Administrator of the Lotus Domino server that you want to crawl

  • Server administrator of the SharePoint Server 2010 crawl server that you want to use to crawl Lotus Notes content

  • Service application administrator for the FAST Search Content Search Service Application

Required software

The following software is required:

  • Lotus C++ API Toolkit for Notes.

  • Lotus Notes client application, available for purchase from IBM.

  • Lotus Notes Domino Server, available for purchase from IBM.

The following table shows combinations of versions of the Lotus Notes Domino server and Lotus Notes client that the Lotus Notes indexing connector integrated in SharePoint Server 2010 will work with.

This server version | With client 5.5.x | With client 6.x | With client 7.x | With client 8.x
Server 5.5.x | YES | YES | NO | NO
Server 6.x | NO | YES | NO | NO
Server 7.x | NO | NO | YES | YES
Server 8.x | NO | NO | NO | YES

User accounts required to crawl Lotus Domino databases

A Domino administrator must grant a Lotus Notes user ID (which represents a Domino user) at least the Reader permission to the Lotus Domino databases and individual documents that you want to crawl. The Domino administrator must also add this Lotus Notes user ID and the Windows domain user account that is assigned to the SharePoint Server Search 14 service (OSearch14) to a mappings database on the Lotus Domino server that you want to crawl.

Note:

Only the user account that is assigned to the OSearch14 service can be used to crawl Lotus Domino databases. You cannot use the default content access account or a crawl rule to specify a different user account to crawl Lotus Domino databases.

The following table summarizes the user accounts that are required to crawl Lotus Domino databases.

Required account Comment Example

Windows domain user account

The user account that is assigned to the OSearch14 service must also be a member of the Administrators group on the SharePoint Server crawl server.

Contoso\User1, where Contoso is the domain name and User1 is the name of the Windows domain user account.

Lotus Notes user ID

Must be granted at least Reader access on the Lotus Domino databases and on the individual documents that you want to crawl. The Domino certificate also contains this Lotus Notes user ID.

User2

Note:

The name of this account and its password do not have to match the Windows domain user account.

More information about this mappings table is provided later in this article.

Install the Lotus Notes client application

Use the following procedure to install the Lotus Notes client application on the SharePoint Server 2010 crawl server in the server farm with which you want to crawl a Lotus Domino database. Note that this client application serves as a protocol handler and is used to configure the Notes.ini file. Both are used by the SharePoint Server 2010 crawler when crawling Lotus Domino databases.

To install Lotus Notes

  1. Verify that the user account that is performing this procedure is a member of the Administrators group on the SharePoint Server 2010 crawl server and has at least Manager permissions on the Domino server.

  2. Copy the Lotus Notes client application to the SharePoint Server 2010 crawl server that you want to use to crawl Lotus Notes documents.

  3. Start the Lotus Notes Installation Wizard.

  4. In the Welcome to the Installation Wizard for Lotus Notes dialog box, click Next.

  5. On the License Agreement page, click I accept the terms in the license agreement, and then click Next to continue.

  6. On the Customer Information page, type a user name in the User Name box and the name of the organization in the Organization box, or accept the default settings, and then click Next.

  7. On the Installation Path Selection page, specify the path that you want to use for the program and data files, or accept the default installation paths, and then click Next.

    Note:

    By default, program files are stored in the <SystemDrive>:\Program Files\lotus\notes\ folder and data files are stored in the <SystemDrive>:\Program Files\lotus\notes\data\ folder, where <SystemDrive> is the drive on which Lotus Notes is installed.

  8. On the Custom Setup page, select the program features that you want to install on the local hard disk drive, and then click Next.

    The following table shows the features and sub-features that are required by the Lotus Notes indexing connector.

    Feature Sub-feature

    Notes Client

    Client Help Files

    Domino Enterprise Connection Services (DECS)

    Domino Designer

    Designer Help

  9. On the Ready to Install the Program page, if you do not want Lotus Notes to be the default e-mail program, clear the selection Make Notes my default e-mail program.

  10. Click Install.

    The Installing Lotus Notes page shows the installation status.

  11. On the Install Wizard Completed page, click Finish.

Grant permissions on the data folder

Use the following procedure to grant Full Control permissions for the WSS_WPG group on the <SystemDrive>:\Program Files\Lotus\Notes\Data folder on the SharePoint Server 2010 crawl server.

To Grant permissions on the data folder

  1. Verify that the user account that is performing this procedure is a member of the Administrators group on the SharePoint Server 2010 crawl server and has at least Manager permissions on the Domino server.

  2. On the SharePoint Server 2010 crawl server, click Start, point to All Programs, click Accessories, and then click Windows Explorer.

  3. In Windows Explorer, go to the <SystemDrive>:\Program Files\Lotus\Notes\Data folder, where <SystemDrive> is the drive on which Lotus Notes is installed.

  4. Right-click the Data folder, and then click Sharing and Security.

  5. In the Properties dialog box, on the Security tab, click Add.

  6. In the Select the object names to select box, do one of the following, and then click OK:

    • If the crawl component of the FAST Search Content Search Service Application is installed on an Active Directory domain controller, type domain\WSS_WPG, where domain is the name of the domain that is associated with the domain controller.

    • If the crawl component of the FAST Search Content Search Service Application is installed on a server that is not an Active Directory domain controller, type server\WSS_WPG, where server is the NetBIOS name of the SharePoint Server 2010 crawl server.

  7. In the Properties dialog box, in the Permissions for WSS_WPG section, select the Allow box in the Full control row, and then click OK.

Configure the Lotus Notes client application

Use the following procedure to configure the Lotus Notes client application. Note that the configuration settings selected in this procedure are written to the Notes.ini file, which the SharePoint Server 2010 crawler uses to discover how to connect to the Lotus Domino server.

To configure Lotus Notes

  1. Verify that the user account that is performing this procedure is a member of the Administrators group on the SharePoint Server 2010 crawl server and has at least Manager permissions on the Domino server.

  2. On the SharePoint Server 2010 crawl server, click Start, point to All Programs, point to Lotus Applications, and then click Lotus Notes.

  3. On the Welcome page, click Next.

  4. On the User Information page, type the user name associated with the Domino certificate in the Your name box.

  5. Type the hierarchical name of the Domino server that you want to crawl in the Domino server box — for example, Contoso/marketing/west.

  6. Ensure that I want to connect to a Domino server is selected, and then click Next.

  7. On the Notes ID File page, click Browse, and then locate where the certificate is stored. Select the certificate, click Open, and then click Next.

  8. Click Yes to copy the certificate to the specified location.

    Note:

    If you are not prompted for a Domino certificate, click Previous, and ensure that you have entered the correct information.

  9. If a dialog box appears that informs you that you are not authorized to access the specified directory, click OK to close the dialog box. Note that this error is expected if the account that you are logged on with does not have access to the mail folder on the Domino server.

  10. On the Instant Messaging Setup page, cancel the selection Setup instant messaging.

  11. Click Next.

  12. On the Additional Services page, click Next.

  13. In the Lotus Notes message box, click OK.

    The Lotus Notes Welcome screen appears.

    Leave the Lotus Notes client application open. You will need it for the next procedure.

Verify access to the Lotus Domino database that you want to crawl

Use the following procedure to verify that the certificate that you installed has access to the database that you want to crawl.

To verify access

  1. Verify that the user account that is performing this procedure is a member of the Administrators group on the SharePoint Server 2010 crawl server and has at least Manager permissions on the Domino server.

  2. In Lotus Notes, click File, point to Database, and then click Open.

  3. In the Open Database dialog box, select the Lotus Domino server that you want to connect to from the Server list.

  4. In the Database list, select the database that you want to connect to, and then click Open.

    The documents that are contained by the database that you selected are displayed in the Document Name section. This means that the SharePoint Server 2010 crawl server has the necessary permissions to crawl these documents.

  5. Repeat steps 1 through 3 for each additional database that you want to verify access to.

  6. On the File menu, click Exit Notes.

Configure security mappings

Use the information in the following table to help you perform the next procedure.

Item Comment

Mappings database name

Name of the Lotus Domino database that is used to map Lotus Notes user IDs to Windows domain user accounts

Lotus Notes field name

Name of the field in the Lotus Domino database file that is used to store Lotus Notes user IDs

Windows user field name

Name of the field in the Lotus Domino database file that is used to store Windows user names

Form name

Name of the form that stores the Lotus Notes field name and Windows user field name fields

View name

Name of the view for the form that stores the Lotus Notes user IDs to Windows user name mappings

Note:

This name is case-sensitive.

Create the mappings database

Use this procedure to create a mappings database by using Domino Designer. You need only one mappings database for each forest of Domino servers that contain databases that you want to crawl.

To create a mappings database

  1. Verify that the user account that is performing this procedure is a member of the Administrators group on the SharePoint Server 2010 crawl server and has at least Manager permissions on the Domino server that you want to crawl.

  2. On the SharePoint Server 2010 crawl server, open Domino Designer.

  3. Click File, point to Database, and then click New.

  4. In the New Database dialog box, do the following:

    1. Select the Domino server from the Server name list.

    2. In the Title box, type a title for the new database.

      This content automatically populates the File Name box, appended with the .nsf file name extension.

      Note that if the title that you chose is more than eight characters long, the file name will be truncated.

    3. Click OK to close the New Database dialog box.

  5. Click Create, point to Design, and then click Form.

  6. Click Create, and then click Field.

  7. In the Field dialog box, in the Name box, type the name that you want to use for this field. Note that this field will be used to store the Lotus Notes user IDs.

  8. Close the dialog box to save the field.

  9. Click Create, and then click Field.

  10. In the Field dialog box, in the Name box, type the name that you want to use for this field. Note that this field will be used to store the Windows domain user accounts.

  11. Close the dialog box to save the field.

  12. Click File, click Save, and then do the following:

    1. Type a name for this form in the Save Form as box.

    2. Click OK to close the dialog box.

  13. On the Create menu, point to Design, and then click View.

  14. In the Create View dialog box, do the following:

    1. Type a name for this view in the View name box.

    2. Select Shared from the View type list.

    3. Click OK to save the view.

  15. Open the view that you created in step 13.

  16. On the Objects tab, select the column that you created in step 10. In the lower-right pane, select Field and then select the field that has the same name.

  17. On the Objects tab, select the column that you created in step 12. In the lower-right pane, select Field and then select the field of the same name.

  18. Click File and then click Save to save the view, and then close Domino Designer.

Add user accounts to the mappings database

Use this procedure to add user accounts to the mappings database using the Lotus Notes client. You should add all accounts which need access to the mappings database and the Domino server.

To add user accounts to the mappings database

  1. Verify that the user account that is performing this procedure is a member of the Administrators group on the SharePoint Server 2010 crawl server and has at least Manager permissions on the Domino server.

  2. On the SharePoint Server 2010 crawl server, open the Lotus Notes client application.

  3. Click File, point to Database, and then click Open.

  4. In the Open Database dialog box, do the following:

    1. Select the Domino server from the Server name list

    2. Select the mappings database that you created earlier.

    3. Click Open.

  5. In the left pane, select the view that you created for this database.

  6. Click Create, and then click the name of the form that you created earlier.

  7. In the form, in the field that you created to store the Lotus Notes user IDs, type a Lotus Notes user ID that you want to map to a Windows domain user account — for example, ContosoUser. Note that this field is case-sensitive.

  8. In the field that you created to map to the Lotus Notes user IDs, type the Windows domain user account that you want to map to the Lotus Notes user ID that you entered in step 6. Note that this must be in the form of domain\user, for example, Contoso\user1.

  9. Click File, and then click Save to save the document.

  10. Repeat steps 6 through 8 if you want to add more mappings. Otherwise, go to step 10.

  11. When finished, save the form and then close the Lotus Notes client application.

Restart the SharePoint Server 2010 crawl server

You must restart the SharePoint Server 2010 crawl server before continuing to the next procedure.

Important:

After the SharePoint Server 2010 crawl server restarts, do not open the Lotus Notes client application again. This is because the Lotus Notes client application might lock files that could cause the following procedures and crawling Lotus Domino databases to fail.

Register Lotus Notes with the SharePoint Server 2010 crawl server

Use the following procedure to register Lotus Notes with the operating system of the SharePoint Server 2010 crawl server.

To register Lotus Notes

  1. Verify that the user account that is performing this procedure is a member of the Administrators group on the SharePoint Server 2010 crawl server.

  2. Run Notessetup.exe on the SharePoint Server 2010 crawl server by using the same credentials that are used to provision the Lotus Notes Connector.

  3. On the SharePoint Server 2010 crawl server, in Windows Explorer, go to the <SystemDrive>:\Program Files\Microsoft Office Servers\14\Bin\1033 folder, where <SystemDrive> is the drive on which Microsoft FAST Search Server 2010 for SharePoint is installed.

  4. Double-click NotesSetup.exe.

  5. On the Welcome to the Lotus Notes Index Setup Wizard page, click Next.

  6. In the Register Lotus Notes for use with SharePoint Server 2010 dialog box, do the following:

    1. In the Location of the notes.ini file box, ensure that the correct path of the Notes.ini file is specified. The default path of this file is <SystemDrive>:\Program Files\lotus\notes\notes.ini, where <SystemDrive> is the drive on which Lotus Notes is installed.

    2. In the Location of the Lotus Notes install directory box, ensure that the correct path of the Lotus Notes installation directory is specified. By default, the path of this directory is <SystemDrive>:\Program Files\lotus\notes.

    3. In the Password box, type the password for the user name that is associated with the Domino certificate.

    4. In the Confirm Password box, retype the password for the user name that is associated with the Domino certificate.

  7. We recommend that you leave the Ignore Lotus Notes security while building the index box cleared. This allows the crawl to include all Lotus Notes documents in the index without restriction. Security for these documents and objects is determined by the mappings table and provides security data without excluding documents from the index.

  8. Click Next.

  9. On the Specify Lotus Notes Owner Field to Windows User Name Mapping page, do the following:

    1. In the Lotus Notes server name text box, type the Lotus Notes Domino server name or the server IP address.

      Note:

      The server has a Windows NetBIOS name and a Lotus Notes Domino server name. If you type the server name instead of the IP address, make sure that you type the Lotus Notes Domino server name rather than the Windows NetBIOS name. If you do not type the Lotus Notes Domino server name or the IP address, the crawler will not be able to crawl Lotus Notes content.

    2. In the Lotus Notes database file name box, type the file name of the Lotus Domino database that maps the Lotus Notes user IDs to Windows domain user accounts. Be sure to include the .nsf file name extension with this name — for example, Mappings.nsf.

    3. In the View name box, type the view name of the Lotus Domino database that stores the Lotus Notes user IDs to Windows user name mappings.

    4. In the Lotus Notes field name column title box, type the name of the column in the Lotus Notes database file that is used to store Lotus Notes user IDs.

    5. In the Windows user name column title box, type the name of the column in the Lotus Notes database file that is used to store the Windows user accounts.

  10. Click Next.

  11. On the Completing the Lotus Notes Index Setup Wizard page, click Finish.

Provision the Lotus Notes indexing connector

Use the following procedure to provision Lotus Notes indexing connector with the operating system of the SharePoint Server 2010 crawl server.

To provision Lotus Notes

  1. Verify that the user account that is performing this procedure is a member of the Administrators group on the SharePoint Server 2010 crawl server.

  2. Open SharePoint Central Administration; in the System Settings section click Manage services on Server.

  3. On the Services on Server page, in the Service column, find the Lotus Notes Connector service.

  4. In the Action column, click Start.

  5. On the Lotus Notes connector settings page, in the application pool section, select Create new application pool, and then enter a name for the new application pool.

  6. In the Configurable drop-down, select or register the same security account used for installation of the NotesSetup.exe.

  7. Click Provision.

The Lotus Notes indexing connector is now provisioned and started.

Restart the OSearch14 service

The server administrator of the SharePoint Server 2010 crawl server must restart the OSearch14 service before a content source can be created for Lotus Domino databases.

Important:

Do not use the Services on Server page on the SharePoint Central Administration Web site to restart this service. Doing so resets the content index, which requires you to do a full crawl of all content to rebuild the index.

Change History

Date | Description
April 02, 2013 | In the "Register Lotus Notes with the crawl server" procedure, added that in the Lotus Notes server name text box you must type the Lotus Notes Domino server name rather than the Windows NetBIOS name.
May 29, 2010 | Initial publication


Start, pause, resume, or stop a crawl (FAST Search Server 2010 for SharePoint)

When you perform a full crawl, all content specified by the content source is crawled even if the content already exists in the index. To perform a full crawl, you must crawl the content defined in a particular content source individually. Note that using the Start all crawls link on the Manage Content Sources page results in all content sources being crawled using an incremental crawl, unless the system detects that a full crawl is required.

You must create at least one content source before a crawl can occur. After you create a content source, you can edit or delete it at any time.

To start, pause, resume, or stop a crawl of a content source

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Content Sources.

  5. On the Manage Content Sources page, in the list of content sources, point to the name of the content source that you want, click the arrow that appears, and then click one of the following menu items:

    • Start Full Crawl. The value in the Status column changes to Crawling Full for the selected content source. The value in the Status column does not automatically change when the crawl is complete. To update the Status column, refresh the Manage Content Sources page by clicking Refresh.

    • Start Incremental Crawl. The value in the Status column changes to Crawling Incremental for the selected content source. The value in the Status column does not automatically change when the crawl is complete. To update the Status column, refresh the Manage Content Sources page by clicking Refresh.

    • Resume Crawl. The value in the Status column changes back to Crawling Full or Crawling Incremental for the selected content source.

    • Pause Crawl. The value in the Status column changes to Paused for the selected content source.

    • Stop Crawl. You must click OK to confirm that you want to stop the crawl.


Manage crawl rules (FAST Search Server 2010 for SharePoint)

You can add a crawl rule to include or exclude specific paths when you crawl content. When you include a path, you can optionally provide alternative account credentials to crawl it. In addition to creating or edit crawl rules, you can test, delete, or reorder existing crawl rules.

In this article:

  • To create or edit a crawl rule

  • To test a crawl rule on a URL

  • To delete a crawl rule

  • To reorder crawl rules

To create or edit a crawl rule

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawl Rules. The Manage Crawl Rules page appears.

  5. To create a new crawl rule, click New Crawl Rule. To edit an existing crawl rule, in the list of crawl rules, point to the name of the crawl rule that you want to edit, click the arrow that appears, and then click Edit.

  6. On the Add Crawl Rule page, in the Path section:

    • In the Path box, type the path to which the crawl rule will apply. You can use standard wildcard characters in the path.

    • Select the Use regular expression syntax for matching this rule check box to use regular expressions instead of wildcard characters.

    • Select the Match case check box if you want the capitalization in the provided path to exactly match the capitalization of the actual path.

  7. In the Crawl Configuration section, select one of the following options:

    • Exclude all items in this path. Select this option if you want to exclude all items in the specified path from crawls. If you select this option, you can refine the exclusion by selecting the following:

      • Exclude complex URLs (URLs that contain question marks - ?). Select this option if you want to exclude URLs that contain parameters that use the question mark (?) notation.

    • Include all items in this path. Select this option if you want all items in the path to be crawled. If you select this option, you can further refine the inclusion by selecting any combination of the following:

      • Follow links on the URL without crawling the URL itself. Select this option if you want to crawl links contained within the URL, but not the starting URL itself.

      • Crawl complex URLs (URLs that contain a question mark - ?). Select this option if you want to crawl URLs that contain parameters that use the question mark (?) notation.

      • Crawl SharePoint content as HTTP pages. Normally, SharePoint sites are crawled by using a special protocol. Select this option if you want SharePoint sites to be crawled as HTTP pages instead. When the content is crawled by using the HTTP protocol, item permissions are not stored.

  8. In the Specify Authentication section, perform one of the following actions:

    Note:

    This option is not available unless the Include all items in this path option is selected in the Crawl Configuration section.

    • To use the default content access account, select Use the default content access account.

    • If you want to use a different account, select Specify a different content access account and then perform the following actions:

      1. In the Account box, type the user account name that can access the paths that are defined in this crawl rule.

      2. In the Password and Confirm Password boxes, type the password for this user account.

      3. To prevent basic authentication from being used, select the Do not allow Basic Authentication check box. The server attempts to use NTLM authentication. If NTLM authentication fails, the server attempts to use basic authentication unless the Do not allow Basic Authentication check box is selected.

    • To use a client certificate for authentication, select Specify client certificate, expand the Certificate menu, and then select a certificate.

    • To use form credentials for authentication, select Specify form credentials, type the form URL (the location of the page that accepts credentials information) in the Form URL box, and then click Enter Credentials. When the logon prompt from the remote server opens in a new window, type the form credentials with which you want to log on. You are prompted if the logon was successful. If the logon was successful, the credentials that are required for authentication are stored on the remote site.

    • To use cookies, select Use cookie for crawling, and then select either of the following options:

      • Obtain cookie from a URL. Select this option to obtain a cookie from a Web site or server.

      • Specify cookie for crawling. Select this option to import a cookie from your local file system or a file share. You can optionally specify error pages in the Error pages (semi-colon delimited) box.

  9. Click OK.

To test a crawl rule on a URL

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawl Rules.

  5. On the Manage Crawl Rules page, in the Type a URL and click test to find out if it matches a rule box, type the URL that you want to test.

  6. Click Test. The result of the test appears below the Type a URL and click test to find out if it matches a rule box.

To delete a crawl rule

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawl Rules.

  5. On the Manage Crawl Rules page, in the list of crawl rules, point to the name of the crawl rule that you want to delete, click the arrow that appears, and then click Delete.

  6. Click OK to confirm that you want to delete this crawl rule.

To reorder crawl rules

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawl Rules.

  5. On the Manage Crawl Rules page, in the list of crawl rules, in the Order column, specify the crawl rule position that you want the rule to occupy. Other values shift accordingly.


Manage crawler impact rules (FAST Search Server 2010 for SharePoint)
See Also

You can increase or decrease the request frequency of the crawl by adding Crawler impact rules.

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawler Impact Rules.

  5. Click Add Rule.

    1. In the Site box, type the name of the site the crawl rule should apply to.

    2. In the Request Frequency box, set the desired number of simultaneous request and/or the desired request interval.

      Warning:

      Do not overload sites with requests. Request few documents at the same time and set a delay between requests.

    3. Click OK.

The request rate depends on the hardware of the server from which the FAST Search Content SSA is running. On high-end servers, you might (by default) request up to 16 documents in parallel from a single server. Set crawler impact rules to throttle the request rate for servers that should not be crawled with a high request rate, such as external Web sites for example.

See Also

Concepts

Plan for crawling and federation (FAST Search Server 2010 for SharePoint)


Verify the crawl (FAST Search Server 2010 for SharePoint)

It is recommended that you view the crawl log to determine whether any crawl errors occurred.

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Content Sources. Click the Refresh button to refresh the crawl status.

  5. On the Search Administration page, under Crawling, click Crawl log. After the crawl of your content source finishes, verify that a zero is listed in the Errors column. If errors occurred, the number in the Errors column will be larger than zero. In that case, click the number in the Errors column to view the crawl log for that content source.

Tip:

If the crawl is running, but your crawls do not finish and you do not get results in the Crawl Log, look in the Windows Event Viewer to determine whether there is a problem. Open the Windows Event Viewer and view the log messages in Windows Logs > Application.


Verify feeding (FAST Search Server 2010 for SharePoint)
See Also

To verify that documents are fed to FAST Search Server 2010 for SharePoint:

  1. Open a Microsoft FAST Search Server 2010 for SharePoint shell on the FAST Search Server 2010 for SharePoint administration server.

  2. Run the command:

    Get-FASTSearchContentCollection -Name "<NameOfContentCollection>"

    Where "<NameOfContentCollection>" is the name of the content collection you want to feed to. The default content collection is named "sp".

  3. Verify that the content collection contains some documents.

    If the content collection does not contain any documents:

    1. Inspect the Windows Event Log on the Microsoft SharePoint Server 2010 machine that hosts the FAST Search Content Search Service Application if there are any issues accessing FAST Search Server 2010 for SharePoint.

    2. Inspect and if necessary correct the FAST Search Content SSA content distributor address.

    3. Verify that SSL communication has been enabled to secure the connection between the Microsoft SharePoint Server 2010 that hosts the FAST Search Content SSA and FAST Search Server 2010 for SharePoint.

See Also

Concepts

Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint)


Remove URLs from search results (FAST Search Server 2010 for SharePoint)
See Also

To remove specific URLs from search result in FAST Search Server 2010 for SharePoint, you must find the Item ID of the URL, remove this Item ID from the FAST Search Server 2010 for SharePoint content collection and create a crawl rule in the SharePoint Server 2010 Central Administration to prevent the URL from being crawled again.

Find the unique Item ID of the URL

Follow this procedure on the SharePoint Server 2010 server.

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click the name of your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawl log.

  5. On the Search Service Application: Crawl Log - Content Source page, click URL.

  6. On the Search Service Application: Crawl Log - URL page, in the Search By section, select the radio button in front of URL or Host name and type the URL in the box, for example, http://www.contoso.com. Select Exact match and then click Search.

  7. Copy or note the Item ID that is displayed.

Remove the Item ID from the content collection

Follow this procedure on the FAST Search Server 2010 for SharePoint server.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    docpush.exe -c <content collection> -l verbose -U -d ssic://<Item ID>

    Where:

    <content collection> is the content collection that the item is found in. The default content collection is sp.

    <Item ID> is the Item ID of the URL you want to remove from the content collection. See the procedure Find the unique Item ID of the URL for information how to obtain the Item ID.

Create a crawl rule

Follow this procedure on the SharePoint Server 2010 server.

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click the name of your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawl rules. The Manage Crawl Rules page appears.

  5. To create a new crawl rule, click New Crawl Rule.

  6. On the Add Crawl Rule page, in the Path section:

    • In the Path box, type the path of the URL to which the crawl rule will apply, for example http://www.contoso.com. You can use standard wildcard characters in the path.

    • Select the Use regular expression syntax for matching this rule check box to use regular expressions instead of wildcard characters.

    • Select the Match case check box if you want the capitalization in the provided path to exactly match the capitalization of the actual path.

  7. In the Crawl Configuration section the following option:

    • Exclude all items in this path. Select this option if you want to exclude all items in the specified path from crawls. If you select this option, you can refine the exclusion by selecting the following:

      • Exclude complex URLs (URLs that contain question marks - ?). Select this option if you want to exclude URLs that contain parameters that use the question mark (?) notation.

  8. Click OK.

See Also

Reference

docpush reference

Concepts

Manage crawl rules (FAST Search Server 2010 for SharePoint)


Delete a file type from the content index (FAST Search Server 2010 for SharePoint)
See Also

Use the following procedure to delete file types from the content index.

To delete a file type from the content index:

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click the name of your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click File Types.

  5. On the Search Service Application: Manage File Types page, click New file type.

  6. In the File name extension field, type the file name extension that you want to exclude.

  7. Click OK.

  8. A confirmation dialog opens; click OK to confirm the exclusion of the file name extension.

The Search Service Application: Manage File Types page refreshes and the file name extension is now excluded.

See Also

Concepts

IFilter and file type reference (FAST Search Server 2010 for SharePoint)


Include a file type in the content index (FAST Search Server 2010 for SharePoint)
See Also

FAST Search Server 2010 for SharePoint crawls and extracts metadata and content from most common file types.

The File Types list that can be viewed in your FAST Search Content Search Service Application is an exclude list of file types that should not be crawled. To include a specific file type in the content index, you must remove the file type from this list or verify that the file type is not excluded by this list as described in Remove a file type from the exclude list.

If a file type is not excluded by the File Types list, it will still be crawled, but the actual content and metadata is only extracted if there an IFilter is registered for that file type. Consult the Crawl Log after a small test crawl with the file type you want to include, and the messages in the Crawl Log will indicate the next steps.

To access the Crawl Log:

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click the name of your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Crawl Log. Look for the following messages:

    • Unknown document format, skipping conversion

      This message indicates that you have to register and install a third-party IFilter. See Configure FAST Search Server 2010 for SharePoint to use a third-party IFilter for more information.

    • No filter available. Enable the Advanced Filter Pack for more filters

      This message indicates that the content and metadata can be extracted by enabling the Advanced Filter Pack. See Enable the Advanced Filter Pack for more information.

    • If none of these messages appear, perform a search and verify that the content is searchable.

In this article:

Remove a file type from the exclude list

Enable the Advanced Filter Pack

Configure FAST Search Server 2010 for SharePoint to use a third-party IFilter

Remove a file type from the exclude list

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click the name of your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click File Types.

  5. On the Search Service Application: Manage File Types page, mouse over the file name extension you want to include in the content index. A drop-down arrow will appear and the file name extension will be highlighted.

  6. Click the drop-down arrow and Delete appears.

  7. Click Delete to delete the file name extension.

  8. A confirmation dialog opens; click OK to confirm the deletion of the file name extension from the exclude list.

The Search Service Application: Manage File Types page refreshes and the selected file name extension is no longer excluded from the content index.

Enable the Advanced Filter Pack

By default, the Advanced Filter Pack is disabled. When you enable it, you will be able to extract metadata and content from several hundred file formats in addition to the document formats that are included in the Microsoft Filter Pack.

Follow the instructions in the article Enable Advanced Filter Pack (FAST Search Server 2010 for SharePoint).

Configure FAST Search Server 2010 for SharePoint to use a third-party IFilter

If you need to use a third-party IFilter, you have to register the IFilter with Windows Search first. After this, update the user_converter_rules.xml configuration file with the file extension, MIME type and format description of the IFilter. For more information, see user_converter_rules.xml reference.

See Also

Concepts

Enable Advanced Filter Pack (FAST Search Server 2010 for SharePoint)
user_converter_rules.xml reference
IFilter and file type reference (FAST Search Server 2010 for SharePoint)


Reset the content index (FAST Search Server 2010 for SharePoint)
See Also

Use the following procedures to reset the content index.

Warning:

The procedures will erase the content index. After a reset, search results will not be available until new crawls have been run.

Reset the content index in SharePoint Server 2010

  1. Verify that the user account that is performing this procedure is a service application administrator for your FAST Search Content Search Service Application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, in the list of service applications, click your FAST Search Content SSA.

  4. On the Search Administration page, under Crawling, click Index Reset.

  5. On the Search Service Application: Index Reset page, click Reset Now.

  6. A confirmation dialog box appears; click OK to confirm the content index reset. The Search Service Application: Search Administration page opens and the System Status is displayed.

Before you start any new crawls, you must manually clear the content from the content collection your FAST Search Content Search Service Application was feeding to. The default collection name is sp. Note that clearing the content collection is irreversible.

To clear a FAST Search Server 2010 for SharePoint content collection, log onto the server that hosts FAST Search Server 2010 for SharePoint and follow this procedure.

Clear the content collection

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    Clear-FASTSearchContentCollection -Name <collection name>

    Where:

    • <collection name> is the content collection you are about to clear.

  6. Wait for the command to finish. This may take some time.

See Also

Reference

Clear-FASTSearchContentCollection


Manage crawling with the FAST Search specific connectors (FAST Search Server 2010 for SharePoint)

There are three indexing connectors that are FAST Search Server 2010 for SharePoint specific. This section describes how to configure and manage crawling with each of these.

In this section:

Crawling Web content with the FAST Search Web crawler

Crawling database content with the FAST Search database connector

Crawling Lotus Notes content with the FAST Search Lotus Notes connector

Manage the crawling impact of the FAST Search specific connectors


Crawling Web content with the FAST Search Web crawler

The FAST Search Web crawler gathers Web items (or Web pages) from Web servers on a network. This is typically a bounded institutional or corporate network, but potentially it can crawl the entire Internet.

The FAST Search Web crawler works, in many ways, like a Web browser downloading content from Web servers. But unlike a Web browser that responds only to user input via mouse clicks or keyboard, the FAST Search Web crawler works from a set of configurable rules it must follow when it requests Web items. This includes, for example, how long to wait between requests for items, and how long to wait before checking for new or updated items.

How the FAST Search Web crawler works

The FAST Search Web crawler starts by comparing the start URL list against include and exclude rules specified in parameters in the XML file containing the crawl configuration. The start URL list is specified with either the start_uris or start_uri_files setting, and the rules via the include_domains and exclude_domains setting. Valid URLs are then requested from their Web servers at a rate determined by the request rate that is configured in the delay setting.

If fetched successfully, the Web item is parsed for hyperlinks and other meta-information, usually by a HTML parser built into the FAST Search Web crawler. The Web item's meta-information is stored in the FAST Search Web crawler meta-database, and the Web item content (the HTML body) is stored in the FAST Search Web crawler store. The hyperlinks are filtered against the crawl rules, and used as the next set of URLs to be downloaded. This process continues until all reachable content has been gathered, until the refresh interval (refresh setting) is complete or until another configuration parameter limiting the scope of the crawl is reached.

There are many different ways to adjust the configuration to suit a specific Web crawling scenario. The table lists some of the fundamental concepts used to set up and control the FAST Search Web crawler.

Concept | Explanation
Crawl collection | A set of Web sites crawled with the same configuration is called a crawl collection. A Web crawler may crawl multiple crawl collections at the same time, and submit these either to a single or separate content collections.
Crawl store | The FAST Search Web crawler stores crawled content locally on disk during crawling. The content is divided into two types; Web item content and meta data.
Include rules | Include rules specify what Web content should be included. However, they do not define where the FAST Search Web crawler should start the crawl.
Exclude rules | Exclude rules specify which hostnames, URLs or URL patterns should not be included in the crawl.
Start URL list | List of URLs to be crawled and collected first, from which additional hyperlinks may be extracted, tested against the rules, and added to work queues for further crawling.
Refresh interval | The time in minutes the FAST Search Web crawler will run before re-crawling the Web sites to check for new or modified content. The refresh interval should be set high enough to ensure the crawler has sufficient time to crawl all content. See the section Determine crawl schedules for information about calculating the refresh interval.
Request rate | The time in seconds between individual requests to a single Web site, configured with the delay setting. This option can be set to 0 to crawl as fast as possible, but requires the Web server owner's permission. For flexibility, different request rates can be specified with the variable delay setting for different times of day, or days of the week.
Concurrent Web sites | The maximum number of Web sites each node scheduler should crawl at the same time. If there are more Web sites to be crawled than this number then the refresh interval must be increased accordingly.
Crawl speed | The rate at which Web items are collected from the Web sites of a given collection. The maximum rate is the number of concurrent Web sites divided by the request rate.
Duplicate documents | A Web item may in some cases have multiple URLs referencing it. In order to avoid indexing the same Web item multiple times a mechanism known as duplicate detection is used to ensure that only one copy of each unique Web item is indexed.

How to operate the FAST Search Web crawler

To start or stop the FAST Search Web crawler, use the Node Controller. The Node Controller is accessed by the command line tool nctrl.exe. Internally the Web crawler is organized as a collection of processes and logical components, which often run on a single server. It is possible to distribute the Web crawler across multiple servers, allowing the FAST Search Web crawler to gather and process a larger number of Web items from a large number of Web sites. The following table lists the components and associated process:

Component | Process | Function
Node Scheduler | crawler.exe | Schedules Web crawling on a single farm server.
Multi-node Scheduler | crawler.exe | Schedules Web crawling across farm servers.
Site Manager | crawler.exe | Performs Web crawling, managed by the Node Scheduler.
Post Process | postprocess.exe | Performs duplicate detection and submits content. Managed by the Node Scheduler, but can also be used separately to resubmit all content to the indexer.
File Server | crawlerfs.exe | Allows item processing to retrieve Web pages from the FAST Search Web crawler. Managed by the Node Scheduler and Post Process.
Duplicate Server | ppdup.exe | Performs duplicate detection across farm servers.
Browser Engine | browserengine.exe | Handles content and hyperlink extraction from Web items. Only used when JavaScript support is turned on.

When the FAST Search Web crawler is deployed on a single server, the primary process is known as the Node Scheduler. It has several tasks, including resolving host names to IP addresses, maintaining the crawl configurations and other global jobs. It is also responsible for routing Web sites to one of the Site Manager processes. The Node Scheduler is started (or stopped) by the Node Controller, and is in turn responsible for starting and stopping other Web crawler processes.

The Site Manager manages work queues per Web site and is responsible for fetching pages, computing the checksum of the Web item content, storing the Web items to disk, and associated activities such as Web site authentication if required.

Post Process maintains a database of Web item checksums in order to determine duplicates and is responsible for submitting Web items for indexing. Small Web items are sent directly to the item processing pipeline but larger Web items are sent with only a URL reference. The File Server process is responsible for supplying the Web item contents to any pipeline stage that requests it.

If the number of Web sites or the total number of Web items to be crawled is large, the FAST Search Web crawler can be scaled by distributing it across multiple servers. In this deployment scenario, additional processes are started. The Multi-node Scheduler is added, which performs hostname to IP resolution, contains centralized configuration and logging, and routes URLs to the appropriate Node Scheduler. Each Node Scheduler continues to have a Post Process locally. But each of these must now submit Web item checksums to the Duplicate Servers, which maintain a global database of URLs and content checksums.

In this section:

  • Determine what Web content to crawl

  • Determine crawl schedules

  • Configure the FAST Search Web crawler

  • Manage crawling with the FAST Search Web crawler


Determine what Web content to crawl

When building a search index, it is critical to exclude duplicate or otherwise less useful content. For example, you will probably want to exclude the empty pages of an online calendar system. Keep in mind what content you want to exclude when configuring the FAST Search Web crawler.

Determine the start URLs

The start URL list provides the initial set of URLs to Web sites/items for the Web crawler to retrieve. As each URL is fetched, the Web crawler parses the Web item to locate additional hyperlinks to the same Web site as well as to other Web sites.

If the start URL list contains URLs to more Web sites than the number of Web sites the Web crawler is allowed to crawl at the same time, as configured by the max_sites setting, then some Web sites will remain queued until another Web site completes crawling, at which point a new Web site can be processed. To ensure all Web sites are crawled within the refresh interval, the max_inter_docs setting can be used to force a different Web site to be scheduled after the specified number of Web items have been downloaded from each Web site.

Note:

This can be expensive with regard to queue structure and the possibility of overflowing file system limits. It is recommended to thoroughly consider the implications on Web scale crawls before enabling the max_inter_docs feature.

Determine include and exclude rules

The first factor to consider is what Web sites should be crawled. If there are no limitations, no rules to restrict it, the FAST Search Web crawler will consider any URL to be valid. In most scenarios, this results in too much data.

Generally, an index is being built for a limited number of known Web sites, identified by their host names. For these Web sites, one or more start URLs are specified, giving the Web crawler a starting point within the Web site. An include rule corresponding to the start URL can be specific, for example, an exact match for www.contoso.org. Or, it can be more general to match all Web sites in a given DNS domain, for example, any host name matching the suffix .contoso.com.

However, there are often exceptions to the general include rules. For example if a hostname within a DNS domain should not be crawled, or particular parts of the Web site should be excluded. These exceptions can be entered in the exclude filters of the configuration file, where you can further specify domains or URLs to be excluded from the crawl.

As the crawler fetches Web items and parses them for new URLs to crawl, it will evaluate each candidate URL against its configured rules. If a URL matches either an include host name or include URL filter rule, while not matching an exclude host name or exclude URL filter rule, then it is considered eligible for further processing, and possibly fetching.

Within a given Web site, you can configure the crawler to gather all pages or only crawl to a limited depth using the crawl_mode setting. Use the max_doc setting to set an overall limit on the number of Web items downloaded from a specific Web site per refresh interval. This can be helpful if a large number of Web sites are being crawled, which results in a large number of Web items being fetched from a "deep" Web site. This would limit the resources available and starve other Web sites. Use the cut_off setting to specify a size limit per Web item. Then, specify consequent behavior for Web items exceeding this threshold with the truncate setting.


Determine crawl schedules

The key variable that affects the load on the crawler, the network, and the remote Web servers is the request rate. This value is determined by the delay setting, which indicates how long the Web crawler should wait, after fetching a Web item, before requesting the next one.

For each Web site that is crawled concurrently, the overall Web item request rate will be a function of the delay and max_pending settings, as well as the response time of the Web server itself. The Web crawler's overall download rate depends on how many Web sites are crawling concurrently. The theoretical maximum speed is defined by dividing the value of the max_sites settings with the delay setting. However, the actual speed will generally be the number of concurrently crawling Web sites divided by the delay setting.

Warning:

Do not overload any Web sites or servers you plan to crawl.

Use the default request rate of 60 seconds (or higher) when crawling Web sites that are not owned by the organization running the crawl. This rate avoids putting a high burden on the Web server from which Web items are being requested. For Web crawls within the same organization or network, lower values may be used, although note that using very low values (for example, less than 5 seconds) can be stressful on the systems involved.

The refresh interval, configured by the refresh setting, determines the overall length of the crawl cycle. The crawl cycle is the period of time over which a crawl should run without revisiting a Web site to see if new or modified Web items exist. Picking an appropriate interval depends on the amount of content to be fetched (which in turn depends both on the number of Web sites and how many Web items each contain) and the update rate or freshness of the Web sites. In some cases, there are many Web sites with very static/stable content, and a few that are updated frequently; these can be configured either as separate crawl collections, or given distinct settings through the use of a sub collection.

The behavior of the Web crawler at the end of the crawl cycle depends upon the refresh_mode and refresh_when_idle settings, as well as and the current level of activity. If the refresh interval is sufficiently large to allow all Web sites to be completely crawled, and the refresh_when_idle setting is disabled, the crawler will remain idle until the refresh interval ends. If the setting is enabled, a new crawl cycle will start immediately. A new crawl cycle is started by adding the start URLs to the crawl work queue.


Configure the FAST Search Web crawler
See Also

The FAST Search Web crawler is highly customizable and is able to scale to large environments, for example when your organization is crawling a large number of external websites.

You install the FAST Search Web crawler by installing FAST Search Server 2010 for SharePoint. Post installation, you need to configure the FAST Search Web crawler for use.

Configuration file templates

The configuration of the FAST Search Web crawler is done by editing a copy of an XML file. You can operate the FAST Search Web crawler through a number of command line tools.

In the <FASTSearchFolder>\etc folder, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch, you will find three configuration templates:

  • CrawlerConfigTemplate-Simple.xml

  • CrawlerConfigTemplate-Advanced.xml

  • CrawlerConfigTemplate-RSS.xml

The simple template contains a minimum set of configuration parameters to be entered. This template is the starting point to set up a FAST Search Web crawler. The advanced template describes most options and contains an extensive number of comments. This file can be used as a reference to enable you to set up complex FAST Search Web crawler environments. The RSS template contains the configuration options needed to set up a crawl of RSS content.

Warning:

Before you set up the first FAST Search Web crawler configuration, copy to the file CrawlerCollectionDefaults.xml.generic.template (located in <FASTSearchFolder>\META\config\profiles\default\templates\installer\) to the <FASTSearchFolder\etc folder. Then rename this file to CrawlerCollectionDefaults.xml.

In case you have a multi-server deployment, copy the CrawlerCollectionDefaults.xml file to the <FASTSearchFolder>\etc folder of each server in the farm that hosts a FAST Search Web crawler component.

If FAST Search Server 2010 for SharePoint was started, you must restart the FAST Search Web crawler process for the changes to take effect.

To set up a basic FAST Search Web crawler configuration, copy the file CrawlerConfigTemplate-Simple.xml that you can find in the <FASTSearchFolder>\etc folder. Give the copied file a new unique name and configure it for your environment. In the same folder, you will find the files CrawlerCollectionDefaults.xml and CrawlerGlobalDefaults.xml. These files contain several default settings for all collections, and do not have to be edited.

Tip:

If you decide not to use certain sections in the configuration file, consider keeping these sections in your configuration file, but with empty values, instead of removing them from the XML. This will make implementing partial configurations at a later stage easier, as you can keep the remaining configuration intact.

Recommended settings

The minimum FAST Search Web crawler configuration should contain information about where the FAST Search Web crawler should start the crawl, what to crawl, how fast and for how long. Specify start URLs first and then set up additional include and exclude rules to limit the number of Web sites being crawled. If you do not set up additional rules, any Web site will be crawled, which is likely to overload the system and crawl unwanted content.

Then, specify how long the FAST Search Web crawler should wait between each download request to a Web site before issuing another request.

The refresh interval depends on the type and amount of content on the Web sites that you are crawling. You can organize highly dynamic Web sites in a separate collection and configure this collection to have a short refresh interval, for example.

To calculate the necessary refresh interval, multiply the expected amount of Web items on the largest Web site with the request rate and divide by 60. For example, a web site that contains 1000 items and is crawled with a request rate of 30 (meaning 30 seconds between retrievals), requires a minimum of (1000*30)/60=500 minutes refresh interval.

Finally, determine how to access the content. For example, you may have to specify a HTTP proxy for the FAST Search Web crawler using the proxy setting. You may also have to configure any authentication scheme and credentials needed using the logins setting. Refer to the article Web Crawler XML configuration reference for more information about advanced configuration settings.

See Also

Concepts

Web Crawler XML configuration reference
Crawling Web content with the FAST Search Web crawler


Manage crawling with the FAST Search Web crawler

The FAST Search Web crawler is a specialized indexing connector. The connector gathers content from Web content sources.

Before you start crawling with the FAST Search Web crawler, consult the topics Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint), Configure the FAST Search Web crawler, Determine what Web content to crawl, and Determine crawl schedules.

In this section:

  • Add, edit or remove a FAST Web crawler configuration

  • Manage crawl rules (FAST Search Web crawler)

  • Start, suspend, resume, or stop a crawl process (FAST Search Web crawler)

  • Customizing the Browser Engine

  • Manage mapping crawled to managed properties (FAST Search Web crawler)


Add, edit or remove a FAST Web crawler configuration
See Also

Add a configuration

  1. Copy the configuration template CrawlerConfigTemplate-Simple.xml that is located in the FAST Search Server 2010 for SharePoint installation etc folder, typically C:\FASTSearch\etc. Give the copied template a new unique name, for example MyCrawlerConfig.xml.

  2. Configure the parameter settings for your environment in a text or XML editor of your choice. Refer to Web Crawler XML configuration for a description of all the available FAST Search Web crawler parameters and their settings.

    Warning:

    Each content collection should have a separate crawl configuration. The default content collection is sp. If you have multiple content collections, create a new crawl configuration for each content collection and indicate the applicable collection name in the crawl configuration file. Add the content collection name in the parameter DomainSpec name.

  3. Save the configuration file, this will make it easier to edit the configuration at a later stage. Make sure the file does not include a Byte-Order Mark (BOM) / UTF-8 signature.

  4. The FAST Search Web crawler configuration is now ready to use. The FAST Search Web crawler needs to run in order for the configuration to be activated. Add the crawl configuration to the FAST Search Web crawler by using the command crawleradmin -f <Path to XML file>, where <Path to XML file> is the path to your FAST Search Web crawler configuration file, for example C:\FASTSearch\etc\MyCrawlerConfig.xml.

Edit a configuration

  1. Open the configuration file that needs to be edited.

  2. Make the necessary edits in a text or XML editor of your choice.

  3. The FAST Search Web crawler needs to run in order to activate the changes in the FAST Search Web crawler configuration. Add the edited crawl configuration to the FAST Search Web crawler by using the command crawleradmin -f <Path to XML file>, where <Path to XML file> is the path to your FAST Search Web crawler configuration file, for example C:\FASTSearch\etc\MyCrawlerConfig.xml.

Remove a configuration

This procedure removes a particular FAST Search Web crawler crawl configuration from all servers that run one or more FAST Search Web crawler components. The FAST Search Web crawler configuration file(s) will not be deleted in this process and can be reused or removed later. This procedure does not remove any data from the content index. Refer to the topic Reset the content index (FAST Search Web crawler) for instructions on how to remove items from the content index.

  1. At the command prompt, navigate to <FASTSearchFolder>\bin, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

  2. At the command prompt, type the following command:

    crawleradmin -d <MyCollection>

    Where <MyCollection> is the file name of the crawler configuration that you want to remove.

  3. Double check that you have entered the correct name of the crawl configuration that you are about to remove and press ENTER. Press ENTER again when you are prompted to confirm that you want to delete the configuration.

  4. Wait for the command to finish. This may take some time.

See Also

Reference

crawleradmin.exe reference
Clear-FASTSearchContentCollection

Concepts

Web Crawler XML configuration reference
Crawling Web content with the FAST Search Web crawler
Start, suspend, resume, or stop a crawl process (FAST Search Web crawler)
Reset the content index (FAST Search Web crawler)


Manage crawl rules (FAST Search Web crawler)
See Also

The FAST Search Web crawler supports a variety of rules to configure what content is and is not crawled. The rules consist of URL and host name based include and exclude rules, extension excludes, MIME type restrictions and HTTP scheme restrictions. Refer to Determine what Web content to crawl for additional guidance.

How to include content in the crawl

Include rules are configured with the include_domains and include_uris settings, and specify what Web content should be included. However, they do not define where the FAST Search Web crawler should start the crawl, which is specified in the start URL list.

The include_domains setting lets you specify what host names the Web crawler is allowed to crawl and the include_uris setting is the URL equivalent. The table shows some examples of how these rules can be used. For a complete overview, see include_domains and include_uris in the Web Crawler XML configuration file reference topic.

Setting | Rule type | Description
include_uris | exact | Include only the page http://www.contoso.com/
 | prefix | Include all content under http://www.contoso.com/public/
 | suffix | Include all URLs ending with ".html"
 | regexp | Include URLs matching the pattern ".*contoso.*"
 | file | Include <path to local file>. This file contains the rule set to use.
include_domains | exact | Include the host name "www.contoso.com".
 | prefix | Include host names that begin with "www.contoso".
 | suffix | Include host names ending with ".contoso.com".
 | regexp | Include all host names starting with www and ending with com: "www\..*\.com".
 | file | Include <path to local file>. This file contains the rule set to use.

If neither include_uris nor include_domains are specified, the FAST Search Web crawler treats all Web sites and URLs as eligible for crawling unless explicitly excluded by other rules.

How to exclude content from the crawl

Important:

Exclude rules override include rules. This means that a Web site or URL specified in the exclude rules will not be crawled, even if it was included by an include rule.

Exclude certain domains and/or sites

Exclude rules are configured with the exclude_domains and exclude_uris settings, and specify what Web content should not be crawled. The rules are configured similarly to the include rules.

Exclude file types

Content can also be excluded based on the file type extension of the Web item. A common example is graphical content. JPEG, GIF and bitmap files are excluded. Links to audio or video content are typically excluded to avoid downloading large amounts of content with no text content, although special multimedia crawls may choose to include this content for further processing.

These restrictions can be implemented using either the exclude_exts settings, which exclude specific file types by their extension (for example, any Web item that ends with ".jpg"), or by restricting the MIME types that are crawled using the allowed_types setting (for example, "image/jpeg"). Note that a MIME type screening requires the crawler to actually fetch part of the document, whereas an extension exclude can be performed without any network access simply by examining the hyperlink itself.

Exclude URL schemes

The FAST Search Web crawler can be restricted to crawl only a certain type of URL schemes by using the allowed_schemes setting. This setting can be used to restrict the Web crawler to, for example, only crawl HTTP or HTTPS content.

See Also

Concepts

Web Crawler XML configuration reference
Determine crawl schedules


Start, suspend, resume, or stop a crawl process (FAST Search Web crawler)
See Also

When you have completed your configuration, use the command line to operate the FAST Search Web crawler.

Start a single server Web crawler process

To add a FAST Search Web crawler configuration, see Add, edit or remove a FAST Web crawler configuration.

To start a single server Web crawler process:

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    nctrl start <process>

    Where:

    • <process> is the process you want to start, either "crawler" or "browserengine"

Stop a single server Web crawler

It may be necessary to stop the FAST Search Web crawler, for example in order to perform certain administrative tasks such as reefed content or run database tools. This is done the same was as you would stop any other FAST Search component, using the nctrl.exe tool. For a single node FAST Search Web crawler deployment there are two processes that always run on the same server in the farm, namely crawler and browserengine. Only the crawler process needs to be stopped when performing "offline" administrative tasks.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    nctrl stop <process>

    Where:

    • <process> is the process you want to stop, either "crawler" or "browserengine"

Start a multi-server Web crawler

To start a FAST Search Web crawler process, start the FAST Search Web crawler on the multinodescheduler server:

Procedure overview

  1. First start all dupserver_primary, dupserver_replica and browserengine processes (if stopped).

  2. Start the multinodescheduler process.

  3. Start all nodescheduler processes.

Stop a multi-server Web crawler

In a multi-server FAST Search Web crawler deployment there are multiple different processes running on different servers in the farm. The multinodescheduler runs on one server, and is the central component of the FAST Search Web crawler. Stopping it will effectively restart all the nodescheduler components and halt any crawling until it is restarted.

Each server responsible for actually crawling will contain one nodescheduler (instead of the crawler in single node) and one browserengine.

In addition, one or more servers will contain a dupserver_primary and optionally a dupserver_replica. These do not have to be stopped when performing any database maintenance (unless the maintenance involves their databases) and must be running for content to be submitted.

The recommended procedure for shutting down a multi node crawler is:

Procedure overview

  1. First stop all nodescheduler processes and wait for them to exit.

  2. Stop the multinodescheduler process and wait for it to exit.

  3. Optionally stop all dupserver_primary, dupserver_replica and browserengine processes.

How to start a process

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    nctrl start <process>

    Where:

    • <process> is the process you want to start.

How to stop a process

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    nctrl stop <process>

    Where:

    • <process> is the process you want to stop.

Suspend and resume a crawl configuration

The FAST Search Web crawler can have multiple crawl collections active at the same time, and each of these can be controlled individually. These operations are performed using the crawleradmin command line tool.

Note that it may take some time, depending on the number of Web sites crawled, for the crawling to be completely suspended or resumed. Refer to crawleradmin for more information on managing the Web crawler.

How to suspend crawling of a given crawl collection:

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    crawleradmin --suspendcollection <MyFASTSearchWebCrawlerCollection>

    Where:

    • <MyFASTSearchWebCrawlerCollection> is the name of the FAST Search Web crawler collection to be suspended.

How to resume crawling of a given crawl collection:

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command:

    crawleradmin --resumecollection <MyFASTSearchWebCrawlerCollection>

    Where:

    • <MyFASTSearchWebCrawlerCollection> is the name of the FAST Search Web crawler collection to be resumed.

For detailed information about the command line options, see the reference topics.

See Also

Reference

crawleradmin.exe reference
crawler.exe reference


Customizing the Browser Engine

How the Browser Engine works

The FAST Search Web crawler uses the Browser Engine component to extract content and hyperlinks from Web items that contain embedded JavaScript(s). The use of this component is controlled by the use_javascript option in the crawl configuration.

When JavaScript support is turned on, the FAST Search Web crawler will examine each HTML Web item to see if it contains one or more <script> tags. If there are any, the Web item is sent to the Browser Engine to be parsed, bypassing the normal HTML hyperlink extraction process.

The Browser Engine functions like an automated Web browser. It loads the Web item, including all referenced cascading style sheets (CSS) and external JavaScripts URLs, and creates a DOM (document object model) from the Web item. As part of this process, all JavaScript code that normally runs upon loading the Web item is ran, and any generated hyperlinks (for example navigation events) are aggregated. Once fully loaded, the Web item is sent through an internal mini pipeline running a set of small extractors responsible for:

  • Simulating user clicks on hyperlinks

  • Filling out simple forms (dropdowns, checkboxes, radio boxes etc.)

  • Aggregating cookies

  • Generating a Web item checksum and a new HTML (for later item processing in the FAST Search item processing stages)

All hyperlinks that are encountered while running the extractors mentioned above are aggregated as well, and the new HTML, checksum and all hyperlinks are sent back to the FAST Search Web crawler.

Unlike the standard FAST Search Web crawler hyperlink extractor, which only looks at the HTML body of a Web item, the Browser Engine also requires external URLs referenced by the Web item. As this increases the load on Web servers being crawled it is possible to adjust the rate at which these are downloaded separately from the normal request rate. This is done through the javascript_delay option. The default is (quite aggressively) set to 0, meaning there is no delay (like a Web browser).

As the Browser Engine processing is quite CPU intensive it is possible to overload the Browser Engine by crawling too fast. If the Browser Engine does not have any available capacity when receiving a processing request it returns an "overload message" to the Web crawler. The Web crawler will attempt to send the request to another Browser Engine, if one is available, or otherwise delay processing until the Browser Engine is again ready to process additional Web items.

Customizing the Browser Engine

The Browser Engine is configured by modifying the BEConfig.xml configuration file, located in the etc folder of the FAST Search Server 2010 for SharePoint installation directory on each farm server where a Browser Engine is deployed. Tuning the Browser Engine is mainly a matter of adjusting the set of timeouts and cache sizes to match how the FAST Search Web crawler is used.

After modifying the BEConfig.xml file, the Browser Engine must be restarted.

Restart the Browser Engine

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click FAST Search Server 2010 for SharePoint.

  4. Click the FAST Search Server 2010 for SharePoint Management Shell.

  5. At the Windows PowerShell command prompt, type the following command:

    nctrl restart browserengine


Manage mapping crawled to managed properties (FAST Search Web crawler)

The Web crawler submits a predefined set of crawled properties in the "enterprise crawler" category to the content index. The following table lists these, and their default mapping to managed properties. The RSS and sitemap properties are only applicable if the relevant features are turned on in the crawl configuration.

Predefined crawled properties

Crawled property | Type | Managed property | Description
links | Text | None | A list of all hyperlinks extracted from the Web item if ds_send_links is enabled
crawltime | Datetime | docdatetime, write | The most recent crawl time of the Web item. Only updated if the Web item has changed since it was last crawled.
mime | Text | contenttype | The MIME type of the Web item
http_lastmodified | Text | write | The Last-Modified HTTP header
size | Integer | size | The size in bytes of the Web item
rss.pubdate | Datetime | write | The RSS item "pubDate" property
rss.title | Text | title | The RSS item "title" property
rss.description | Text | description | The RSS item "description" property
rss.* | Text | None | Any other RSS item property
rss.feeds | Text | None | The URLs of linked RSS feeds
rss.feed.* | Text | None | The RSS feed properties
sitemap.title | Text | title | The sitemap "title" property
sitemap.* | Text | None | Any other sitemap property


Reset the content index (FAST Search Web crawler)

Published: March 17, 2011

This topic explains how to completely remove content from the index that was previously crawled by the FAST Search Web crawler.

You can choose to remove all current crawl configurations and reset the content index. In that case, building an index afterward requires a full crawl. Follow the procedure Reset the content index and remove a FAST Search Web crawler configuration if you want to completely remove previously crawled FAST Search Web crawler content and rebuild the index from scratch.

Or, you can choose to delete the content index and then re-feed previously crawled content to the index. You can use this procedure if you have to rebuild the index because you are adding an index column, for example. Follow the procedure Reset the content index and re-feed content.

Reset the content index and remove a FAST Search Web crawler configuration

Remove a configuration

This procedure removes a particular FAST Search Web crawler crawl configuration from all servers that run one or more FAST Search Web crawler components. The FAST Search Web crawler configuration file(s) will not be deleted in this process and can be reused or removed later.

  1. At the command prompt, navigate to <FASTSearchFolder>\bin, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

  2. At the command prompt, type the following command:

    crawleradmin -d <MyCollection>

    Where <MyCollection> is the file name of the crawler configuration that you want to remove.

  3. Double check that you have entered the correct name of the crawl configuration that you are about to remove and press ENTER. Press ENTER again when you are prompted to confirm that you want to delete the configuration.

  4. Wait for the command to finish. This may take some time.

Clear the content collection

You are now ready to reset the content index by clearing the contents of the content collection that you were feeding to.

Warning:

This procedure will delete all items in the content collection. Any other content from other content sources that was fed to the same content collection is also cleared.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    Clear-FASTSearchContentCollection -Name <collection name>

    Where:

    • <collection name> is the content collection you are about to clear.

  6. Wait for the command to finish. This may take some time.

    Important:

    After clearing the content collection, all the relevant content will have been removed from the search engine. It will not be searchable until the content from the FAST Search Web crawler is (re)crawled by creating a new FAST Search Web crawler collection.

Reset the content index and re-feed content

  1. Stop the FAST Search Web crawler. See Start, suspend, resume, or stop a crawl process (FAST Search Web crawler).

  2. Clear the content collection.

    Warning:

    This procedure will delete all items in the content collection. Any other content from other content sources that was fed to the same content collection is also cleared.

    1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

    2. On the Start menu, click All Programs.

    3. Click Microsoft FAST Search Server 2010 for SharePoint.

    4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

    5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

      Clear-FASTSearchContentCollection -Name <collection name>

      Where:

      • <collection name> is the content collection you are about to clear.

    6. Wait for the command to finish. This may take some time.

      Important:

      After clearing the content collection, all the relevant content will have been removed from the search engine. It will not be searchable until the content from the FAST Search Web crawler is re-fed.

  3. Re-feed FAST Search Web crawler content by using the postprocess.exe command-line tool.

    In a single server deployment, use the postprocess.exe command-line tool on the server that runs the FAST Search Web crawler.

    In a multi-server FAST Search Web crawler deployment, use the postprocess.exe command-line tool on each server that has a nodescheduler.

    1. At the command prompt, navigate to <FASTSearchFolder>\bin, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    2. At the command prompt, type the following command:

      postprocess -R <crawl_collections>

      Where:

      • <crawl_collections> specifies a single collection or a comma-separated list of collections (without any white spaces).

  4. Restart the FAST Search Web crawler. See Start, suspend, resume, or stop a crawl process (FAST Search Web crawler).


Crawling database content with the FAST Search database connector

The FAST Search database connector is a specialized indexing connector. The connector gathers content from database content sources.

Before you start crawling with the FAST Search database connector, consult the topic Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint).

In this section:

  • Configure the FAST Search database connector

  • Add, edit or remove a FAST Search database connector configuration

  • Manage crawl rules (FAST Search database connector)

  • Test, start or stop a crawl (FAST Search database connector)

  • Schedule crawls (FAST Search database connector)

  • Manage mapping crawled to managed properties (FAST Search database connector)


Configure the FAST Search database connector
See Also

The FAST Search database connector is a specialized indexing connector. The connector gathers content from database content sources.

You install the FAST Search database connector by installing FAST Search Server 2010 for SharePoint. After installation, you need to configure an XML file and map crawled to managed properties in the Microsoft SharePoint Server 2010 Central Administration to enable search on the content gathered by the connector.

Before you begin

Before you configure the FAST Search database connector, ensure you meet the following requirements.

Tip:

If you have deployed FAST Search Server 2010 for SharePoint on multiple servers, you need to fulfill the requirements on the server that will run the connector, not on all the machines.

  • Install the applicable database drivers to connect to the database of your choice. The drivers should be placed into the \lib directory of your FAST Search Server 2010 for SharePoint installation directory.

    Note:

    The SQL Server driver is already included.

  • Enable TCP/IP on your source database to give the database client access.

  • Install the 32 bit Java Runtime Environment (JRE) version 1.6 or newer.

  • Use a SQL Server 2008 Enterprise (or newer) database for tracking the state of items and the connector, instead of a SQL Server Express database, if you plan to crawl more than roughly 2 million items.

    Tip:

    The system generates the file dbconfig.xml in the \etc folder in the FAST Search Server 2010 for SharePoint installation directory when you configure or reconfigure FAST Search Server 2010 for SharePoint. Should you have to change the database after installation, you need to update the connection string in this file.

  • Create a SQL query to use for crawling your database content. You may need the assistance of the database administrator for this.

Set the JAVA_HOME system environment variable

The following system environment variable should be set in order for the FAST Search database connector to work correctly.

You can set environment variables in Windows Server 2008 by clicking Start, right click on Computer, select Properties, then Advanced System Settings (click Continue if prompted to provide access), click Environment Variables and click New below System variables.

Tip:

Use Windows Explorer to verify or find the actual file path in your environment.

Variable | Description | Value
JAVA_HOME | Point this variable to the folder where the Java Runtime Environment (JRE) is installed. Note: If you have installed Java Development Kit (JDK), you should point JAVA_HOME to the bundled JRE instead. | Without JDK installed:Typically C:\Program Files (x86)\Java\jre1.6 With JDK installed: Typically C:\Program Files (x86)\Java\jdk1.6.0_17\jre

Configuring the connector

To set up a basic configuration, copy the file jdbctemplate.xml that you can find in the FAST Search Server 2010 for SharePoint installation etc folder, typically located in C:\FASTSearch\etc. Give the copied file a new unique name, for example MyJDBCConfig.xml, and configure it for your environment in an XML editor of your choice. The configuration file should be stored in UTF-8 format. Place the new file back in the etc directory after configuration.

The table lists a number of parameters from this file that are required to set up a basic configuration.

Parameter group Parameter Description

Input

JDBCDriver

JDBC driver class name, defined by the JDBC driver provider.

Input

JDBCUrl

JDBC URL, specified by the JDBC driver provider. The URL may vary for different JDBC drivers.

Input

JDBCUsername

Database username.

May be omitted when using Windows integrated authentication to access SQL Server.

Input

JDBCPassword

Database password.

May be omitted when using Windows integrated authentication to access SQL Server.

Note:

This password must be encrypted using the connectorpwgen.bat tool. See the section Encrypt passwords.

Input

JDBCSQL

Specifies the SQL crawl query. The selected columns in the SQL query will be available as crawled properties in FAST Search Server 2010 for SharePoint .

If you use time stamp updates, this attribute requires one of the following:

  • %TIMESTAMP%: Last crawl time is specified as a datetime value.

  • %TIMESTAMPSEC%: Last crawl time is specified as the number of seconds since the epoch.

Transformation

JDBCPrimaryKeyField

The database view/table primary key. Used to create the item ID. Case-sensitive: must match your SELECT statement.

FASTSearchSubmit

Collection

Name of the destination content collection.

The default content collection for FAST Search Server 2010 for SharePoint is sp.

Encrypt passwords

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    connectorpwgen.bat

  6. Enter the password that you want to encrypt twice.

    Warning:

    The encryption/decryption key is dependent on the user that performs the encryption/decryption. The user who will run the connector must run the connectorpwgen.bat tool.

  7. Copy the password (not including any new lines or white spaces) and paste it into Notepad. Remove the new line character and starting text.

  8. Copy the password from Notepad to the appropriate parameter in the configuration file.

See Also

Reference

connectorpwgen.bat reference

Concepts

jdbctemplate.xml reference


Add, edit or remove a FAST Search database connector configuration
See Also

Add a configuration

  1. Copy the configuration template jdbctemplate.xml that is located in the FAST Search Server 2010 for SharePoint installation etc folder, typically C:\FASTSearch\etc. Give the copied template a new unique name, for example MyJDBCConfig.xml. Keep the file in the etc directory.

  2. Configure the parameter settings for your environment in an XML editor of your choice.

  3. Save the configuration file in UTF-8 format. Make sure the file is located in the etc directory of the FAST Search Server 2010 for SharePoint installation folder.

  4. The FAST Search database connector configuration is now added. The connector has to be run in order for the configuration to be activated.

    Warning:

    Do not rename the configuration file after you have run the connector. The configuration file name is used to identify the change detection tables for this configuration. If you rename the file, the change detection information for the configuration will be lost.

Edit a configuration

  1. Open the configuration file that needs to be edited from the etc directory of the FAST Search Server 2010 for SharePoint installation folder.

  2. Make the necessary edits in an XML editor of your choice, and save the changes in UTF-8 format.

  3. The FAST Search database connector has to be run in order to activate the changes in the configuration.

Remove a configuration

Follow the steps in the topic Reset the content index (FAST Search database connector) to remove a FAST Search database connector configuration. This topic also explains how to completely remove previously crawled database content from the index.

See Also

Concepts

Configure the FAST Search database connector
Test, start or stop a crawl (FAST Search database connector)
Reset the content index (FAST Search database connector)
jdbctemplate.xml reference


Manage crawl rules (FAST Search database connector)
See Also

Specify which content should be indexed

The FAST Search database connector crawls content by following SQL select statements. These SQL select statements can be specified either in the JDBCSQL parameter or in a file that you point to by specifying the JDBCSQLFile parameter. Specify one of these parameters in the Input group of the connector configuration file. All selected columns will be indexed by default.

To manage which content should be indexed, select specific columns in the select statements. For example:

Copy Code
SELECT Product.rowguid ID, Product.name Name, Product.weight weight
FROM Product

Specify which content should be transformed

More advanced result sets may require more advanced result set handling. The following parameters from the configuration file can be used to indicate what content should be transformed, and how.

Filter category | Parameter | Short description
Transformation | UseMultiRowNormalizer | If set to true, values for columns of records with identical identifiers are merged. In order for this feature to work, the SELECT statement must have an ORDER BY clause on the primary key/identifier field.
Transformation | JDBCMultiValueIncludeFields | Semicolon (;) separated list of column names that should be concatenated in the output. This field overrides any settings to JDBCMultiValueSkipFields.. Do not use this setting together with JDBCAttributeColumn and JDBCValueColumn
Transformation | JDBCMultiValueSkipFields | Semicolon (;) separated list of column names that have the same values in each row, and that should not be repeated in the output. Do not use this setting together with JDBCMultiValueIncludeFields.
Transformation | JDBCAttributeColumn | For normalized tables, specifies the column that contains attribute names. Do not use this setting together with JDBCMultiValueIncludeFields and JDBCMultiValueSkipFields.
Transformation | JDBCValueColumn | For normalized tables, specifies the column that contains attribute values. Do not use this setting together with JDBCMultiValueIncludeFields and JDBCMultiValueSkipFields.

How to use the multi row normalizer

When joining tables, item IDs may be duplicated over several rows. This leads to several rows containing the same identifier. Use the multi row normalizer to collapse these rows into one item.

When you set the parameter UseMultiRowNormalizer to true, text from multiple rows with the same JDBCPrimaryKeyField value are concatenated. If a column contains a non-value, like NULL for example, this sometimes causes the string "NULL" to appear in the result set. To prevent this, use the isnull function when you use SQL Server (or a similar function for a different database type) in the parameter JDBCSQL. For example: SELECT ID, intro, body, isnull( comments, '' ) comments FROM cms.articles;

How to use the parameters JDBCAttributeColumn and JDBCValueColumn

Items extracted from a database may have the desired item attribute names in one column, and the column values in another column. Use these two parameters to specify which column contains the attribute names, and which column contains the attribute value.

How to connect to the source database

Configuring the FAST Search database connector to connect to the source database requires a JDBC connection string to be created for the source database. Different databases use different connection strings. Specify the connection string to the source database in the parameter JDBCURL.

Examples of JDBC connection strings for common databases:

Microsoft Microsoft SQL Server 2000:

jdbc:microsoft:sqlserver://<YourServerHere>:1433;DatabaseName=<YourDatabaseNameHere>

Where:

  • <YourServerHere> is the SQL Server host name

  • <YourDatabaseNameHere> is the name of the source database that you are connecting to

Microsoft Microsoft SQL Server 2005 and later versions

jdbc:sqlserver://<YourServerHere>:1433;database=<YourDatabaseNameHere>

Where:

  • <YourServerHere> is the SQL Server host name

  • <YourDatabaseNameHere> is the name of the source database that you are connecting to

IBM DB2

jdbc:db2://<YourServerHere>:50000/<YourDatabaseNameHere>

Where:

  • <YourServerHere> is the IBM DB2 server host name

  • <YourDatabaseNameHere> is the name of the source database that you are connecting to

Oracle

jdbc:oracle:thin:@<YourServerHere>:1521:<YourDataBaseNameHere>

Where:

  • <YourServerHere> is the Oracle server host name

  • <YourDatabaseNameHere> is the name of the source database that you are connecting to

Connecting to databases from providers other than Microsoft also requires the JDBC driver from the specific provider to be put in the /lib directory of your FAST Search Server 2010 for SharePoint installation folder. Refer to the database provider to find out how to obtain the JDBC driver.

See Also

Concepts

jdbctemplate.xml reference


Test, start or stop a crawl (FAST Search database connector)
See Also

When you have completed your configuration, use the command line to operate the FAST Search database connector.

Test the connection

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    jdbcconnector.bat testconnections -f ..\etc\<connector configuration filename>

    Where:

    • <connector configuration filename> is the name of the FAST Search database connector configuration file.

Start a crawl

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    jdbcconnector.bat start -f ..\etc\<connector configuration filename>

    Where:

    • <connector configuration filename> is the name of the FAST Search database connector configuration file.

Stop a crawl

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    jdbcconnector.bat stop -f ..\etc\<connector configuration filename>

    Where:

    • <connector configuration filename> is the name of the FAST Search database connector configuration file.

For detailed information about the command line options, see the jdbcconnector.bat reference topic.

See Also

Reference

jdbcconnector.bat reference

Concepts

Add, edit or remove a FAST Search database connector configuration
Configure the FAST Search database connector


Schedule crawls (FAST Search database connector)

Use this procedure to schedule crawls for the FAST Search database connector. To schedule crawls, use the Windows Task Scheduler.

Create a new scheduled task using the Windows Task Scheduler (Windows Server 2008)

  1. On the Start menu, click Administrative tools.

  2. Click Task Scheduler.

  3. In the Actions pane, click Create Basic Task.

  4. Give the task a unique name and click Next.

  5. In the Trigger menu, mark the radio button in front of the preferred interval and click Next. Provide any additional input if prompted and click Next.

    Warning:

    Make sure there is enough time for a full crawl to complete between the intervals.

  6. In the Action menu, mark the radio button in front of Start a Program and click Next.

    1. In the Program/script field, type <FAST Search Server 2010 for SharePoint installation directory>\bin\jdbcconnector.bat, where <FAST Search Server 2010 for SharePoint installation directory> is the installation directory of FAST Search Server 2010 for SharePoint, usually C:\FASTSearch\.

    2. In the Add arguments field, type start -f ..\etc\<name of the connector configuration file>, where <name of the connector configuration file> is the configuration file for the FAST Search database connector.

    3. In the Start in field, type <FAST Search Server 2010 for SharePoint installation directory>\bin, where <FAST Search Server 2010 for SharePoint installation directory> is the installation directory of FAST Search Server 2010 for SharePoint, usually C:\FASTSearch\.

  7. In the Summary dialogue, review your settings and click Finish to create the task.

Allow scheduled tasks to run when the user is not logged on

By default, scheduled task(s) are not allowed to run when the user is not logged on. To allow the scheduled task(s) to run when the user is not logged on:

  1. In the Windows Task Scheduler left navigation pane, click the Task Scheduler Library.

  2. Right click on the scheduled task(s) that you created and click Properties.

  3. In the General tab, select the radio button in front of Run whether user is logged on or not and then click OK.

  4. Type the user name and password of the user who is supposed to run the task and then click OK.


Manage mapping crawled to managed properties (FAST Search database connector)

Once you have run the FAST Search database connector, crawled content will show up in the crawled properties in the Microsoft SharePoint Server 2010 Central Administration. All crawled properties in the JDBC category are searchable by default. To make the properties appear in the search result, they must be mapped to managed properties. No properties are mapped by default for JDBC content.

Columns or aliases from the SELECT statement are available as crawled properties.

Beware that some databases return column names in uppercase, while the index profile only accepts lower case letters. In such cases you can use aliases in order to lower case the column names. For example:

SELECT ID as "id", NAME as "name", ADDRESS as "address" FROM table

How to map a crawled to a managed property

To open the Managed Properties page:

  1. On the Central Administration page, select General Applications Settings.

  2. On the General Application Settings page, under Search, select Farm Search Administration.

  3. Select Search Service Application, and select your Query Search Service Application (for example, fastsearchquery).

  4. Select FAST Search Administration from the menu on the left.

  5. On the FAST Search Administration page, under Property Management, click Managed Properties.

To add a crawled property to a managed property:

  1. On the Managed Properties page, click Crawled Properties Categories.

  2. On the Crawled Property Categories page, find the JDBC category and click it.

  3. On the Crawled properties page, click the crawled property name that you want to edit.

  4. On the Edit Crawled Property page, in the Mappings to managed properties section, click Add Mapping.

  5. In the Select managed properties to map to dialog box, configure the settings. In the Available managed properties list, search for or select a managed property, and then click Add. Repeat if you want to add more managed properties to the same crawled property. Click OK.

  6. On the Edit Crawled Property page, if you want to include values for the crawled property in the search index, select the Include values for this property in the search index check box.

    Select the check box to enable querying against the values of this crawled property. This will place these values in the search index. For example, if the crawled property is "author," simple queries such as "Smith" will return both items that contain the word "Smith" and items whose author property is "Smith." When unchecked, users will have to query against the managed property "author:Smith" to find the same items. Including unnecessary properties may have a negative effect on search relevance and performance.

  7. Click OK.

  8. Rerun the connector for the changes to take effect.


Reset the content index (FAST Search database connector)
See Also

Published: March 17, 2011

This topic explains how to completely remove previously crawled database content from the index. First you must clear the change detection database, and then you should decide whether or not you want to delete the FAST Search database connector configuration file, and lastly you must clear the content collection that the database content was fed to.

Clear the change detection database

  1. Open the FAST Search database connector configuration file. You can find this file in the etc folder of the FAST Search Server 2010 for SharePoint installation folder.

    1. Set the parameter ChangeDetection/ChangeDBPurge to true.

      Note:

      Setting the parameter ChangeDBPurge to true is only relevant if you used checksum based change detection.

    2. Set the parameter Input/JDBCSQL to SELECT 1 WHERE 1 > 2. This will cause the connector to not find any content to crawl.

    3. Save the file.

  2. Run the FAST Search database connector. This will clear all the change detection database tables and free space in the SQL Server database.

    1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

    2. On the Start menu, click All Programs.

    3. Click Microsoft FAST Search Server 2010 for SharePoint.

    4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

    5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

      jdbcconnector.bat start -f ..\etc\<connector configuration filename>

      Where:

      • <connector configuration filename> is the name of the FAST Search database connector configuration file.

  3. Wait for the connector to clear its change detection database tables and to stop running. Since you provided a non-existent result set in step 1b, the connector will fail with an error message that you can ignore. The change detection database is now cleared.

If you want to delete the FAST Search database connector configuration file, continue with Remove the configuration (optional). If you want to keep the configuration file for later use, continue with Clear the content collection.

Remove the configuration (optional)

Skip these steps if you want to reuse the FAST Search database connector configuration file later.

  1. Navigate to the FAST Search database connector configuration file that you want to remove. You can find this file in the etc folder of the FAST Search Server 2010 for SharePoint installation folder.

  2. Open it to take note of the content collection that is specified in the parameter FASTSearchSubmit/Collection if you have not already done this.

  3. Delete the file.

The FAST Search database connector configuration file is now removed. Continue with Clear the content collection.

Clear the content collection

You are now ready to reset the content index by clearing the contents of the content collection that you were feeding to.

Warning:

This procedure will delete all items in the content collection. Any other content from other content sources that was fed to the same content collection is also cleared.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    Clear-FASTSearchContentCollection -Name <collection name>

    Where:

    • <collection name> is the content collection you are about to clear.

  6. Wait for the command to finish. This may take some time.

    Important:

    After clearing the content collection, all the relevant content will have been removed from the search engine. It will not be searchable until the FAST Search database connector is rerun.

See Also

Reference

Clear-FASTSearchContentCollection

Concepts

Configure the FAST Search database connector
Test, start or stop a crawl (FAST Search database connector)
jdbctemplate.xml reference


Crawling Lotus Notes content with the FAST Search Lotus Notes connector

The FAST Search Lotus Notes connector is a specialized indexing connector that consists of two parts: a content connector and a user directory connector. The content connector gathers content from a Lotus Notes content source. The user directory connector ensures that the end users can only search Lotus Notes content that they have access to. The user directory connector maps the Active Directory user directory with the Lotus Notes user accounts and is closely integrated with FAST Search Authorization (FSA).

Before you start crawling with the FAST Search Lotus Notes content or user directory connector, consult the topic Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint).

In this section:

  • Configure the FAST Search Lotus Notes connector

  • Manage crawling with the FAST Search Lotus Notes content connector

  • Manage crawling with the FAST Search Lotus Notes user directory connector


Configure the FAST Search Lotus Notes connector
See Also

The FAST Search Lotus Notes connector is a specialized indexing connector that consists of two parts: a user directory connector and a content connector. The connector gathers content from the Lotus Notes content source. It fully supports Lotus Notes security, including roles, and can index Lotus Notes databases as attachments.

You install the FAST Search Lotus Notes connector by installing FAST Search Server 2010 for SharePoint. After installation, but before you crawl content, you need to configure two XML files. You configure one file for the content connector and one for the user directory connector. Then, you apply security by configuring FAST Search Authorization (FSA), by creating a local cache and creating a mapping between Lotus Domino users and Active Directory users. After you run user directory connector, you should import the mapping between Domino and Active Directory users into the connector setup. After you run the content connector, you can map additional crawled to managed properties in the Microsoft SharePoint Server 2010 Central Administration.

Before you begin

Before you configure the FAST Search Lotus Notes connector, ensure you meet the following requirements. This topic assumes that you have installed FAST Search Server 2010 for SharePoint and Microsoft SharePoint Server 2010.

Tip:

If you have deployed FAST Search Server 2010 for SharePoint on multiple servers, you need to fulfill the requirements on the server that will run the connector, not on all the machines.

  • Install the 32 bit Java Runtime Environment (JRE) version 1.6 or newer.

    Tip:

    Note the path to the installation directory, you will need it during the configuration.

  • Install a 32 bit Lotus Notes Client version 6.5.6 or newer.

    Important:

    Note the path to the installation directory, you will need it during the configuration.

  • Start the Lotus Notes Client and connect it to the Domino domain at least once before running the connector for the first time. The Active Directory user who runs the FAST Search Lotus Notes connector should access the Lotus Notes content with the permissions of a Domino user who has sufficient permissions to crawl Lotus Notes content.

    Important:

    It is not sufficient that the user has read permissions on the documents in the database. The user must also have permissions to look up metadata (for example security) on the database, its views and its documents.

    To ensure the Active Directory user running the connector uses the correct Domino user permissions to crawl content:

    1. Copy the ID file of a Domino user with sufficient permissions to the \etc folder of the FAST Search Server 2010 for SharePoint installation directory.

    2. Start the Lotus Notes Client. The logon dialog will appear. Expand the User drop down menu and select Other user.

    3. The file browser dialog appears. Locate the Domino user ID file that you copied to the \etc folder and click Open.

    4. Continue to log on to the Lotus Notes Client using the selected ID file, and then exit the Lotus Notes Client.

  • Use a SQL Server 2008 Enterprise (or newer) database for tracking the state of items and the connector, instead of a SQL Server Express database, if you plan to crawl more than approximately 2 million items.

    Tip:

    The system generates the file dbconfig.xml in the \etc folder in the FAST Search Server 2010 for SharePoint installation directory when you configure or reconfigure FAST Search Server 2010 for SharePoint. If you have to change the database after installation, you must update the connection string in this file.

  • The Lotus Domino Administrator should make sure that all the Active Directory users are added to the user documents in the Domino directory to enable mapping between the Domino user and the Active Directory user who performs the search query from SharePoint.

    This should be done before running the user directory connector. The Active Directory user names should be put in the multi-value field that is named User name in the format DOMAIN\user_name and it should be the bottom value in that field.

    Important:

    Do not map the Active Directory user running the FAST Search Query Search Service Application to a Domino user document. If you enable mapping for this user, all other user queries will return search results with the permission settings of that user.

    Important:

    Do not map multiple Domino users to the same Active Directory user or one Domino user to multiple Active Directory users. This is not supported and will lead to errors when you import the mappings.

  • Running the FAST Search Lotus Notes connector against a Lotus Notes client that is using the Client Single Logon Feature is not supported. If the feature is enabled, you can disable it in the Lotus Notes Client. Start the Lotus Notes Client, and open File > Security > User Security. In the dialog that appears, clear the check box before Log in to Notes using your operating system logon. Then exit the Lotus Notes Client. The next time that you start the client, you are prompted for a user name and password.

  • Running the FAST Search Lotus Notes connector with an ID file that contains an embedded password so that you are not prompted to log on when you start the Lotus Notes Client, is not supported. You have to use a user who has an ID file without an embedded password.

Set the system environment variables

The following system environment variables should be set for the FAST Search Lotus Notes connector to work correctly.

You can set environment variables in Windows Server 2008 by clicking Start, right-click Computer, select Properties, and then Advanced System Settings (click Continue if prompted to provide access), click Environment Variables and then click New under System variables. Set each variable individually.

Tip:

Use Windows Explorer to verify or find the actual file path in your environment.

Variable | Description | Value
Notes_ExecDirectory | Point this variable to the installation path of the Lotus Notes Client. | Typically C:\Program Files (x86)\lotus\notes
NOTES_JAR_LIB | Point this variable to the folder where the Notes Java API (Notes.jar) is located. | Typically C:\Program Files (x86)\lotus\notes\jvm\lib\ext
JAVA_HOME | Point this variable to the folder where the Java Runtime Environment (JRE) is installed. Note: If you have installed Java Development Kit (JDK), you should point JAVA_HOME to the bundled JRE instead. | Without JDK installed:Typically C:\Program Files (x86)\Java\jre1.6 With JDK installed: Typically C:\Program Files (x86)\Java\jdk1.6.0_17\jre

See Also

Concepts

Configure the FAST Search Lotus Notes content connector
Configure the FAST Search Lotus Notes user directory connector
Prepare FAST Search Authorization for use with the FAST Search Lotus Notes connector


Configure the FAST Search Lotus Notes content connector
See Also

To set up a basic configuration, copy the file lotusnotestemplate.xml that you can find in the FAST Search Server 2010 for SharePoint installation etc folder, typically located in C:\FASTSearch\etc. Give the copied file a new unique name, for example MyLotusNotesContentConfig.xml, and configure it for your environment in an XML editor of your choice. The configuration file should be stored in UTF-8 format. Place the new file back in the \etc directory after configuration.

Basic configuration parameters

The following table lists several parameters from this file that are used to set up a basic configuration. The two parameters in the Connection group are required to get started. If these parameters are set, the content connector will crawl everything on the server and feed the content to the default content collection that is named sp.

Parameter group | Parameter | Description
Connection | Hostname | Enter the host name of the Domino server. This can be a host name, a fully qualified domain name or "localdisk". The reserved keyword "localdisk" instructs the connector to look for databases on the local Lotus Notes client.
Connection | Password | Copy the encrypted password that is used to authenticate to the Lotus Notes client. This must match the password that you last used to log on to the Lotus Notes Client on this computer. The password must be encrypted, encrypt it with the connectorpwgen.bat utility. See the section Encrypt passwords for more information.

Encrypt passwords

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    connectorpwgen.bat

  6. Enter the password that you want to encrypt twice.

    Warning:

    The encryption/decryption key depends on the user who performs the encryption/decryption. The user who will run the connector must run the connectorpwgen.bat tool.

  7. Copy the password (without including any new lines or white spaces) and paste it into Notepad. Remove the new line character and starting text.

  8. Copy the password from Notepad to the appropriate parameter in the configuration file.

The following table lists more parameters that, although not required for the content connector to run, could be useful to specify:

Parameter group | Parameter | Description
Filters | DataBase | A list of database paths to crawl. May contain view names. Separate multiple databases with a semicolon (;). Example: mail\user1.nsf;help\help_client.nsf Specify a view in the format: <database path>#viewname. When you specify a view, only the search formula of the view is used.
XMLExport | ExportToXML | Set to true or false. If set to true, exports all items to XML. This parameter is used only for debugging, and should generally not be used.
ConnectorExecution | Workfolder | Specifies the work folder (either a relative or absolute file path) for all file output, such as log files and XML output files. The folder is automatically created if it does not exist. Multiple connector configurations should have different values for this parameter.
ConnectorExecution | NumAdapters | Specifies how many databases the connector should crawl in parallel. Increasing this value may increase performance of the connector but it will also increase memory consumption.
StateTracker | PurgeAtStart | Set to true or false. If set to true, all data in the state tracker tables is purged at startup, telling the connector to retrieve everything, even if it was crawled previously. Use when testing your configuration. Set to false when in production.

Tuning your configuration

There are several parameters in the content connector configuration that can be used to tune the behavior, such as performance, memory consumption, and the load exerted on the Domino server or the FAST Search Server 2010 for SharePoint back-end.

The following table lists the configuration parameters that affect the behavior of the content connector and describes how they affect it:

Parameter group Parameter Description

Filters

MaxAttachmentSize

This size filter enables you to define the maximum size of Lotus Notes attachments that you want to crawl. Increasing the value of this filter will, if you have larger attachments, cause the memory consumption of the content connector to increase. This is because a copy of the attachment is kept in memory.

It will also cause crawl performance (documents per second) to drop, because more time will be spent extracting the large attachments from Domino in addition to converting and indexing them.

Decreasing the value for this parameter will have the opposite effect, but large attachments will not be searchable.

ConnectorExecution

QueueSize

This is the size of the internal memory queue of documents that have been extracted from Domino but are waiting to be submitted for indexing.

Increasing this value may in some cases (but not all) increase performance at the cost of increased memory consumption in the content connector.

ConnectorExecution

NumAdapters

This parameter specifies how many Lotus Notes databases should be processed in parallel.Increasing this parameter is likely to have a positive effect on extraction rate.

However, if the bottleneck is the indexing back-end, scaling up on the extraction side is unlikely to help. In fact, it is likely to cause internal queues to fill up so that the content connector uses more memory.

Increasing the number of threads also causes the connector to use more memory.

ConnectorExecution

AdapterThrottleSleepMS

This parameter says how many milliseconds the connector should sleep between each document that it extracts from Domino.

This is a way to throttle the extraction rate if it is necessary.

You would typically not use this parameter, as the connector does not add a significant load on the Domino server.

You should first consider reducing the parameter ConnectorExecution/NumAdapters before you use this parameter to throttle the connector.

ConnectorExecution

EnableStatustracker

Set to true or false to turn document level status tracking on or off.

If you do not want to monitor the state of individual documents, set this parameter to false.

Setting this parameter to false will relieve the state database (SQL Server) from some additional traffic, as the connector will insert one row per document in a status table. This table also contributes to the overall size of the state database.

See Also

Reference

connectorpwgen.bat reference

Concepts

lotusnotestemplate.xml reference
Crawling Lotus Notes content with the FAST Search Lotus Notes connector


Configure the FAST Search Lotus Notes user directory connector
See Also

To set up a basic configuration, copy the file lotusnotessecuritytemplate.xml that you can find in the FAST Search Server 2010 for SharePoint installation etc folder, typically located in C:\FASTSearch\etc. Give the copied file a new unique name, for example MyLotusNotesSecurityConfig.xml, and configure it for your environment in an XML editor of your choice. The configuration file should be stored in UTF-8 format. Place the new file back in the \etc directory after configuration.

Basic configuration parameters

The following table lists several parameters from this file that are used to set up a basic user directory connector configuration.

Parameter group Parameter Description

NotesConnection

NotesNameServer

Enter the host name of the Domino server.

NotesConnection

NotesNamePassword

The password that is used to authenticate to the Lotus Notes Client.

This must match the password that you last used to log on to the Lotus Notes client on this computer. The password must be encrypted, encrypt it with the connectorpwgen.bat utility.

See the section Encrypt passwords for more information.

FSAOutput

UserStoreHost

Enter the host name of the server where the Lotus Notes User Store is located. This should be the node where the FSA module resides.

Note:

When this is a single server installation of FAST Search Server 2010 for SharePoint, you do not have to change this from "localhost".

When this is a multi-server installation, this parameter should specify the host name of the administration server that was specified while installing FAST Search Server 2010 for SharePoint.

FSAOutput

UserStorePort

Enter the port to the CCTK Server that you enabled with the Windows PowerShell cmdlet Set-FASTSearchSecurityCCTKServer -Enable 1. This is usually 13271.

You can check the port number by running the Windows PowerShell cmdlet Get-FASTSearchSecurityCCTKServer.

Encrypt passwords

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    connectorpwgen.bat

  6. Enter the password that you want to encrypt twice.

    Warning:

    The encryption/decryption key depends on the user who performs the encryption/decryption. The user who will run the connector must run the connectorpwgen.bat tool.

  7. Copy the password (without including any new lines or white spaces) and paste it into Notepad. Remove the new line character and starting text.

  8. Copy the password from Notepad to the appropriate parameter in the configuration file.

See Also

Concepts

lotusnotessecuritytemplate.xml reference
Crawling Lotus Notes content with the FAST Search Lotus Notes connector


Prepare FAST Search Authorization for use with the FAST Search Lotus Notes connector
See Also

The user directory connector pushes Lotus Notes security information (user, group and group membership) into a FAST Search Authorization (FSA) Lotus Notes user store, essentially creating a copy/cache of the Lotus Domino user directory to be used by FAST Search Server 2010 for SharePoint.

The purpose of this is that, when an Active Directory user performs a search in FAST Search Server 2010 for SharePoint, the system can expand the user to all its Domino groups (and certifiers). This additional search filter makes sure that the search results only contain Lotus Notes documents that the user has permission to read.

Create a Lotus Notes user store, enable the CCTK server and create an XMLAliaser

Before you run the user directory connector, you have to prepare FAST Search Authorization by creating a Lotus Notes user store, enabling the CCTK server and creating an XMLAliaser. This is done through Windows PowerShell cmdlets.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following commands to create the new Lotus Notes user store:

    New-FASTSearchSecurityLotusNotesUserStore -id lnx

    Verify if the command returns without errors. If there are any errors, the command will return red text. If the command is successful, you will see a number of parameters appear.

    Set-FASTSearchSecurityCCTKServer -Enable 1

    Verify if the command returns without errors. If there are any errors, the command will return red text. If the command is successful, you see the port number and an acknowledgement that the CCTK server is enabled. Note this port number as you will need it during the configuration.

  6. Type the following command to create the XMLAliaser:

    New-FASTSearchSecurityXMLAliaser -id win2lnx -InputUserStoreId win -OutputUserStoreIds lnx -InputPropertyName '$PRINCIPAL_REFERENCE_ALIAS'

    Verify if the command returns without errors. If there are any errors, the command will return red text. If the command is successful, you see several parameters appear.

Configure aliasing

To be able to search Lotus Notes content through the SharePoint front-end, you have to map the Active Directory users to the Domino users. This process is known as aliasing. You do this by using the SSOMapping feature of the user directory connector together with the XMLAliasing feature of FAST Search Authorization (FSA).

You turn on the feature in the user directory connector by setting the parameter SSOMapping/UseSSOMapping to true. You may also have to configure the parameter SSOMapping/ADUserNameField to refer to the correct field in the user document in Domino that holds the Active Directory user name.

Parameter group | Parameter | Description
SSOMapping | UseSSOMapping | Turn the generation of the SSO Mapping XML file on (true) or off (false). This parameter should be set to true if you have populated the Domino user documents with the corresponding Active Directory users in the format DOMAIN\user_name.
SSOMapping | ADUserNameField | If you have populated the Domino user documents with the Active Directory user names as the bottom value in the User name field, this parameter should have the value FullName(-1). This tells the connector that the last value in the multi-value field FullName (the internal API name for the User name field) contains the Active Directory user name.

After turning this feature on, you can run the FAST Search Lotus Notes user directory connector. Verify that the file specified in the parameter SSOMapping/XMLOutputFileName is created in the folder that is specified in the parameter FSAOutput/OutputDirectory and that it contains the correct mappings. It should contain mappings from the Active Directory user to the Domino distinguished name.

Upload the SSO mapping file to the XMLAliaser

After you have run the user directory connector (and generated a new ssomapping.xml file), upload this file to the XMLAliaser.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command to upload the ssomapping.xml file to the XMLAliaser:

    Set-FASTSearchSecurityXMLAliaser -id win2lnx -PathToXMLFile 'C:\FASTSearch\var\lotusnotesconnector\security\ssomapping.xml'

    Caution:

    The path in this command is different if you have not installed FAST Search Server 2010 for SharePoint in the default directory C:\FASTSearch.

    Verify if the command returns without errors. If there are any errors, the command will return red text. If the command is successful, you see several parameters appear.

To verify that the import went well, enter the following command:

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    Get-FASTSearchSecurityXMLAliaser -id win2lnx

Verify that the parameter XmlFileName was assigned a randomly generated XML file name:

  • PathToXMLFile :

  • XmlFileName : win2lnx_c3289e98-d0d1-4e84-8f83-97767eaf74df.xml

  • InputPropertyName : $PRINCIPAL_REFERENCE_ALIAS

  • Identity : win2lnx

  • InputUserStoreId : win

  • OutputUserStoreIds : {lnx}

You should now be ready to search through the front-end and get results from Domino. These results should reflect the expected output according to the permissions of the Domino user mapped to the Active Directory user who searches.

Troubleshooting

Unable to upload the SSO mapping file to the XMLAliaser

Issue: The command Set-FASTSearchSecurityXMLAliaser returns a stack trace that contains the message Caused By: An item with the same key has already been added.

Cause: You have mapped more than one Domino user to the same Active Directory user, or the other way around.

Resolution:

  1. Inspect the file that you referred to in the FAST Search Lotus Notes user directory connector configuration file parameter SSOMapping/XMLOutputFileName and find the duplicate entry/entries.

    For example:

    Copy Code
    <user name="AD\user1">
<domain prefix="lnx" username="cn=<Domino User 1>/ou=department/o=company"/>
</user>
<user name="AD\user1">
<domain prefix="lnx" username="cn=<Domino User 2>/ou=department/o=company"/>
</user>

  2. Open Domino Administrator and edit the user document for one of the users who were discussed earlier (<Domino User 1> or <Domino User 2>) and remove the mapping for that user.

  3. Save the user document and rerun the FAST Search Lotus Notes user directory connector to reproduce the ssomapping.xml file.

  4. Rerun the command Set-FASTSearchSecurityXMLAliaser, see Upload the SSO mapping file to the XMLAliaser, and verify that the stack trace error message has disappeared.

  5. Run a query and check whether it returns the expected results.

A query does not return the expected security trimmed results

Issue: Running a query does not return the expected security trimmed search results from Lotus Notes. Only anonymous Lotus Notes content is being returned.

Cause: The format of the Active Directory user in the security claim may be different from the Active Directory user format in the Domino user document.

Resolution: Set the FAST Search Authorization (FSA) security log level to "info" to check whether the security claim format for the Active Directory user in the FSA authorization worker log file corresponds to the Active Directory user format in the Domino user document. If the format is different, edit the Domino user document.

  1. Set the log level to info. Open a Microsoft FAST Search Server 2010 for SharePoint command prompt as an administrator and run the following command:

    Copy Code
    Set-FASTSearchSecurityLogLevel -DefaultLogLevel Info

  2. Rerun the search.

  3. Go to <FASTSearchFolder>\var\log\syslog,

    where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

  4. Open the log file authorization-worker_<computer>.txt, where <computer> is the computer name of the query processing node. If there are multiple query processing nodes in the system, the information can be in any log file for any of the query processing nodes, depending on which query processing node serviced the request.

  5. Search for the system message from Claims.dll:GetClaimsPrincipal.

  6. Check if the format of the Active Directory user is the same as the format in the bottom value of the User name field in the Domino user document.

  7. If the format is different, correct the format in the Domino user document to match the format in the security claim.

    1. Open Domino Administrator and edit the Domino user document. Specify the Active Directory user name in the correct format in the bottom value of the multi-value User name field. Save the changes.

    2. Rerun the FAST Search Lotus Notes user directory connector to reproduce the ssomapping.xml file.

  8. Rerun the command Set-FASTSearchSecurityXMLAliaser. See Upload the SSO mapping file to the XMLAliaser.

  9. Run a query and check whether it returns the expected results.

See Also

Concepts

Configure the FAST Search Lotus Notes connector
Crawling Lotus Notes content with the FAST Search Lotus Notes connector
Start a crawl (FAST Search Lotus Notes user directory connector)
lotusnotessecuritytemplate.xml reference


Manage crawling with the FAST Search Lotus Notes content connector

Before you start crawling with the FAST Search Lotus Notes content connector, consult the topics Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint) and Configure the FAST Search Lotus Notes connector.

In this section:

  • Add, edit, or remove a FAST Search Lotus Notes content connector configuration

  • Manage crawl rules (FAST Search Lotus Notes connector)

  • Test, start or stop a crawl (FAST Search Lotus Notes content connector)

  • Schedule crawls (FAST Search Lotus Notes content connector)

  • Manage mapping crawled to managed properties (FAST Search Lotus Notes connector)


Add, edit, or remove a FAST Search Lotus Notes content connector configuration
See Also

Add a configuration

  1. Copy the configuration template lotusnotestemplate.xml that is located in the FAST Search Server 2010 for SharePoint installation etc folder, typically C:\FASTSearch\etc. Give the copied template a new unique name, for example MyLotusNotesContentConfig.xml. Keep it in the etc folder.

  2. Configure the parameter settings for your environment in an XML editor of your choice.

  3. Save the configuration file in UTF-8 format. Make sure the file is located in the etc folder of the FAST Search Server 2010 for SharePoint installation folder.

  4. The FAST Search Lotus Notes content connector configuration is now added. The content connector has to be run for the configuration to be enabled.

    Warning:

    Do not rename the configuration file after you have run the connector. The configuration file name is used to identify the state tracking tables for this configuration. If you rename the file, the state information for the configuration will be lost.

Edit a configuration

  1. Open the configuration file that you want to edit. You can find this file in the etc folder of the FAST Search Server 2010 for SharePoint installation folder.

  2. Make the necessary edits in an XML editor of your choice and save the changes in UTF-8 format.

  3. The content connector has to be run in order to enable the changes in the configuration.

Remove a configuration

Follow the steps in the topic Reset the content index (FAST Search Lotus Notes connector) to remove a FAST Search Lotus Notes content connector configuration. This topic also explains how to completely remove previously crawled Lotus Notes content from the index.

See Also

Concepts

Configure the FAST Search Lotus Notes content connector
Test, start or stop a crawl (FAST Search Lotus Notes content connector)
Reset the content index (FAST Search Lotus Notes connector)
lotusnotestemplate.xml reference


Manage crawl rules (FAST Search Lotus Notes connector)

The FAST Search Lotus Notes connector offers a number of ways to specify which Domino content to crawl.

Available filtering mechanisms

To add filters for which databases to crawl, you can:

  • Explicitly list one or more databases to crawl;

  • Specify one or more regular expressions to include databases in search;

  • Specify one or more regular expressions to exclude databases from search.

To add filters to select which Lotus Notes documents to crawl, you can:

  • Specify a database view to use the search formula of that view to select which Lotus Notes documents in the database to crawl;

  • Specify a search formula to select which Lotus Notes documents in the database to crawl.

To add filters to select which attachments to crawl, you can:

  • List which attachment document extensions are to be crawled (for example ppt, doc, xls and pdf);

  • List which attachment document extensions are to be excluded from the crawl (for example exe, jpg and zip);

  • Specify a maximum size for attachments to be crawled.

The parameter group Filters in the FAST Search Lotus Notes content connector configuration file is used to specify filters. The table below lists the parameters from the configuration file and how they are used. For a full description and example values, refer to the technical reference.

Filter category Parameter Short description

Databases

DataBase

Semicolon separated list of database paths to be crawled. These paths need to match the path (and file name) of the database on the Domino server exactly.

This parameter may contain views. To include a view, append the view to the database path, separating it with a #.

If a view is specified, only the search formula of that view is used. If no view is specified, the default database view is used. If there is no default view, all Lotus Notes documents will be crawled.

Databases

IncludeDatabases

Semicolon separated list of one or more include filters, regular expressions, for database names/paths.

Use when the DataBase parameter is empty, and in combination with the ExcludeDatabases filter.

Databases

ExcludeDatabases

Semicolon separated list of one or more exclude filters, regular expressions, for database names/paths.

Use when the DataBase parameter is empty, and in combination with the IncludeDatabases filter.

This parameter overrides the include filter; use it to narrow included content.

Lotus Notes documents

ViewName

The name of a view to be used for all databases that are crawled. If the view cannot be found in a database, all documents will be crawled.

To use different views for different databases, use the DataBase parameter and list all the databases to be indexed along with the view name for each.

Lotus Notes documents

SearchQuery

A search formula written in the Notes Query Language that will be used for all databases that do not have the view specified in the ViewName parameter AND do not have a default view.

Attachments

IncludeAttachmentExtensions

Semicolon separated list of filename extensions for attachments that should be crawled.

Attachments

ExcludeAttachmentExtensions

Semicolon separated list of filename extensions for attachments that should NOT be crawled. This parameter is ignored if the IncludeAttachmentExtensions parameter is set.

Attachments

MaxAttachmentSize

Maximum size of attachments (in KB). All attachments that are larger than this size will not be crawled. Use this parameter to increase performance and/or to prevent the connector from sending documents to the search engine that are too large.

Considerations when modifying the filters

Be careful when you modify any of the filters after the initial crawl has taken place. This because when the connector performs incremental crawls, it will only pick up databases and Lotus Notes documents that have been modified since the previous time it ran.

Warning:

You can increase the number of items the content connector crawls, but not reduce the number of items in the crawl.

Scenario 1 describes what happens if you try to reduce the number of items to be crawled by modifying the filters after the initial run, resulting in an incorrect behavior. Scenario 2 describes an example how a filter change can increase the number of items in the crawl, resulting in correct behavior.

Scenario 1

You run the connector with a DataBase parameter that lists three databases to be crawled: Database1.nsf;Database2.nsf;Database3.nsf. Each of these databases contains 1000 Lotus Notes documents, so 3000 items are crawled in total.

Now, you remove Database3.nsf from the DataBase parameter before an incremental run of the connector takes place. The connector will not detect that the parameter has changed, and will not detect what the value of the parameter was before the change. The connector will crawl Database1.nsf and Database2.nsf incrementally, detecting changes in these databases. The 1000 notes from Database3.nsf will remain unmodified in the search engine. Unless you re-add Database3.nsf to the configuration, the Lotus Notes documents in this database will remain unmodified in the search engine forever. This is the case even if you delete the entire database from the Domino server, which will lead to erroneous search results.

This applies to all of the parameters in the Filter group of the configuration file.

If you need to reconfigure the connector to no longer crawl content that you previously crawled, follow this procedure:

  1. Make the necessary modifications to your FAST Search Lotus Notes content connector configuration file and set the configuration parameter StateTracker/PurgeAtStart to true.

  2. Clear the contents of the content collection that you are feeding to, as specified in the configuration parameter FASTSearchSubmit/Collection.

    Warning:

    This will delete all the items in the content collection, possibly including content from other sources, like the shared SharePoint indexing connector(s), the FAST Search Web crawler and/or the FAST Search database connector.

    1. On the Start menu, click All Programs.

    2. Click Microsoft FAST Search Server 2010 for SharePoint.

    3. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

    4. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

      Clear-FASTSearchContentCollection -Name <collection name>

      Where:

      <collection name> is the content collection you are about to clear.

      Wait for the command to finish. This may take some time.

  3. Run the FAST Search Lotus Notes content connector to re-feed all the documents that match the new configuration. Because you set StateTracker/PurgeAtStart to true, the connector will not perform an incremental crawl, but instead purge its state and run a full crawl. This may take a long time, depending on how much content is to be crawled.

  4. When the full crawl has finished, set the parameter StateTracker/PurgeAtStart to false in order to activate incremental crawls.

Important:

After clearing the content collection, all the relevant content will have been removed from the search engine. It will not be searchable until the FAST Search Lotus Notes content connector is rerun.

To reduce the impact of having to delete all content from the search engine, consider to split the connector configuration into multiple configurations, where each connector configuration uses a separate content collection. This will reduce the amount of content that is (temporarily) deleted from the search engine. When deciding how to break up the configuration, use the following guidelines:

  • If you have Lotus Notes databases that you know will never get deleted, crawl them with a separate configuration (you should never need to perform the procedure above for this configuration)

  • If you have Lotus Notes databases that are more important to always have searchable than others, crawl them with a separate configuration and keep the amount of content for this configuration reasonably small so that the time that the content is not searchable is minimized. For example, if you are crawling a thousand e-mail databases, and it is important that this content is searchable, consider crawling the mailboxes with ten different configurations (each crawling one hundred databases). That way, when a user quits the company, only one hundred users are affected by this, and their search-downtime will be shorter.

  • Even if you conclude that it is OK to have a single configuration for the FAST Search Lotus Notes connector, you should consider having one separate content collection for Lotus Notes content and one for other content (for example SharePoint sites, databases and Web) so that the procedure above does not delete the content from those sources.

Scenario 2

You run the connector with a DataBase parameter that lists 3 databases to be crawled: Database1.nsf;Database2.nsf;Database3.nsf. Each of these databases contains 1000 Lotus Notes documents, so 3000 items are crawled in total.

Now, you add Database4.nsf to the DataBase parameter before an incremental run of the connector occurs. The connector will not detect that the parameter value has changed, but will crawl all four databases and detect any changes to databases Database1.nsf, Database2.nsf and Database3.nsf. In addition, it will crawl and index all content from Database4.nsf. This is because the connector has no state for this database from a previous run. In the next incremental run, all four databases will be crawled incrementally.


Test, start or stop a crawl (FAST Search Lotus Notes content connector)
See Also

When you have completed your configuration, use the command line to operate the FAST Search Lotus Notes content connector.

Test the connection

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    lotusnotesconnector.bat testconnections -f ..\etc\<content connector configuration filename>

    Where:

    • <content connector configuration filename> is the name of the FAST Search Lotus Notes content connector configuration file.

Start a crawl

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    lotusnotesconnector.bat start -f ..\etc\<content connector configuration filename>

    Where:

    • <content connector configuration filename> is the name of the FAST Search Lotus Notes content connector configuration file.

Stop a crawl

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    lotusnotesconnector.bat stop -f ..\etc\<content connector configuration filename>

    Where:

    • <content connector configuration filename> is the name of the FAST Search Lotus Notes content connector configuration file.

For detailed information about the command line options, see the lotusnotesconnector.bat reference topic.

See Also

Reference

lotusnotesconnector.bat reference

Concepts

Add, edit, or remove a FAST Search Lotus Notes content connector configuration
Configure the FAST Search Lotus Notes content connector


Schedule crawls (FAST Search Lotus Notes content connector)

Use this procedure to schedule crawls for the FAST Search Lotus Notes content connector. To schedule crawls, use the Windows Task Scheduler.

Create a new scheduled task using the Windows Task Scheduler (Windows Server 2008)

The FAST Search Lotus Notes connector should have a task for both the content connector as well as the user directory connector.

Warning:

Do not schedule the FAST Search Lotus Notes content connector and FAST Search Lotus Notes user directory connector to start crawling at the same time. The same ID file is used to connect to Domino, and if both connectors start at the same time, errors could occur. The scheduled crawls may crawl at the same time, as long as the scheduled start time of the crawl is not the same.

  1. On the Start menu, click Administrative tools.

  2. Click Task Scheduler.

  3. In the Actions pane, click Create Basic Task.

  4. Give the task a unique name and click Next.

  5. In the Trigger menu, mark the radio button in front of the preferred interval and click Next. Provide any additional input if prompted and click Next.

    Warning:

    Make sure there is enough time for a full crawl to complete between the intervals.

  6. In the Action menu, mark the radio button in front of Start a Program and click Next.

    1. In the Program/script field, type <FAST Search Server 2010 for SharePoint installation directory>\bin\lotusnotesconnector.bat, where <FAST Search Server 2010 for SharePoint installation directory> is the installation directory of FAST Search Server 2010 for SharePoint, usually C:\FASTSearch\.

    2. In the Add arguments field, type start -f ..\etc\<name of the connector configuration file>, where <name of the connector configuration file> is the configuration file for the FAST Search Lotus Notes content connector.

    3. In the Start in field, type <FAST Search Server 2010 for SharePoint installation directory>\bin, where <FAST Search Server 2010 for SharePoint installation directory> is the installation directory of FAST Search Server 2010 for SharePoint, usually C:\FASTSearch\.

  7. In the Summary dialog, review your settings and then click Finish to create the task.

Allow scheduled tasks to run when the user is not logged on

By default, scheduled task(s) are not allowed to run when the user is not logged on. To allow the scheduled task(s) to run when the user is not logged on:

  1. In the Windows Task Scheduler left navigation pane, click the Task Scheduler Library.

  2. Right click on the scheduled task(s) that you created and then click Properties.

  3. In the General tab, select the radio button in front of Run whether user is logged on or not and then click OK.

  4. Type the user name and password of the user who is supposed to run the task and then click OK.


Manage mapping crawled to managed properties (FAST Search Lotus Notes connector)

Once you have run the FAST Search Lotus Notes content connector, crawled content will show up in the crawled properties in the Microsoft SharePoint Server 2010 Central Administration. In order to make this content searchable, you must map the crawled properties to managed properties. A number of properties are mapped by default. If you want to add an additional crawled property to a managed property, you must either map it to an existing Managed Property or create a new Managed Property for the crawled property you want to create a mapping for.

Predefined crawled properties

The FAST Search Lotus Notes connector submits a predefined set of crawled properties in the "Lotus Notes" category to the content index. The following table lists these, and their default mapping to managed properties if applicable.

Crawled property | Type | Managed property | Description
subject | Text | title | Typically the title of a note. There may be databases that do not have a "subject" property, but most do.
lastmodified | Datetime | datetime | The time that this note was last modified.
dbtitle | Text | None | The title of the Lotus Notes database where the document resides.
lastaccessed | Text | None | The time that this note was last accessed.
size | Integer | size | The size in bytes of the note.
created | Datetime | created | The time that the document was created.
dbtemplate | Text | None | The database template used to create the database where the document resides.
dbpath | Text | None | The file path (on the Domino server) to the database where the document resides
dbcategories | Text | None | The categories under which the database (where the document resides) appears in the Database Library
docpath | Text | None | A unique hierarchical string representing the document ID of a document. The string is hierarchical because Lotus Notes databases can be nested arbitrarily deep.
uid | Text | None | The universal ID, which uniquely identifies a document across all replicas of a database.
notesurl | Text | None | The Domino URL to the document. Typically used to open the document from the Lotus Notes client.
form | Text | None | The form used to create the database where the document resides.
database | Text | None | The file name of the database where the document resides.
dbserver | Text | None | The name of the Domino server where the document resides.
nestinglevel | Integer | None | Identifies the nesting level of the document. For example, if the document is in a database that is attached to a note in a different database, the nestinglevel value would be 2.
viewurl | Text | None | The URL to the database view that this document belongs to. If no view was specified in the configuration, the default view of the database is used.

You may choose to map one or more of the crawled properties from this table to managed properties.

One mapping that does not appear in the predefined mapping table and that you should consider adding is the crawled property dbtitle. Map this property to the managed property sitename to improve the search experience when searching Lotus Notes content. Mapping dbtitle to sitename will populate the sitename refiner in the search front-end with Lotus Notes database title for all Lotus Notes documents.

How to map crawled to managed properties

To open the Managed Properties page:

  1. On the Central Administration page, select General Applications Settings.

  2. On the General Application Settings page, under Search, select Farm Search Administration.

  3. Select Search Service Application, and select your Query Search Service Application (for example, fastsearchquery).

  4. Select FAST Search Administration from the menu on the left.

  5. On the FAST Search Administration page, under Property Management, click Managed Properties.

To add a crawled property to a managed property:

  1. On the Managed Properties page, click Crawled Properties Categories.

  2. On the Crawled Property Categories page, find the Lotus Notes category and click it.

  3. On the Crawled properties page, click the crawled property name that you want to edit.

  4. On the Edit Crawled Property page, in the Mappings to managed properties section, click Add Mapping.

  5. In the Select managed properties to map to dialog box, configure the settings. In the Available managed properties list, search for or select a managed property, and then click Add. Repeat if you want to add more managed properties to the same crawled property. Click OK.

  6. On the Edit Crawled Property page, if you want to include values for the crawled property in the search index, select the Include values for this property in the search index check box.

    Select the check box to enable querying against the values of this crawled property. This will place these values in the search index. For example, if the crawled property is "author," simple queries such as "Smith" will return both items that contain the word "Smith" and items whose author property is "Smith." When unchecked, users will have to query against the managed property "author:Smith" to find the same items. Including unnecessary properties may have a negative effect on search relevance and performance.

  7. Click OK.

  8. Rerun the FAST Search Lotus Notes content connector for the changes to take effect.

    Tip:

    Incremental crawls will not pick up the new mapping. In order to activate the new mapping, it is necessary to run a full crawl.


Reset the content index (FAST Search Lotus Notes connector)
See Also

Published: March 17, 2011

This topic explains how to completely remove previously crawled Lotus Notes content from the index. First you must clear the state tracker database, and then you should decide whether or not you want to delete the FAST Search Lotus Notes content connector configuration file, and lastly you must clear the content collection that the Lotus Notes content was fed to.

Clear the state tracker database

  1. Open the FAST Search Lotus Notes content connector configuration file. You can find this file in the etc folder of the FAST Search Server 2010 for SharePoint installation folder.

    1. Set the parameter StateTracker/PurgeAtStart to true.

    2. Set the parameter Filters/Database to a non-existent database name, for example nosuchdatabaseexists. This will cause the connector to not find any content to crawl.

    3. Save the file.

  2. Run the FAST Search Lotus Notes content connector. This will clear all the state tracker database tables and free space in the SQL Server database.

    1. Verify that you meet the following minimum requirement: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

    2. On the Start menu, click All Programs.

    3. Click Microsoft FAST Search Server 2010 for SharePoint.

    4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

    5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

      lotusnotesconnector.bat start -f ..\etc\<content connector configuration filename>

      Where:

      • <content connector configuration filename> is the name of the FAST Search Lotus Notes content connector configuration file.

  3. Wait for the connector to clear its state tracker database tables and to stop running. Since you provided a non-existent database name in step 1b, the connector will fail with an error message that you can ignore. The state tracker database is now cleared.

If you want to delete the FAST Search Lotus Notes content connector configuration file, continue with Remove the configuration (optional). If you want to keep the configuration file for later use, continue with Clear the content collection.

Remove the configuration (optional)

Skip these steps if you want to reuse the FAST Search Lotus Notes content connector configuration file later.

  1. Navigate to the FAST Search Lotus Notes content connector configuration file that you want to remove. You can find this file in the etc folder of the FAST Search Server 2010 for SharePoint installation folder.

  2. Open the configuration file and note the name of the content collection that is specified in the parameter FASTSearchSubmit/Collection if you have not already done this.

  3. Delete the file.

The FAST Search Lotus Notes content connector configuration file is now removed. Continue with Clear the content collection.

Clear the content collection

You are now ready to reset the content index by clearing the contents of the content collection that you were feeding to.

Warning:

This will delete all items in the content collection. Any other content from other content sources that was fed to the same content collection is also cleared.

  1. Verify that you meet the following minimum requirement: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    Clear-FASTSearchContentCollection -Name <collection name>

    Where:

    • <collection name> is the content collection you are about to clear.

  6. Wait for the command to finish. This may take some time.

    Important:

    After clearing the content collection, all the relevant content will have been removed from the search index. It will not be searchable until the FAST Search Lotus Notes content connector is rerun.

See Also

Reference

Clear-FASTSearchContentCollection

Concepts

Configure the FAST Search Lotus Notes content connector
Test, start or stop a crawl (FAST Search Lotus Notes content connector)
lotusnotestemplate.xml reference


Manage crawling with the FAST Search Lotus Notes user directory connector

Before you run the FAST Search Lotus Notes user directory connector, consult the topics Plan to deploy FAST Search specific connectors (FAST Search Server 2010 for SharePoint), Configure the FAST Search Lotus Notes connector and Prepare FAST Search Authorization for use with the FAST Search Lotus Notes connector.

In this section:

  • Add, edit, or remove a FAST Search Lotus Notes user directory connector configuration

  • Start a crawl (FAST Search Lotus Notes user directory connector)

  • Schedule crawls (FAST Search Lotus Notes user directory connector)


Add, edit, or remove a FAST Search Lotus Notes user directory connector configuration
See Also

Add a configuration

  1. Copy the configuration template lotusnotessecuritytemplate.xml that is located in the FAST Search Server 2010 for SharePoint installation etc folder, typically C:\FASTSearch\etc. Give the copied template a new unique name, for example MyLotusNotesSecurityConfig.xml. Keep the file in the etc directory.

  2. Configure the parameter settings for your environment in an XML editor of your choice.

  3. Save the configuration file in UTF-8 format. Make sure the file is located in the etc directory of the FAST Search Server 2010 for SharePoint installation folder.

  4. The FAST Search Lotus Notes user directory connector configuration is now added. The user directory connector has to be run in order for the configuration to be activated.

Edit a configuration

  1. Open the configuration file that needs to be edited from the etc directory of the FAST Search Server 2010 for SharePoint installation folder.

  2. Make the necessary edits in an XML editor of your choice, and save the changes in UTF-8 format.

  3. The user directory connector has to be run in order to activate the changes in the configuration.

Remove a configuration

  1. Navigate to the configuration file in the etc directory of the FAST Search Server 2010 for SharePoint installation folder that needs to be removed. Delete the file.

See Also

Concepts

Configure the FAST Search Lotus Notes user directory connector
Start a crawl (FAST Search Lotus Notes user directory connector)
lotusnotessecuritytemplate.xml reference


Start a crawl (FAST Search Lotus Notes user directory connector)
See Also

When you have completed your configuration, use the command line to operate the FAST Search Lotus Notes user directory connector.

Start a crawl

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click the Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Microsoft FAST Search Server 2010 for SharePoint shell command prompt, type the following command:

    lotusnotessecurity.bat start -f ..\etc\<user directory connector configuration filename>

    Where:

    • <user directory connector configuration filename> is the name of the FAST Search Lotus Notes user directory connector configuration file.

For detailed information about the command line options, see the lotusnotessecurity.bat reference topic.

See Also

Reference

lotusnotessecurity.bat reference

Concepts

Add, edit, or remove a FAST Search Lotus Notes user directory connector configuration
Configure the FAST Search Lotus Notes user directory connector


Schedule crawls (FAST Search Lotus Notes user directory connector)

Use this procedure to schedule crawls for the FAST Search Lotus Notes user directory connector. To schedule crawls, use the Windows Task Scheduler.

Create a new scheduled task using the Windows Task Scheduler (Windows Server 2008)

The FAST Search Lotus Notes connector should have a task for both the content connector as well as the user directory connector.

Warning:

Do not schedule the FAST Search Lotus Notes content connector and FAST Search Lotus Notes user directory connector to start crawling at the same time. The same ID file is used to connect to Domino, and if both connectors start at the same time, errors could occur. The scheduled crawls may crawl at the same time, as long as the scheduled start time of the crawl is not the same.

  1. On the Start menu, click Administrative tools.

  2. Click Task Scheduler.

  3. In the Actions pane, click Create Basic Task.

  4. Give the task a unique name and click Next.

  5. In the Trigger menu, mark the radio button in front of the preferred interval and click Next. Provide any additional input if prompted and click Next.

    Warning:

    Make sure there is enough time for a full crawl to complete between the intervals.

  6. In the Action menu, mark the radio button in front of Start a Program and click Next.

    1. In the Program/script field, type <FAST Search Server 2010 for SharePoint installation directory>\bin\lotusnotessecurity.bat, where <FAST Search Server 2010 for SharePoint installation directory> is the installation directory of FAST Search Server 2010 for SharePoint, usually C:\FASTSearch\.

    2. In the Add arguments field, type start -f ..\etc\<name of the connector configuration file>, where <name of the connector configuration file> is the configuration file for the FAST Search Lotus Notes user directory connector.

    3. In the Start in field, type <FAST Search Server 2010 for SharePoint installation directory>\bin, where <FAST Search Server 2010 for SharePoint installation directory> is the installation directory of FAST Search Server 2010 for SharePoint, usually C:\FASTSearch\.

  7. In the Summary dialogue, review your settings and click Finish to create the task.

Allow scheduled tasks to run when the user is not logged on

By default, scheduled task(s) are not allowed to run when the user is not logged on. To allow the scheduled task(s) to run when the user is not logged on:

  1. In the Windows Task Scheduler left navigation pane, click the Task Scheduler Library.

  2. Right click on the scheduled task(s) that you created and click Properties.

  3. In the General tab, select the radio button in front of Run whether user is logged on or not and then click OK.

  4. Type the username and password of the user that is supposed to run the task and click OK.


Manage the crawling impact of the FAST Search specific connectors

This topic explains how to manage the crawling impact of the indexing connectors that are specific to FAST Search Server 2010 for SharePoint: the FAST Search database connector, the FAST Search Lotus Notes connector, and the FAST Search Web crawler.

Managing the impact of the FAST Search database connector

The FAST Search database connector itself does not have a throttling mechanism to reduce the impact of a crawl on the source database or on the search engine.

However, it is unlikely that the FAST Search database connector will put a significant load on the source database. The connector is able to extract database rows from the source database at a much higher rate than that the search engine is able to index them. Only when the SELECT query used by the FAST Search database connector is very complex there may be an increased load on the database.

In some types of SQL/JDBC implementations, selecting a large dataset can cause a large part of the memory to be allocated to the FAST Search database connector. In some cases, the complete dataset will be transferred to the FAST Search database connector client before it can be processed. You can avoid this by having a server-side cursor. If you are using SQL Server, add ";selectMethod=cursor" to the JDBCURL parameter in FAST Search database connector configuration file. Note that this will increase the memory consumption on the database server side, since the result set is held in memory there until it is transferred to the connector.

Managing the impact of the FAST Search Lotus Notes connector

A single instance of the FAST Search Lotus Notes connector is typically able to extract documents from the Domino server significantly faster than the search engine is capable of indexing them. Because of this, the load on the Domino server is not likely to significantly increase when the FAST Search Lotus Notes connector is running.

However, the connector automatically slows down the extraction rate to match the feed rate to the search engine. This means that the FAST Search Lotus Notes connector could increase the load on the search engine backend.

To throttle the extraction rate from the Domino server, you can configure the AdapterThrottleSleepMS parameter in the FAST Search Lotus Notes content connector configuration file. The value of this parameter, located in the ConnectorExecution group, sets how many milliseconds the connector should sleep between each Note it extracts from the Domino Server.

Note that the sleep interval counts for each adapter thread in the connector. If you use multiple adapter threads, specified in the configuration parameter ConnectorExecution/NumAdapters, the maximum extraction rate for the connector increases. For example: you set the parameter ConnectorExecution/AdapterThrottleSleepMS to 200. This means that each adapter thread can extract a maximum of 5 documents per second from Domino. If you use the default value for the parameter ConnectorExecution/NumAdapters, which is 3, the maximum number of documents that the connector can extract per second is 15. The actual number will be lower since, in addition to completing the sleep interval, the extraction itself takes some time as well.

Managing the impact of the FAST Search Web crawler

While configuring the FAST Search Web crawler, make sure to avoid potentially overloading Web servers. There are a number of settings that can be considered in order to reduce the load of the FAST Search Web crawler.

Each Node Scheduler crawls a number of Web sites (and servers) at the same time, as configured by the max_sites setting. A request will be issued every number of seconds to these Web sites and servers, as configured in the delay setting, with a maximum number of requests pending to any server at any time as configured in the max_pending setting. This combination of settings can result in a significant load, especially if multiple Web sites are hosted on the same server.

Enabling JavaScript support (with the use_javascript setting) causes an additional load, as both JavaScript and CSS dependencies have to be downloaded. It is not uncommon for a Web item to refer around ten to thirty external JavaScripts. In order to improve performance on Web items that contain many external JavaScripts, the FAST Search Web crawler will by default use a request delay of 0 seconds for these dependencies. This delay can be increased using the javascript_delay setting. If you increase the javascript_delay setting, make sure that you adjust the processing timeouts in the Browser Engine too, to avoid timeouts while it is downloading dependencies.


Manage federated locations (FAST Search Server 2010 for SharePoint)

Published: October 7, 2010

Federated locations provide information that exists outside of your internal network to your end-users. Federation in Microsoft FAST Search Server 2010 for SharePoint is very similar to federation in Microsoft SharePoint Server 2010 search. However, there is a difference: FAST Search Server 2010 for SharePoint uses two separate Search Service Applications: the FAST Search Content Search Service Application (Content SSA) and the FAST Search Query Search Service Application (Query SSA). The Content SSA is used to crawl content and feed it into the FAST Search Server 2010 for SharePoint back-end, and the Query SSA is used to provide query results from the content that is crawled by the Content SSA. Federated locations are configured in the FAST Search Query Search Service Application.

Add, edit, or delete a federated location

To go to the Manage Federated Locations page

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Query Search Service application.

  2. On the Home page of the SharePoint Central Administration Web site, in the Application Management section, click Manage service applications.

  3. On the Service Applications page, click the name of your FAST Search Query Search Service Application.

  4. In the Quick Launch on your FAST Search Query Search Service Application Search Administration page, in the Queries and Results section, click Federated Locations.

To add a federated location

  1. On the Manage Federated Locations page, click New Location.

  2. On the Add Federated Location page, in the Location Name section, in the Location Name box, type a name for the new federated location. This name will appear to service application administrators and developers. In the Display Name box, type a name that will identify the federated location to service application administrators and site owners.

  3. In the Description section, in the Description box, type a detailed description for this federated location.

  4. In the Author section, in the Author box, type the name of the author that you want to appear to users of this location.

  5. In the Version section, in the Version box, type a version number for this location.

  6. In the Trigger section, select one of the following options:

    • Always: Query should always match.

    • Prefix: Query must begin with a specified prefix. If you select this option, type the prefix in the Add Prefix box.

    • Pattern: Query must match a specified pattern. If you select this option, type the pattern in the Add Pattern box.

  7. Expand the Location Information section.

    1. In the Location Type section, select one of the following options:

      • Search Index on this Server

      • FAST Index

      • OpenSearch 1.0/1.1

    2. In the Query Template section, in the Query Template box, specify the template for passing queries on the selected location.

    3. In the "More Results" Link Template section, in the "More Results" Link Template box, type the URL of the page that displays results for a search query.

  8. Expand the Display Information section, and then configure the following options:

    1. In the Federated Search Results Display Metadata section, clear the Use Default Formatting check box to do the following:

      • In the XSL box, edit the default XSL if necessary.

      • In the Properties box, edit the default properties if necessary.

      • In the Sample Data box, edit the default sample data as necessary.

    2. In the Core Search Results Display Metadata section, clear the Use Default Formatting check box to do the following:

      • In the XSL box, edit the default XSL if necessary.

      • In the Properties box, edit the default properties if necessary.

      • In the Sample Data box, edit the default sample data as necessary.

    3. In the Top Federated Results Display Metadata section, clear the Use Default Formatting check box to do the following:

      • In the XSL box, edit the default XSL if necessary.

      • In the Properties box, edit the default properties if necessary.

      • In the Sample Data box, edit the default sample data as necessary.

  9. Expand the Restrictions and Credentials Information section

    1. In the Restrict Usage section, select one of the following options:

      • No restriction: All sites can use this location,

      • Use restriction: Only allowed sites can use this location. If you select this option, in the Allowed Sites box, type the start addresses, using semicolons to separate the URLs if necessary.

To edit a federated location

  1. On the Manage Federated Locations page, point to the location that you want to edit, click the arrow that appears, and then click Edit Location.

  2. After you have made the changes that you want, click OK.

To copy a federated location

  1. On the Manage Federated Locations page, point to the location that you want to copy, click the arrow that appears, and then click Copy Location.

  2. Select the federated location you wish to copy, and then click OK.

To delete a federated location

  1. On the Manage Federated Locations page, point to the location that you want to delete, click the arrow that appears, and then click Delete Location.

  2. Click OK to confirm that you want to proceed with the deletion.

To import a federated location

  1. On the Manage Federated Locations page, click Import Location.

  2. On the Import Federated Location page, in the Location Definition File section, in the Federated Location Definition File box, type the path of a federated location definition (.fld) file or an OpenSearch description (.osdx) file, or click the Browse button to find an .fld or .osdx file on the local file system or the network.

  3. Click OK.

Note:

The .osdx file type in FAST Search Server 2010 for SharePoint is the successor to the .fld file type from Office SharePoint Server 2007. When you create and export a federated location in FAST Search Server 2010 for SharePoint, the location information is saved as an .osdx file.


Manage people search (FAST Search Server 2010 for SharePoint)

Published: December 9, 2010

This section explains how to enable and configure people search for Microsoft FAST Search Server 2010 for SharePoint and how to crawl people profiles by crawling the user profile store.

In this section:

  • Configure people search (FAST Search Server 2010 for SharePoint)

  • Crawl the user profile store (FAST Search Server 2010 for SharePoint)


Configure people search (FAST Search Server 2010 for SharePoint)

Published: December 9, 2010

To enable and configure people search for use with FAST Search Server 2010 for SharePoint, follow the instructions in this article.

People search in FAST Search Server 2010 for SharePoint is very similar to people search in Microsoft SharePoint Server 2010. There is one important difference: FAST Search Server 2010 for SharePoint uses two separate Search Service Applications. The content source for people search is added to the FAST Search Query Search Service Application.

In this article:

Enable people search

Configure Web application and site collections

Add user profiles to the User Profile service application

Configure My Site settings

Enable people search

People search requires that the following service applications were created and that the services on which they depend were started.

  • FAST Search Query Search Service Application

  • User Profile

  • Managed Metadata

The User Profile service application is an important part of people search because it stores much of the information that appears in results for people search. For example, information in My Site sites and a lightweight directory access protocol (LDAP) store, such as Active Directory Domain Services (AD DS) are stored by the User Profile service application.

For more information about the User Profile service application and how to manage it, refer to the SharePoint Server 2010 topic User Profile Service administration (SharePoint Server 2010).

In addition to the service applications, people search requires a Search Center site that is created by using the Enterprise Search Center template. Unlike the Basic Search Center, the FAST Search Center has a tab that is used to search for people.

Ensure service applications are running

This section explains how to verify the existence of the service applications that people search requires, helps you to create the service applications if they do not exist, and explains how to ensure that dependent services are started.

To verify that service applications exist

  1. Verify that the user account that is performing this procedure is a member of the Farm Administrators SharePoint group.

  2. On the Home page of the Central Administration Web site, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, verify that the following are listed in the Name column:

    • Managed Metadata Service

    • FAST Search Query Search Service Application (This service application is specific to FAST Search Server 2010 for SharePoint and has a unique name. Make sure that you know the name of the FAST Search Query Search Service Application.)

    • User Profile Service Application

  4. If the Managed Metadata Service or the User Profile Service Application does not exist, follow the steps in the To create a Managed Metadata or User Profile Service Application section in this article.

    If the FAST Search Query Search Service Application does not exist, you may not have completed the deployment of FAST Search Server 2010 for SharePoint. For more information, see Deployment for FAST Search Server 2010 for SharePoint for an overview of all the required steps and Create and set up the Query Search Service Application (FAST Search Server 2010 for SharePoint) for specifics.

    If all the service applications exist, continue with To ensure that the Managed Metadata Service application is started.

To create a Managed Metadata or User Profile Service Application

  1. On the Home page of the Central Administration Web site, in the Quick Launch, click Configuration Wizards.

  2. On the Configuration Wizards page, click Launch the Farm Configuration Wizard.

  3. On the Help Make SharePoint Better page, if it appears, click one of the following options, and then click OK:

    • Yes, I am willing to participate (Recommended)

    • No, I don't want to participate

  4. On the Configure your SharePoint farm page, select Yes, walk me through the configuration of my farm using this wizard, and then click Start the Wizard.

  5. On the Configure your SharePoint farm page, in the Service Account section, select Use an existing managed account.

  6. Select the check-boxes for the service applications that you need to create, and then click Next.

  7. On the Create Site Collection page, click Skip.

  8. On the Configure your SharePoint farm page, click Finish.

To ensure that the Managed Metadata Service application is started

  1. In the Application Management section, click Manage service applications.

  2. Ensure that the Status column for the Managed Metadata Service application shows Started. If the farm configuration wizard was not used to create this service application, the Managed Metadata Web Service might not be running. If this is the case, continue to step 3. Otherwise, skip to Configure Web application and site collections.

  3. On the Central Administration Home page, in the System Settings section, click Manage services on server.

  4. On the Services on Server page, in the Managed Metadata Web Service row, click the Start link in the Action column. (If the link in the Action column is Stop, the Managed Metadata Web Service is already running.)

Configure Web application and site collections

In a production environment, typically you create a separate Web application to host the site collection for My Site sites. For evaluation purposes, you can use the same Web application to host separate site collections for your SharePoint sites, FAST Search Center site, and My Site sites.

This section assumes that you will use the same Web application to host the site collections for SharePoint sites, the FAST Search Center site, and My Site sites. The server farm should already have at least two Web applications: one for the Central Administration Web site and one for SharePoint sites. This assumes that you used the procedures in Single server with SQL server (SharePoint Server) to install SharePoint Server 2010.

Ensure that a Web application was created for SharePoint sites, the FAST Search Center site, and My Site sites

  1. Verify that the user account that is performing this procedure is a member of the Farm Administrators SharePoint group.

  2. On the Home page of the Central Administration Web site, in the Application Management section, click Manage web applications.

  3. On the Web Applications Management page, all Web applications that have been created for this server farm are listed.

    The steps in Single server with SQL server (SharePoint Server) create a Web application that uses port 80. You can use that Web application to host the site collections for your SharePoint sites, FAST Search Center site, and My Site sites.

    Tip:

    If you performed a Standalone installation of SharePoint Server 2010, a Web application named SharePoint - 80 is automatically created on port 80 and a site collection is created in that Web application by using the Team Site template. This Web application could have also been created after a Server Farm installation if a site collection was created by using the Farm Configuration Wizard.

  4. If a Web application does not exist for your SharePoint sites and My Site sites, go to step 5. Otherwise, skip to Ensure service applications are associated with Web application.

  5. On the Web applications Management page, on the ribbon, in the Contribute group, click New.

  6. In the Create New Web Application dialog box, select Create a new IIS web site and optionally type a name for the site in the Name box.

  7. In the Port box, type the port number that you want this Web application to use.

  8. Specify any other settings, and then click OK.

  9. On the Application Created page, click OK.

Ensure that service applications are associated with a Web application

  1. On the Web Applications Management page, in the Name column, click the row that contains the Web application that you want to ensure is associated with the required service applications. Typically, this Web application is named SharePoint - 80.

    Note:

    You must ensure that the Web applications for your SharePoint sites, FAST Search Center site, and My Site sites are associated with the following service applications:

    • Managed Metadata Service

    • User Profile

    • FAST Search Query Search Service Application

  2. On the ribbon, in the Manage group, click Service Connections.

  3. In the Configure Service Application Associations dialog box, ensure that the following check boxes are selected: Managed Metadata Service, User Profile Service Application, <the name of your FAST Search Query Search Service Application>.

  4. Click OK.

Add user profiles to the User Profile service application

Before you can get meaningful people search results, you must add user profiles to the User Profile service application. There are two ways to do this:

  • Add user profiles manually

  • Synchronize with a directory service or external data source, such as AD DS, third party LDAP, or the Business Data Connectivity service

Important:

For a test environment, we recommend that you do not synchronize the profile store to a directory service or other external data source that is in a production environment. Instead, make a copy of the directory service with which to synchronize the User Profile service application.

View a list of user profiles

  1. On the Manage Service Applications page, click the row that contains the User Profile service application, and then on the ribbon, click Manage.

  2. On the Manage Profile Service page, in the People section, click Manage User Profiles.

  3. On the Manage User Profiles page, in the Find profiles box, type the name of the domain to which the users are a member, and then click Find.

    For example, if your users are members of the Contoso.com domain, type Contoso in the Find profiles box. Do not type the fully qualified domain name.

  4. Click Find.

Configure My Site settings

My Site settings are configured, by default, when the Farm Configuration Wizard is used to create the User Profile service application. Perform the following steps if you want to change the default settings.

  1. On the Manage Service Applications page, click the row that contains the User Profile service application.

  2. In the Operations group of the ribbon, click Manage.

  3. On the Manage Profile Service page, in the My Site Settings section, click Setup My Sites.

  4. In the Preferred Search Center section, in the Preferred Search Center box, ensure that the path of the FAST Search Center site is correct.

  5. In the Search scope for finding people list, ensure that People is selected.

  6. In the Search scope for finding documents list, select the search scope that you want to use for finding documents.

    We recommend that you select the All Sites scope.

  7. In the My Site Host section, in the My Site Host location box, ensure that the path to your My Site host is correct.

  8. In the Personal Site Location section, in the Location box, type the location at which to create personal sites.

    By default, the path is my/personal.

  9. In the Site Naming Format section, select the format you want to use to name new personal sites.

    Note that User name (do not resolve conflicts) is selected by default.

  10. In the Language Options section, optionally choose whether to allow users to choose the language of their personal site and whether the user profile will be edited using the language of their personal site.

  11. In the Read Permission Level section, optionally enter one or more accounts that will be granted the Read permission level in the personal site when it is created.

    By default, all authenticated users are granted this permission.

  12. In the My site E-mail Notifications section, optionally specify a different name that will appear in all e-mail notifications sent from all My Site sites.

    By default, the sender's name is My Site.

  13. Click OK.

Add information to My Site sites

A My Site stores its information in the User Profile service application. Therefore, the more information you add to a My Site site, the more interesting people search results will be. For example, we recommend that you provide information about user's expertise and interest in My Site sites to see how they appear on search results pages.

To add information to My Site sites, log on as a user for whom a user profile has been created in the User Profile service application and go to the URL of your My Site. To do this, in a Web browser, go to http://hostname/my/. If your My Site has not been created, one will be automatically created for you.


Crawl the user profile store (FAST Search Server 2010 for SharePoint)

Published: December 9, 2010

After you have followed the instructions in Configure people search (FAST Search Server 2010 for SharePoint), you can crawl the user profile store by adding a content source for people to the FAST Search Query Search Service Application.

Crawl the profile store

Before you crawl the profile store, we recommend that you ensure that the user account that is assigned to the default content access account is an administrator for the User Profile service application.

To add an administrator to the User Profile service application

  1. On the Manage Service Applications page, click the row that contains the User Profile service application, and then in the ribbon, click Administrators.

  2. In the Administrators for User Profile Service Application dialog box, in the To add an account… box, type the user account you want to add in the form of domain\user name.

  3. Click Add.

  4. In the Permissions list box, select the Full Control check box.

  5. Click OK.

Create a content source and crawl the profile store

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Query Search Service Application.

  2. On the Home page of the Central Administration site, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, click the name of your FAST Search Query Search Service Application.

  4. On the Search Administration page, under Crawling, click Content sources.

  5. On the Manage Content Sources page, click New Content Source.

  6. On the Add Content Source page, in the Name section, type a name for this content source.

  7. In the Content Source Type section, ensure that SharePoint Sites is selected.

  8. In the Start Addresses section, type sps3://hostname, where host name is the host name of the server farm.

  9. In the Crawl Schedules section, optionally create schedules for full and incremental crawls.

  10. In the Start Full Crawl section select the Start full crawl of this content source check box, and then click OK.

Verify the crawl of the profile store

  1. On the Search Administration page of the FAST Search Query Search Service Application, in the Quick Launch, click Search Administration.

    Note:

    We recommend that you view the crawl log to determine if any crawl errors occurred.

  2. Under Crawling, click Crawl log.

  3. In the Crawl History section, click the Refresh button to refresh the crawl status.

    After the crawl of the content source completes, verify that a zero is listed in the Errors column. If errors occurred, the number in the All Errors column will be greater than zero. In that case, click the number in the Errors column to view the crawl log for that content source. Otherwise, if no errors occurred, proceed to Optional steps.

    Note:

    If the crawl log indicates that an authentication error occurred, you can resolve the problem by following the steps in To add an administrator to the User Profile service application or by authenticating the crawler by creating a crawl rule to use the account of an existing administrator of the User Profile service application. For information about creating a crawl rule, see Create a crawl rule.

Optional steps

The following procedure is optional.

Create a crawl rule

If the account you assigned as the default content access account is not an administrator on the User Profile service application, we recommend that you create a crawl rule that the crawler can use to authenticate to the User Profile service application.

Note:

This procedure is not necessary if you performed the steps shown in the To add an administrator to the User Profile service application section earlier in this article.

  1. On the Home page of the Central Administration Web site, in the Application Management section, click Manage service applications.

  2. On the Manage Service Applications page, click the name of your FAST Search Query Search Service Application.

  3. On the Search Administration page, under Crawling, click Crawl rules. The Manage Crawl Rules page appears.

  4. On the Manage Crawl Rules page, click New Crawl Rule.

  5. In the Path section, in the Path box, type the start address for the User Profile service application in the form sps3://hostname, where hostname is the host name of the server farm.

  6. In the Crawl Configuration section, select Include all items in this path.

  7. In the Specify Authentication section, select Specify a different content access account.

  8. In the Account box that appears, type the name of user account that is an administrator on the User Profile service application in the form domain\user name.

  9. Type the password in the Password and Confirm Password boxes.

  10. Click OK.


Monitor FAST Search Server 2010 for SharePoint

FAST Search Server 2010 for SharePoint publishes monitoring data by using a number of different of technologies and APIs. This chapter contains information about monitoring sources and applications that can be used to locate and view this data.

With a few exceptions, the same information is available regardless of which tools and technologies an administrator prefers to use.

Recommended monitoring solution

The recommended monitoring solution for FAST Search Server 2010 for SharePoint is Microsoft System Center Operations Manager (SCOM) 2007 SP1 or R2.

Monitoring and logging sources

Logs:

  • Windows event logs: Applications and Services logs\FAST Search (Event Viewer)

  • SharePoint Unified Logging Service (ULS) Trace Logs: provider FASTSearch

  • Log files: <FAST Search Server 2010 for SharePoint installation folder>\var\log\[syslog]

Windows Management Instrumentation (WMI):

  • WMI namespace: root\Microsoft\FASTSearch

Performance counters:

  • Performance counter categories: FAST Search <x> (Performance monitor/perfmon.exe)

In this section:

  • FASTSearchMonitoring service and log servers

  • Monitor FAST Search Server 2010 for SharePoint with SCOM

  • Monitor FAST Search Server 2010 for SharePoint using WMI

  • Monitor FAST Search Server 2010 for SharePoint using performance counters

  • Read FAST Search Server 2010 for SharePoint logs

  • Monitor crawling in FAST Search Server 2010 for SharePoint


FASTSearchMonitoring service and log servers
See Also

The FAST Search Server 2010 for SharePoint monitoring windows service (FASTSearchMonitoring) runs on all search nodes. The service is not started or stopped by the FAST Node Controller, because the monitoring service should run regardless of the state of the remaining FAST Search Server 2010 for SharePoint processes in order to monitor the correct state of the search system at all times.

The output of the monitoring service can be consumed by System Center Operations Manager 2007 (SCOM), built-in Windows tools like Performance Monitor and Event Viewer, scripts and third party monitoring tools.

The monitoring service contains three sub-modules:

  • Metrics services - publishes monitoring data to performance counters and WMI

  • Local log server - publishes logs to the Windows event log and SharePoint ULS

  • Main log server - publishes logs to the Windows event log (farm mode) and log files (syslog)

Metrics services

This module polls statistics-, performance- and other health information from the FAST Search Server 2010 for SharePoint processes and publishes them into performance counters and WMI. It reads its main configuration from etc\MonitoringService.xml. MonitoringServiceConfig.exe is used to install the performance counters and WMI objects in the operation system.

This service also reads data from NodeConf.xml, Node.xml and LogServerConfig.xml, all found in the "etc" folder. The FASTSearchMonitoring service must be manually restarted if any of these files change.

Local log server

The local log server receives log messages from individual FAST Search Server 2010 for SharePoint processes on the same machine. This module runs on all nodes, and it forwards logs to the main log server running on the administration server. It also publishes the local logs to the Windows event log "FAST Search" folder and the SharePoint ULS (Unified Logging Service) trace log.

Main log server

The main log server is only active on the FAST Search Server 2010 for SharePoint administration server. It receives log messages from all local log servers and outputs aggregated logs for the whole system into the " FAST Search Farm" event log folder and individual log files in the <FASTSearchFolder>\var\log\syslog folder. WCF communication parameters used for server to server communication can be tuned in the standard .NET configuration file MonitoringService.exe.config.

See the article Read FAST Search Server 2010 for SharePoint logs for more information regarding the various log outputs of the log servers, and logserverconfig.xml reference for configuration information.

See Also

Reference

monitoringserviceconfig.exe reference

Concepts

Configure the FASTSearchMonitoring service
monitoringservice.xml reference


Configure the FASTSearchMonitoring service

The metrics service within the FASTSearchMonitoring Windows service poll statistics-, performance- and other health information from the FAST Search Server 2010 for SharePoint processes and publishes them into performance counters and WMI. In the rare occasion it should be needed, the FASTSearchMonitoring service can be configured by editing the monitoringservice.xml file, located in the \etc directory of the FAST Search Server 2010 for SharePoint installation folder.

Edit the FASTSearchMonitoring service

Edit the monitoringservice.xml file to configure WMI classes and performance counters published by the Microsoft FAST Search Server 2010 for SharePoint monitoring service.

For more information, refer to monitoringservice.xml reference.

Apply the new FASTSearchMonitoring service configuration

Use the monitoringserviceconfig.exe utility to apply file changes and to synchronize the configuration, WMI classes, and performance counters registered in the Windows operating system.

For more information, refer to monitoringserviceconfig.exe reference.


Monitor FAST Search Server 2010 for SharePoint with SCOM
See Also

The recommended monitoring solution for FAST Search Server 2010 for SharePoint is Microsoft System Center Operations Manager (SCOM) 2007 SP1 or R2. Various processes in FAST Search Server 2010 for SharePoint, including the FAST Search Server 2010 for SharePoint Monitoring service provide monitoring data using standardized interfaces in Windows Server 2008 and SharePoint Server. This data is consumed by SCOM but can also be utilized by other management applications and services.

Refer to the FAST Search Server 2010 for SharePoint Management Pack guide for information regarding management pack classes, discovery, views, rules, monitors and tasks. The guide is available as a download from the System Center Management Pack Catalog Website. Direct link to the FAST Search Server 2010 for SharePoint management pack and the FAST Search Server 2010 for SharePoint Management Pack guide: http://go.microsoft.com/fwlink/p/?LinkID=182110.

Refer to the topics Plan to monitor the health of the search system (FAST Search Server 2010 for SharePoint) and Deploy System Center Operations Manager (SCOM) 2007 (FAST Search Server 2010 for SharePoint) for SCOM 2007 deployment information.

See Also

Concepts

Best practices for monitoring FAST Search Server 2010 for SharePoint with SCOM


Best practices for monitoring FAST Search Server 2010 for SharePoint with SCOM

This topic contains tips and recommendations for efficient use of System Center Operations Manager 2007 with the FAST Search Server 2010 for SharePoint management pack.

Use Maintenance Mode

Always use Maintenance Mode when you install, reconfigure, reboot, uninstall or perform other kinds of maintenance. The consequences of not using maintenance mode can be temporary incorrect discovery, many alerts being triggered and monitors reporting critical and warning states.

Back up/export the default management pack

Even though new objects should be put in custom user-created management packs, it is easy to accidentally add new objects or references to the default management pack.

Management packs cannot be deleted until all references to them are deleted. Deleting references manually can be cumbersome. If you have a backup, the default management pack can be deleted and re-imported in order to quickly remove all references to other management packs.

Management packs are listed in Administration > Management Packs. Use Look for: default and then right-click and select Export Management Pack.

Customize rule behavior

Use overrides to customize monitor/rule behavior and disable individual monitors/rules. A typical use for overrides is, for example, to disable a monitor that repeatedly triggers bogus alerts or other errors.

Important:

New objects/overrides should always be added to a custom management pack, by using the Create New option.

Set up a periodic search query with the Web Application wizard

Use the Web Application wizard in Authoring > Management Pack Templates to set up a periodic search query towards your user- or customer-facing search front-end. This simple monitor will provide wide test coverage making sure that the search solution is working end-to-end.

Set up a distributed application to view the overall health status of a solution

Use the Distributed Application wizard in Authoring > Management Pack Templates to set up a distributed application that contains all SearchNode objects together with associated IIS, SQL Server, SharePoint and Windows Server objects from their respective management packs. The distributed application object will accumulate and reflect the true health status of the whole solution.

Customize the view by changing the axis layout

The QRServer and Indexer performance views in the Overview dashboard are small and sometimes trigger an unwanted layout of the X Axis timestamps, especially for shorter time ranges. Consider hiding the X Axis, by deselecting Personalize View > Chart Area > Show X Axis.

Group SearchNode objects

In multi-node search systems, group SearchNode objects together by using the InstanceID attribute.

Reset multiple event log monitors at the same time

The FAST Search Server 2010 for SharePoint management pack has many event log monitors. These event log monitors are triggered by warning and error log events from their respective components. There are usually no corresponding log events available that can be used to indicate a healthy state and automatically reset the event log monitor.

In some situations multiple event log monitors can be triggered by the same operational incident. This might occur, for example, when an administrator restarts nodes without first enabling Maintenance mode.

With many event log monitors in a warning or critical state, it can be cumbersome to reset each of them manually. Use the task Reset Log Monitors on a SearchNode object to reset all log monitors on a search node automatically. Use multi-select to perform the operation on many nodes at the same time.

Tip:

Consider adding an Override to disable log monitors that trigger too often.


Monitor FAST Search Server 2010 for SharePoint using WMI

WMI data can be accessed from System Center Operations Manager (SCOM) 2007, COM, Windows Scripting Host, Windows PowerShell, .NET, Win32 APIs and third party utilities and monitoring suites. Various GUI tools are available to access WMI, you can download, for example, WMI Administrative Tools from the Microsoft Download Center.

Refer to the topic WMI Overview for an extensive overview of the Windows Management Instrumentation (WMI) technology.

Reference information about the FAST Search WMI values

The FAST Search Server 2010 for SharePoint WMI classes are available in the ROOT\Microsoft\FASTSearch namespace.

For an overview of all the FAST Search Server 2010 for SharePoint WMI classes, refer to WMI classes (FAST Search Server 2010 for SharePoint).

Examples of how to retrieve WMI information

The following two examples show how you can monitor FAST Search Server 2010 for SharePoint using WMI.

Visual Basic script to retrieve process status

Visual Basic Script Copy Code
Dim oWMIService
Set oWMIService = GetObject("winmgmts:\\.\ROOT\Microsoft\FASTSearch")

Dim oResult
Set oResult = oWMIService.ExecQuery("SELECT * FROM NodeConfProcess")

For Each oWmiEntry In oResult
    WScript.Echo "Name: " & oWmiEntry.Id & ", State: " & oWmiEntry.Status
Next

Windows built-in tool WMIC.EXE command line to retrieve process status

Copy Code
C:\>wmic /NAMESPACE:\\ROOT\Microsoft\FASTSearch Path NodeConfProcess


Monitor FAST Search Server 2010 for SharePoint using performance counters

Performance counters can be accessed using the Windows Reliability and Performance Monitor tool (perfmon), System Center Operations Manger 2007, scripting, .NET, Windows PowerShell, Win32 APIs, etc.

Most FAST Search Server 2010 for SharePoint performance counter category names use the prefix FAST Search, for example FAST Search Indexer. The performance counters related to the indexing connectors used by the FAST Search Content Search Service Application are represented by the OSS Search FAST Content Plugin performance counter category.

For an overview of all the FAST Search Server 2010 for SharePoint performance counters, refer to Performance counters (FAST Search Server 2010 for SharePoint).

Examples of how to retrieve performance counter information

The following two examples show how you can monitor FAST Search Server 2010 for SharePoint using performance counters.

How to get a real-time overview of FAST Search performance counters in perfmon

  1. Open the Windows Reliability and Performance Monitor (perfmon).

  2. Remove any existing counters from the view.

  3. Add counters, with the green "+" icon (or use Ctrl+i) for one or more related performance counter categories.

  4. Click Action, then click Properties, then click General and set to Sample every to 3-10 seconds.

  5. Switch to Report view. Click Action, then click Properties, then click Graph, then click View (or Ctrl+G twice) to quickly get an overview of the data.

  6. Click Use File and then Save As to create an .MMC file with this configuration.

  7. Repeat this process to create views for different sets of performance counters.

You can double-click the saved .MMC files later to open the saved performance counter configuration/view.

How to read performance counters using Windows PowerShell 2.0

You can use the Windows PowerShell 2.0 Get-Counter commandlet to read performance counter data.

  1. On the Start menu, click All Programs.

  2. Click a Microsoft Windows PowerShell 2.0 shell.

  3. At the Windows PowerShell command prompt, type the command(s) in the syntax as found in the Get-Counter commandlet reference:

    Copy Code
    Get-Counter -Counter "\<Performance Counter category name>\<Performance Counter>" [-ComputerName <string[]>] [-Continuous][-MaxSamples <Int64>][-SampleInterval <int>] [<CommonParameters>]

    Where:

    • <Performance Counter category name> is the name of the performance counter category.

    • <Performance Counter> is the name of the performance counter.

    See Performance counters (FAST Search Server 2010 for SharePoint) for reference information for performance counter categories and individual counter names.

For example, type the following command to read the performance counter for the number of queries per second from FAST Search Server 2010 for SharePoint QRServer instances:

Copy Code
Get-Counter -Counter "\FAST Search QRServer(*)\# of Queries/sec" -SampleInterval 5 -MaxSamples 1


Read FAST Search Server 2010 for SharePoint logs

This section describes the various log outputs supported by FAST Search Server 2010 for SharePoint.

In this section:

  • Windows event log (FAST Search Server 2010 for SharePoint)

  • SharePoint Unified Logging Service (ULS) (FAST Search Server 2010 for SharePoint)

  • FAST Search Server 2010 for SharePoint specific logs


Windows event log (FAST Search Server 2010 for SharePoint)

The local log server outputs log messages from local processes to the FAST Search Windows event log folder. The main log server running on the FAST Search Server 2010 for SharePoint administration server outputs aggregated logs from all nodes to the FAST Search Farm event log folder. The event log messages from the farm are prefixed with the host name of the log message source.

The built-in Windows tool Event Viewer displays these event log folders within the Application and Services Logs node. To quickly get an overview of a multi-node search installation, log in to the administration server and inspect the log messages in the FAST Search Farm folder, which will contain log messages from all search servers.

The Windows event log is accessible using System Center Operations Manager 2007, scripting, .NET, Windows PowerShell, Win32 APIs, third party tools, etc.

Log level conversion table:

FAST Search Server 2010 for SharePoint Log Level | Windows Event Log Level
CRITICAL | Error
ERROR | Error
WARNING | Warning
INFO | Information
VERBOSE (not enabled by default) | Information
DEBUG (not enabled by default) | Information


SharePoint Unified Logging Service (ULS) (FAST Search Server 2010 for SharePoint)

FAST Search Server 2010 for SharePoint uses the ULSCommonCore component to interface with the SharePoint Unified Logging Service (ULS). The local log server emits ULS trace logs on all hosts, using FASTSearch as the Area value.

The default log file output path is: %ProgramFiles%\Common Files\Microsoft Shared\ULS\14\Logs.

Log level conversion table:

FAST Search Server 2010 for SharePoint Log Level | ULS Log Level
CRITICAL | Unexpected
ERROR | Monitorable
WARNING | High
INFO | Medium
VERBOSE | Verbose
DEBUG | VerboseEx


FAST Search Server 2010 for SharePoint specific logs
See Also

On the FAST Search Server 2010 for SharePoint administration server, aggregated logs from all servers are written as text files in the <FASTSearchFolder>\var\log\syslog folder. The logs use UTF-8 character encoding, including an UTF-8 Byte Order Mark (BOM) at the start of each log file.

The file "all.log" contains all log messages from all nodes and all modules.

The file "lasterror.log" contains the last log message having a severity of either ERROR or CRITICAL. Only one log messages will be present in this file.

The "archive" folder is used to store old log data when the current log file in the syslog folder reaches a certain size. The default archive size threshold is 500 000 bytes. Use the configuration file etc\LogServerConfig.xml to reconfigure this setting. NTFS folder compression is automatically enabled for this folder to save disk space.

See Also

Concepts

logserverconfig.xml reference


Monitor crawling in FAST Search Server 2010 for SharePoint

This section explains how to monitor crawling in Microsoft FAST Search Server 2010 for SharePoint.

In this section:

  • Monitoring the integrated indexing connectors (FAST Search Server 2010 for SharePoint)

  • Monitoring the FAST Search Lotus Notes and database connector

  • Monitoring the FAST Search Web crawler


Monitoring the integrated indexing connectors (FAST Search Server 2010 for SharePoint)

Published: June 17, 2010

To monitor crawls and queries that are made through the FAST Search Content Search Service Application and the FAST Search Query Search Service Application, you can use reports and logs that are provided by SharePoint Server 2010.

In this section:

View search administration reports (FAST Search Server 2010 for SharePoint)

Use search administration reports (FAST Search Server 2010 for SharePoint)

Best practices for using crawl logs (FAST Search Server 2010 for SharePoint)


View search administration reports (FAST Search Server 2010 for SharePoint)

Published: June 17, 2010

SharePoint Server 2010 administrative reports are pre-built reports that use usage data to analyze various aspects of search, such as query and crawl performance, for crawls made through the FAST Search Content SSA.

Note:

You cannot use Windows PowerShell to view administrative report data.

View administrative reports by using Central Administration

You can view administrative reports by using Central Administration.

To view administrative reports by using Central Administration

  1. Log on as a member of the Farm Administrators group.

  2. In Central Administration, on the Home page, click Monitoring.

  3. On the Monitoring page, in the Reporting section, click View administrative reports.

  4. On the Administrative Report Library page, click the name of the report folder.

  5. On the report folder page, click the name of the report to view the report.

  6. To view different results, change the filter settings and click Apply Settings.

    Note:

    The report might have settings that you can use to narrow the results to a specific scope or time range.


Use search administration reports (FAST Search Server 2010 for SharePoint)

Published: June 17, 2010

Search administration reports help you to determine the health of Microsoft FAST Search Server 2010 for SharePoint search service applications on a SharePoint farm. There are three types of search administration reports:

  • Basic search administration reports. Enabled by default, these reports show high-level monitoring data aggregated from all components for the selected search service application.

  • Advanced search administration reports. Enabled by default, these reports show more in-depth monitoring data aggregated from all components for the selected search service application.

  • Verbose search administration reports. After enabling verbose query monitoring for search, this trend report uses per-query data to derive query latency percentiles.

Note:

To obtain search administration reports, you must have a Microsoft SharePoint Server 2010, Microsoft Search Server 2010, or Microsoft FAST Search Server 2010 for SharePoint environment that contains crawled data and a Search Center. You also have to ensure that the State Service has been enabled through the Farm Configuration Wizard.

Note:

The search administration reports only provide information relating to crawls run with the SharePoint Server 2010 crawler. For more information on how to monitor crawls with the FAST Search Server 2010 for SharePoint specific indexing connectors, see Monitoring the FAST Search Lotus Notes and database connector and Monitoring the FAST Search Web crawler.

The following table shows the four basic search administration reports and the metrics they provide. For each report, you can filter the results by FAST Search search service application and time. You can also filter the Crawl Rate per Content Source by content source.

Report Description Metrics Reported Filter On

Crawl Rate per Content Source

Provides a view of recent crawl activity, aggregated by content source. Anchor crawl appears as a separate (virtual) content source. You can filter the report by search service application, time, and content source.

Average items crawled per minute, separated by content source. Accompanying table shows crawl start times, duration, and crawl rate.

FAST Search Content SSA

Crawl Rate per Type

Provides a view of recent crawl activity, sorted by items and actions for a given URL. These items and actions include modified items, deleted items, retries, errors, and others. You can filter the report by search service application and time.

Average items per minute, separated by type of crawl transaction:

  • Modified: Shows items that were modified and re-crawled.

  • Not modified: Shows items that were not modified and not crawled.

  • Security only: shows items for which the security attributes were changed.

  • Deleted: Shows items that were deleted from the content source and must be deleted from the index.

  • Not indexed: Shows items that were crawled but not indexed.

  • Anchor text: Shows anchor text items crawled.

  • Retry: Shows crawl retries.

  • Error: Shows crawl errors.

Accompanying table shows totals for each type of crawl task.

FAST Search Content SSA

Overall Query Latency

Provides a view of recent query activity, showing latency from the major segments of the query pipeline and query averages per minute. You can filter the report by search service application and time.

  • Average query count per minute (along the right side of the graph), represented by a light purple line. Usually fluctuates throughout the day.

  • Average overall query latency per minute (on the left side of the graph), represented by the height of the stacked bars for the major segments of the query pipeline:

    • Server Rendering: Shows time spent for the various web parts in the Web front-end.

    • Object Model: Shows the time spent for communication between the Web front-end and the search back-end.

    • Back-end: Shows the time spent querying all index and property databases, removing duplicates, and returning the results to the Object Model.

FAST Search Query SSA

SharePoint Back-end Query Latency

Provides a view of recent query activity, showing latency details from the index and property database portion of the query pipeline and query averages per minute. You can filter the report by search service application and time. This report is only applicable to queries that go to the Microsoft SharePoint Server 2010 or Microsoft Search Server 2010 back-end.

Average back-end query count per minute (on the right side of the graph), represented by a light purple line. Usually fluctuates throughout the day.

Average back-end query latency per minute (on the left side of the graph) for the following back-end tasks:

  • Results Population: Average time spent creating the in memory table to be passed back to the Object Model.

  • High Confidence Results: Average time spent to retrieve high confidence results for queries.

  • Multiple Results Retrieval: Average time spent retrieving document metadata, such as title or author, to appear in the query results.

  • Best Bets: Average time spent determining whether there are best bets available for the query terms.

  • Final Sort: Average times spent sorting results.

  • Security Trimming: Average time spent removing items the user doesn't have access to. You can increase the security cache to decrease the values.

  • Duplicate Removal: Average time spent removing duplicates.

  • Property Store Query: Average time spent querying the property database for property-based queries.

  • Full-text Query: Average time spent querying the full-text index for results. Sustained high values for your environment mean you should probably partition the index.

FAST Search Query SSA

The following table shows the three advanced search administration reports and the metrics they provide.

Report Description Metrics Reported Filter On

Crawl Processing per Activity

Provides a view of where crawl processing occurs in the pipeline. The timings per component are grouped by activity, such as filtering or word breaking. You can filter the report by FAST Search Content search service application and time.

Crawl activity in seconds, separated by major task:

  • Waiting: Time spent waiting.

  • Standard Properties: Time spent retrieving standard properties.

  • Protocol Handlers: Time spent running protocol handlers.

  • Initializing Filter: Time spent initializing filters.

  • Filtering: Time spent filtering.

  • Chunk Processing: Time spent on processing chunks.

  • Initializing Document Filtering: Time spent on initializing document filtering.

  • Completing Document Filtering: Time spent on completing document filtering.

FAST Search Content SSA

Crawl Processing per Component

Provides a view of where crawl processing occurs in the pipeline, per minute. The timings are grouped by component, such as File Protocol Handler or HTTP Protocol Handler.

Crawl activity in seconds, separated by component:

  • Waiting: Time spent on waiting.

  • BCS Protocol Handler: Crawl processing time for the Business Connectivity Services (BCS) protocol handler.

  • File Protocol Handler: Crawl processing time for the file protocol handler.

  • HTTP Protocol Handler: Crawl processing time for the HTTP protocol handler.

  • Lotus Protocol Handler: Crawl processing time for the Lotus protocol handler.

    Warning:

    This does not apply to the FAST Search Lotus Notes connector.

  • SPS Protocol Handler: Crawl processing time for the Microsoft Office SharePoint Portal Server 2003 protocol handler.

  • STS3 Protocol Handler: Crawl processing time for the Microsoft Office SharePoint Server 2007 protocol handler.

  • STS4 Protocol Handler: Crawl processing time for the SharePoint Server 2010 protocol handler.

  • Other Protocol Handler: Crawl processing time for other protocol handlers.

  • Initializing Filter: Crawl processing time for the filter initializer.

  • Filtering: Crawl processing time for the filtering component.

  • Gatherer Plug-in: Crawl processing time for the gatherer plug-in.

  • Other: Crawl processing time for other components.

FAST Search Content SSA

Crawl Queue

Provides a view of the state of the crawl queue, displaying incoming links to process and outgoing transactions queued.

  • Links to Process: Incoming links to process.

  • Transactions Queued: Outgoing transactions queued.

FAST Search Content SSA

The following table shows the verbose search administration report and the metrics it provides.

Report Description Metrics Reported Filter On

Query Latency Trend

This report provides a view of server-side query latency breakdown by percentile, in addition to the current crawl rate.

  • Query latency (in milliseconds), separated by percentile

FAST Search Query SSA

Note:

To enable the Query Latency Trend report, you must run the following Windows PowerShell cmdlets:

  • $app = Get-SPEnterpriseSearchServiceApplication "<application name>"

  • $app = Set-SPEnterpriseSearchServiceApplication -VerboseQueryMonitoring "True"

  • $app = Get-SPEnterpriseSearchServiceApplication "<application name>"

  • $app.Update()

In this article:

  • Filter data for search administration reports

Filter data for search administration reports

To view administrative reports, see View search administration reports (FAST Search Server 2010 for SharePoint). You can filter data for search administration reports by using Central Administration. By default, search administration reports show data for all search applications over the last 12 hours. You can filter report data in the following ways:

  • Restrict a report to one or more FAST Search search service applications

  • Restrict a report to a specific time period

To restrict a report to one or more FAST Search search service applications

  1. Verify that you have the following administrative credentials:

    • To view reports in the Search administration reports folder, you must be a member of the Farm Administrators SharePoint group on the computer running the SharePoint Central Administration Web site.

  2. On the report page, click the filter icon next to the Application box.

  3. In the Select Filter Value(s) dialog box, select the FAST Search search service applications that you want, and then click OK.

  4. Click the Apply Filters button to update the report.

To restrict a report to a specific time period

  1. Verify that you have the following administrative credentials:

    • To view reports in the Search administration reports folder, you must be a member of the Farm Administrators SharePoint group on the computer running the SharePoint Central Administration Web site.

  2. On the report page, click the Start Date and End Date calendars and times to specify the time period that you want.

  3. Click the Apply Filters button to update the report.


Best practices for using crawl logs (FAST Search Server 2010 for SharePoint)

Published: June 17, 2010

The crawl log tracks information about the status of content that was crawled through the FAST Search Content Search Service Application with the indexing connectors integrated in SharePoint Server 2010.

Note:

The crawl log does not contain information about the FAST Search specific indexing connectors (the FAST Search Web crawler, the FAST Search Lotus Notes connector and/or the FAST Search database connector).

This log lets you determine whether crawled content was successfully added to the index, whether it was excluded because of a crawl rule, or whether indexing failed because of an error. The crawl log also contains more information about crawled content, including the time of the last successful crawl, the content sources, and whether any crawl rules were applied. You can use the crawl log to diagnose problems with the search experience.

In this article:

  • To view the crawl log

  • Crawl log views

  • Crawl log timer job

  • Troubleshoot common problems

To view the crawl log

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content Search Service application.

  2. In Central Administration, in the Quick Launch, click Application Management.

  3. On the Application Management page, under Service Applications, click Manage service applications.

  4. On the Service Applications page, in the list of service applications, click your FAST Search Content Search Service Application.

  5. On the Search Administration page, in the Quick Launch, under Crawling, click Crawl Log.

  6. On the Crawl Log - Content Source page, click the view that you want.

Crawl log views

The following table shows the different views that you can select for viewing the status of crawled content.

View Description

Content Source

Summarizes items crawled per content source. Shows successes, warnings, errors, top-level errors, and deletes. The data in this view represent the current status of items that are already present in the index per content source. The Object Model provides the data for this view.

Host Name

Summarizes items crawled per host. Shows successes, warnings, errors, deletes, top-level errors, and total. The data in this view represent the current status of items that are already present in the index per host. If your environment has multiple crawl databases, the data is shown per crawl database. The Search Administration database provides the data for this view. You can filter the results by typing a URL in the Find URLs that begin with the following hostname/path: box.

URL

Lets you search the crawl logs by content source or URL or host name and view details of all items that are present in the index. The MSSCrawlURLReport table in the crawl database provides the data for this view. You can filter the results by setting the Status, Message:, Start Time, and End Time fields.

Crawl History

Summarizes crawl transactions that were completed during a crawl. There can be multiple crawl transactions per item in a single crawl, so the number of transactions can be larger than the total number of items. This view shows data for three kinds of crawls:

  • Full. Crawls all items in a content source.

  • Incremental. Crawls items that have been changed since the last full or incremental crawl. This kind of crawl only runs if it is scheduled.

  • Delete. If start addresses are removed from a content source, a delete crawl removes items associated with the deleted start address from the index before a full or incremental crawl runs. This kind of crawl cannot be scheduled.

The Search Administration database provides the data for this view. You can filter the results by content source.

Error Message

Provides aggregates of errors per content source or host name. The MSSCrawlURLReport table in the crawl database provides the data for this view. You can filter by content source or host.

Note:

The filter drop-down box only shows content sources that contain errors. If there is an error against an item that does not appear in the index, the error does not appear in this view.

The Content Source and Host Name views show data in the following columns:

  • Successes. Items that were successfully crawled and searchable.

  • Warnings. Items that might not have been successfully crawled and might not be searchable.

  • Errors. Items that were not successfully crawled and might not be searchable.

  • Deletes. Items that were removed from the index and are no longer searchable.

  • Top Level Errors. Errors in top-level documents, including start addresses, virtual servers, and content databases. Every top-level error is counted as an error, but not all errors are counted as top-level errors. Because the Errors column includes the count from the Top Level Errors column, top-level-errors are not counted again in the Host Name view.

The Crawl history view shows the following additional columns:

  • Crawl started. Shows the date and the time when the crawl began.

  • Crawl completed. Shows the date and the time when the crawl finished.

  • Crawl duration. Shows how long it took to complete the crawl.

  • Type. Shows what type of crawl was completed: full, incremental or delete.

  • Not Modified. Items that were modified between crawls.

  • Security Update. Items whose security settings were crawled because they were modified.

Crawl log timer job

By default, the data for each view in the crawl log is refreshed every five minutes by the timer job Crawl Log Report for Search Application <FAST Search Content Search Service Application name>. You can change the refresh rate for this timer job, but in general, this setting should remain as is.

Tip:

If you think the crawl log is not showing fresh data, make sure that the timer job has not been paused and has recently run.

To check the status of the crawl log timer job

  1. Verify that the user account that is performing this procedure is a member of the Farm Administrators SharePoint group.

  2. In Central Administration, in the Monitoring section, click Check job status.

  3. On the Timer Job Status page, click Job History.

  4. On the Job History page, find Crawl Log Report for Search Application <FAST Search Content Search Service Application name> for the Search service application that you want and review the status.

To change the refresh rate for the crawl log timer job

  1. Verify that the user account that is performing this procedure is a member of the Farm Administrators SharePoint group.

  2. In Central Administration, in the Monitoring section, click Check job status.

  3. On the Timer Job Status page, click Job History.

  4. On the Job History page, click Crawl Log Report for Search Application <FAST Search Content Search Service Application name> for the FAST Search Content Search Service Application that you want.

  5. On the Edit Timer Job page, in the Recurring Schedule section, change the timer job schedule to the interval that you want.

  6. Click OK.

Troubleshoot common problems

This section provides information about common crawl log errors, crawler behavior, and actions to take to maintain a healthy crawling environment.

When an item is deleted from the index

When a crawler cannot find an item that exists in the index because the URL is obsolete or it cannot be accessed due to a network outage, the SharePoint Server 2010 crawler reports an error for that item in that crawl. If this continues during the next three crawls, the item is deleted from the index. For file-share content sources, items are immediately deleted from the index when they are deleted from the file share.

"Object could not be found" error for a file share

This error can result from a crawled file-share content source that contains a valid host name but an invalid file name. For example, with a host name and file name of \\ValidHost\files\file1, \\ValidHost exists, but the file file1 does not. In this case, the crawler reports the error "Object could not be found" and deletes the item from the index. The Crawl History view shows:

  • Error: 1

  • Deletes: 1

  • Top Level Errors: 1 (\\ValidHost\files\file1 shows as a top-level error because it is a start address)

The Content Source view shows:

  • Errors: 0

  • Deletes: 0

  • Top Level Errors: 0

The Content Source view will show all zeros because it only shows the status of items that are in the index, and this start address was not entered into the index. However, the Crawl History view shows all crawl transactions, whether or not they are entered into the index.

"Network path for item could not be resolved" error for a file share

This error can result from a crawled file-share content source that contains an invalid host name and an invalid file name. For example, with a host name and file name of \\InvalidHost\files\file1, both \\InvalidHost and the file file1 do not exist. In this case, the crawler reports the error "Network path for item could not be resolved" and does not delete the item from the index. The Crawl History view shows:

  • Errors: 1

  • Deletes: 0

  • Top Level Errors: 1 (\\ValidHost\files\file1 shows as a top-level error because it is a start address)

The Content Source view shows:

  • Error: 0

  • Deletes: 0

  • Top Level Errors: 0

The item is not deleted from the index, because the crawler cannot determine if the item really does not exist or if there is a network outage that prevents the item from being accessed.

Obsolete start addresses

The crawl log reports top-level errors for top-level documents, or start addresses. To ensure healthy content sources, you should take the following actions:

  • Always investigate non-zero top-level errors.

  • Always investigate top-level errors that appear consistently in the crawl log.

  • Otherwise, we recommend that you remove obsolete start addresses every two weeks after contacting the owner of the site.

To troubleshoot and delete obsolete start addresses

  1. Verify that the user account that is performing this procedure is an administrator for the FAST Search Content Search Service Application.

  2. When you have determined that a start address might be obsolete, first determine whether it exists or not by pinging the site. If you receive a response, determine which of the following issues caused the problem:

    • If you can access the URL from a browser, the crawler could not crawl the start address because there were problems with the network connection.

    • If the URL is redirected from a browser, you should change the start address to be the same as the new address.

    • If the URL receives an error in a browser, try again at another time. If it still receives an error after multiple tries, contact the site owner to ensure that the site is available.

  3. If you do not receive a response from pinging the site, the site does not exist and should be deleted. Confirm this with the site owner before you delete the site.

Access Denied

When the crawl log continually reports an "Access Denied" error for a start address, the content access account might not have Read permissions to crawl the site. If you are able to view the URL with an administrative account, there might be a problem with how the permissions were updated. In this case, you should contact the site owner to request permissions. For information about how to set permissions for a crawler, see Manage crawl rules (FAST Search Server 2010 for SharePoint).

Numbers set to zero in Content Source view during host distribution

During a host distribution, the numbers in all columns in Content Source view are set to zero. This happens because the numbers in Content Source view are sourced directly from the crawl database tables. During a host distribution, the data from these tables are being moved, so the values remain at zero during the duration of the host distribution.

After the host distribution is complete, run an incremental crawl of the content sources in order to restore the original numbers.

Showing file-share deletes in Content Source view

When documents are deleted from a file-share content source that was successfully crawled, they are immediately deleted from the index during the next full or incremental crawl. These items will show as errors in the Content Source view of the crawl log, but will show as deletes in other views.

Stopping or restarting the SharePoint Server Search service causes crawl log transaction discrepancy

The SharePoint Server Search service (OSearch14) might be reset or restarted due to administrative operations or server functions. When this occurs, a discrepancy in the crawl history view of the crawl log can occur. You may notice a difference between the number of transactions reported per crawl and the actual number of transactions performed per crawl. This can occur because the OSearch14 service stores active transactions in memory and writes these transactions after they are completed. If the OSearch14 service is stopped, reset, or restarted before the in-memory transactions have been written to the crawl log database, the number of transactions per crawl will be shown incorrectly.


Monitoring the FAST Search Lotus Notes and database connector

You can view log messages for the FAST Search connectors for Lotus Notes and database content by:

  • Consulting the local log file

  • Viewing the real time log from the console

  • Looking at the Windows Event Viewer

  • Monitoring the connectors through the System Center Operations Manager (SCOM)

  • Monitoring indexed items

Consulting the local log file

The FAST Search database connector and the FAST Search Lotus Notes connector both log monitoring information to a local log file. You can inspect this file for log messages of the type "FATAL", "ERROR" and (possibly) "WARNING". The log file can be accessed by navigating to the log file location as indicated in the following table.

Connector | Log file location
FAST Search database connector | <FASTSearchFolder>\var\jdbcconnector\<collection>\log <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.
FAST Search Lotus Notes content connector | <FASTSearchFolder>\var\lotusnotesconnector\<collection>\log
FAST Search Lotus Notes user directory connector | <FASTSearchFolder>\var\lotusnotesconnector\security\log

Viewing the real time log from the console

If you are running a connector from the console, you can monitor the logs directly from the console the connector is running in.

Looking at the Windows Event Viewer

In addition to logging to a local file and the console, the same log messages are sent to the Windows Event Log.

To open the Windows Event Viewer:

  1. Start Server Manager.

  2. Navigate to the node Server Manager.

  3. Go to Diagnostics, select Event Viewer, select Applications and Services Logs and open FAST Search.

The FAST Search database connector log entries are identified with Source="FAST Search JDBC Connector" and the FAST Search Lotus Notes connector log entries with Source="FAST Search Lotus Notes Connector".

Monitoring the connectors through the System Center Operations Manager (SCOM)

If you are using System Center Operations Manager (SCOM) and have installed the FAST Search management pack, the log entries from the FAST Search database and FAST Search Lotus Notes connectors are also available there. The health of the connectors is registered in SCOM and it is included in the overall health of the FAST Search Server 2010 for SharePoint system.

Note that the FAST Search database connector and the FAST Search Lotus Notes connectors tend to log a bit too aggressively for the liking of the event log. You may see a "Log Flooding detected" event generated by the event log for one or both connectors. This will result in the health of the connector turning from "Healthy" to "Warning". This can safely be ignored. Because of this, you should ignore a "Warning" health for these connectors and only take action if the health status becomes "Critical".

A common way to monitor the connectors in a production setting is to use SCOM to monitor the health of the connectors. If the health turns to Critical, consult the local log file to investigate the cause.

Monitoring indexed items

Items that are indexed by the FAST Search database and FAST Search Lotus Notes connector are logged in the status tracker. You can access and monitor the status tracker with the SQL Server 2008 Report Builder.

  1. Open the SQL Server 2008 Report Builder.

  2. From the Report Builder, open the file <FASTSearchFolder>\bin\ConnectorStatus.rdl. If prompted for a Report Server string, click Cancel.

  3. Right-click on the ConnectorStatus DB symbol and select Data Source Properties.

  4. Mark the Use a connection embedded in my report radio button and click Build….

  5. Select the FAST Search Admin database and click Run in the Report Builder to generate a report.

The report will provide an overview of indexed items. You will be able to sort, filter and search on document ID and index status.

Note:

After clicking the Run button, it may take some time for the report to be generated.


Monitoring the FAST Search Web crawler

You can monitor the FAST Search Web crawler using SCOM, the crawleradmin tool, the beadmin tool or by analyzing logs.

SCOM and the command line tools give overall status and statistics information on the FAST Search Web crawler and its crawl collections. While the FAST Search Web crawler logs and the command line tools provide information on URL granularity and provide debug information.

Essential information you should monitor is:

  • The overall Web item request rate;

  • The number of Web items stored;

  • The number of Web items crawled per cycle;

  • The overall health of the FAST Search Web crawler and the Browser Engine.

Important:

It is important to monitor a crawl collection in its first cycle, to ensure that the crawl collection is configured and behaving as intended.

In this section:

  • Monitoring the FAST Search Web crawler with SCOM

  • Monitoring the FAST Search Web crawler with command line tools

  • Monitoring the FAST Search Web crawler log files


Monitoring the FAST Search Web crawler with SCOM

The FAST Search Management pack in SCOM provides views and reports on the FAST Search Web crawler and the Browser Engine. For more information about obtaining and installing the FAST Search Management pack in SCOM, refer to Monitor FAST Search Server 2010 for SharePoint with SCOM.

The Overview view provides a general status view of the FAST Search Web crawler and of the Browser Engine. Here you can see if the components are healthy. This view also provides a view of all important FAST Search Server 2010 for SharePoint events, including events from the FAST Search Web crawler and the Browser Engine.

The Performance folder has two individual views for the Browser Engine (BrowserEngine Performance), and the FAST Search Web crawler (CrawlerCollection Performance). Each view has a set of rules each with an explanation of the rule. The statistics of the CrawlerCollection Performance view is per crawl collection, the Target column specifies the name of the crawl collection. Additional reporting and status information can be retrieved from the Crawler Collection state within the State View folder.

The Windows Events folder contains two views; BrowserEngine Events and Crawler Events. These views provide useful information, warning and error events produced by the Browser Engine and the FAST Search Web crawler.

Note:

All statistics are reset each time the Browser Engine is restarted.


Monitoring the FAST Search Web crawler with command line tools

The FAST Search Web crawler has two monitoring command line tools, the crawleradmin tool (crawleradmin.exe) and the beadmin tool (beadmin.exe). They are respectively used for monitoring the FAST Search Web crawler and the Browser Engine.

crawleradmin.exe command line tool

The crawleradmin tool has multiple statistics and status commands. Refer to crawleradmin.exe reference for a description of the output of the statistics commands. The detail of information shown depends on the command that is used.

beadmin.exe command line tool

The beadmin tool has two monitoring commands status and statistics. The status command returns information of all active threads in the Browser Engine. The statistics command gives overall statistics information.

In addition, the statistics command outputs information on the Browser Engine extractors. An extractor is used to extract links and construct HTML from the processed Web item. For each extractor the average, maximum and minimum processing time is displayed as well as the number of total links the extractor has found.

For detailed information, refer to beadmin.exe reference.


Monitoring the FAST Search Web crawler log files

The main single server FAST Search Web crawler log is located at <FAST Search Server 2010 for SharePoint installation folder>\var\log\crawler\node\crawler.log. This log contains overall status information and exceptional conditions, for example when the FAST Search Web crawler starts/stops, when it starts/stops storage compaction and general feeding information. When debugging an issue, look in this log first.

The Browser Engine log is located at <FAST Search Server 2010 for SharePoint installation folder>\var\log\browserengine\BrowserEngine.log. This log contains overall status messages and URL processing information.

Additional FAST Search Web crawler logs, which provide per crawl collection information, are located in the <FAST Search Server 2010 for SharePoint installation folder>\var\log\crawler\node\ folder. Each folder represents a particular log type, and contains a set of folders that represent a particular crawl collection. The number of log folders depends on what the crawl collection is configured to report. A collection can contain these log folders:

Log type | Description
dns | Contains log files from DNS resolutions.
dsfeed | Contains log files of the status of each URL submitted to Web item processing. Deletes are also logged. This log is useful to find out why a Web item was not indexed even though it was crawled.
fetch | Contains log files for every URL fetched by the Web crawler.
header | Contains logs of HTTP request/response exchanges, separated into directories by Web site name. The header log is disabled by default. This log can be useful for analyzing a forms login session.
PP | Contains log files from Post Process. Post Process performs duplicate detection of downloaded Web items, and process content. The Post Process log contains the URLs and referrer URL to every unique Web item together with their size, MIME type and URLs to any duplicates found.
screened | Contains log files of all URLs processed by the Web crawler and details for any given URL on whether or not it will be crawled. This log is used to find out why a Web item was excluded from a crawl.
site | Contains log files crawler Web site events. The site log contains information on when the Web crawler starts/stops to crawl the Web site, when it refreshes the Web site or when it becomes idle on the Web site.

In a FAST Search Web crawler multi-node deployment, the duplicate server logs are located at <FAST Search Server 2010 for SharePoint installation folder>\var\log\crawler\node\dupserver.log. This log contains the overall status information and exceptional conditions of the duplicate server.

On the multi-node scheduler, FAST Search Web crawler log is located at <FAST Search Server 2010 for SharePoint installation folder>\var\log\crawler\multinode.log. On the node schedulers, the log is called var\log\crawler\node.log.


Manage property extraction (FAST Search Server 2010 for SharePoint)

The property extraction feature identifies key information such as person names, company names, and geographic names/locations in documents. These properties are extracted from documents based on word lists and context rules, and help find out the "who", "what", "when" and "where" of each document.

The properties can be used to improve the search experience, for example by providing search result refinement based on the properties, or by highlighting the properties in the result summary or displaying meta information about the property in the result summary. For each property extractor, the administrator can maintain an include list and an exclude list.

Note:

This section describes how to manage property extraction by using Windows PowerShell. For information about managing property extraction by using the graphical user interface, see Manage Property Extraction (Office.com).

In this section:

  • Add or remove items in include and exclude lists by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Display all items in include and exclude lists by using Windows PowerShell (FAST Search Server 2010 for SharePoint)


Add or remove items in include and exclude lists by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

This topic explains how to add or remove items in include and exclude lists in Microsoft FAST Search Server 2010 for SharePoint by using Windows PowerShell.

In this topic:

  • Add an item to the property extraction include list by using Windows PowerShell

  • Add an item to the property extraction exclude list by using Windows PowerShell

  • Add multiple items to the property extraction include list by using Windows PowerShell

  • Add multiple items to the property extraction exclude list by using Windows PowerShell

  • Remove all items from the property extraction include list by using Windows PowerShell

  • Remove all items from the property extraction exclude list by using Windows PowerShell

Add an item to the property extraction include list by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $entityExtractorContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.EntityExtractorContext
$entityExtractors = $entityExtractorContext.TermEntityExtractors
$entityExtractor
foreach ($extractor in $entityExtractors) { if ($extractor.Name -eq "<ExtractorTypeName>") { $entityExtractor = $extractor } }
$entityExtractor.Inclusions.Add("<InclusionTerm>")

    Where:

    • <ExtractorTypeName> is the kind of term property extractor that is to be populated with a new term. Three term property extractor types are available: companies, locations, and personnames.

    • <InclusionTerm> is the term to be included in entity extraction.

See also

  • Microsoft.SharePoint.Search.Extended.Administration Namespace

Add an item to the property extraction exclude list by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $entityExtractorContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.EntityExtractorContext
$entityExtractors = $entityExtractorContext.TermEntityExtractors
$entityExtractor
foreach ($extractor in $entityExtractors) { if ($extractor.Name -eq "<ExtractorTypeName>") { $entityExtractor = $extractor } }
$entityExtractor.Exclusions.Add("<ExclusionTerm>")

    Where:

    • <ExtractorTypeName> is the kind of term property extractor that is to be populated with a new term. Three term property extractor types are available: companies, locations, and personnames.

    • <ExclusionTerm> is the term that should excluded from entity extraction.

See also

  • Microsoft.SharePoint.Search.Extended.Administration Namespace

Add multiple items to the property extraction include list by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $entityExtractorContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.EntityExtractorContext
$entityExtractors = $entityExtractorContext.TermEntityExtractors
$entityExtractor
foreach ($extractor in $entityExtractors) { if ($extractor.Name -eq "<ExtractorTypeName>") { $entityExtractor = $extractor } }
foreach ($entityToAdd in $entitiesToAdd) { $extractor.Inclusions.Add( $entityToAdd ) }

    Where:

    • <ExtractorTypeName> is the kind of term property extractor that is to be populated with a new term. Three term property extractor types are available: companies, locations, and personnames.

See also

  • Microsoft.SharePoint.Search.Extended.Administration Namespace

Remove all items from the property extraction include list by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $entityExtractorContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.EntityExtractorContext
$entityExtractors = $entityExtractorContext.TermEntityExtractors
$entityExtractor
foreach ($extractor in $entityExtractors) { if ($extractor.Name -eq "<ExtractorTypeName>") { $entityExtractor = $extractor } }
$entityExtractor.Inclusions.Clear()

    Where:

    • <ExtractorTypeName> is the kind of term property extractor the items will be removed from. Three term property extractor types are available: companies, locations, and personnames.

See also

  • Microsoft.SharePoint.Search.Extended.Administration Namespace

Add multiple items to the property extraction exclude list by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $entityExtractorContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.EntityExtractorContext
$entityExtractors = $entityExtractorContext.TermEntityExtractors
$entityExtractor
foreach ($extractor in $entityExtractors) { if ($extractor.Name -eq "<ExtractorTypeName>") { $entityExtractor = $extractor } }
foreach ($entityToAdd in $entitiesToAdd) { $extractor.Exclusions.Add( $entityToAdd ) }

    Where:

    • <ExtractorTypeName> is the kind of term property extractor that is to be populated with a new term. Three term property extractor types are available: companies, locations, and personnames.

See also

  • Microsoft.SharePoint.Search.Extended.Administration Namespace

Remove all items from the property extraction exclude list by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $entityExtractorContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.EntityExtractorContext
$entityExtractors = $entityExtractorContext.TermEntityExtractors
$entityExtractor
foreach ($extractor in $entityExtractors) { if ($extractor.Name -eq "<ExtractorTypeName>") { $entityExtractor = $extractor } }
$entityExtractor.Exclusions.Clear()

    Where:

    • <ExtractorTypeName> is the kind of term property extractor the items will be removed from. Three term property extractor types are available: companies, locations, and personnames.

See also

  • Microsoft.SharePoint.Search.Extended.Administration Namespace


Display all items in include and exclude lists by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

This topic explains how to display all items in a property extraction include or exclude list in Microsoft FAST Search Server 2010 for SharePoint by using Windows PowerShell.

In this topic:

  • Display all items in a property extraction include list by using Windows PowerShell

  • Display all items in a property extraction exclude list by using Windows PowerShell

Display all items in a property extraction include list by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $entityExtractorContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.EntityExtractorContext
$entityExtractors = $entityExtractorContext.TermEntityExtractors
foreach ($entityExtractor in $entityExtractors) { $entityExtractor.Inclusions }

See also

  • Microsoft.SharePoint.Search.Extended.Administration Namespace

Display all items in a property extraction exclude list by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $entityExtractorContext = New-Object -TypeName Microsoft.SharePoint.Search.Extended.Administration.EntityExtractorContext
$entityExtractors = $entityExtractorContext.TermEntityExtractors
foreach ($entityExtractor in $entityExtractors) { $entityExtractor.Exclusions }

See also

  • Microsoft.SharePoint.Search.Extended.Administration Namespace


Manage content collections (FAST Search Server 2010 for SharePoint)

A content collection is a group of items crawled by one or more indexing connectors and stored in a search index for inclusion in a FAST Search Server 2010 for SharePoint search system. A content collection contains items from one or more content sources (for example SharePoint, Lotus Notes, etc.).

The default content collection is called sp. This default collection should be the only content collection to feed to when you use the FAST Search Content Search Service Application.

In order to feed content to the FAST Search Server 2010 for SharePoint content collection using the indexing connectors integrated in SharePoint Server 2010, you must configure the FAST Search Content SSA, configure SSL enabled communication and add at least one content source to your FAST Search Content SSA. See Create and set up the Content Search Service Application (FAST Search Server 2010 for SharePoint) and Manage crawling with the FAST Search Content SSA (FAST Search Server 2010 for SharePoint) for more information.

The FAST Search specific indexing connectors (the FAST Search Web crawler, the FAST Search Lotus Notes connector and the FAST Search database connector) do not depend on the FAST Search Content SSA. These indexing connectors feed their content directly into a content collection. This could be the default content collection named sp, but may also be a separate content collection.

In this section:

  • Display all content collections by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Display the number of documents in a content collection by using Windows PowerShell (FAST Search Server 2010 for SharePoint)


Display all content collections by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

This topic explains how to display all content collections in Microsoft FAST Search Server 2010 for SharePoint by using Windows PowerShell.

In this section:

  • Display all content collections by using Windows PowerShell

Display all content collections by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    Get-FASTSearchContentCollection

See also

  • Get-FASTSearchContentCollection


Display the number of documents in a content collection by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

This topic explains how to display the number of documents in a content collection in Microsoft FAST Search Server 2010 for SharePoint by using Windows PowerShell.

In this section:

  • Display the number of documents in a content collection by using Windows PowerShell

Display the number of documents in a content collection by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $collection = Get-FASTSearchContentCollection -Name "<CollectionName>"
$collection.DocumentCount

    Where:

    • <CollectionName> is the name of the collection.

See also

  • Get-FASTSearchContentCollection


Manage index schema (FAST Search Server 2010 for SharePoint)
See Also

The FAST Search Server 2010 for SharePoint index schema controls which managed properties are indexed, searchable, and available for viewing. The index schema Windows PowerShell cmdlets work with metadata extracted from content sources. Metadata can be structured content (such as the title or the author from a Word document), or unstructured content based on the content of an item (such as a detected language or extracted keywords).

When managing metadata, you work with crawled properties and managed properties:

  • Crawled properties are metadata extracted from content sources when you run a crawler or connector.

  • Managed properties are a subset of crawled properties that are searchable. To make metadata searchable, you map one or more crawled properties to a managed property. You can also create a full-text index and map the content of a managed property into the full-text index.

Query refinement uses aggregated statistical data, such as the distribution of managed property values in query results, to filter results. It is typically applied to item metadata - such as creation date, author, and person names - appearing in the item. For example, users can refine queries to items created in a certain time period, or to items referencing a specific person. For numeric properties, refiners provide drill down options in the form of refinement bins, where each bin represents a set of values in the statistical distribution.

FAST Search Server 2010 for SharePoint supports two kinds of query refiners:

  • Deep refiners: Query refinement is based on the aggregation of managed property statistics for all of the results of a search query.

  • Shallow refiners: Query refinement is based on the aggregation of managed property statistics for the top 50 results for a search.

Refinement data is created during results processing.

Note:

This section describes how to manage properties by using Windows PowerShell. For information about how to work with managed and crawled properties by using the graphical user interface, see Property Management (Office.com).

For details about the index schema object model and important index schema properties, see Index Schema (MSDN)

In this section:

  • Manage crawled properties by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Manage managed properties by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Manage full-text indexes by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

See Also

Concepts

The index schema and its features (FAST Search Server 2010 for SharePoint)


Manage crawled properties by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

This topic explains how to work with crawled properties in Microsoft FAST Search Server 2010 for SharePoint by using Windows PowerShell.

In this section:

  • Configure category to discover new crawled properties by using Windows PowerShell

  • Re-order crawled properties mappings by using Windows PowerShell

  • Look up a crawled property if you know the full name of the property by using Windows PowerShell

  • Look up a crawled property if you know part of the name of the property by using Windows PowerShell

  • Look up all crawled properties in a category by using Windows PowerShell

  • Map a crawled property to a managed property by using Windows PowerShell

Configure category to discover new crawled properties by using Windows PowerShell

Crawled properties are automatically extracted from crawled content and grouped by category based on the property-set of each crawled property. Each category can be configured individually to discover new crawled properties, or not.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    Set-FASTSearchMetadataCategory -Name "<CategoryName>" -DiscoverNewProperties "<Flag>"

    Where:

    • <CategoryName> is the name of the category to update.

    • <Flag> is a Boolean indicating whether or not new crawled properties should be discovered for this category. Use $true if crawled properties should be discovered, $false if not.

If a crawled property is discovered and it has a property-set that is not mapped to an existing category, a new category will be created for the property-set as well as the new crawled property. The new category will be named Category # (where # is a category number).

See also

  • Set-FASTSearchMetadataCategory

Re-order crawled properties mappings by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $mp = Get-FASTSearchMetadataManagedProperty -Name "<ManagedPropertyName>"
#Fetch all the mappings for the managed property:
$mappings = $mp.GetCrawledPropertyMappings()
#Get the crawled properties to be stored in position 1 and position 2:
$cp1 = Get-FASTSearchMetadataCrawledProperty -Name "<CrawledPropertyName1>"
$cp2 = Get-FASTSearchMetadataCrawledProperty -Name "<CrawledPropertyName2>"
#Remove the properties from their current position in the list:
$mappings.Remove($cp1)
$mappings.Remove($cp2)
#Insert the crawled properties in their new positions:
$mappings.Insert(0, $cp1)
$mappings.Insert(1, $cp2)
#Call SetCrawledPropertyMappings(...) to make the update permanent:
$mp.SetCrawledPropertyMappings($mappings)

    Where:

    • <ManagedPropertyName> is the name of the managed property.

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

  • Get-FASTSearchMetadataManagedProperty

Look up a crawled property if you know the full name of the property by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $cp = Get-FASTSearchMetadataCrawledProperty -Name "<Name>"

    Where:

    • <Name> is the full name of the crawled property.

See also

  • Get-FASTSearchMetadataCrawledProperty

Look up a crawled property if you know part of the name of the property by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $cp = Get-FASTSearchMetadataCrawledProperty -Filter "<Filter>"

    Where:

    • <Filter> is part of name of the crawled property.

See also

  • Get-FASTSearchMetadataCrawledProperty

Look up all crawled properties in a category by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $category = Get-FASTSearchMetadataCategory -Name "<CategoryName>"
$unmappedCPs = $category.GetUnmappedCrawledProperties()
$unmappedCPs[$unmappedCPs.Count-1]

    Where:

    • <CategoryName> is the name of the category.

Note:

Crawled properties are added to the UnmappedCrawledProperties list in the order they are reported. The most recently reported properties will be the last items in the list.

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

  • Get-FASTSearchMetadataManagedProperty

Map a crawled property to a managed property by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $mp = Get-FASTSearchMetadataManagedProperty -Name "<ManagedPropertyName>"
$cp = Get-FASTSearchMetadataCrawledProperty -Name "<CrawledPropertyName>"
New-FASTSearchMetadataCrawledPropertyMapping -ManagedProperty $mp -CrawledProperty $cp

    Where:

    • <ManagedPropertyName> is the name of the managed property.

    • <CrawledPropertyName> is the name of the crawled property.

Note:

If there are multiple crawled properties with the same name, a list of crawled properties will be returned instead of a single crawled property object. In such cases you will have to inspect the list to find the crawled property that you want to map to the managed property.

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

  • Get-FASTSearchMetadataManagedProperty

  • New-FASTSearchMetadataCrawledPropertyMapping


Manage managed properties by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

This article explains how to work with managed properties in Microsoft FAST Search Server 2010 for SharePoint by using Windows PowerShell.

In this article:

  • Display the properties of a managed property by using Windows PowerShell

  • Make a managed property queryable by using the Set-FASTSearchMetadataManagedProperty Windows PowerShell cmdlet

  • Make a managed property queryable by updating the properties on the managed property object

  • Enable merging of crawled property values by using Windows PowerShell

  • Set size properties for a managed property by using Windows PowerShell

Display the properties of a managed property by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    Get-FASTSearchMetadataManagedProperty -Name "<ManagedPropertyName>"

    Where:

    • <ManagedPropertyName> is the name of the managed property you want to update.

Note:

Some of these properties are set when the managed property is created and are not possible to update, such as the managed property Name or Type.

See also

  • Get-FASTSearchMetadataManagedProperty

Make a managed property queryable by using the Set-FASTSearchMetadataManagedProperty Windows PowerShell cmdlet

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    Set-FASTSearchMetadataManagedProperty -Name "<ManagedPropertyName>" -Queryable <QueryableFlag>

    Where:

    • <ManagedPropertyName> is the name of the managed property.

    • <QueryableFlag> is a Boolean indicating whether the managed property should be queryable or not. Set to $true if the managed property should be queryable, if not set to $false.

See also

  • Set-FASTSearchMetadataManagedProperty

Make a managed property queryable by updating the properties on the managed property object

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $mp = Get-FASTSearchMetadataManagedProperty -Name "<ManagedPropertyName>"
$mp.Queryable = <QueryableFlag>
$mp.Update()

    Where:

    • <ManagedPropertyName> is the name of the managed property.

    • <QueryableFlag> is a Boolean indicating whether the managed property should be queryable or not. Set to $true if the managed property should be queryable, if not set to $false.

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

  • Get-FASTSearchMetadataManagedProperty

Enable merging of crawled property values by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    Set-FASTSearchMetadataManagedProperty -Name "<ManagedPropertyName>" -MergeCrawledProperties <Flag>

    Where:

    • <ManagedPropertyName> is the name of the managed property.

    • <Flag> is a Boolean indicating whether mapped crawled properties should be merged, or not. Use $true to merge, $false if not.

See also

  • Set-FASTSearchMetadataManagedProperty

Set size properties for a managed property by using Windows PowerShell

A managed property has two size-related properties, MaxIndexSize and MaxResultSize. These properties define the maximum number of kilobytes used for indexing the managed property, and the maximum number of kilobytes returned by the managed property as part of the search result. The following example doubles the value for these properties:

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $mp = Get-FASTSearchMetadataManagedProperty -Name Title
Set-FASTSearchMetadataManagedProperty -Name Title -MaxIndexSize ($mp.MaxIndexSize*2) -MaxResultSize ($mp.MaxResultSize*2)

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

  • Get-FASTSearchMetadataManagedProperty

  • Set-FASTSearchMetadataManagedProperty


Manage full-text indexes by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

A full-text index is used to index groups of managed properties, which allows a search query to be run across multiple properties at the same time. You must use a full-text index to achieve ranked results for queries.

In this article:

  • Define a full-text index with a corresponding rank profile by using Windows PowerShell

  • Modify the list of full-text index rank components for a rank profile by using Windows PowerShell

Define a full-text index with a corresponding rank profile by using Windows PowerShell

The following procedure creates a new full-text index, creates a new rank profile, and associates the full-text index to the rank profile by using the FullTextIndexRankCollection interface.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following commands:

    Copy Code
    $NewFTI = New-FASTSearchMetadataFullTextIndex -Name "<NewFullTextIndexName>" -Description "<NewFullTextIndexDescription>"
$NewRP = New-FASTSearchMetadataRankProfile -Name "<NewRankProfileName>"
$RCList = $NewRP.GetFullTextIndexRanks()
$RCList.Create($NewFTI)

    Where:

    • $NewFTI is the full-text index object of the new full-text index.

    • <NewFullTextIndexName> is the name of the new full-text index.

    • <NewFullTextIndexDescription> is a textual description of the new full-text index.

    • $NewRP is the rank profile object of the new rank profile.

    • <NewRankProfileName> is the name of the new rank profile.

    • $RCList is the list of existing full-text index rank components for the new rank component.

Note:

When you create a new full-text index, it will be created with the default values for its properties. If you want to set properties to non-default values, you will have to do that after you have created the full-text index.

Note:

When you create a new rank profile, it will be created with the default values for its properties. If you want to set properties to non-default values, you will have to do that after you have created the rank profile. When you create a new rank profile, you can also provide an existing rank profile as a template specifying the "-Template" parameter when you use the New-FASTSearchMetadataRankProfileWindows PowerShell cmdlet.

Note:

When you create a new rank profile, it has the default full-text index as its full-text index rank component. If you want to associate your rank profile with another full-text index, you must create a new full-text index rank component in your rank profile. It will be added as a second full-text index rank component to the rank profile. Then you delete the default full-text index rank component after you have created the new one.

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

  • Manage rank profiles (FAST Search Server 2010 for SharePoint)

  • Index Schema Object Model Reference

  • New-FASTSearchMetadataRankProfile

  • New-FASTSearchMetadataFullTextIndex

Modify the list of full-text index rank components for a rank profile by using Windows PowerShell

If you want to modify the list of full-text index rank components for a rank profile, you must traverse the list of full-text index rank components and modify the individual full-text index rank component.

The following procedure shows how you can delete the full-text index rank component that is associated with the default full-text index.

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following commands:

    Copy Code
    $NewRP = Get-FASTSearchMetadataRankProfile -Name "<NewRankProfileName>"
$RCList = $NewRP.GetFullTextIndexRanks()
foreach ($FTIRankComponent in $RCList)
{
   if($FTIRankComponent.FullTextIndexReference.Name.Equals("<DefaultFTIName>"))
   {
      $FTIRankComponent.Delete()
   }
}

    Where:

    • <NewRankProfileName> is the name of the new rank profile.

    • $NewRP is the rank profile object of the new rank profile.

    • $RCList is the list of existing full-text index rank components for the rank profile.

    • <DefaultFTIName> is the name of the default full-text index (in a default configuration this is named "content").

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Schema Namespace

  • Index Schema Object Model Reference

  • Get-FASTSearchMetadataRankProfile


Manage search scopes (FAST Search Server 2010 for SharePoint)

Published: July 8, 2010

A search scope defines a subset of information in the search index. Scopes can for example direct a query to particular sites, or to items marked with particular property values.

Users can apply the search scope by selecting a pre-defined scope from a drop-down list in the search application, or they can specify the search scope directly in the Keyword Syntax query string by using the special property filter scope:scopename, where scope is a pre-defined property name and scopename is the name of the defined search scope.

Note:

In FAST Search Server 2010 for SharePoint, users can search two or more scopes by using "AND" only. Linking scopes with "OR" and "AND NOT" is not supported. This is regardless of whether users search by using Keyword Syntax or if they use the Only the scope(s) option in Advanced Search. The search will only return results that are found in all selected scopes.

In FAST Search Server 2010 for SharePoint, a search scope is a pre-defined property filter expression that is specified in FAST Query Language (FQL) syntax. If you create the search scope by using the SharePoint graphical user interface, the FAST scope filter is auto-generated based on the rules created for the scope.

The number of items that are returned by the scope, will not be displayed in the View Scopes list. To see the number of items, you can run a query using the scope on the Search Center site, and then view the number of results in the result list.

If you need to specify more complex rules in the scope than what is available in the scope rule user interface, you can create or edit the scope by using Windows PowerShell. You specify the scope filter as a string according to the FQL query syntax. A scope filter can consist of any FQL syntax. For more information about FQL, see FAST Query Language (FQL) Syntax Reference.

A search scope is a query-time feature, and does not affect the index or index schema.

Note:

Scope matching does not include stemming and synonyms.

In this section:

  • Create and view search scopes (FAST Search Server 2010 for SharePoint)


Create and view search scopes (FAST Search Server 2010 for SharePoint)

In FAST Search Server 2010 for SharePoint a search scope is a pre-defined property filter expression that is specified in FAST Query Language (FQL) syntax. If you create the search scope by using the SharePoint graphical user interface, the FAST scope filter is auto-generated based on the rules created for the scope. If you have to specify more complex rules in the scope than what is available in the scope rule user interface, you can create or edit the scope by using Windows PowerShell. You specify the scope filter as a string according to the FQL query syntax. A scope filter can consist of any FQL syntax.

Note:

In FAST Search Server 2010 for SharePoint, users can search two or more scopes by using "AND" only. Linking scopes with "OR" and "AND NOT" is not supported. This is regardless of whether users search by using Keyword Syntax or if they use the Only the scope(s) option in Advanced Search. The search will only return results that are found in all selected scopes.

The number of items that are returned by the scope, will not be displayed in the View Scopes list. To see the number of items, you can run a query using the scope on the Search Center site, and then view the number of results in the result list.

A search scope is a query-time feature, and does not affect the index or index schema.

Note:

Scope matching does not include stemming and synonyms.

In this section:

  • Create search scopes by using the graphical user interface

  • Create a search scope at the search service application level by using Windows PowerShell

  • Create a site-level search scope by using Windows PowerShell

  • Create a site-level search scope that searches a full-text index by using Windows PowerShell

  • View the FAST Query Language (FQL) filter for a search scope at the search service application level by using Windows PowerShell

  • View the FAST Query Language (FQL) filter for a site level search scope by using Windows PowerShell

  • Make scopes available on the Search Center site

Create search scopes by using the graphical user interface

You can set search scopes both at the Search service application level (farm-level) and at the site administration level. This topic describes how to create a search scope and add a scope rule at the service application level for a FAST Query SSA by using the graphical user interface.

Note:

If you create the scope by using the user interface and then manually change the auto-generated filter by using Windows PowerShell, your changes will not be reflected in the scope rules shown in the user interface.

To create a search scope at the search service application level using the graphical user interface

  1. Verify that the user account that is performing this procedure is a service application administrator for the FAST Query Search service application (FAST Query SSA).

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, in the list of service applications, click FAST Query SSA (the name of the FAST Query SSA).

  4. On the Search Administration page, in the Quick Launch, under Queries and Results, click Scopes.

  5. To create a new scope, click New Scope.

  6. In the Title and Description section, type a title and description for the search scope.

  7. In the Target Results Page section, select one of the following options:

    • Use the default Search Results Page. Select this option if you want search results from this search scope to be presented by using the standard Search Results page (Searchresults.aspx).

    • Specify a different page for searching this scope. Select this option if you want search results from this search scope to be presented on a custom webpage. If you select this option, in the Target results page box, type the URL for the custom search results page.

  8. Click OK to create the scope and return to the View Scopes page.

To add scope rules to the scope

  1. On the View Scopes page, in the list of scopes, point to the name of the scope that you want to edit, click the arrow that appears, and then click Edit Properties and Rules.

  2. To add a new rule, on the Scope Properties and Rules page, in the Rules section, click New Rule.

  3. In the Scope Rule Type section, select one of the following options:

    • Web Address. Select this option if you want the scope to include or exclude content from any resource in the search index that can be identified either by a URL (such as Web sites, file shares, and Microsoft Exchange Server public folders) or by a host name, domain name, or subdomain name.

    • Property Query. Select this option if you want the scope to include or exclude content that has a managed property that has a particular value. For example, Author="John Doe".

      If the property that you want to use in the scope is not available in the Property Query list, see Make FAST Search Server 2010 for SharePoint properties available for use in scopes.

      If you want the scope to include or exclude content that was crawled by using a particular content source, click Property Query and select ContentSource from the list. Enter a value for the content source property, for example Local SharePoint sites.

    • All Content. Select this option if the rule should not restrict the scope (the scope will include or exclude all content in the index).

  4. In the Behavior section, select an option to specify how this rule combines with other rules to define the scope:

    • Include - Any item that matches this rule will be included, unless the item is excluded by another rule. Use this option to apply an "OR" rule.

    • Require - Every item in the scope must match this rule. Use this option to apply an "AND" rule.

    • Exclude - Items matching this rule will be excluded from the scope. Use this option to apply an "AND NOT" rule.

  5. Click OK.

Make FAST Search Server 2010 for SharePoint properties available for use in scopes

The managed properties that are listed in the Property Query list under scope rules are the managed properties that are defined in the Search Server schema. Out of the box, FAST Search Server 2010 for SharePoint and Search Server have the same properties, and the FAST Search Server 2010 for SharePoint properties will also be available in the Property Query list.

If you create a new managed property for FAST Search Server 2010 for SharePoint to be used in scopes, you must also add the same managed property in Search Server. The Search Server property will only be a placeholder that is used to populate the Property Query list.

To make managed properties available for scopes, also make sure that the Allow this property to be used in scopes option is selected for the property.

To add a placeholder managed property to populate the Property Query list

  1. Verify that the user account that is performing this procedure is an administrator for the Search service application.

  2. In Central Administration, in the Application Management section, click Manage service applications.

  3. On the Manage Service Applications page, in the list of service applications, click FAST Query SSA (the name of the FAST Query SSA).

  4. On the Search Administration page, in the Quick Launch, under Queries and Results, click Metadata Properties.

  5. On the Metadata Property Mappings page, click New Managed Property.

  6. On the New Managed Property page, in the Property name box in the Name and type section, type the name of the managed property.

  7. In the Description box, type a description of the property, for example that this is a placeholder property only.

  8. On the New Managed Property page, in the Use in scopes section, select the Allow this property to be used in scopes check box.

  9. Click OK. You do not have to fill in any of the other options on this page as the purpose of adding this managed property is to populate the Query Property list.

Create a search scope at the search service application level by using Windows PowerShell

  1. Verify that you have Farm Administrator permissions.

  2. On Microsoft SharePoint Server, open a SharePoint 2010 Management Shell:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Click SharePoint 2010 Management Shell.

  3. At the command prompt, type the following command:

    Copy Code
    New-SPEnterpriseSearchQueryScope -SearchApplication "<SearchApplication>" -Name <ScopeName> -Description "<ScopeDescription>"
-DisplayInAdminUI 1 -ExtendedSearchFilter "<FQL_Expression>"

    Where:

    • <SearchApplication> is the name of the Query Search Service Application (SSA).

    • <ScopeName> is the name of the new scope.

    • <ScopeDescription> is a description of the scope.

    • <FQL_Expression> is the FAST Query Language (FQL) query expression that represents the scope filter.

      Note:

      Any managed property names that are used in the FQL expression must be written in lowercase, even if the name of the managed property contains uppercase letters.

When a scope is created, it may take some minutes before the scope is available. To verify that the scope is active, run a query using the scope in the query. If the scope is not available, the query will return an error message.

See also

FAST Query Language (FQL) Syntax Reference

Create a site-level search scope by using Windows PowerShell

This example creates a site-level scope and adds a FAST Query Language (FQL) filter that limits the result set to items from two specific sites.

  1. Verify that you have Farm Administrator permissions.

  2. On Microsoft SharePoint Server, open a SharePoint 2010 Management Shell:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Click SharePoint 2010 Management Shell.

  3. At the command prompt, type the following command:

    Copy Code
    New-SPEnterpriseSearchQueryScope -SearchApplication "<SearchApplication>" -Name <ScopeName> -Description "<ScopeDescription>"
-DisplayInAdminUI 1 -OwningSiteUrl "<OwningSiteURL>"
$scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "<SearchApplication>" " -Identity "<ScopeName>"
$scope.Filter = 'site:or("<ScopeSite1URL>", "<ScopeSite2URL>"'
$scope.Update()

    Where:

    • <SearchApplication> is the name of the Query Search Service Application (SSA).

    • <ScopeName> is the name of the new scope.

    • <ScopeDescription> is a description of the scope.

    • <OwningSiteURL> is the URL of the site you want to define the scope for.

    • <ScopeSite1URL> is the URL of the first site to include in the search scope.

    • <ScopeSite2URL> is the URL of the second site to include in the search scope.

When a scope is created, it may take some minutes before the scope is available. To verify that the scope is active, run a query using the scope in the query. If the scope is not available, the query will return an error message.

See also

FAST Query Language (FQL) Syntax Reference

Create a site-level search scope that searches a full-text index by using Windows PowerShell

This example creates a scope with a FAST Query Language (FQL) filter on titles that contain the word "vista" searching a full-text index.

  1. Verify that you have Farm Administrator permissions.

  2. On Microsoft SharePoint Server, open a SharePoint 2010 Management Shell:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Click SharePoint 2010 Management Shell.

  3. At the command prompt, type the following command:

    Copy Code
    New-SPEnterpriseSearchQueryScope -SearchApplication "<SearchApplication>" -Name <ScopeName> -Description "<ScopeDescription>"
-DisplayInAdminUI 1 -ExtendedSearchFilter "title:vista,FullTextIndex=<FullTextIndex>" -OwningSiteUrl "<OwningSiteURL>"

    Where:

    • <SearchApplication> is the name of the Query Search Service Application (SSA).

    • <ScopeName> is the name of the new scope.

    • <ScopeDescription> is a description of the scope.

    • <FullTextIndex> is the name of the full-text index you want to include in the search scope.

    • <OwningSiteURL> is the URL of the site you want to define the scope for.

See also

FAST Query Language (FQL) Syntax Reference

View the FAST Query Language (FQL) filter for a search scope at the search service application level by using Windows PowerShell

  1. Verify that you have Farm Administrator permissions.

  2. On Microsoft SharePoint Server, open a SharePoint 2010 Management Shell:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Click SharePoint 2010 Management Shell.

  3. At the command prompt, type the following command:

    Copy Code
    $scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "<SearchApplication>" "<ScopeName>"
$scope.Filter

    Where:

    • <SearchApplication> is the name of the Query Search Service Application (SSA).

    • <ScopeName> is the name of the scope.

View the FAST Query Language (FQL) filter for a site level search scope by using Windows PowerShell

  1. Verify that you have Farm Administrator permissions.

  2. On Microsoft SharePoint Server, open a SharePoint 2010 Management Shell:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Click SharePoint 2010 Management Shell.

  3. At the command prompt, type the following command:

    Copy Code
    $scope = Get-SPEnterpriseSearchQueryScope -SearchApplication "<SearchApplication>" "<ScopeName>" -Url "<SiteURL>"
$scope.Filter

    Where:

    • <SearchApplication> is the name of the Query Search Service Application (SSA).

    • <ScopeName> is the name of the scope.

    • <SiteURL> is the URL of the site for which the scope is defined.

Make scopes available on the Search Center site

After a scope is defined, a site collection administrator can make the scope available to users on the Search Center site.

To define which scopes that will be available

  1. Verify that you have site collection administrator permissions.

  2. On the top level site, click Site Actions, and then click Site Settings.

  3. Under Site Collection Administration, click Search scopes.

  4. Click Display Groups.

  5. Click Search Dropdown.

  6. In the Scopes section, select which scopes that you want to display in the dropdown, and specify the order in which the scopes will be displayed.

  7. Click OK.

To display scopes in the Search Box Web Part on the Search Center site

  1. On the Search Center site, click the Site Actions menu, and then click Edit Page.

  2. Click the arrow in the Search Box Web Part menu, and then click Edit Web Part.

  3. Under Scopes Dropdown, select Show scopes dropdown.

  4. In the Dropdown label box, type the text you want to be displayed as the name for the scope drop-down list on the Search Center site.

  5. Click OK.

  6. Repeat these steps for the Search Box Web Part on the Search Results page.


Security and protection (FAST Search Server 2010 for SharePoint)

Before setting up item level security in Microsoft FAST Search Server 2010 for SharePoint, review the information in Security considerations for indexing (FAST Search Server 2010 for SharePoint) and Security considerations for the server farm (FAST Search Server 2010 for SharePoint).

This section provides information about item level security in FAST Search Server 2010 for SharePoint.

In this section:

  • FAST Search Authorization (FSA) overview

  • FAST Search Authorization (FSA) data flow

  • About multiple user stores

  • Manage FAST Search Authorization (FSA)


FAST Search Authorization (FSA) overview

Most Microsoft FAST Search Server 2010 for SharePoint systems provide various content sources for searching. These include public documents and proprietary or classified items. FAST Search Server 2010 for SharePoint includes a security module named FAST Search Authorization (FSA) which contributes to creating a secure search system with item level security and security trimming. FSA limits user access to items in search results that they are authorized to see.

In this article:

  • Use and benefits of FSA

  • FSA components

Use and benefits of FSA

Item level security operates in two phases:

  • Phase 1, Indexing: Content repositories are traversed and indexes are created. Authorization information is added to each item's authorization managed properties (the item's ACL, or access control list), identifying users and groups that are granted or denied access to the item.

  • Phase 2, Searching: A user submits a query and the indexes find search results. In this phase, the query processing service rewrites the user's query so that the user only sees items that he is authorized to see. This security trimming is performed with the help of a user search security filter, which is created by FSA based on the user's specific content permissions. By checking item managed properties (item ACLs which define who has permissions to view each item) against that user's search security filter, unsuitable search results are filtered or trimmed out.

Note:

FSA does not authenticate users. Authentication is performed by the SharePoint Server search front-end. See Plan authentication (SharePoint Server 2010).

To generate a user's security filter, the FSA worker component uses the information that is contained in the user's claim. Claims-based authentication is a set of operations that establishes trust relationships between claims providers and applications. When a claim arrives, the user has already been authenticated, and FSA makes an access control decision based on that claim, which is provided by the SharePoint front-end. See Configure claims authentication (SharePoint Server 2010) and Create a FAST Search Center site (FAST Search Server 2010 for SharePoint) for more information.

FSA components

FSA has two components: the FSA manager service (one per FAST Search Server 2010 for SharePoint system) and the FSA worker (one on each server that processes queries).

  • The FSA manager service receives security changes from indexing connectors and pushes the updates to all the query processing nodes in the system. The FSA manager also keeps the security-related configuration of these nodes consistent by administering the FSA workers and synchronizing changes across the nodes.

  • The FSA worker is part of the Query and Result Service (query processing node). FSA workers generate user search security filters based on user credentials. To do this, the FSA worker obtains group membership information for users from the FSA user stores and uses principal aliasing to map users/groups from one store to another.

When you install FAST Search Server 2010 for SharePoint, the FSA manager and worker services are installed automatically as Windows Services on the administration server:

  • FAST Search for SharePoint Sam Admin

  • FAST Search for SharePoint Sam Worker

When you configure an additional query processing node, another FSA worker service is started on the new node.


FAST Search Authorization (FSA) data flow
See Also

The FAST Search Authorization (FSA) solution for securing search content includes the following components and processes.

FSA data flow

FAST Search Authorization data flow overview

FSA helps secure search content at item processing time and during search activities.

Phase 1: Item processing and indexing

A content connector extracts items, access control information (ACLs), and user directory information from a source system, such as SharePoint. The content and item ACL are fed into an item processing pipeline for indexing. During the indexing phase, the document ACL is indexed exactly like any other field in the document and lists the users and groups that can read the documents. Depending on the security model of the source system, you can also include deny-lists.

A user store is a logical grouping of users, groups, and content permissions that serves as a security gateway to a third-party content repository to help protect its content from unauthorized access. If your search implementation includes content from several different providers, you may have to create multiple security user stores. A claims user store is created as the default user store when you work with claims authentication.

Phase 2: Searching

On the search side, FSA enforces security trimming, limiting search results based on the identity of the user who submitted the query. The Query and Result Service (query processing node) accepts a user's query together with the user's claim. When an incoming search request passes through the query transformation pipeline, FSA expands the request with group membership information for that query user, building a user search security filter that specifies content permissions for that specific search user. The QRServer intersects the query with the user's search security filter, checking item ACLs against the filter to weed out items that the user is not allowed to access. When the search is run, it returns a subset of items which the user is authorized to see.

Lotus Notes content

To include Lotus Notes content in the FAST Search Server 2010 for SharePoint index, you must first create a Lotus Notes user store. The FAST Search Lotus Notes connector reads users and groups from a Lotus Notes system and populates the FSA Lotus Notes user store with user credentials and content permissions for Lotus Notes database collections. The query processing server uses this information to create a user security filter that lets Lotus Notes indexed content to be returned to the user at query time. You must match the ID of the Lotus Notes user store with the ID that you specify in the FAST Search Lotus Notes connector configuration. See Configure the FAST Search Lotus Notes connector for more information.

Principal aliasing

FSA uses a process that is known as principal aliasing to map information in the user's claim to their equivalent IDs in the Lotus Notes user store. For example, you can map a user's claim with the value of jbrown to be the same as the user brownj in the Lotus Notes user store. The FAST Search Lotus Notes connector generates a mapping file which you can upload to FSA. The Lotus Notes mapping file is an XML file that instructs the FSA worker how to map a claim value to its equivalent ID in the Lotus Notes system. This mapping helps create a user's security filter that lets indexed Lotus Notes items to be returned to the user at query time. See Prepare FAST Search Authorization for use with the FAST Search Lotus Notes connector for more information.

See Also

Concepts

Security cmdlets (FAST Search Server 2010 for SharePoint)


About multiple user stores
See Also

In many systems, you will have content coming from multiple sources, fed by several different indexing connectors. When the indexed content requires item level security, you must create multiple FSA user stores. Any content originating in Microsoft sources (Outlook, Word documents, SharePoint databases, etc.) uses claims authentication for the user's claims token and item level security. Another common user store is a Lotus Notes user store, which caches credentials for Lotus Notes users.

Multiple user stores generally require principal aliasing. If a Lotus Notes database is publically available to all Lotus Notes users (that is to say, read access for the Lotus Notes Default user but not for the Lotus Notes Anonymous user) you would still have to map Windows aliases to Lotus Notes users to make the content searchable if you use the FAST Search Lotus Notes connector.

In this section:

  • About principal aliasing

  • Remove a principal aliaser by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

See Also

Concepts

Create a FAST Search Center site (FAST Search Server 2010 for SharePoint)
Prepare FAST Search Authorization for use with the FAST Search Lotus Notes connector


About principal aliasing
See Also

In a system with multiple content sources, users may have identities in multiple user stores (e.g., a Windows user may also have a Lotus Notes account that has a different ID). To generate the user's security filter for narrowing query results, the FSA worker component must know the user's ID and groups in all stores. FSA uses principal aliasing to map users and groups from one user store to their equivalents in another user store.

Principal aliasing uses an XML file to map each specific security ID to its equivalent security ID in another store. Use the New-FASTSearchSecurityXMLAliaser cmdlet to create XML alias mappings.

When using FAST Search Center to perform queries, credentials will usually be authenticated by SharePoint Server 2010 claims based authentication. You must set up a claims security user store and make it the default user store. Typically, the default store is named win. By using this configuration, users may query on content sources crawled using the FAST Search connector. They will be unable to query documents crawled using the FAST Search Lotus Notes connector, because the document ACL from a Lotus Notes database will not match a claims domain ID.

See Also

Reference

New-FASTSearchSecurityXMLAliaser

Concepts

Prepare FAST Search Authorization for use with the FAST Search Lotus Notes connector


Remove a principal aliaser by using Windows PowerShell (FAST Search Server 2010 for SharePoint)
See Also

This topic explains how to remove a principal aliaser in Microsoft FAST Search Server 2010 for SharePoint by using Windows PowerShell.

In this section:

  • Remove a principal aliaser by using Windows PowerShell

Remove a principal aliaser by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    Remove-FASTSearchSecurityAliaser -Identity <aliaser>

    Where:

    • <aliaser> is the name of the aliaser to remove.

See Also

Reference

Remove-FASTSearchSecurityAliaser


Manage FAST Search Authorization (FSA)

When you install FAST Search Server 2010 for SharePoint, the FSA manager and worker services are installed automatically as Windows Services on the administration server. Whenever you configure an additional FAST Search Server 2010 for SharePoint server that has QRServer on it, another FSA worker service is started on that host. Once the FSA services are installed and started, there is little you have to manage on a regular basis. The most common administration tasks are described in this article:

  • Restart FSA

  • View and monitor the FSA log files

  • Change the FSA log settings by using Windows PowerShell

  • Determine if a FSA user store exists by using Windows PowerShell

  • Use FSA cmdlets in Windows PowerShell

Restart FSA

Under ordinary usage, you should not have to restart the FSA manager or worker services. On rare occasions, you may have to restart the system:

  • FSA manager

    • The Xxx-FASTSearchSecurityXxx cmdlets fail because they cannot communicate with the FSA manager

    • If user/group information upload requests from the FAST Search Lotus Notes connector fail

  • FSA worker

    • If query is not working on a specific query processing node

    • If the FSA worker is using too much memory

    • If you change a FSA configuration setting

    • If synchronization from the FSA manager is unsuccessful

Use the FAST Search Server 2010 for SharePoint node controller (nctrl) to stop and start FSA if you have to restart FSA. The node controller will restart the FSA services if it detects that they stopped prematurely.The correct order is as follows:

  1. Stop the FSA manager (samadmin service)

  2. Stop all FSA worker processes (samworker services)

  3. Start all FSA worker processes (samworker services)

  4. Start the FSA manager (samadmin service)

For more information, see nctrl.exe reference.

View and monitor the FSA log files

FSA log files are written to the administration host where the FSA manager resides, under <FASTSearchFolder>\var\log\syslog, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch. Here you will find two kinds of log files, one for the FSA manager and one for each FSA worker:

  • authorization-admin_<host_name>.log

  • authorization-worker_<host_name>.log

Change the FSA log settings by using Windows PowerShell

Use the Set-FASTSearchSecurityLogLevel cmdlet to configure the FSA log level. The logging levels from most verbose to least verbose are as follows:

  • DEBUG

  • INFO

  • WARNING

  • ERROR

  1. To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. Open a Windows PowerShell session as an Administrator.

  3. Run the Set-FASTSearchSecurityLogLevel cmdlet. The following example sets the log level for the Microsoft namespace to the Warning level:

    Copy Code
    Set-FASTSearchSecurityLogLevel -WarningNameSpaceLogLevel Microsoft
    Note:

    The Set-FASTSearchSecurityLogLevel cmdlet changes the log levels in all the FSA manager and worker logs, not merely one particular log file.

For more information, see Set-FASTSearchSecurityLogLevel.

Determine if a FSA user store exists by using Windows PowerShell

  1. To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. Open a Windows PowerShell session as an Administrator.

  3. To check whether a user store exists, run the Get-FASTSearchSecurityUserStore cmdlet. For example:

    Copy Code
    Get-FASTSearchSecurityUserStore -Identity

    If you do not specify one user store id with the -Identity parameter, the cmdlet will retrieve all user stores.

For more information, see Get-FASTSearchSecurityUserStore.

Use FSA cmdlets in Windows PowerShell

  1. On a FAST Search server, open the Microsoft FAST Search Server 2010 for SharePoint command prompt as an administrator. This will open a Windows PowerShell window and load the Microsoft.FASTSearch.Powershell cmdlet snap-ins.

  2. If you have to run Windows PowerShell script files, you must set the execution policy to RemoteSigned. For example:

    Copy Code
    Set-ExecutionPolicy RemoteSigned

    Type Get-Help Set-ExecutionPolicy for more information about the Windows PowerShell execution policy.

Tip:

Windows PowerShell has an auto-complete feature that can help in selecting FSA cmdlets. You can test this by typing GET-FASTSearchSecurity and pressing the Tab key to have Windows PowerShell complete your cmdlet.


Troubleshooting (FAST Search Server 2010 for SharePoint)

This section describes troubleshooting steps for common issues in FAST Search Server 2010 for SharePoint. As new solutions become available, they will be added to this section.

In this section:

  • Administration pages: Cannot access the pages (FAST Search Server 2010 for SharePoint)

  • Crawling: Slow crawling and slow copying between the indexer and the search nodes (FAST Search Server 2010 for SharePoint)

  • FSA: The search request was unable to connect to the Search Service (FAST Search Server 2010 for SharePoint)

  • FSA: Queries always fail on a specific query processing node (FAST Search Server 2010 for SharePoint)

  • FSA: Queries are returning fewer documents than expected (FAST Search Server 2010 for SharePoint)

  • FSA: Queries are returning more documents than expected (FAST Search Server 2010 for SharePoint)

  • Indexer server: Disk usage increases unexpectedly (FAST Search Server 2010 for SharePoint)

  • Installation: Configuration wizard fails (FAST Search Server 2010 for SharePoint)


Administration pages: Cannot access the pages (FAST Search Server 2010 for SharePoint)
See Also

Published: May 16, 2012

Issue: Cannot access the pages for Central Admin, Site Settings and Site Collections or the administration pages for keywords and managed properties in Microsoft FAST Search Server 2010 for SharePoint.

Summary: If you are having difficulties accessing keywords, managed properties or administrator pages, check the following causes:

  • Cause 1: Certificates have not been installed correctly, have expired, or have been revoked

  • Cause 2: User accounts have not been added to the FASTSearchAdministrators group

  • Cause 3: SharePoint Publishing features have not been enabled for non-publishing sites

  • Cause 4: Invalid IIS application pool credentials

Cause 1: Certificates have not been installed correctly, have expired, or have been revoked

Resolution: Check the status of your certificates

Each server in a FAST Search Server 2010 for SharePoint farm potentially has three certificates which serve different functions and which must be configured (and replaced) separately. In case of errors, you should check the status of your certificates.

For more information about managing certificates, see: Manage certificates (FAST Search Server 2010 for SharePoint).

Cause 2: User accounts have not been added to the FASTSearchAdministrators group

Resolution: Add one or several user accounts to the FASTSearchAdministrators group

For more information about setting permissions, reviewing software requirements and security settings, see:

  • Review hardware and software requirements (FAST Search Server 2010 for SharePoint)

  • Administration cmdlets (FAST Search Server 2010 for SharePoint)

  • Account permissions and security settings (SharePoint Server 2010)

  • User roles required for deployment tasks

Cause 3: SharePoint Publishing features have not been enabled for non-publishing sites

Resolution: Enable the SharePoint Publishing features

SharePoint Publishing features are all the features that are part of a preconfigured publishing site or that are added when publishing is enabled at the site collection and site level. When publishing is enabled, all publishing features are automatically enabled.

For more information about enabling SharePoint Publishing features, see:

  • Enable publishing features

  • Publishing features overview (SharePoint Server 2010)

Cause 4: Invalid IIS application pool credentials

Resolution: Set the application pool identity and restart the application pool

Web sites and Web applications depend on the availability of Internet Information Services (IIS) application pools. IIS application pools in turn depend on the Windows Process Activation Service (WAS). If WAS is not running or errors occur during the startup or shutdown of an application pool, Web sites and Web applications may not be available.

For more information about Internet Information Services (IIS) application pools, see http://technet.microsoft.com/library/cc735179(WS.10).aspx

See Also

Other Resources

Manage FAST Search keywords


Crawling: Slow crawling and slow copying between the indexer and the search nodes (FAST Search Server 2010 for SharePoint)

Published: February 21, 2012

Issue: Crawling in SharePoint is very slow when indexing content on a multi-server FAST Search Server 2010 for SharePoint farm.

Summary: FAST Search Server 2010 for SharePoint relies on a fast and stable network connection for internal communication. This communication is secured with IPSec and it is configured during the FAST Search Server 2010 for SharePoint post-configuration. If you experience unexpected slow crawling and indexing, this article explains the probable cause and how to resolve this issue.

Symptom: Slow crawling and slow copying between the indexer and the search nodes

Note:

The issue is typically seen in virtualized environments, but can also occur in physical environments.

Cause: The Task offload setting may be enabled on the FAST Search Server 2010 for SharePoint servers.

Resolution: Check the status of the Task offload setting and disable it if it is enabled

To check the status of the Task offload setting:

  1. On all FAST Search servers, open a command promt as an administrator, and type the following commands(s):

    Copy Code
    For IPv4, type: "netsh int ipv4 show global"
For IPv6, type: "netsh int ipv6 show global"

To disable the Task offload setting:

  1. On all FAST Search servers, open a command promt as an administrator, and type the following command:

    Copy Code
    "netsh int ip set global taskoffload=disabled"

    where ip is the protocol version, either ipv4 or ipv6, where ipv4 is default

  2. Reinitialize the network adapter or restart the server.


FSA: The search request was unable to connect to the Search Service (FAST Search Server 2010 for SharePoint)

Issue: The search request was unable to connect to the Search Service.

Summary: If search does not work, it might be that the communication channels are not cleaned up correctly in some scenarios, the Microsoft FAST Search Server 2010 for SharePoint query processing services are not running, the Microsoft SharePoint Server certificate is not present on the node running query processing or SharePoint Server has not been configured to point to FAST Search Server 2010 for SharePoint query processing. We recommend that you work through each possible resolution in the given order.

Symptom: The search request was unable to connect to the Search Service.

Causes: This symptom can be the result of the following causes:

  • Cause 1: The communication channels are not cleaned up correctly

  • Cause 2: The FAST Search Server 2010 for SharePoint query processing services are not running

  • Cause 3: The SharePoint certificate is not present on one or more of the FAST Search Server 2010 for SharePoint query processing servers

  • Cause 4: SharePoint has not been configured to point to the FAST Search Server 2010 for SharePoint query processing node

If these suggestions do not resolve the problem, there may be a problem between SharePoint Server and the QRProxy, or between QRProxy and FSA. Follow the suggestions in FSA: Queries always fail on a specific query processing node (FAST Search Server 2010 for SharePoint).

Cause 1: The communication channels are not cleaned up correctly

Resolution: Retry the search

If searching was working previously, the problem may result from a known issue in which the communication channels are not cleaned up correctly in some scenarios. Retry the search and it should succeed.

Cause 2: The FAST Search Server 2010 for SharePoint query processing services are not running

Resolution: Restart FSA

For each FAST Search Server 2010 for SharePoint query processing server:

  1. Log onto the server and open a FAST Search Server 2010 for SharePoint command prompt.

  2. Run the command: nctrl status.

  3. Verify that these processes are running: samworker, qrproxy, and qrserver.

If any of these processes are not running, restart FSA. For more information, refer to Restart FSA.

Cause 3: The SharePoint certificate is not present on one or more of the FAST Search Server 2010 for SharePoint query processing servers

Resolution: Export the SharePoint Server certificate to the given FAST Search Server 2010 for SharePoint server

As part of the setup for connecting SharePoint Server to the FAST Search Server 2010 for SharePoint query processing node, a certificate must be copied to enable communication between the two servers. To verify that the certificate is located on the FAST Search Server 2010 for SharePoint server, look at the certificates in the Local Computer, Trusted People certificates. You should find a certificate named SharePoint Security Token Service and it should have a thumbprint ID that matches the certificate located on the SharePoint Server computer at Local Computer, SharePoint, SharePoint Security Token Service.

To view the certificates on a server, follow these steps.

  1. Run MMC from the Windows Start menu.

  2. Select File > Add or Remove Snapin.

  3. Select Certificates in Available snap-ins and Add.

  4. Select Computer account then Local computer. Click Finish and OK.

  5. Expand Certificates (local Computer).

  6. Expand Trusted People.

  7. Expand Certificates.

  8. Double-click SharePoint Security Token Service.

  9. On Details tab, click Thumbprint to get the thumbprint id.

If the certificate is missing on a server, refer to Enable queries from Microsoft SharePoint Server (FAST Search Server 2010 for SharePoint) for information about how to export the certificate.

Cause 4: SharePoint has not been configured to point to the FAST Search Server 2010 for SharePoint query processing node

Resolution: Configure SharePoint Server to point to the FAST Search Server 2010 for SharePoint query processing node.

To verify the configuration of SharePoint Server, follow these steps:

  1. Open the SharePoint ServerCentral Admininstration Web site.

  2. Go to the Application Management page and select Manage Web applications.

  3. Highlight the row of the Web site from which searches are being performed.

  4. From the ribbon under the Web Applications tab, select Service Connections.

  5. Find the name of the service application that has a Type of Search Service Application Proxy and is selected for use with the Web site where searches are performed. This name will be needed to look up the settings for communications between SharePoint Server and FAST Search Server 2010 for SharePoint.

  6. Close the window and go back to the Central Administration home page.

  7. Select Manage service applications under the Application Management section.

  8. Highlight the name of the service application that has the name that you previously found for the Search Service Application (step 5). Be sure to highlight the row; do not click the name link.

  9. From the ribbon under Service Applications, select Properties.

  10. Verify that the FAST Search Query radio button is set to the FAST Service Application. Verify that the Query Service Location is pointing to the FAST Search Server 2010 for SharePoint query processing node that you expect. The format of this field is protocol://FQDN:port, with a semicolon (;) separating multiple servers. For example:

    Copy Code
    http://my-server.domain.com:13287;http://my-server2.domain.com:13287

    If you are unsure of the server name and port, verify it by going to the FAST Search Server 2010 for SharePoint query processing node. Open the file <FASTSearchFolder>\Install_Info.txt, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch. Look for the Query Service Location HTTP setting. The other settings in the Service Applications, Properties window are important but they do not affect search. The correct values for these settings are also in the Install_Info.txt file.


FSA: Queries always fail on a specific query processing node (FAST Search Server 2010 for SharePoint)

Issue: Queries always fail on a specific query processing node.

Summary: If Microsoft FAST Search Server 2010 for SharePoint does not always work, determine whether queries are failing on one specific query processing node. If that is true, the cause might be that the FAST Search Server 2010 for SharePoint query processing service is not running on the given node, the FAST Search Server 2010 for SharePoint query processing node is not set up correctly to communicate with Microsoft SharePoint Server 2010, the FSA worker process is not up to date with configuration changes, or the QRProxy service cannot communicate with FSA. We recommend that you work through each possible resolution in the given order.

Symptom: Queries always fail on a specific query processing node.

Causes: This symptom can be the result of the following causes:

  • Cause 1: The FAST Search Server 2010 for SharePoint query processing service is not running

  • Cause 2: The FAST Search Server 2010 for SharePoint query processing node might not be set up correctly to communicate with SharePoint

  • Cause 3: The FSA worker process is not up to date with configuration changes

  • Cause 4: The QRProxy service cannot communicate with FSA

Cause 1: The FAST Search Server 2010 for SharePoint query processing service is not running

Resolution: Restart FSA

Verify that FAST Search Server 2010 for SharePoint is running:

  1. Log onto the server and open a FAST Search Server 2010 for SharePoint command prompt.

  2. Run the command: nctrl status.

  3. Verify that these processes are running: samworker, qrproxy, and qrserver.

If any of these processes are not running, restart FSA. For more information, refer to Restart FSA.

Cause 2: The FAST Search Server 2010 for SharePoint query processing node might not be set up correctly to communicate with SharePoint

To verify the setup of FAST Search Server 2010 for SharePoint, follow the guidance in FSA: The search request was unable to connect to the Search Service (FAST Search Server 2010 for SharePoint)

Cause 3: The FSA worker process is not up to date with configuration changes

Resolution: Ensure that all configuration changes have been propagated to all query processing nodes.

When configuration changes are made to FSA, the changes must be copied to all the query processing nodes.

Configuration changes occur when you run the Windows PowerShell cmdlets with the prefix verb-FASTSearchSecurity. For example, the Set-FASTSearchSecurityLogLevel cmdlet updates the configuration with the log level setting. When a FAST Search for SharePoint Sam Worker is restarted, it must wait for the FAST Search for SharePoint Sam Admin process to communicate with it and to verify that it has all the configuration changes. This process may take several minutes depending on how many query processing nodes are in the system and how many changes occurred while the node was down.

To verify that all the worker nodes have the most recent configuration changes, follow these steps:

  1. Open a FAST Search Server 2010 for SharePoint command prompt

  2. Run Get-FASTSearchSecurityWorkerNode

  3. Verify that the status for all nodes is active

If any node is not active, wait for several minutes and recheck. If it is still not active after 5 to 10 minutes, increase the log file levels and look in the logs for both the FAST Search for SharePoint Sam Worker and FAST Search for SharePoint Sam Admin.

Cause 4: The QRProxy service cannot communicate with FSA

Does the event viewer contain a message for the FAST Search QRProxy that says, "An unhandled exception occurred: Unable to connect to SAM"? You may receive this error message because the FAST Search for SharePoint Sam Worker is not started, because the FAST Search for SharePoint Sam Worker does not have the most recent configuration changes, or because a WCF error is preventing it from communicating. If the FAST Search for SharePoint Sam Worker is running and has the most recent configuration, restart QRProxy. If the problem still exists, verify that the FAST Search for SharePoint Sam Worker is ready to receive requests. If restarting the FSA worker and QRProxy does not resolve the problem, and the FAST Search for SharePoint Sam Worker is ready to receive requests, enable WCF tracing to diagnose the error.

Resolution 1: Restart QRProxy

  1. Open a FAST Search Server 2010 for SharePoint command prompt.

  2. Run the command: nctrl restart qrproxy.

  3. After the QRProxy service is started, retry the search.

Resolution 2: Ensure that the FAST Search for SharePoint Sam Worker is ready to receive requests

Follow these steps on the node where the QRProxy and FAST Search for SharePoint Sam Worker communication is not working.

  1. Open the FAST Search Server 2010 for SharePoint command prompt as an administrator and run the following commands:

    Copy Code
    set-alias installutil $env:windir\Microsoft.NET\Framework64\v2.0.50727\installutil
    Copy Code
    installutil Microsoft.SharePoint.Search.Extended.Security.Worker.PowerShell.Commands.dll
    Copy Code
    add-pssnapin FASTSearchSecurityWorkerSnapIn

  2. Modify the configuration file base port value.

    There is a known issue which requires a minor change to the file at <FASTSearchFolder>\bin\Microsoft.SharePoint.Search.Extended.Security.Worker.Powershell.Commands.dll.config. This file includes a base_port XML element. Add one to the value and save the file. For example, if the value is 13000, change the value to 13001 and save the file.

  3. Run the cmdlet:

    Copy Code
    Get-FASTSearchSecurityConfigurationStatus

If the return value from this cmdlet is false, increase the log levels and look at the FSA worker logs.

Resolution 3: Enable WCF tracing to diagnose the error

To enable WCF tracing between QRProxy and the FSA worker, follow these steps.

  • Modify <FASTSearchFolder>\bin\Microsoft.SharePoint.Extended.Security.WorkerService.exe.config and <FASTSearchFolder>\bin\QRProxyService.exe.config. Remove the comments <!-- and --> from around the <diagnostics> and <system.diagnostics> XML elements in both files.

  • Save the files.

  • Restart both the FSA worker and QRProxy. This will create two files in the <FASTSearchFolder>\bin with extension .svclog.

You can find more information about how to view WCF trace files by searching TechNet.


FSA: Queries are returning fewer documents than expected (FAST Search Server 2010 for SharePoint)

Issue: Queries are returning fewer documents than expected.

Summary: If a Microsoft FAST Search Server 2010 for SharePoint query is returning fewer documents than you expect to see, it could be that the search term is not present in the documents you expected it to be in, or it could indicate one of several item-level security issues.

Symptom: Queries are returning fewer documents than expected.

Causes: We recommend that you work through the possible causes in the given order:

  • Cause 1: The search term is not present in the documents that you expected it to be in

  • Cause 2: The query is performed by the wrong user

  • Cause 3: The user did not have access rights to the documents when they were indexed

  • Cause 4: The group that the user belongs to does not have access to the documents

Cause 1: The search term is not present in the documents that you expected it to be in

Resolution: Ensure that the term is present in the documents

If possible, verify that the search term is in the documents outside Microsoft SharePoint Server. Open the files and search for the term. If the documents cannot be viewed outside SharePoint Server, try searching for a different term that should also be in the documents.

Cause 2: The query is performed by the wrong user

Resolution: Log in as the correct user

Security around search results is based on the user who is logged into SharePoint Server when the search is performed. Verify that the correct user is logged into SharePoint Server by looking at the name of the user in the upper-right corner of the web page.

Cause 3: The user did not have access rights to the documents when they were indexed

Resolution: Re-index the documents

If the permissions on the document have changed, security on the search will not reflect those changes until the documents are re-indexed.

Cause 4: The group that the user belongs to does not have access to the documents

Resolution: Determine the user's permissions and ensure that the user has access to see the documents

  1. Set the log level to info. Open a Microsoft FAST Search Server 2010 for SharePoint command prompt as an administrator and run the following command:

    Copy Code
    Set-FASTSearchSecurityLogLevel -DefaultLogLevel Info

  2. Re-run the search.

  3. Go to <FASTSearchFolder>\var\log\syslog,

    where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

  4. Open the log file that uses the name format authorization-worker_<computer>.txt, where <computer> is the computer name of the query processing node. If there are multiple query processing nodes in the system, the information can be in any log file for any of the query processing nodes, depending on which query processing node that serviced the request.

  5. In the log file for the FSA worker (FAST Search for SharePoint Sam Worker), search for a message from Claims.dll:GetClaimsPrincipal. This message includes the ID of the user making the request and the number of claims for this user, and it then lists the data for each claim that is associated with the user. For the claims with a claim type of http://schemas.microsoft.com/ws/2008/06/identity/claims/primarygroupsid, the value will be the Windows SID (Security Identifier) of a group that the user is a member of.

There is a known issue that causes the logging mechanism to truncate log messages when it has reached a certain size. If the user is a member of many groups, not all of the information will be displayed in the log file. If the log file is truncated, you can create a new log file:

  1. Restart the FAST Search for SharePoint Sam Worker.

    This can cause the query processing node to be unavailable until the FSA worker is functioning and up to date with configuration.

  2. Go to the query processing node where you want to create a new log file.

  3. Open the file <FASTSearchFolder>\bin\Microsoft.SharePoint.Search.Extended.Security.WorkerService.exe.config and search for <dependency name="FileLogWriter" />.

  4. Remove the comments <!-- and --> from around this XML element and save the file.

  5. Restart the FSA worker:

    Open a Microsoft FAST Search Server 2010 for SharePoint command prompt and run the following command: nctrl restart samworker.

    This will create a new log file here: <FASTSearchFolder>\components\sam\worker\samworkerservice.txt.

  6. Re-run the search. The complete log message and all of the security information should be included in the newly created log file.

  7. When you are finished with the log file, set the log level back to the default of Warning and comment out the FileLogWriter that allows the second log file to be generated.


FSA: Queries are returning more documents than expected (FAST Search Server 2010 for SharePoint)

Issue: Queries are returning more documents than expected.

Summary: If a Microsoft FAST Search Server 2010 for SharePoint query is returning more documents than you expect to see, it could be because the documents are public for all users or that the user performing the query is a member of a group that has permissions to see the documents.

Symptom: Queries are returning more documents than expected.

Causes: This symptom can result from one or both of the following causes:

  • Cause 1: The documents are public or can be accessed by all users

  • Cause 2: The user performing the query is a member of a group that has permissions to see the documents

Cause 1: The documents are public or can be accessed by all users

Resolution: Ensure that the document permissions correspond to your intentions

The user and group permissions must match the security permissions of the indexed documents.

Cause 2: The user performing the query is a member of a group that has permissions to see the documents

Resolution: Ensure that group memberships do not expose the documents too widely

To determine the security permissions of the user performing the search, follow the guidance in FSA: Queries are returning fewer documents than expected (FAST Search Server 2010 for SharePoint), section 'The group that the user belongs to does not have access to the documents'. By knowing the security permissions for the user and of a document, you should be able to determine which permissions that provided access to the document.


Indexer server: Disk usage increases unexpectedly (FAST Search Server 2010 for SharePoint)

Published: February 17, 2011

Issue: Indexer server: Disk usage increases unexpectedly.

Summary: Typically, the disk usage on an indexer server will not increase significantly unless new content sources are added or existing ones have increased in size. If you experience an unexpected increase in disk usage, the probable causes are an unplanned increase in crawled content, or that the query matching component has been stopped by the administrator without suspending indexing.

Symptom: Disk usage on the indexer server is increasing unexpectedly, and you risk running out of disk space.

Causes: This symptom can be a result of one or both of the following causes:

  • Cause 1: Too many indexed items in one index column caused by unplanned increase in crawled content or changed type of content

  • Cause 2: You have a query matching component installed on the index server, but the query matching component has been stopped

Cause 1: Too many indexed items in one index column caused by unplanned increase in crawled content or changed type of content

Choose the resolution that best matches your system installation.

Resolution 1: Reduce the amount of crawled content

  1. Reduce the disk usage by limiting the amount of crawled content. How to do this depends on the type of indexing connector you are using.

    If you are using the FAST Search Content Search Service Application, you can exclude content by adding crawl rules in the Central Administration. For more information, see Manage crawl rules (FAST Search Server 2010 for SharePoint).

    If you are using one or more FAST Search Server 2010 for SharePoint specific indexing connectors, you can exclude content by configuring certain parameters in the connector configuration file. For more information, see Manage crawl rules (FAST Search Web crawler), Manage crawl rules (FAST Search Lotus Notes connector), and / or Manage crawl rules (FAST Search database connector).

Resolution 2: Add an additional indexer column

  1. Increase the indexing capacity by adding an additional indexer column. For information about how to do this, see Add an index column (FAST Search Server 2010 for SharePoint).

Cause 2: You have a query matching component installed on the index server, but the query matching component has been stopped

When the indexer and query matching components are running on the same server, the query matching component is responsible for deleting old index sets as it is activating new sets. If the query matching component has been stopped but the administrator has not suspended indexing, the indexer will continue to produce new index sets based on crawled content. As the query matching component is unable to delete the old index sets, the disk usage will increase over time.

Resolution: Start the query matching component to remove old and unused index sets

  1. From a command prompt, start the search process (query matching role) by running the following command:

    <FASTSearchFolder>\bin\nctrl start search-1

    where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

    When the query matching component is running, new index sets will become active and the old index sets will be deleted. Disk usage will decrease accordingly.


Installation: Configuration wizard fails (FAST Search Server 2010 for SharePoint)

Published: March 17, 2011

Issue: Installation: Configuration wizard fails

Summary: During installation, you use the FAST Search Server 2010 for SharePoint Configuration Wizard to configure both single server and multiple server deployments. If the configuration wizard should fail, this article lists some of the probable causes and how to resolve the issues.

Symptom: The configuration of the deployment is unsuccessful, and you will see an error message.

Causes: Choose the cause that best matches the error message you are seeing, or work through the causes in sequential order:

  • Cause 1: The Windows Firewall is disabled

  • Cause 2: The deployment.xml file has an invalid character at the beginning of the file

  • Cause 3: The Windows PowerShell execution policy is incomplete

  • Cause 4: You have entered an incorrect database connection string

  • Cause 5: You have increased the size of the SQL database from its default value

Cause 1: The Windows Firewall is disabled

One common reason why the configuration fails is that the Windows Firewall is disabled on one or more of the servers.

Resolution: Enable Windows Firewall

  1. Enable the Windows Firewall on all servers. See Verify that Windows Firewall is turned on on all servers in the FAST Search Server 2010 for SharePoint farm.

  2. Re-run the FAST Search Server 2010 for SharePoint Configuration Wizard.

Cause 2: The deployment.xml file has an invalid character at the beginning of the file

The configuration wizard reads the deployment.xml file to obtain information about the topology of the FAST Search Server 2010 for SharePoint deployment. If you have opened and saved the deployment.xml file in an editor (for example Visual Studio) that adds an UTF-16 Byte Order Mark (BOM) to the beginning of files, the configuration wizard fails to read the deployment.xml file properly.

A typical indication of this cause is this error message:

"Set-FASTSearchIPSec : XML Validation error: Data at the root level is invalid. Line 1, position 1. At line:1 char:20"

Resolution: Save the deployment.xml file in a text editor that does not implement UTF-16

  1. Open and save the deployment.xml file in a text editor that does not implement UTF-16, for example Notepad.

  2. Re-run the FAST Search Server 2010 for SharePoint Configuration Wizard.

Cause 3: The Windows PowerShell execution policy is incomplete

A typical indication of this cause is this error message:

"psconfig.ps1 cannot be loaded because the execution scripts is disabled on this system".

The reason is probably that the execution policy for Windows PowerShell scripts does not allow scripts to run remotely.

Resolution: Enable Windows PowerShell scripts to run as Administrator and to run remotely

  1. Ensure that Windows PowerShell scripts are running as Administrator, and that the Windows PowerShell scripts are allowed to run remotely. See Enable Windows PowerShell scripts to run for more information.

  2. Re-run the FAST Search Server 2010 for SharePoint Configuration Wizard.

Cause 4: You have entered an incorrect database connection string

A typical indication of this cause is an error message that states: "Database validation failed" when you configure the administration server.

The reason is probably that an incorrect database connection string has been entered in the configuration wizard.

Resolution: Verify that the database connection string is correct

  1. On the server where the SQL Server is installed, open SQL Server Configuration Manager from the Start menu. Under SQL Server Services, you see the SQL Server main service. The service name contains the connection string in parentheses.

  2. Re-run the FAST Search Server 2010 for SharePoint Configuration Wizard.

Cause 5: You have increased the size of the SQL database from its default value

If the the initial database size that is defined by the MS SQL Server master database is increased from its default size, the creation of the FASTSearchAdminDatabase can fail because of a time-out.

An indication of this cause is an error message that states "Configuration failed" when you configure the administration server. To determine if the SQL database size has caused a time-out, you must examine the configuration wizard log file (psconfig.log) for details. This log file is typically located in:

<FASTSearchFolder>\var\log\installer\

where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

Resolution: Change the initial database size back to default

  1. From the SQL Server Management Studio, reset the default initial size of the database to 2MB.

  2. Re-run the FAST Search Server 2010 for SharePoint Configuration Wizard.


Technical reference (FAST Search Server 2010 for SharePoint)

Technical reference includes information about the FAST Search Server 2010 for SharePoint Windows PowerShell cmdlets, the FAST Search Server 2010 for SharePoint command-line tools, the FAST Search Server 2010 for SharePoint configuration files, and overviews of the WMI classes and performance counters for monitoring.

In this section:

  • About Windows PowerShell cmdlets (FAST Search Server 2010 for SharePoint)

  • About command-line tools (FAST Search Server 2010 for SharePoint)

  • About configuration files (FAST Search Server 2010 for SharePoint)

  • About WMI classes and performance counters (FAST Search Server 2010 for SharePoint)

  • User roles required for deployment tasks


About Windows PowerShell cmdlets (FAST Search Server 2010 for SharePoint)

Windows PowerShell is a powerful command-line shell and scripting language that helps you perform administrative tasks.

Using FAST Search for SharePoint cmdlets

Windows PowerShell cmdlets for FAST Search Server 2010 for SharePoint are specialized .NET classes that implement specific system administration actions in the areas of search management, index schema management, installation, security, and administration. Typical tasks include uploading a compiled dictionary to the resource store, creating a security user store, and mapping metadata to indexable fields. Some administrative tasks can be done on a Central Administration page, whereas other tasks can only be run with Windows PowerShell cmdlets.

FAST Search Server 2010 for SharePoint uses a master Windows PowerShell snap-in called Microsoft.FASTSearch.PowerShell.dll. This DLL includes all the code for over 80 cmdlets.

For basic information about how to use Windows PowerShell cmdlets and scripts, see Windows PowerShell.

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Categories of cmdlets

FAST Search Server 2010 for SharePoint Windows PowerShell cmdlets cover these basic areas:

  • Administration cmdlets (FAST Search Server 2010 for SharePoint)

  • Index schema cmdlets (FAST Search Server 2010 for SharePoint)

  • Installation cmdlets (FAST Search Server 2010 for SharePoint)

  • Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)

  • Security cmdlets (FAST Search Server 2010 for SharePoint)


Administration cmdlets (FAST Search Server 2010 for SharePoint)
See Also

Administration cmdlets provide control over the search system. This functionality is in addition to the search configuration tasks that you can complete with the management pages in Central Administration. Common Windows PowerShell tasks include managing configuration files and binary resources, and managing content collections.

In this section:

  • Manage resource stores

  • Manage search setting groups

  • Manage content collections

  • Manage search scopes

Manage resource stores

The FAST Search Server 2010 for SharePoint resource store is a centralized file storage area for compiled dictionaries used in search tasks such as spell checking and stemming. The resource store controls the updating and propagation of these files across FAST Search Server 2010 for SharePoint servers. You can move local files to and from the resource store with these cmdlets.

Resource cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to manage your resource files:

Task | Cmdlet
Upload a local resource to the FAST Search Server 2010 for SharePoint resource store | Add-FASTSearchResource
Download a resource from the FAST Search Server 2010 for SharePoint resource store | Get-FASTSearchResource
Delete a file from the FAST Search Server 2010 for SharePoint resource store | Remove-FASTSearchResource

Manage search setting groups

A search setting group is a collection of keywords, search settings, and user contexts. Keywords can be associated with search settings which determine how a query for a keyword is transformed before the query is executed. A search setting can be a synonym, Best Bet, Visual Best Bet, document promotion, or document demotion. For example, you could have a "product_terms" search setting group that includes keywords specific to a newly released software product. Every time that a user searches for one of those keywords, a Visual Best Bet banner could display, offering a trial copy of the product. A search setting can also be associated with a specific user context (a subset of users grouped by a common attribute in their SharePoint user profiles, such as an employee's department), to target the search settings to a specific audience or special interest group. For example, the new product banner could be displayed only for non-employees, to provide a personalized search experience for different groups of users.

To use search setting groups, first define new keywords at Site Settings > Site Collection Administration > FAST Search keywords. Associate synonyms, Best Bets, Visual Best Bets, document promotions, or document demotions with specific keywords on the FAST Search keywords page. Create user contexts on the Manage User Context page, accessed from Site Settings > Site Collection Administration. You can associate a user context with a specific keyword's search setting on the FAST Search keywords page. Finally, use the New-FASTSearchSearchSettingGroup cmdlet to name search setting groups and to associate keywords with those groups.

Note:

You will rarely have to use the search setting group cmdlets. Usually, there is a one-to-one correspondence between site collections in SharePoint and search setting groups in FAST Search Server 2010 for SharePoint, created automatically by the SharePoint front-end.

Search setting group cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to manage search setting groups:

Task | Cmdlet
Retrieve one or all FAST Search Server 2010 for SharePoint search setting groups | Get-FASTSearchSearchSettingGroup
Create a search setting group | New-FASTSearchSearchSettingGroup
Delete a search setting group | Remove-FASTSearchSearchSettingGroup

Manage content collections

A content collection is a group of items crawled by an indexing connector and stored in a search index for inclusion in a FAST Search Server 2010 for SharePoint search system. Each content collection is associated with a content source (e.g., SharePoint, Lotus Notes, etc.). When you use the integrated indexing connectors, you must have one collection per connector instance (FAST Search Content Search Service Application (SSA)). When you use the FAST Search Server 2010 for SharePoint specific connectors, you must have one collection per connector configuration. You can create, clear, delete, or retrieve collections with these cmdlets.

Note:

Be careful when you are using Clear-FASTSearchContentCollection and Remove-FASTSearchContentCollection. These cmdlets will permanently delete all items of the specified collection from the index. Instead, delete content from a collection by changing the crawl configuration for the associated indexing connector.

Content collection cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to manage your content collections:

Task | Cmdlet
Retrieve one or more FAST Search Server 2010 for SharePoint content collections | Get-FASTSearchContentCollection
Create a new content collection | New-FASTSearchContentCollection
Clear a collection of all content: delete all collection items from the index without removing the collection | Clear-FASTSearchContentCollection
Delete a content collection and all the items in the collection | Remove-FASTSearchContentCollection

Item processing cmdlets

You rarely have to use the Get-FASTSearchDocumentProcessingPipeline cmdlet because there is only one item processing pipeline defined in the system. The pipeline is named Office14 (webcluster).

Task | Cmdlet
Retrieve a FAST Search Server 2010 for SharePoint item preprocessing pipeline | Get-FASTSearchDocumentProcessingPipeline

Manage search scopes

A search scope can specify content that is important to your organization, such as all items that relate to a specific product or organizational group. Or a search scope can filter on a specific content collection or language. Search scopes are offered as drop-down selections in the search application. Search scopes are available for both Microsoft SharePoint Server and FAST Search Server 2010 for SharePoint search service applications (SSA), set at the site level.

Create search scopes

To create search scopes, define query filters with the Microsoft SharePoint Server cmdlet, New-SPEnterpriseSearchQueryScope.

To create a search scope specific to FAST Search Server 2010 for SharePoint, use the -ExtendedSearchFilter <fql expression> parameter with the New-SPEnterpriseSearchQueryScope cmdlet.

Search scope cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the Farm-level Administrators local group on the computer where Microsoft SharePoint Server is installed.

Use these SharePoint Server cmdlets to manage your search scopes:

Task | Cmdlet
Retrieve a FAST Search Server 2010 for SharePoint search scope | Get-SPEnterpriseSearchQueryScope
Create a FAST Search Server 2010 for SharePoint search scope | New-SPEnterpriseSearchQueryScope

See Also

Concepts

Index schema cmdlets (FAST Search Server 2010 for SharePoint)
Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)


Add-FASTSearchResource
See Also Example
[image]

Published: May 12, 2010

Uploads a resource to the resource store.

Add-FASTSearchResource [-Path] <String> [-FilePath] <String>

Parameters

Parameter | Required | Type | Description
Path | Required | System.String | Partial URI within the resource store where the file will be uploaded.
FilePath | Required | System.String | Path of the local resource to be uploaded.

Detailed Description

This cmdlet uploads a specified file, located in the local environment, to the Microsoft FAST Search Server 2010 for SharePoint resource store. The location of the resource store is typically specified in <FASTSearchFolder>/etc/Admin.config (where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.) Specify a unique location of the resource in the upload URI.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Add-FASTSearchResource dictionaries\spellcheck\check_en.aut c:\check_en.aut

This example uploads the file c:\check_en.aut to dictionaries\spellcheck\check_en.aut to the resource store.

See Also

Reference

Get-FASTSearchResource
Remove-FASTSearchResource


Clear-FASTSearchContentCollection
See Also Example
[image]

Published: May 12, 2010

Clears the content of a Microsoft FAST Search Server 2010 for SharePoint content collection.

Clear-FASTSearchContentCollection [-Name] <String> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the content collection to clear.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), you will not be prompted. In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).
Force | Optional | System.Management.Automation.SwitchParameter | Overrides any user prompting settings so the user is not asked to confirm the operation.
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet deletes from the index all items that are associated with the specified collection without removing the collection.

Note:

If you are crawling using the integrated indexing connectors (Content SSA), you should only run this command as part of the procedure as specified in http://technet.microsoft.com/en-us/library/ff191228.aspx.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Clear-FASTSearchContentCollection -Name newscollection

This example clears the content of the "newscollection" content collection.

---------------EXAMPLE 2-----------------

Copy Code
Clear-FASTSearchContentCollection -Name newscollection -Force

This example clears the content of the "newscollection" content collection. The user does not have to confirm the operation.

See Also

Reference

New-FASTSearchContentCollection
Remove-FASTSearchContentCollection
Get-FASTSearchContentCollection


Get-FASTSearchContentCollection
See Also Example
[image]

Published: May 12, 2010

Retrieves Microsoft FAST Search Server 2010 for SharePoint content collections.

Get-FASTSearchContentCollection [[-Name] <String>]

Parameters

Parameter | Required | Type | Description
Name | Optional | System.String | The name of the FAST Search Server 2010 for SharePoint collection to retrieve.

Detailed Description

This cmdlet retrieves all FAST Search Server 2010 for SharePoint content collections. When a collection name is specified, a single collection is retrieved, if it exists.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchContentCollection

This example retrieves all FAST Search Server 2010 for SharePoint content collections.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchContentCollection -Name newscollection

This example retrieves the "newscollection" content collection.

See Also

Reference

Clear-FASTSearchContentCollection
New-FASTSearchContentCollection
Remove-FASTSearchContentCollection


Get-FASTSearchDocumentProcessingPipeline
Example
[image]

Published: May 12, 2010

Retrieves FAST Search Server 2010 for SharePoint item processing pipelines.

Get-FASTSearchDocumentProcessingPipeline [[-Name] <String>]

Parameters

Parameter | Required | Type | Description
Name | Optional | System.String | The name of the item processing pipeline to retrieve.

Detailed Description

An item processing pipeline consists of multiple processing stages that an item goes through before being indexed for searching. Item processing includes adding/removing/modifying metadata, linguistic processing, language detection. This cmdlet retrieves all FAST Search Server 2010 for SharePoint item processing pipelines. When one pipeline name is specified, a single pipeline is retrieved if it exists.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchDocumentProcessingPipeline

This example retrieves all FAST Search Server 2010 for SharePoint item processing pipelines.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchDocumentProcessingPipeline -Name "Office14 (webcluster) "

This example retrieves a single FAST Search Server 2010 for SharePoint item processing pipeline named "Office14 (webcluster)".


Get-FASTSearchResource
See Also Example
[image]

Published: May 12, 2010

Downloads a resource from the resource store.

Get-FASTSearchResource [-Path] <String> [-FilePath] <String>

Parameters

Parameter | Required | Type | Description
Path | Required | System.String | Partial URI within the resource store where the file needs to be downloaded from.
FilePath | Required | System.String | Path to download the resource.

Detailed Description

This cmdlet downloads a specified file located in the Microsoft FAST Search Server 2010 for SharePoint resource store to the local environment. The location of the resource store is typically specified in <FASTSearchFolder>/etc/Admin.config (where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.)

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchResource dictionaries\spellcheck\check_en.aut c:\check_en.aut

This example downloads dictionaries\spellcheck\check_en.aut from the resource store to c:\check_en.aut.

See Also

Reference

Remove-FASTSearchResource
Add-FASTSearchResource


Get-FASTSearchSearchSettingGroup
See Also Example
[image]

Published: May 12, 2010

Retrieves Microsoft FAST Search Server 2010 for SharePoint search setting groups.

Get-FASTSearchSearchSettingGroup [-Name <String>]

Parameters

Parameter | Required | Type | Description
Name | Optional | System.String | The name of the search setting group to retrieve.

Detailed Description

This cmdlet retrieves one or all FAST Search Server 2010 for SharePoint search setting groups. When a group name is specified, a single group is retrieved, if it exists.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSearchSettingGroup

This example retrieves all FAST Search Server 2010 for SharePoint search setting groups.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSearchSettingGroup -Name marketinggroup

This example retrieves the "marketinggroup" search setting group.

See Also

Reference

New-FASTSearchSearchSettingGroup
Remove-FASTSearchSearchSettingGroup


New-FASTSearchContentCollection
See Also Example
[image]

Published: May 12, 2010

Creates a new Microsoft FAST Search Server 2010 for SharePoint content collection.

New-FASTSearchContentCollection [-Name] <String> [[-Description] <String>] [[-Pipeline] <String>]

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the FAST Search Server 2010 for SharePoint content collection to add.
Description | Optional | System.String | A description of the content collection to add.
Pipeline | Optional | System.String | The FAST Search Server 2010 for SharePoint item processing pipeline used to feed content to the content collection. In FAST Search Server 2010 for SharePoint the only pipeline defined is "Office 14 (webcluster)". Since this is also the default pipeline, you do not need to specify this parameter when creating a new content collection.

Detailed Description

This cmdlet creates a new FAST Search Server 2010 for SharePoint content collection.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchContentCollection -Name newscollection

This example creates a new FAST Search Server 2010 for SharePoint content collection named "newscollection".

---------------EXAMPLE 2-----------------

Copy Code
New-FASTSearchContentCollection -Name newscollection -Description "A collection for news content"

This example creates a new content collection named "newscollection", and attaches a description to it.

Note:

Creating an additional content collection uses memory. You should consider an additional memory consumption of 500 MB per content collection per running indexer node. For example, in an environment with two indexers and two content collections, the total memory consumption will be 2x(2X500MB)=2000MB, or 2 GB, divided across all indexers.

See Also

Reference

Clear-FASTSearchContentCollection
Remove-FASTSearchContentCollection
Get-FASTSearchContentCollection


New-FASTSearchSearchSettingGroup
See Also Example
[image]

Published: May 12, 2010

Creates a new Microsoft FAST Search Server 2010 for SharePoint search setting group.

New-FASTSearchSearchSettingGroup -Name <String>

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the search setting group to add.

Detailed Description

This cmdlet creates a new FAST Search Server 2010 for SharePoint search setting group.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchSearchSettingGroup -Name marketinggroup

This example creates a new FAST Search Server 2010 for SharePoint search setting group named "marketinggroup".

See Also

Reference

Get-FASTSearchSearchSettingGroup
Remove-FASTSearchSearchSettingGroup


Remove-FASTSearchContentCollection
See Also Example
[image]

Published: May 12, 2010

Deletes a Microsoft FAST Search Server 2010 for SharePoint content collection.

Remove-FASTSearchContentCollection [-Name] <String> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the content collection to delete.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), you will not be prompted. In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).
Force | Optional | System.Management.Automation.SwitchParameter | Overrides any user prompting settings so the user is not asked to confirm the operation.
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet deletes a FAST Search Server 2010 for SharePoint content collection. All items in the collection will be deleted.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchContentCollection -Name newscollection -Force

This example removes the content collection named "newscollection" without asking the user to confirm the operation.

---------------EXAMPLE 2-----------------

Copy Code
Remove-FASTSearchContentCollection -Name newscollection -WhatIf

This example describes what would happen if you executed the Remove-FASTSearchContentCollection cmdlet.

See Also

Reference

Clear-FASTSearchContentCollection
New-FASTSearchContentCollection
Get-FASTSearchContentCollection


Remove-FASTSearchResource
See Also Example
[image]

Published: May 12, 2010

Deletes a resource from the resource store.

Remove-FASTSearchResource [-Path] <String>

Parameters

Parameter | Required | Type | Description
Path | Required | System.String | Partial URI within the resource store of the file which will be deleted.

Detailed Description

This cmdlet deletes a specified file located in the Microsoft FAST Search Server 2010 for SharePoint resource store. The location of the resource store is typically specified in <FASTSearchFolder>/etc/Admin.config (where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.)

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchResource dictionaries\spellcheck\check_en.aut

This example deletes the resource dictionaries\spellcheck\check_en.aut from the resource store.

See Also

Reference

Get-FASTSearchResource
Add-FASTSearchResource


Remove-FASTSearchSearchSettingGroup
See Also Example
[image]

Published: May 12, 2010

Deletes a Microsoft FAST Search Server 2010 for SharePoint search setting group.

Remove-FASTSearchSearchSettingGroup -Name <String> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the search setting group to delete.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), you will not be prompted. In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).
Force | Optional | System.Management.Automation.SwitchParameter | Overrides any user prompting settings so the user is not asked to confirm the operation.
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet deletes a FAST Search Server 2010 for SharePoint search setting group.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchSearchSettingGroup -Name marketinggroup

This example deletes the "marketinggroup" search setting group.

---------------EXAMPLE 2-----------------

Copy Code
Remove-FASTSearchSearchSettingGroup -Name marketinggroup -Force

This example deletes the "marketinggroup" search setting group without being prompted to confirm the operation.

---------------EXAMPLE 3-----------------

Copy Code
Remove-FASTSearchSearchSettingGroup -Name marketinggroup -WhatIf

This example describes what would happen if you executed the Remove-FASTSearchSearchSettingGroup cmdlet.

See Also

Reference

New-FASTSearchSearchSettingGroup
Get-FASTSearchSearchSettingGroup


Index schema cmdlets (FAST Search Server 2010 for SharePoint)

Microsoft FAST Search Server 2010 for SharePoint provides a full set of cmdlets to customize the administration of metadata extracted from content sources and to make that metadata searchable.

Index schema management

The FAST Search Server 2010 for SharePoint index schema controls which managed properties are indexed, searchable, and available for viewing. The index schema cmdlets work with metadata extracted from content sources. Metadata can be structured content (such as a title or author from a Word document) or unstructured content based on the content of an item (such as a detected language or extracted keywords). When managing metadata, you work with crawled properties and managed properties:

  • A crawled property is metadata extracted from content sources when you run a crawler or connector.

  • A managed property is a subset of crawled properties that are searchable. To make metadata searchable, map one or more crawled properties to a managed property. You can also create a full-text index and map the content of a managed property into the full-text index.

Crawled properties can be grouped into a category when they share common attributes. The category properties then control the default behavior of the member crawled properties. For example, you may group crawled properties extracted from SharePoint items into one category.

With schema cmdlets, you can also create rank profiles to control the relevancy calculation of queries, and to adjust the components of a rank profile for improving search result relevance. A rank profile defines the sort order of a result set.

Note:

You can configure and map managed properties either with cmdlets or in the UI at Central Administration > Search Service Application: FAST Search Administration > Property Management.

In this section:

  • Manage crawled properties

  • Manage categories

  • Manage mapped properties

  • Manage managed properties

  • Manage full-text indexes

  • Manage full-text indexes mapping

  • Manage rank profiles

Manage crawled properties

A crawled property is a piece of metadata, either structured or unstructured, that is extracted from content repositories. The main purpose of discovering crawled properties is to discover content that should be searchable. Crawled properties that are searchable are mapped to managed properties.

Crawled properties are automatically discovered and reported to the index schema model when content that contains the property is fed into the system. If you want to map a crawled property before it is discovered (that is, before items are indexed), you can create the crawled property using the New-FASTSearchMetadataCrawledProperty cmdlet. This is useful if you are creating a complete deployment script for a new installation before you start a crawl, or when you are moving a deployment configuration from a staging environment to production.

There are two ways to make a crawled property searchable:

  • Use the Set-FASTSearchMetadataCrawledProperty cmdlet to specify that the crawled property will be mapped to the default full-text index.

  • Map the crawled property to a managed property. This method provides additional configuration options. They include the following:

    • Map the property to other full-text index fields.

    • Configure the managed property for search result sorting.

    • Configure the property for query refinement.

    • Make the property searchable as metadata.

    • Return the property as part of the query hit list.

Crawled property cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to specify crawled properties:

Task | Cmdlet
Retrieve one or more crawled properties from FAST Search Server 2010 for SharePoint | Get-FASTSearchMetadataCrawledProperty
Create a new crawled property from extracted metadata | New-FASTSearchMetadataCrawledProperty
Specify if a crawled property is indexed as part of the default full-text index named content | Set-FASTSearchMetadataCrawledProperty

Manage categories

A category contains several crawled properties that are grouped together for specifying common properties. The properties of the category control the default behavior of the member crawled properties. For example, you can group crawled properties extracted from Lotus Notes databases into one category.

A category can be mapped to a full-text index just as you would map individual crawled properties to a full-text index. This simplifies making all properties from one content repository searchable without mapping each property individually. This also ensures that new properties from a repository are automatically discovered and indexed.

Note:

You cannot map a category to one managed property.

You will not have to create categories often, because they are usually defined by the protocol handler of the connector. However, one common scenario for creating a new category is when you want to handle the same properties from multiple repositories differently, even if the properties were retrieved by the same connector. For example, you may have to map categories to different managed properties.

Category cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to define and manage categories:

Task | Cmdlet
Retrieve one or more categories from FAST Search Server 2010 for SharePoint | Get-FASTSearchMetadataCategory
Create a new category for crawled properties | New-FASTSearchMetadataCategory
Change the settings of a crawled property category and map a category to the content full-text index | Set-FASTSearchMetadataCategory
Delete a crawled property category from your installation | Remove-FASTSearchMetadataCategory

Manage mapped properties

A managed property is content or metadata associated with an item that may be searched or used in other ways, such as being displayed in search results or used for query refinement. To define the content of managed properties, you create mappings from one or more crawled properties to a managed property. For example, if you have two crawled properties called author and editor, you may want to map them into a name property so that the source properties are searchable as one field.

You can also map properties in the UI at Central Administration > Search Service Application: FAST Search Administration > Property Management.

Mapped property cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to associate crawled properties with managed properties:

Task | Cmdlet
Retrieve a list of crawled properties that are mapped to a managed property | Get-FASTSearchMetadataCrawledPropertyMapping
Map a crawled property to one or more managed properties | New-FASTSearchMetadataCrawledPropertyMapping
Replace a managed property's crawled property mappings with a different set of mappings | Set-FASTSearchMetadataCrawledPropertyMapping
Remove the mapping of a crawled property to a managed property while still keeping both properties in FAST Search Server 2010 for SharePoint | Remove-FASTSearchMetadataCrawledPropertyMapping

Manage managed properties

A managed property is content or metadata associated with an item that may be searched or used in other ways, such as being displayed in search results or used for query refinement. To make metadata searchable, first define the managed properties. Then, map the crawled properties that should be searchable to a managed property. At this point, the crawled metadata is searchable as a field. To search this metadata automatically as a general part of the item, map the content of a managed property into the default full-text index (known as content) for querying. You can create additional full-text indexes if you only want this mapping to apply to selected queries.

You can also configure managed properties in the UI with Central Administration > Search Service Application: FAST Search Administration > Property Management.

Managed property cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to specify managed properties:

Task | Cmdlet
Retrieve one or more managed properties | Get-FASTSearchMetadataManagedProperty
Create a new managed property in FAST Search Server 2010 for SharePoint | New-FASTSearchMetadataManagedProperty
Update the features of an existing managed property | Set-FASTSearchMetadataManagedProperty
Remove a managed property, its full-text index mappings, and its crawled property mappings from FAST Search Server 2010 for SharePoint | Remove-FASTSearchMetadataManagedProperty

Manage full-text indexes

Full-text indexes are the main searchable content of a FAST Search Server 2010 for SharePoint system (a search system can also include fielded searching).

To specify the content that should be included in the indexes, map managed properties into the full-text indexes.

Full-text index cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to manage full-text indexes:

Task | Cmdlet
Retrieve one or more full-text index objects | Get-FASTSearchMetadataFullTextIndex
Create a new full-text index in FAST Search Server 2010 for SharePoint | New-FASTSearchMetadataFullTextIndex
Enable and disable stemming in a full-text index; edit the description of a full-text index | Set-FASTSearchMetadataFullTextIndex
Remove a full-text index and its content from FAST Search Server 2010 for SharePoint while keeping the managed properties that were mapped to the full-text index | Remove-FASTSearchMetadataFullTextIndex

Manage full-text indexes mapping

To specify the metadata content that should be included in a full-text index for searching, define mappings from managed properties to full-text indexes. You can group properties into a full-text index so that searchers can query over multiple properties at the same time, and so that multiple properties are presented as one unique name in your search application. By mapping multiple properties into one index, you can also facilitate dynamic relevancy ranking of queries.

The default full-text index (known as content) is the target for free-text queries, accessed when a searcher does not specify a property or index in a query. You can create additional full-text indexes if you want selected queries to target a set of properties other than those specified in the default full-text index.

You can also configure individual managed properties for fielded searching. Fielded searching does not contribute to ranking, and is primarily used by searchers to filter or refine a search result set.

Full-text index mapping cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to map managed properties to full-text indexes:

Task | Cmdlet
Retrieve mappings from managed properties to full-text indexes | Get-FASTSearchMetadataFullTextIndexMapping
Add the content of a managed property to a full-text index by mapping the managed property into the full-text index | New-FASTSearchMetadataFullTextIndexMapping
Change the importance level of an existing full-text index mapping | Set-FASTSearchMetadataFullTextIndexMapping
Remove the mapping of a managed property to a full-text index while keeping both the managed property and the full-text index in FAST Search Server 2010 for SharePoint | Remove-FASTSearchMetadataFullTextIndexMapping

Manage rank profiles

A rank profile defines how relevancy calculations are performed when you search a full text index. A rank profile consists of several components which are weighted when calculating an item's relevance. You can adjust the weights of a profile's components to improve search result relevance. Rank profile components include the following:

  • Freshness: how the age of an item affects rank

  • Proximity: how the distance between query terms affects rank

  • Authority: how links between Web documents affect rank

  • Query authority: how user selections in query results affect rank

  • Context: how different managed properties within the associated full text indexes contribute to the rank

When a user conducts a search, if no rank profile is specified in the query, the default rank profile is used. Searchers can sort search results by a selected rank profile.

Rank profile cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to manage rank profiles:

Task | Cmdlet
Retrieve rank profiles by name | Get-FASTSearchMetadataRankProfile
Create a rank profile; use default values or base the rank profile on the properties of an existing rank profileFAST Search Server 2010 for SharePoint | New-FASTSearchMetadataRankProfile
Adjust the relevancy components of a rank profile to improve search result relevance | Set-FASTSearchMetadataRankProfile
Delete a rank profile from the FAST Search Server 2010 for SharePoint system (the default rank profile cannot be deleted) | Remove-FASTSearchMetadataRankProfile


Get-FASTSearchMetadataCategory
See Also Example
[image]

Published: May 12, 2010

Retrieves a crawled property category.

Get-FASTSearchMetadataCategory [[-Name] <String>]

Parameters

Parameter | Required | Type | Description
Name | Optional | System.String | The name of the category to retrieve. If no category is specified, all categories are returned.

Detailed Description

This cmdlet retrieves a category object from the installation. If no name is specified, it retrieves all categories in the system.

A category contains several crawled properties that are grouped together for setting properties that are common to them all.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchMetadataCategory

This example returns all available categories. This is the default behavior if you do not specify any parameters.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchMetadataCategory -Name SharePoint

This example returns one specific category, the "SharePoint" category.

---------------EXAMPLE 3-----------------

Copy Code
Get-FASTSearchMetadataCategory -Name m*

This example returns all categories with a name that begins with "m". Matching is case-insensitive.

---------------EXAMPLE 4-----------------

Copy Code
$category = Get-FASTSearchMetadataCategory -Name &quot;Enterprise Crawler&quot;
$category.GetAllCrawledProperties()|ForEach-Object {$_.Name}

This example returns all crawled properties that are mapped to the category named "Enterprise Crawler". The example first retrieves the category, and then calls the category's GetAllCrawledProperties() function to list its crawled properties. It then iterates over the list of crawled properties and selects the name of each crawled property.

See Also

Reference

Remove-FASTSearchMetadataCategory
Set-FASTSearchMetadataCategory
New-FASTSearchMetadataCategory


Get-FASTSearchMetadataCrawledProperty
See Also Example
[image]

Published: May 12, 2010

Retrieves one or more crawled properties from Microsoft FAST Search Server 2010 for SharePoint.

Get-FASTSearchMetadataCrawledProperty [[-Name] <String>] [-Count <Nullable>] [-Filter <String>] [-Offset <Nullable>]

Parameters

Parameter Required Type Description

Name

Optional

System.String

The name of crawled property to retrieve.

The Name parameter supports the following wildcard characters:

  • The [*] wildcard match any string of zero or more characters

  • The [?] wildcard match any single character

  • The [`] (backtick) character can be put in front of a wildcard character to indicate that it should be interpreted as a literal character and not as a wildcard character

You can combine the Name parameter with the Offset and Count parameters to control the range that is searched.

Count

Optional

System.Nullable

The maximum number of crawled properties to return (starting from the first returned crawled property, or from the index given by the Offset parameter).

If the Filter or Name parameter is specified, these will only be applied after the result set has been narrowed down to the range [Offset, Offset + Count].

Filter

Optional

System.String

The term used to search for configured crawled properties.

The Filter parameter does not support wildcards. Matching is case-insensitive, and matches a string anywhere within the name attribute of the crawled property.

Offset

Optional

System.Nullable

The offset in the result set from which to start returning results.

Detailed Description

This cmdlet retrieves one or more crawled properties from the FAST Search Server 2010 for SharePoint system. If the cmdlet is called without any parameters, it returns all crawled properties in the system. The order of the returned crawled property list is fixed, and multiple calls return the same ordered list as long as no new crawled properties have been added.

Note:

This cmdlet returns one or more matching crawled properties. To match exactly one crawled property, use Set-FASTSearchMetadataCrawledproperty instead. This may be needed if multiple crawled properties have the same name but are part of different categories.

Get-FASTSearchMetadataCrawledProperty supports several ways of looking up properties. If the cmdlet is run with a single parameter or with a named parameter (using Name), the cmdlet retrieves crawled properties with that exact name, if any exist. The Name parameter supports wildcard characters. Alternatively, a filter can be specified (using the Filter parameter) to search the crawled properties and to return any crawled property with a name that includes the filter value. The filter option does not support wildcard characters.

To paginate a set of results from this cmdlet, use an Offset and Count to control the number of returned results.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchMetadataCrawledProperty

Calling Get-FASTSearchMetadataCrawledProperty without any parameters returns all known crawled properties in the FAST Search Server 2010 for SharePoint system.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchMetadataCrawledProperty -Name viewurl

This example retrieves a crawled property with the exact name "viewurl".

---------------EXAMPLE 3-----------------

Copy Code
Get-FASTSearchMetadataCrawledProperty -filter 4

This example retrieves and returns any crawled property that includes a "4" in its name.

---------------EXAMPLE 4-----------------

Copy Code
Get-FASTSearchMetadataCrawledproperty -Name *url -count 2

This example retrieves the first two crawled properties that have a name ending with "url".

---------------EXAMPLE 5-----------------

Copy Code
Get-FASTSearchMetadataCrawledproperty |ForEach-Object{ if ($_.IsMappedToContents) {$_.Name} }

This example returns the names of all crawled properties that have the IsMappedToContents parameter set to "true". The example first retrieves all configured crawled properties, and then iterates over them to find those with IsMappedToContents set to "true"

---------------EXAMPLE 6-----------------

Copy Code
Get-FASTSearchMetadataCrawledproperty -offset 0 -count 2
Get-FASTSearchMetadataCrawledproperty -offset 2 -count 2
Get-FASTSearchMetadataCrawledproperty -offset 4 -count 2

This example generates a list of two crawled properties each time the cmdlet is run. The Offset is increased by the Count value to start returning crawled properties where the last run ended.

---------------EXAMPLE 7-----------------

Copy Code
Get-FASTSearchMetadataCrawledProperty -filter url| where-object {($_.Name -eq "url") -and ($_.VariantType -eq 31) -and ($_.Propset -eq "63e90878-0292-490d-8b7c-f3905a8b65fd")}

This example lists all potential matches to a named crawled property, and then selects the exact crawled property by iterating over the result set and picking out the result that also matches both the requested property set and variant type. The command returns the "url" crawled property in the "Enterprise Crawler" category.

See Also

Reference

New-FASTSearchMetadataCrawledProperty
Set-FASTSearchMetadataCrawledProperty


Get-FASTSearchMetadataCrawledPropertyMapping
See Also Example
[image]

Published: May 12, 2010

Retrieves a list of crawled properties mapped to a managed property.

Get-FASTSearchMetadataCrawledPropertyMapping -ManagedProperty <ManagedProperty>
Get-FASTSearchMetadataCrawledPropertyMapping [-Name] <String>

Parameters

1
Parameter | Required | Type | Description
ManagedProperty | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty | The managed property object for finding crawled properties.

2
Parameter | Required | Type | Description
Name | Required | System.String | The name of a managed property for finding crawled properties.

Detailed Description

This cmdlet retrieves a list of the crawled properties that are mapped to a managed property. It contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the ManagedProperty versus the Name parameter. The only difference is when using the Name parameter you look up the managed property each time, while when using the ManagedProperty parameter you use the object you already have a reference to (object as returned by Get-FASTSearchMetadataManagedProperty). However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchMetadataCrawledPropertyMapping -Name title

This example returns a list of crawled properties that are mapped to the managed property named "title".

---------------EXAMPLE 2-----------------

Copy Code
$managedproperty = Get-FASTSearchMetadataManagedProperty -name title
Get-FASTSearchMetadataCrawledPropertyMapping -ManagedProperty $managedproperty

This example first retrieves a ManagedPropertyImpl object, and then uses it to look up the corresponding crawled property mappings. It returns all crawled properties that are set up to map content into this managed property.

---------------EXAMPLE 3-----------------

Copy Code
$crawledproperty = Get-FASTSearchmetadataCrawledProperty -name title
$crawledproperty.GetMappedManagedProperties()

These commands are the opposite of Get-FASTSearchMetdataCrawledPropertyMapping; they find all managed properties that have a mapping for a specific crawled property. This is useful when assessing how modifying a crawled property will impact the search index.

See Also

Reference

New-FASTSearchMetadataCrawledPropertyMapping
Remove-FASTSearchMetadataCrawledPropertyMapping
Set-FASTSearchMetadataCrawledPropertyMapping
Get-FASTSearchMetadataCrawledProperty
Get-FASTSearchMetadataManagedProperty


Get-FASTSearchMetadataFullTextIndex
See Also Example
[image]

Published: May 12, 2010

Retrieves a full text index.

Get-FASTSearchMetadataFullTextIndex [[-Name] <String>]

Parameters

Parameter | Required | Type | Description
Name | Optional | System.String | The name of the full text index to retrieve.

Detailed Description

This cmdlet retrieves a full text index object. The full text indexes are the searchable content of a Microsoft FAST Search Server 2010 for SharePoint system. Although it is also possible to search on a specific managed property without using a full text index, full relevancy calculation is only done when searching in full text indexes.

If no name is specified, all full text indexes are retrieved.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchMetadataFullTextIndex -Name content

This example retrieves the full text index named "content".

See Also

Reference

New-FASTSearchMetadataFullTextIndex
Remove-FASTSearchMetadataFullTextIndex
Set-FASTSearchMetadataFullTextIndex


Get-FASTSearchMetadataFullTextIndexMapping
See Also Example
[image]

Published: May 12, 2010

Retrieves mappings between managed properties and the full text indexes.

Get-FASTSearchMetadataFullTextIndexMapping [-FullTextIndex <FullTextIndex>] [-ManagedProperty <ManagedProperty>]

Parameters

Parameter | Required | Type | Description
FullTextIndex | Optional | Microsoft.SharePoint.Search.Extended.Administration.Schema.FullTextIndex | The full text index specified to retrieve mappings.
ManagedProperty | Optional | Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty | The managed property specified to retrieve mappings.

Detailed Description

This cmdlet retrieves mappings from managed properties to full text indexes. These mappings control which managed properties are included in each full text index.

Running Get-FASTSearchMetadataFullTextIndexMapping with only a managed property as a parameter (the ManagedProperty parameter) returns all mappings to full text indexes in which the managed property is included.

Running Get-FASTSearchMetadataFullTextIndexMapping with only a full text index as a parameter (the FullTextIndex parameter) returns all mappings from managed properties that are included in the full text index

Running Get-FASTSearchMetadataFullTextIndexMapping with both the ManagedProperty and the FullTextIndex parameters returns a single FullTextIndexMappingImpl object for that exact mapping, if it exists in the Microsoft FAST Search Server 2010 for SharePoint system.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$fulltextindex = Get-FASTSearchMetadataFullTextIndex -name content
Get-FASTSearchMetadataFullTextIndexMapping -fulltextindex $fulltextindex| ForEach-Object {$_.ManagedProperty.Name}

This example retrieves the full text index mappings for the full text index "content". It then iterates over the mappings and outputs the names of the managed properties included in the full text index.

---------------EXAMPLE 2-----------------

Copy Code
$title = Get-FASTSearchMetadataManagedProperty -name title
Get-FASTSearchMetadataFullTextIndexMapping -managedproperty $title

This example retrieves all full text indexes where the managed property named "title" is included.

The specified managed property is mapped to both "content" and "thirdfulltextindex", showing that there is no limit to how many full text indexes a managed property can be part of.

See Also

Reference

New-FASTSearchMetadataFullTextIndexMapping
Remove-FASTSearchMetadataFullTextIndexMapping
Set-FASTSearchMetadataFullTextIndexMapping
Get-FASTSearchMetadataFullTextIndex
Get-FASTSearchMetadataManagedProperty


Get-FASTSearchMetadataManagedProperty
See Also Example
[image]

Published: May 12, 2010

Retrieves a managed property with the specified name.

Get-FASTSearchMetadataManagedProperty [[-Name] <String>]

Parameters

Parameter | Required | Type | Description
Name | Optional | System.String | The name of the managed property to retrieve. This option supports wildcards, and the matching is case-insensitive.

Detailed Description

This command retrieves managed properties that match the input parameter. If wildcards are used, the command may return more than one managed property.

A managed property is a piece of information (such as title, author, etc.) that is stored for each item in the index.

See Set-FASTSearchMetadataManagedProperty for details about the managed property parameters and how to configure them.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchMetadataManagedProperty -name title

This example retrieves a managed property named "title". It returns an object representing the managed property.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchMetadataManagedProperty -name content*

This example returns all managed properties with names starting with "content". It returns a list of managed property objects.

See Also

Reference

New-FASTSearchMetadataManagedProperty
Set-FASTSearchMetadataManagedProperty
Remove-FASTSearchMetadataManagedProperty


Get-FASTSearchMetadataRankProfile
See Also Example
[image]

Published: May 12, 2010

Retrieves rank profiles from the Microsoft FAST Search Server 2010 for SharePoint system.

Get-FASTSearchMetadataRankProfile [[-Name] <String>]

Parameters

Parameter | Required | Type | Description
Name | Optional | System.String | The name of the rank profile to retrieve. This option supports wildcards, and matching is case-insensitive.

Detailed Description

This cmdlet retrieves rank profiles which match the Name parameter. A rank profile is a weighting scheme for calculating relevance, specifying how each relevance component should be weighted when calculating an item's relevance.

See Set-FASTSearchMetadataRankProfile for details about rank profile parameters.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchMetadataRankProfile -Name newdefault

This example retrieves the rank profile named "newdefault".

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchMetadataRankProfile -Name *?default

This example retrieves all rank profiles that match the pattern "*?default". It will not match the default rankprofile named "default".

See Also

Reference

New-FASTSearchMetadataRankProfile
Remove-FASTSearchMetadataRankProfile
Set-FASTSearchMetadataRankProfile


New-FASTSearchMetadataCategory
See Also Example
[image]

Published: May 12, 2010

Creates a new category for crawled properties.

New-FASTSearchMetadataCategory -Name <String> -Propset <Guid>

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the new category.
Propset | Required | System.Guid | The property set Global Unique Identifier (GUID) assigned to this category. A GUID is a 128-bit integer that has a very low probability of being duplicated. The GUID can be specified either in the form of a System.Guid object, or as a 128-bit integer with the format "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx". A crawled property can only be mapped to one category at a time.

Detailed Description

This cmdlet creates a new category for crawled properties. A category is identified by its name and its property set global unique identifier (GUID).

All crawled properties that are members of a category share the same property set GUID as the category.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$guid = [guid]::NewGuid()
New-FASTSearchMetadataCategory -Name ExampleCategory -Propset $guid

This example creates a category named "ExampleCategory" with a new GUID generated by the system. New crawled properties can then be mapped to this category by specifying the GUID when calling New-FASTSearchMetadataCrawledProperty or Set-FASTSearchMetadataCrawledProperty.

See Set-FASTSearchMetadataCategory for descriptions of category properties.

See Also

Reference

Get-FASTSearchMetadataCategory
Remove-FASTSearchMetadataCategory
Set-FASTSearchMetadataCategory
New-FASTSearchMetadataCrawledProperty
Set-FASTSearchMetadataCrawledProperty


New-FASTSearchMetadataCrawledProperty
See Also Example
[image]

Published: May 12, 2010

Creates a new crawled property.

New-FASTSearchMetadataCrawledProperty -Name <String> -Propset <Guid> -VariantType <Int32>

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the new crawled property.
Propset | Required | System.Guid | The property set GUID of the new crawled property. The GUID controls which category the crawled property belongs to. A GUID (Global Unique Identifier) is a 128-bit integer that has a very low probability of being duplicated. The GUID can be specified either as a System.Guid object, or as an integer having the format "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx".
VariantType | Required | System.Int32 | An integer value specifying the data type of the crawled property. See http://go.microsoft.com/fwlink/p/?LinkId=163232 for a list of common variant types.

Detailed Description

This cmdlet creates a new crawled property. A crawled property is a piece of metadata extracted from content sources and fed to the Microsoft FAST Search Server 2010 for SharePoint system. For example, the title of a Web page would normally be a crawled property.

A crawled property is uniquely defined by three properties:

-- The name of the crawled property.

-- The property set to which the crawled property is mapped, specified as a GUID (Global Unique Identifier). The GUID identifies which category the crawled property belongs to. A crawled property must be a member of a category.

-- The variant type of the crawled property, which specifies the crawled property's data type.

To include content in the full text indexes, crawled properties must be mapped to managed properties. If a crawled property has IsMappedToContents set to "true", then the crawled property will automatically be added to the default searchable index (the full text index called "content"). A newly created crawled property has its IsMappedToContents parameter set to the same value as the mapped-to category's MapToContents parameter.

See New-FASTSearchMetadataCategory for information about creating a new category for crawled properties.

See Set-FASTSearchMetadataCrawledProperty for information about changing the properties of an existing crawled property.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$guid = [guid]::NewGuid()
New-FASTSearchMetadataCategory -Name MoreWeb -Propset $guid
New-FASTSearchMetadataCrawledProperty -Name ExtendedTitle -Propset $guid -Varianttype 31

This example first creates a new category named "MoreWeb", and then creates a new crawled property "ExtendedTitle" that is mapped to the "MoreWeb" category (because it has the same property set value). The data type of the crawled property is a Unicode string (VT_LPWSTR, variant type 31).

The new category gets the property set value "e20d8de2-0b5c-46f8-993d-6cdacbda5164", which was randomly generated by the System.GUID.NewGuid() method.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchMetadataCrawledProperty | Where-Object -filterscript { $_.CategoryName -eq "Web"}
New-FASTSearchMetadataCrawledProperty -Name Internaltitle -varianttype 31 -Propset d1b5d3f0-c0b3-11cf-9a92-00a0c908dbf1

This example adds a new crawled property to an existing catalog of crawled properties. The new crawled property is named "Internaltitle", added to the "Web" category. It is variant type 31, a Unicode string in the user default locale.

The "Web" category has several property set GUIDs. Investigating with the command:

Get-FASTSearchMetadataCrawledProperty | Where-Object -filterscript { $_.CategoryName -eq "Web"}

reveals that the "DESCRIPTION" crawled property, which is a type similar to the crawled property to add, has the property set value "d1b5d3f0-c0b3-11cf-9a92-00a0c908dbf1". This property set is chosen for the new crawled property.

To use this crawled property, you must configure the connector or feeding application to send a value with the same name, variant type, and property set.

See Also

Reference

Set-FASTSearchMetadataCrawledProperty
Get-FASTSearchMetadataCrawledProperty


New-FASTSearchMetadataCrawledPropertyMapping
See Also Example
[image]

Published: May 12, 2010

Creates a new mapping from a crawled property to a managed property.

New-FASTSearchMetadataCrawledPropertyMapping -CrawledProperty <CrawledProperty> -ManagedProperty <ManagedProperty>

Parameters

Parameter | Required | Type | Description
CrawledProperty | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.CrawledProperty | The crawled property that is mapped to a managed property.
ManagedProperty | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty | The managed property to which a crawled property is mapped.

Detailed Description

This cmdlet maps a crawled property to a managed property and stores the crawled content in the Microsoft FAST Search Server 2010 for SharePoint index.

You can map a crawled property to multiple managed properties. If you map multiple crawled properties to a single managed property, use the managed property's MergeCrawledProperties parameter to control whether or not the crawled properties are merged into the managed property or just overwritten. See Set-FASTSearchMetadataManagedProperty for more information.

When you map a crawled property into a managed property, the variant type of the crawled property is converted into the available data types in FAST Search Server 2010 for SharePoint. See http://go.microsoft.com/fwlink/p/?LinkId=163233 for details about this conversion, and which variant types can be converted into native FAST Search Server 2010 for SharePoint types.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$sitetitle_managedproperty = Get-FASTSearchMetadataManagedProperty -Name sitetitle
$internaltitle_crawledproperty = Get-FASTSearchMetadataCrawledProperty -name internaltitle
New-FASTSearchMetadataCrawledPropertyMapping -Managedproperty $sitetitle_managedproperty -crawledproperty $internaltitle_crawledproperty

This example adds a mapping from the crawled property "internaltitle" (here part of the Web category) to the managed property "sitetitle".

Note that both the crawled property and the managed property already exist, and these commands only add the mapping. As a result, the crawled property "internaltitle" will now be mapped to "sitetitle" and stored in the FAST Search Server 2010 for SharePoint system. The managed property can then be included in a full text index to make it searchable.

---------------EXAMPLE 2-----------------

Copy Code
$extended_title = New-FASTSearchMetadataManagedproperty -name ExtendedTitle -Type 1
$title = Get-FASTSearchMetadataManagedProperty -name title
$title.GetCrawledPropertyMappings() | ForEach-Object { New-FASTSearchMetadataCrawledPropertyMapping -ManagedProperty $extended_title -CrawledProperty $_ }

This example adds a new managed property named "ExtendedTitle", and maps to "ExtendedProperty" all crawled properties that are normally mapped to the managed property named "title". The original "title" managed property is not changed, to avoid changing the current behavior of the search system.

This example first generates a list of crawled property objects mapped to the original "title" managed property. Then the example iterates over this list using ForEach-Object to add each of them to the new "ExtendedTitle" managed property. More crawled properties can be added to this new managed property later without impacting the "title" managed property.

See Also

Reference

Get-FASTSearchMetadataCrawledPropertyMapping
Remove-FASTSearchMetadataCrawledPropertyMapping
Set-FASTSearchMetadataCrawledPropertyMapping


New-FASTSearchMetadataFullTextIndex
See Also Example
[image]

Published: May 12, 2010

Creates a new full text index.

New-FASTSearchMetadataFullTextIndex -Description <String> -Name <String>

Parameters

Parameter | Required | Type | Description
Description | Required | System.String | Description of the full text index, including the type of content or which connector feeds the data.
Name | Required | System.String | Name of the new full text index.

Detailed Description

This cmdlet creates a new full text index in Microsoft FAST Search Server 2010 for SharePoint. A full text index can be searched by specifying the full text index name in a query. For example: "extracontent:car" searches for the term "car" in the full text index named "extracontent". If no full text index is specified, the default full text index is used. A relevancy score is calculated when searching the full text index, according to the rank profile specified in the query (or the default rank profile, if no rank profile is specified).

See Set-FASTSearchMetadataFullTextIndex for details about changing the default full text index for searching.

See Set-FASTSearchMetadataRankProfile for more details about rank profiles.

Use New-FASTSearchMetadataFullTextIndexMapping to create mappings from managed properties to a new full text index.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchMetadataFullTextIndex -Name CommonSharePoint -Description "Content from the common SharePoint site."

This example creates a new full text index named "CommonSharePoint". After it has been created, you can map managed properties into the full text index. See New-FASTSearchMetadataFullTextIndexMapping for details about mapping properties into a full text index.

See Also

Reference

Remove-FASTSearchMetadataFullTextIndex
Set-FASTSearchMetadataFullTextIndex
Get-FASTSearchMetadataFullTextIndex
Set-FASTSearchMetadataRankProfile
New-FASTSearchMetadataFullTextIndexMapping


New-FASTSearchMetadataFullTextIndexMapping
See Also Example
[image]

Published: May 12, 2010

Creates a new mapping from a managed property into the full text index.

New-FASTSearchMetadataFullTextIndexMapping -FullTextIndex <FullTextIndex> -Level <Int32> -ManagedProperty <ManagedProperty>

Parameters

Parameter | Required | Type | Description
FullTextIndex | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.FullTextIndex | The full text index to add the mapping to.
Level | Required | System.Int32 | The importance level of this managed property. Allowed values are 1 through 7.
ManagedProperty | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty | The managed property being mapped to the full text index.

Detailed Description

This cmdlet adds the content of a managed property to a full text index by mapping the managed property into the full text index. The mapping is only applicable to new content; existing content must be re-fed to appear in this full text index.

The managed properties are ordered by a concept called importance level. If the number of matching documents for a query term is greater than the StopWordThreshold for the rank profile in use, the Microsoft FAST Search Server 2010 for SharePoint system retries the query, limiting the managed properties searched to those from a higher importance level.

Importance level 1 (the first level that will be used in a query) includes managed properties from all levels 1 and up. Importance level 2 includes levels 2 and up. The maximum level is 7.

This process is called drilling. See Set-FASTSearchMetadataRankProfile for more information about StopWordThreshold.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$title = Get-FASTSearchMetadataManagedProperty -name title
$fulltextindex = Get-FASTSearchMetadataFullTextIndex -Name thirdfulltextindex
New-FASTSearchMetadataFullTextIndexMapping -ManagedProperty $title -FullTextIndex $fulltextindex -Level 3

This example maps the managed property called "title" into the full text index named "thirdfulltextindex" at level 3.

After this mapping is made, the content must be re-fed to populate the full text index.

See Also

Reference

Remove-FASTSearchMetadataFullTextIndexMapping
Set-FASTSearchMetadataFullTextIndexMapping
Get-FASTSearchMetadataFullTextIndexMapping
Set-FASTSearchMetadataRankProfile
Get-FASTSearchMetadataManagedProperty
Get-FASTSearchMetadataFullTextIndex


New-FASTSearchMetadataManagedProperty
See Also Example
[image]

Published: May 12, 2010

Creates a new managed property.

New-FASTSearchMetadataManagedProperty -Name <String> -Type <Int32> [-Description <String>]

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the new managed property.
Type | Required | System.Int32 | An integer representing the data type of the new managed property. Valid values are: -- 1 (Text) -- 2 (Integer) -- 3 (Boolean) -- 4 (Float) -- 5 (Decimal) -- 6 (Datetime)
Description | Optional | System.String | A text description of the managed property explaining what the managed property is used for.

Detailed Description

This cmdlet creates a new managed property in the Microsoft FAST Search Server 2010 for SharePoint system. The managed property can be one of the following data types:

-- Text: UTF-8 encoded text string

-- Integer: Signed 64 bit integer

-- Boolean: true / false

-- Float: Double-precision 64-bit floating point, as specified in IEEE-754

-- Decimal: Integer with up to 27 decimals (number of decimals is set through the DecimalPlaces parameter of the managed property; see Set-FASTSearchMetadataManagedProperty).

-- Datetime: Date range in ISO8601 format. Valid date range is -29000-01-01T00:00:00,000 to 29000-12-31T23:59:59,999. Managed properties which have the type datetime always have sorting enabled.

See http://go.microsoft.com/fwlink/p/?LinkId=163234 for more detailed information about the valid value ranges for each of these data types, and how illegal values are handled.

For IEEE-754, see http://go.microsoft.com/fwlink/p/?LinkID=188655.

For details about the options that can be set on a managed property, see Set-FASTSearchMetadataManagedProperty.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchMetadataManagedProperty -name integervalue -type 2 -description "A managed property for integer values"

This example creates a managed property called "integervalue" of type 2 (integer). To further change the settings on the managed property, see Set-FASTSearchMetadataManagedProperty.

---------------EXAMPLE 2-----------------

Copy Code
$managedproperty = New-FASTSearchMetadataManagedProperty -Name webtitle -type 1 -description "Titles from RSS and Sitemap Files"
$rsstitle = Get-FASTSearchMetadataCrawledProperty -Name rss.title
$sitetitle = Get-FASTSearchMetadataCrawledProperty -Name sitemap.title
New-FASTSearchMetadataCrawledPropertyMapping -ManagedProperty $managedproperty -CrawledProperty $rsstitle
New-FASTSearchMetadataCrawledPropertyMapping -ManagedProperty $managedproperty -CrawledProperty $sitetitle
$content_fulltextindex = Get-FASTSearchMetadataFullTextIndex -name content
New-FASTSearchMetadataFullTextIndexMapping -FullTextIndex $content_fulltextindex -ManagedProperty $managedproperty -level 7

This example shows a complete work-flow for making a new managed property and mapping a crawled property to it. It creates a new managed property called "webtitle" to which it maps the crawled properties "rss.title" and "sitemap.title". The "webtitle" managed property is then mapped into the full text index "content" at importance level 7.

After this, you can feed documents (not shown here) with the crawled properties "rss.title" and "sitemap.title", and they will appear in the "webtitle" managed property. This managed property is then searchable as part of the full text index named "content".

To make the managed property searchable on its own, see Set-FASTSearchMetadataManagedProperty -Queryable.

See Also

Reference

Get-FASTSearchMetadataManagedProperty
Set-FASTSearchMetadataManagedProperty
Remove-FASTSearchMetadataManagedProperty


New-FASTSearchMetadataRankProfile
See Also Example
[image]

Published: May 12, 2010

Creates a new rank profile to calculate relevance in queries.

New-FASTSearchMetadataRankProfile -Name <String> [-Template <RankProfile>]

Parameters

Parameter | Required | Type | Description
Name | Required | System.String | The name of the new rank profile.
Template | Optional | Microsoft.SharePoint.Search.Extended.Administration.Schema.RankProfile | An object representing the rank profile that will serve as a template in creating the new rank profile.

Detailed Description

A rank profile controls the relevancy calculation for a query. It consists of several components which are weighted so that the relevancy calculation fits the content. See Set-FASTSearchMetadataRankProfile for more details.

A rank profile can either be created with default values, or based on another existing rank profile using the Template parameter.

The rank profile is typically specified as part of a query. If no rank profile is specified in the query, the default rank profile is used.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchMetadataRankProfile -Name newrankdistribution

This example adds a new rank profile based on the default settings. The new rank profile is called "newrankdistribution".

---------------EXAMPLE 2-----------------

Copy Code
$rankprofile_template = Get-FASTSearchMetadataRankProfile -Name newrankdistribution
New-FASTSearchMetadataRankProfile -Template $rankprofile_template -Name ExtraRankProfile

This example adds a new rank profile based on an existing rank profile called "newrankdistribution".

---------------EXAMPLE 3-----------------

Copy Code
$new_rank_profile = New-FASTSearchMetadataRankProfile -Name ImprovedDefault
$new_rank_profile.makeDefault()

This example specifies a new rank profile to use as the default rank profile for the FAST Search Server 2010 for SharePoint.

See Also

Reference

Remove-FASTSearchMetadataRankProfile
Set-FASTSearchMetadataRankProfile
Get-FASTSearchMetadataRankProfile


Remove-FASTSearchMetadataCategory
See Also Example
[image]

Published: May 12, 2010

Removes a crawled property category from the installation.

Remove-FASTSearchMetadataCategory -Category <Category> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]
Remove-FASTSearchMetadataCategory [-Name] <String> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]

Parameters

1
Parameter | Required | Type | Description
Category | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.Category | A category object to be deleted.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm $false), confirmation prompting is not used. In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).
Force | Optional | System.Management.Automation.SwitchParameter | Overrides any user prompting settings, so that confirmation of the operation is not required.
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

2
Parameter | Required | Type | Description
Name | Required | System.String | The name of the category to delete.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm $false), confirmation prompting is not used. In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).
Force | Optional | System.Management.Automation.SwitchParameter | Overrides any user prompting settings, so that confirmation of the operation is not required.
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet deletes a crawled property category from your installation. It contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the Category versus the Name parameter. The only difference is when using the Name parameter you look up the category each time, while when using the Category parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

All crawled properties mapped to this category are also removed. To avoid deleting crawled properties belonging to a category that you are about to remove, first map all crawled properties that you wish to save to a different property set using the Set-FASTSearchMetadataCrawledProperty cmdlet.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchMetadataCategory -name Sharepoint

This example deletes the category named "Sharepoint". Confirmation of the operation is requested. All crawled properties mapped to the "SharePoint" category are also removed.

---------------EXAMPLE 2-----------------

Copy Code
$category = Get-FASTSearchMetadataCategory -name Office
Remove-FASTSearchMetadataCategory -category $category -force

This example deletes a crawled property category (stored as $category). Since Force is specified, confirmation of the operation is not required.

See Also

Reference

Get-FASTSearchMetadataCategory
Set-FASTSearchMetadataCategory
New-FASTSearchMetadataCategory
Set-FASTSearchMetadataCrawledProperty


Remove-FASTSearchMetadataCrawledPropertyMapping
See Also Example
[image]

Published: May 12, 2010

Removes a crawled-property-to-managed-property mapping.

Remove-FASTSearchMetadataCrawledPropertyMapping -CrawledProperty <CrawledProperty> -ManagedProperty <ManagedProperty> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]

Parameters

Parameter Required Type Description

CrawledProperty

Required

Microsoft.SharePoint.Search.Extended.Administration.Schema.CrawledProperty

The crawled property which is having a mapping removed. The object can be retrieved by using Get-FASTSearchMetadataCrawledProperty, or by using the GetCrawledPropertyMappings() method call on a managed property object.

ManagedProperty

Required

Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty

The managed property object which is having a mapping removed. The object can be retrieved by using the Get-FASTSearchMetadataManagedPropertycmdlet.

Confirm

Optional

System.Management.Automation.SwitchParameter

Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), confirmation prompting is not used.

In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).

Force

Optional

System.Management.Automation.SwitchParameter

Overrides any user prompting settings, so that confirmation of the operation is not required.

WhatIf

Optional

System.Management.Automation.SwitchParameter

Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet removes the mapping of a crawled property to a managed property. Both the managed property and the crawled property still exist after the mapping is removed.

The cmdlet parameters are both objects, and must be retrieved separately.

The content must be recrawled/re-fed for the search index to reflect that the mapping was removed.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$title = Get-FASTSearchMetadataManagedProperty -name title
$crawledproperties = $title.GetCrawledPropertyMappings()
$crawledproperties # To List the output before the removal
Remove-FASTSearchMetadataCrawledPropertyMapping -managedproperty $title -crawledproperty $crawledproperties[4]
$crawledproperties # To list the output after the removal

This example removes the mapping of the crawled property "sitemap.title" to the managed property "title".

This example calls the GetCrawledPropertyMappings() method call exposed by the managed property, but you could also find the crawled property by using Get-FASTSearchMetadataCrawledProperty. The return value from GetCrawledPropertyMappings() is an array of crawled properties, of which the last is the "sitemap.title" crawled property mapping. The last of the crawled properties (array index 4) has the mapping removed from it by calling Remove-FASTSearchMetadataCrawledPropertyMapping.

See Also

Reference

Get-FASTSearchMetadataCrawledPropertyMapping
New-FASTSearchMetadataCrawledPropertyMapping
Set-FASTSearchMetadataCrawledPropertyMapping
Get-FASTSearchMetadataCrawledProperty
Get-FASTSearchMetadataManagedProperty


Remove-FASTSearchMetadataFullTextIndex
See Also Example
[image]

Published: May 12, 2010

Removes a full text index from the Microsoft FAST Search Server 2010 for SharePoint system.

Remove-FASTSearchMetadataFullTextIndex -FullTextIndex <FullTextIndex> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]
Remove-FASTSearchMetadataFullTextIndex -Name <String> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]

Parameters

1

Parameter

Required

Type

Description

FullTextIndex

Required

Microsoft.SharePoint.Search.Extended.Administration.Schema.FullTextIndex

A full text index object representing the full text index to remove.

Confirm

Optional

System.Management.Automation.SwitchParameter

Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), confirmation prompting is not used.

In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).

Force

Optional

System.Management.Automation.SwitchParameter

Overrides any user prompting settings, so that confirmation of the operation is not required.

WhatIf

Optional

System.Management.Automation.SwitchParameter

Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

2

Parameter

Required

Type

Description

Name

Required

System.String

The name of the full text index to remove.

Confirm

Optional

System.Management.Automation.SwitchParameter

Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), confirmation prompting is not used.

In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).

Force

Optional

System.Management.Automation.SwitchParameter

Overrides any user prompting settings, so that confirmation of the operation is not required.

WhatIf

Optional

System.Management.Automation.SwitchParameter

Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet removes a full text index. The content of the index is removed and is no longer searchable. The managed properties mapped to the full text index are still available and can be mapped to other or new full text indexes.

This cmdlet contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the FullTextIndex versus the Name parameter. The only difference is when using the Name parameter you look up the full text index each time, while when using the FullTextIndex parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

Note:

Do not delete and re-create a full text index with the same name but a different configuration. The re-created full text index will initially contain all items that were indexed in this full text index prior to deletion. The items that no longer belong in the full text index will appear in query results until the items are updated or deleted.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchMetadataFullTextIndex -name secondarycontent

This example removes the full text index named "secondarycontent" from the FAST Search Server 2010 for SharePoint system. The cmdlet asks for confirmation as long as neither the Force parameter is used or the $ConfirmPreference environment is set to a higher value than HIGH.

---------------EXAMPLE 2-----------------

Copy Code
$index=Get-FASTSearchMetadataFullTextIndex -name secondarycontent
Remove-FASTSearchMetadataFullTextIndex -FullTextIndex $index -force

This example removes the full text index represented by the $index variable. Since the Force parameter is specified, confirmation of the index removal is not used.

See Also

Reference

New-FASTSearchMetadataFullTextIndex
Set-FASTSearchMetadataFullTextIndex
Get-FASTSearchMetadataFullTextIndex


Remove-FASTSearchMetadataFullTextIndexMapping
See Also Example
[image]

Published: May 12, 2010

Removes a managed-property-to-full-text-index mapping.

Remove-FASTSearchMetadataFullTextIndexMapping [-Mapping] <FullTextIndexMapping>

Parameters

Parameter | Required | Type | Description
Mapping | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.FullTextIndexMapping | An object representing the full text index mapping to be removed.

Detailed Description

This cmdlet removes the mapping of a managed property to a full text index. The managed properties are not affected, and can still be mapped to other full text indexes.

The content must be recrawled/re-fed to reflect that managed properties have been removed from full text indexes.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$fulltextindex = Get-FASTSearchMetadataFullTextIndex -Name content
$managedproperty = Get-FASTSearchMetadataManagedProperty -Name CreatedBy
$fulltextindexmapping = Get-FASTSearchMetadataFullTextIndexMapping -ManagedProperty $managedproperty -FullTextIndex $fulltextindex
Remove-FASTSearchMetadataFullTextIndexMapping -mapping $fulltextindexmapping

This example removes the mapping of the managed property "CreatedBy" from the full text index "content". Since the cmdlet takes an object of the mapping, the example first retrieves the mapping using Get-FASTSearchMetadataFullTextIndexMapping.

See Also

Reference

New-FASTSearchMetadataFullTextIndexMapping
Set-FASTSearchMetadataFullTextIndexMapping
Get-FASTSearchMetadataFullTextIndexMapping


Remove-FASTSearchMetadataManagedProperty
See Also Example
[image]

Published: May 12, 2010

Removes a managed property from the Microsoft FAST Search Server 2010 for SharePoint system.

Remove-FASTSearchMetadataManagedProperty -ManagedProperty <ManagedProperty> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]
Remove-FASTSearchMetadataManagedProperty [-Name] <String> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]

Parameters

1

Parameter

Required

Type

Description

ManagedProperty

Required

Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty

An object that represents the managed property to remove.

Confirm

Optional

System.Management.Automation.SwitchParameter

Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), confirmation prompting is not used.

In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).

Force

Optional

System.Management.Automation.SwitchParameter

Overrides any user prompting settings, so that confirmation of the operation is not required.

WhatIf

Optional

System.Management.Automation.SwitchParameter

Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

2

Parameter

Required

Type

Description

Name

Required

System.String

The name of the managed property to remove.

Confirm

Optional

System.Management.Automation.SwitchParameter

Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), confirmation prompting is not used.

In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is equal to or greater than the ConfirmImpact setting of the cmdlet (HIGH).

Force

Optional

System.Management.Automation.SwitchParameter

Overrides any user prompting settings, so that confirmation of the operation is not required.

WhatIf

Optional

System.Management.Automation.SwitchParameter

Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This command removes a managed property and its content from the FAST Search Server 2010 for SharePoint system. If the managed property is mapped into any full text indexes, the mapping is also removed. If there are any crawled property mappings to this managed property, those mappings are also removed.

This cmdlet contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the ManagedProperty versus the Name parameter. The only difference is when using the Name parameter you look up the managed property each time, while when using the ManagedProperty parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchMetadataManagedProperty -Name title

This example removes the managed property named "title" from the system. It will ask for confirmation before going through with the removal unless Force is specified.

---------------EXAMPLE 2-----------------

Copy Code
$createdby = Get-FASTSearchMetadataManagedProperty -name CreatedBy
Remove-FASTSearchMetadataManagedProperty -ManagedProperty $createdby -Force

This example removes the managed property represented by the object $managedproperty. The object is retrieved by calling Get-FASTSearchMetaDataManagedProperty, but any other call that returns a ManagedPropertyImpl object is acceptable.

No confirmation is required, since the Force parameter is specified.

See Also

Reference

Get-FASTSearchMetadataManagedProperty
New-FASTSearchMetadataManagedProperty
Set-FASTSearchMetadataManagedProperty


Remove-FASTSearchMetadataRankProfile
See Also Example
[image]

Published: May 12, 2010

Removes a rank profile from the Microsoft FAST Search Server 2010 for SharePoint system.

Remove-FASTSearchMetadataRankProfile -RankProfile <RankProfile> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]
Remove-FASTSearchMetadataRankProfile [-Name] <String> [-Confirm [<SwitchParameter>]] [-Force <SwitchParameter>] [-WhatIf [<SwitchParameter>]]

Parameters

1
Parameter | Required | Type | Description
RankProfile | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.RankProfile | An object representing the rank profile to remove.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), confirmation prompting is not used. In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is greater than or equal to the ConfirmImpact setting of the cmdlet (HIGH).
Force | Optional | System.Management.Automation.SwitchParameter | Overrides any user prompting settings, so that confirmation of the operation is not required.
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

2
Parameter | Required | Type | Description
Name | Required | System.String | The name of the rank profile to remove.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated. If Confirm is false (-Confirm:$false), confirmation prompting is not used. In cases where Confirm is not specified, the cmdlet will prompt if the $ConfirmPreference shell variable is greater than or equal to the ConfirmImpact setting of the cmdlet (HIGH).
Force | Optional | System.Management.Automation.SwitchParameter | Overrides any user prompting settings, so that confirmation of the operation is not required.
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet removes a rank profile from the FAST Search Server 2010 for SharePoint system. It contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the RankProfile versus the Name parameter. The only difference is when using the Name parameter you look up the rank profile each time, while when using the RankProfile parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

A rank profile controls relevance calculation when searching a full text index. If a deleted rank profile is specified when searching, then the default rank profile is used.

The default rank profile cannot be deleted.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchMetadataRankProfile -Name webcontent

This example removes the rank profile "webcontent".

---------------EXAMPLE 2-----------------

Copy Code
$rankprofile = Get-FASTSearchMetadataRankProfile -Name newrankprofile
Remove-FASTSearchMetadataRankProfile -Force -RankProfile $rankprofile

This example deletes the rank profile object which represents the rank profile named "newrankprofile".

No confirmation is required, since the Force parameter is specified.

See Also

Reference

New-FASTSearchMetadataRankProfile
Set-FASTSearchMetadataRankProfile
Get-FASTSearchMetadataRankProfile


Set-FASTSearchMetadataCategory
See Also Example
[image]

Published: May 12, 2010

Updates the settings of a crawled property category.

Set-FASTSearchMetadataCategory -Category <Category> [-DiscoverNewProperties <Nullable>] [-MapToContents <Nullable>] [-NewName <String>]
Set-FASTSearchMetadataCategory -Name <String> [-DiscoverNewProperties <Nullable>] [-MapToContents <Nullable>] [-NewName <String>]

Parameters

1
Parameter | Required | Type | Description
Category | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.Category | Specifies the category to be changed.
MapToContents | Optional |  | If this parameter is set to true, all newly created or discovered properties (if automatic discovery is enabled) are added to the "content" full text index. The name of each crawled property is added to the "crawledpropertynames" managed property.
DiscoverNewProperties | Optional |  | If this parameter is set to true, previously unknown crawled properties mapped to this category (having the same property set identifier) are created and added to the category.
NewName | Optional | System.String | The new name for the category.

2
Parameter | Required | Type | Description
Name | Required | System.String | The name of the category to update.
MapToContents | Optional |  | If this parameter is set to true, all newly created or discovered properties (if automatic discovery is enabled) are added to the "content" full text index. The name of each crawled property is added to the "crawledpropertynames" managed property.
DiscoverNewProperties | Optional |  | If this parameter is set to true, previously unknown crawled properties mapped to this category (having the same property set identifier) are created and added to the category.
NewName | Optional | System.String | The new name for the category.

Detailed Description

This cmdlet changes the settings of a crawled property category. A category is a group of crawled properties, and the properties of the category control the default behavior of the member crawled properties.

This cmdlet contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the Category versus the Name parameter. The only difference is when using the Name parameter you look up the category each time, while when using the Category parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

If MapToContents is set to "true" on a crawled property category, new crawled properties added to a category are mapped to the "content" full text index. The names of the crawled properties mapped to the "content" full text index are added to the crawledpropertynames managed property. You can override this action by setting the crawled property's IsMappedToContents parameter to "false".

You can call the cmdlet with either the name of the category (Name parameter), or with a category object (Category parameter).

You can also use the cmdlet to rename a category.

Note:

You can only map a crawled property to one category at a time.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchMetadataCategory -Name Titles -DiscoverNewProperties 1 -MapToContents 1

With this example, all new crawled properties for the category "Titles" are automatically discovered and added to the category. A crawled property is considered a member of the category if it has the same property set as the category.

This example also sets MapToContents so that new crawled properties are mapped to the "content" full text index. If you do not want a newly discovered crawled property made available through the full text index, override this action by setting the specific crawled property's IsMappedToContents parameter to false.

---------------EXAMPLE 2-----------------

Copy Code
Set-FASTSearchMetadataCategory -Name ExtendedTitle -Newname Titles

This example changes the name of the category "ExtendedTitle" to "Titles".

See Also

Reference

Get-FASTSearchMetadataCategory
Remove-FASTSearchMetadataCategory
New-FASTSearchMetadataCategory


Set-FASTSearchMetadataCrawledProperty
See Also Example
[image]

Published: May 12, 2010

Updates the properties of a crawled property.

Set-FASTSearchMetadataCrawledProperty -CrawledProperty <CrawledProperty> [-IsMappedToContents <Nullable>] [-Propset <Guid>] [-VariantType <Nullable>]
Set-FASTSearchMetadataCrawledProperty -Name <String> [-IsMappedToContents <Nullable>] [-Propset <Guid>] [-VariantType <Nullable>]

Parameters

1
Parameter | Required | Type | Description
CrawledProperty | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.CrawledProperty | An object representing the crawled property to update.
Propset | Optional | System.Guid | The property set GUID of the crawled property which is being updated. The GUID controls which category the crawled property is part of. A GUID (Global Unique Identifier) is a 128-bit integer that has a very low probability of being duplicated. The accepted GUID format is "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" or a System.Guid object.
VariantType | Optional |  | An integer value specifying the variant type of the updated crawled property. See http://go.microsoft.com/fwlink/p/?LinkId=163232 for a list of common variant types.
IsMappedToContents | Optional |  | This Boolean parameter specifies whether or not the content of the crawled property should be automatically mapped into the default full text index.

2
Parameter | Required | Type | Description
Name | Required | System.String | The name of the crawled property to update.
Propset | Optional | System.Guid | The property set GUID of the crawled property which is being updated. The GUID controls which category the crawled property is part of. A GUID (Global Unique Identifier) is a 128-bit integer that has a very low probability of being duplicated. The accepted GUID format is "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx" or a System.Guid object.
VariantType | Optional |  | An integer value specifying the variant type of the updated crawled property. See http://go.microsoft.com/fwlink/p/?LinkId=163232 for a list of common variant types.
IsMappedToContents | Optional |  | This Boolean parameter specifies whether or not the content of the crawled property should be automatically mapped into the default full text index.

Detailed Description

This cmdlet specifies if a crawled property should be automatically indexed as part of the default full text index, "content".

When the IsMappedToContents parameter is set to "true", the crawled property is added to the default searchable index.

To find the crawled property to update, specify a crawled property object, or the combination of name, property set, and variant type.

This cmdlet contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the CrawledProperty versus the Name parameter. The only difference is when using the Name parameter you look up the crawled property each time, while when using the CrawledProperty parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

Input Types

Return Types

Crawled property.

Example

---------------EXAMPLE 1-----------------

Copy Code
$description = Get-FASTSearchMetadataCrawledProperty -Name Description
$description # Before
Set-FASTSearchMetadataCrawledProperty -CrawledProperty $description -IsMappedToContents 1
$description # After

This example automatically maps a crawled property to the full text index "content", making the content of the crawled property searchable.

See Also

Reference

New-FASTSearchMetadataCrawledProperty
Get-FASTSearchMetadataCrawledProperty


Set-FASTSearchMetadataCrawledPropertyMapping
See Also Example
[image]

Published: May 12, 2010

Specifies which crawled properties map into specific managed properties.

Set-FASTSearchMetadataCrawledPropertyMapping -CrawledPropertyMapping <List> -ManagedProperty <ManagedProperty>
Set-FASTSearchMetadataCrawledPropertyMapping -CrawledPropertyMapping <List> -Name <String>

Parameters

1
Parameter | Required | Type | Description
CrawledPropertyMapping | Required | System.Collections.Generic.List | The list of crawled property mappings applied to the specified managed property. The order in the list controls which crawled property is used when multiple crawled properties exist. The list must consist of one or more Microsoft.SharePoint.Search.Extended.Administration.Schema.CrawledPropertyImpl objects.
ManagedProperty | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty | An object representing a managed property.

2
Parameter | Required | Type | Description
CrawledPropertyMapping | Required | System.Collections.Generic.List | The list of crawled property mappings applied to the specified managed property. The order in the list controls which crawled property is used when multiple crawled properties exist. The list must consist of one or more Microsoft.SharePoint.Search.Extended.Administration.Schema.CrawledPropertyImpl objects.
Name | Required | System.String | The name of the managed property whose crawled property mappings you are updating.

Detailed Description

This cmdlet replaces a managed property's crawled property mappings with a set of mappings specified as a parameter (CrawledPropertyMapping). All other existing crawled property mappings for the managed property are removed.

This cmdlet contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the ManagedProperty versus the Name parameter. The only difference is when using the Name parameter you look up the managed property each time, while when using the ManagedProperty parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$title = Get-FASTSearchMetadataManagedProperty -Name title
$extendedtitle = Get-FASTSearchMetadataManagedProperty -name extendedtitle
$mappings_for_title = $title.GetCrawledPropertyMappings()
Set-FASTSearchMetadataCrawledPropertyMapping -CrawledPropertyMapping $mappings_for_title -ManagedProperty $extendedtitle

This example adds the crawled property mappings from the managed property "title" to the managed property "extendedtitle".

The example first retrieves a list of the crawled property mappings for the managed property named "title" using the GetCrawledPropertyMappings(). It then calls Set-FASTSearchMetadataCrawledPropertyMapping with this list as a parameter. This replaces any existing crawled property mappings for the managed property "extendedtitle".

See Also

Reference

Get-FASTSearchMetadataCrawledPropertyMapping
New-FASTSearchMetadataCrawledPropertyMapping
Remove-FASTSearchMetadataCrawledPropertyMapping


Set-FASTSearchMetadataFullTextIndex
See Also Example
[image]

Published: May 12, 2010

Updates the properties of a full text index.

Set-FASTSearchMetadataFullTextIndex -FullTextIndex <FullTextIndex> [-Description <String>] [-StemmingEnabled <Nullable>]
Set-FASTSearchMetadataFullTextIndex -Name <String> [-Description <String>] [-StemmingEnabled <Nullable>]

Parameters

1
Parameter | Required | Type | Description
FullTextIndex | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.FullTextIndex | An object representing the full text index to be updated.
Description | Optional | System.String | A new description to use for the full text index.
StemmingEnabled | Optional | System.Nullable | Enables or disables stemming for this full text index.

2
Parameter | Required | Type | Description
Name | Required | System.String | Name of the full text index to be updated.
Description | Optional | System.String | A new description to use for the full text index.
StemmingEnabled | Optional | System.Nullable | Enables or disables stemming for this full text index.

Detailed Description

This cmdlet enables/disables stemming and changes the description of a full text index. This cmdlet contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the FullTextIndex versus the Name parameter. The only difference is when using the Name parameter you look up the full text index each time, while when using the FullTextIndex parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

Stemming is the process of reducing words to their base form or expanding to all known forms to increase recall. Whether stemming is done through reduction or expansion depends on the language in which the content is written.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchMetadataFullTextIndex -name content -description "Central content repository"

This example updates the description of the existing full text index named "content".

---------------EXAMPLE 2-----------------

Copy Code
Set-FASTSearchMetadataFullTextIndex -Name content -StemmingEnabled 1

This example enables stemming on the full text index named "content". The content of the full text index must be recrawled/re-fed for this change to take effect.

See Also

Reference

New-FASTSearchMetadataFullTextIndex
Remove-FASTSearchMetadataFullTextIndex
Get-FASTSearchMetadataFullTextIndex


Set-FASTSearchMetadataFullTextIndexMapping
See Also Example
[image]

Published: May 12, 2010

Updates the importance level of an existing full text index mapping.

Set-FASTSearchMetadataFullTextIndexMapping -Level <Int32> -Mapping <FullTextIndexMapping>

Parameters

Parameter | Required | Type | Description
Level | Required | System.Int32 | The importance level to use for this mapping. Valid values are 1 through 7.
Mapping | Required | Microsoft.SharePoint.Search.Extended.Administration.Schema.FullTextIndexMapping | The full text index object to update.

Detailed Description

This cmdlet changes the importance level of an existing full text index mapping.

Managed properties are mapped according to an importance level. If the number of matching documents for a query term is higher than the StopWordThreshold for the rank profile in use, the Microsoft FAST Search Server 2010 for SharePoint system retries the query while limiting the search to managed properties belonging to a higher importance level than the one previously tried.

It repeats this process until it finds a level where the query term count is lower than StopWordThreshold.

Importance level 1 (the first that will be used) includes all levels from 1 and up. Importance level 2 includes all levels 2 and up. The maximum level is 7.

This process is called drilling. See Set-FASTSearchMetadataRankProfile for more information about StopWordThreshold.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$content = get-fastsearchmetadatafulltextindex -name content
$sitetitle = Get-FASTSearchMetadataManagedProperty -name sitetitle
$sitetitle_fulltextindexmapping = Get-FASTSearchMetadataFullTextIndexMapping -managedproperty $sitetitle -fulltextindex $content
Set-FASTSearchMetadataFullTextIndexMapping -mapping $sitetitle_fulltextindexmapping -level 7

This example looks up a full text index mapping for a managed property named "SiteTitle" and moves it from importance level 1 to importance level 7.

The example first retrieves the full text index object for "content" and the managed property object for "sitetitle". It then uses these objects to get the full text index mapping object with the Get-FASTSearchMetadataFullTextIndexMapping cmdlet. Finally, the example calls the Set-FASTSearchMetadataFullTextIndexMapping cmdlet to update the level.

Note:

You have to re-feed the content in order for the importance level change to take effect.

See Also

Reference

New-FASTSearchMetadataFullTextIndexMapping
Remove-FASTSearchMetadataFullTextIndexMapping
Get-FASTSearchMetadataFullTextIndexMapping


Set-FASTSearchMetadataManagedProperty
See Also Example
[image]

Published: May 12, 2010

Updates the features of a managed property.

Set-FASTSearchMetadataManagedProperty -ManagedProperty <ManagedProperty> [-DecimalPlaces <Nullable>] [-Description <String>] [-IsMapped <Nullable>] [-MaxIndexSize <Nullable>] [-MaxResultSize <Nullable>] [-MergeCrawledProperties <Nullable>] [-Queryable <Nullable>] [-RefinementEnabled <Nullable>] [-SortableType <Nullable>] [-StemmingEnabled <Nullable>] [-SummaryType <Nullable>]
Set-FASTSearchMetadataManagedProperty -Name <String> [-DecimalPlaces <Nullable>] [-Description <String>] [-IsMapped <Nullable>] [-MaxIndexSize <Nullable>] [-MaxResultSize <Nullable>] [-MergeCrawledProperties <Nullable>] [-Queryable <Nullable>] [-RefinementEnabled <Nullable>] [-SortableType <Nullable>] [-StemmingEnabled <Nullable>] [-SummaryType <Nullable>]

Parameters

1

Parameter

Required

Type

Description

ManagedProperty

Required

Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty

The managed property object to modify.

Description

Optional

System.String

A text string describing the managed property's use.

MergeCrawledProperties

Optional

Boolean

This Boolean value controls whether or not multiple crawled properties mapped to this managed property are merged when multiple crawled properties are present in the input.

If not set, only the first element of the first non-empty crawled property is stored in the managed property. The one that is chosen is the crawled property that was mapped to the managed property first.

The content must be re-fed/recrawled for this change to take effect.

This property must also be set to true to include all values from a multivalued crawled property. If set to false, only the first value from a multivalued crawled property is included.

This property must also be set to true to include all values from a multivalued crawled property. If set to false, only the first value from a multivalued crawled property is included.

Queryable

Optional

Boolean

This Boolean value specifies whether or not the managed property is directly searchable on its own (and not only as part of a full text index).

If not set, it can only be searched when included in a full text index.

The content must be recrawled/re-fed again for this change to take effect.

RefinementEnabled

Optional

Boolean

This Boolean value specifies whether or not a refiner is available for the managed property.

Refiners group search results based on the values of individual managed properties from the complete result set of a query. Refiners are often used to navigate the result set in the front-end application.

After enabling refinement on a managed property, the content must be recrawled/re-fed for the change to take effect.

SortableType

Optional

The sorting type of the managed property. Valid values are:

-- 0 ("sorting disabled") specifies that the managed property is not sortable.

-- 1 ("sorting enabled") specifies that the managed property is sortable.

-- 2 ("latent sorting") specifies that the managed property is prepared for sorting, but must be changed to "sorting enabled" for sorting to be supported.

In contrast to switching from "sorting disabled" to "sorting enabled", this does not require reprocessing of data.

Setting sorting to latent on a managed property can help reduce memory resource usage, while keeping the option of enabling sorting quickly at a later date.

If a managed property is changed from "sorting disabled" to "sorting enabled", the sort value will default to 0 for all documents that have not been recrawled/re-fed.

Managed properties of the type datetime always have sorting enabled.

SummaryType

Optional

The type of summary to return for the managed property. Valid values are:

-- 0 (summary disabled), no content is returned for this managed property.

-- 1 (static), the full value of the managed property is returned.

-- 2 (dynamic), hit highlighting is done on the content of the managed property based on the query terms, and a hit highlighted summary is returned.

The content must be recrawled/re-fed for any change in this setting to take effect, except when disabling summary generation.

StemmingEnabled

Optional

Enables or disables stemming for the managed property.

Stemming is the process of reducing words to their base form, or expanding to all known forms, to increase recall. Whether stemming is done through reduction or expansion depends on the content language.

The content must be recrawled/re-fed for this to take effect.

DecimalPlaces

Optional

This integer parameter specifies how many decimal places should be used for the managed property.

A higher number of decimals provides a more precise result, but reduces the minimum and maximum number size that can be stored in the managed property.

MaxIndexSize

Optional

This integer parameter sets the maximum number of kilobytes indexed for the managed property.

Valid values range from 0 through 2097151 (2GB).

The measured value equals the data stored prior to indexing and is slightly larger than the effective number of kilobytes that is searchable for the managed property.

MaxResultSize

Optional

This integer parameter sets the maximum number of kilobytes that can be returned for this managed property.

If a managed property's content is longer than this, then the returned version of the managed property is truncated to the specified value.

Valid values are from 0 through 2097151 (2GB).

If the summary type of the managed property is dynamic, then this size refers to the size of the source data used for generating the hit highlighted summary.

IsMapped

Optional

Boolean

This Boolean parameter specifies whether or not the managed property is mapped to support Alternate Access Mappings in SharePoint.

2

Parameter

Required

Type

Description

Name

Required

System.String

The name of the managed property to modify.

Description

Optional

System.String

A text string describing the managed property's use.

MergeCrawledProperties

Optional

Boolean

This Boolean value controls whether or not multiple crawled properties mapped to this managed property are merged when multiple crawled properties are present in the input.

If not set, only the first element of the first non-empty crawled property is stored in the managed property. The one that is chosen is the crawled property that was mapped to the managed property first.

The content must be re-fed/recrawled for this change to take effect.

This property must also be set to true to include all values from a multivalued crawled property. If set to false, only the first value from a multivalued crawled property is included.

This property must also be set to true to include all values from a multivalued crawled property. If set to false, only the first value from a multivalued crawled property is included.

Queryable

Optional

Boolean

This Boolean value specifies whether or not the managed property is directly searchable on its own (and not only as part of a full text index).

If not set, it can only be searched when included in a full text index.

The content must be recrawled/re-fed again for this change to take effect.

RefinementEnabled

Optional

Boolean

This Boolean value specifies whether or not a refiner is available for the managed property.

Refiners group search results based on the values of individual managed properties from the complete result set of a query. Refiners are often used to navigate the result set in the front-end application.

After enabling refinement on a managed property, the content must be recrawled/re-fed for the change to take effect.

SortableType

Optional

The sorting type of the managed property. Valid values are:

-- 0 ("sorting disabled") specifies that the managed property is not sortable.

-- 1 ("sorting enabled") specifies that the managed property is sortable.

-- 2 ("latent sorting") specifies that the managed property is prepared for sorting, but must be changed to "sorting enabled" for sorting to be supported.

In contrast to switching from "sorting disabled" to "sorting enabled", this does not require reprocessing of data.

Setting sorting to latent on a managed property can help reduce memory resource usage, while keeping the option of enabling sorting quickly at a later date.

If a managed property is changed from "sorting disabled" to "sorting enabled", the sort value will default to 0 for all documents that have not been recrawled/re-fed.

Managed properties of the type datetime always have sorting enabled.

SummaryType

Optional

The type of summary to return for the managed property. Valid values are:

-- 0 (summary disabled), no content is returned for this managed property.

-- 1 (static), the full value of the managed property is returned.

-- 2 (dynamic), hit highlighting is done on the content of the managed property based on the query terms, and a hit highlighted summary is returned.

The content must be recrawled/re-fed for any change in this setting to take effect, except when disabling summary generation.

StemmingEnabled

Optional

Enables or disables stemming for the managed property.

Stemming is the process of reducing words to their base form, or expanding to all known forms, to increase recall. Whether stemming is done through reduction or expansion depends on the content language.

The content must be recrawled/re-fed for this to take effect.

DecimalPlaces

Optional

This integer parameter specifies how many decimal places should be used for the managed property.

A higher number of decimals provides a more precise result, but reduces the minimum and maximum number size that can be stored in the managed property.

MaxIndexSize

Optional

This integer parameter sets the maximum number of kilobytes indexed for the managed property.

Valid values range from 0 through 2097151 (2GB).

The measured value equals the data stored prior to indexing and is slightly larger than the effective number of kilobytes that is searchable for the managed property.

MaxResultSize

Optional

This integer parameter sets the maximum number of kilobytes that can be returned for this managed property.

If a managed property's content is longer than this, then the returned version of the managed property is truncated to the specified value.

Valid values are from 0 through 2097151 (2GB).

If the summary type of the managed property is dynamic, then this size refers to the size of the source data used for generating the hit highlighted summary.

IsMapped

Optional

Boolean

This Boolean parameter specifies whether or not the managed property is mapped to support Alternate Access Mappings in SharePoint.

Detailed Description

This cmdlet updates the properties/features of an existing managed property. This cmdlet contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the ManagedProperty versus the Name parameter. The only difference is when using the Name parameter you look up the managed property each time, while when using the ManagedProperty parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

Several of these changes require that the content be recrawled/re-fed to take full effect. If, for example, a managed property is made searchable (Queryable) it will only become searchable after the content is updated. During the recrawl/re-feed, the new content will be searchable, while the old will not.

For properties such as whether the managed property should have sorting enabled (SortableType), the managed property defaults to the value '0' until the content has been re-fed/recrawled.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
$title = Get-FASTSearchMetadataManagedProperty -name title
Set-FASTSearchMetadataManagedProperty -ManagedProperty $title -Refinement 1

This example enables refinement for the managed property "title". Content must be re-fed/recrawled to populate the refinement. Until it is populated, the refinement defaults to having no value.

---------------EXAMPLE 2-----------------

Copy Code
$title = Get-FASTSearchMetadataManagedProperty -name title
Set-FASTSearchMetadataManagedProperty -ManagedProperty $title -Queryable 1

This example enables searching against the "title" managed property. A front-end application can search in the specified managed property using the syntax: "title:<search terms>".

Relevancy is not calculated for such queries (unlike searches against the full text indexes), and only the quality component of the relevancy score is available.

The content must be recrawled/re-fed for this to take effect.

---------------EXAMPLE 3-----------------

Copy Code
$title = Get-FASTSearchMetadataManagedProperty -name title
Set-FASTSearchMetadataManagedProperty -ManagedProperty $title -SummaryType 1

This example changes the summary type for the managed property named "title" from dynamic to static. A static summary type means that the entire contents of the managed property is returned as part of the item summary.

The content must be recrawled/re-fed for this to take effect.

See Also

Reference

Get-FASTSearchMetadataManagedProperty
New-FASTSearchMetadataManagedProperty
Remove-FASTSearchMetadataManagedProperty


Set-FASTSearchMetadataRankProfile
See Also Example
[image]

Published: May 12, 2010

Reconfigures a rank profile.

Set-FASTSearchMetadataRankProfile -RankProfile <RankProfile> [-AuthorityWeight <Nullable>] [-FreshnessManagedPropertyReference <ManagedProperty>] [-FreshnessResolution <Nullable>] [-FreshnessWeight <Nullable>] [-PositionStopWordThreshold <Nullable>] [-QualityWeight <Nullable>] [-QueryAuthorityWeight <Nullable>] [-RankModelName <String>] [-StopWordThreshold <Nullable>]
Set-FASTSearchMetadataRankProfile -Name <String> [-AuthorityWeight <Nullable>] [-FreshnessManagedPropertyReference <ManagedProperty>] [-FreshnessResolution <Nullable>] [-FreshnessWeight <Nullable>] [-PositionStopWordThreshold <Nullable>] [-QualityWeight <Nullable>] [-QueryAuthorityWeight <Nullable>] [-RankModelName <String>] [-StopWordThreshold <Nullable>]

Parameters

1

Parameter

Required

Type

Description

RankProfile

Required

Microsoft.SharePoint.Search.Extended.Administration.Schema.RankProfile

An object representing a rank profile.

AuthorityWeight

Optional

System.Nullable

This integer parameter sets the authority weight component of the rank profile.

Ranking on authority takes the connections between items into account when calculating a relevancy score. Items that are central and frequently linked to receive a higher authority relevancy score than items that are hard to reach and less frequently linked to.

The authority relevancy score is calculated regularly by the Webanalyzer component.

FreshnessManagedPropertyReference

Optional

Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty

This parameter specifies which managed properties to use when calculating freshness.

The managed property must be represented with a ManagedPropertyImpl object (e.g., as returned from Get-FASTSearchMetadataManagedProperty).

The managed property must be of the type "datetime" and have sorting enabled.

The value of the managed property is used when calculating how fresh the document is. Items with an older datetime value in the specified managed property get a lower freshness relevancy score.

FreshnessResolution

Optional

System.Nullable

This integer parameter sets the size of the freshness boost tick.

The tick is the smallest amount of time the freshness relevancy considers to be a different value.

If set to an hour, all documents with a freshness managed property value within the same hour receive the same freshness relevancy score.

If set to a day, documents with the same day value in the freshness managed property are considered equally fresh.

Valid values are:

0 (Second)

1 (Minute)

2 (Hour)

3 (Day)

4 (Year)

FreshnessWeight

Optional

System.Nullable

This integer parameter sets the weight of the freshness relevancy component when calculating the relevance of a result.

The freshness rank value goes down the older the document is.

See the FreshnessManagedPropertyReference parameter to change the managed property used for calculating freshness.

Set this value to 0 to ignore item freshness when calculating relevancy.

PositionStopWordThreshold

Optional

System.Nullable

This integer parameter sets the position stop word threshold.

If a query term occurs more often than position-stop-word-threshold (independent of the number of items it occurs in), then proximity relevancy calculations are not done for that term.

If the query term count does not exceed the position stop word threshold, an extra rank score is added if query terms are positioned close to each other in the managed properties.

If you do not want to use proximity as part of the relevancy model, set this parameter to 0 to disable proximity calculation. This will decrease CPU use when searching.

QualityWeight

Optional

System.Nullable

This integer parameter sets the weight of the quality component of the relevancy model.

Quality denotes the assigned importance of a document. The quality metrics are assigned to individual documents without taking into account the query being evaluated.

An excessively high quality weight makes item ranking less dependent on the user query terms, which can lead to a poor search experience, since items with a high quality relevancy score could outweigh items that are more relevant to the query terms.

The quality relevancy score is calculated dynamically, based on one or more quality components. The GetQualityComponents() method available on the RankProfileImpl object lists the managed properties used for this.

QueryAuthorityWeight

Optional

System.Nullable

This integer parameter sets the weight of the query authority relevancy component.

The query authority increases for an item when a user clicks through to it from search results. The more popular the item is for a given query, the higher the query authority relevancy score.

RankModelName

Optional

System.String

The name of a rank profile to modify.

StopWordThreshold

Optional

System.Nullable

This integer parameter sets the stop word threshold of the rank profile.

A stop word is a search term that is so common in the result set that it is not counted as part of the relevancy calculation.

When a query term exceeds this threshold, FAST Search Server 2010 for SharePoint retries the query with a higher full text index importance level until it can find a level where the query term is not a stop word (see Set-FASTSearchMetadataFullTextIndexMapping for details about importance levels).

If this is not possible, the query term is not included in the result set's relevancy. A low StopWordThreshold value gives better search performance, but a lower result set relevancy (since there is a bigger chance that a query term does not influence which items are in the result set).

2

Parameter

Required

Type

Description

Name

Required

System.String

The name of a rank model configuration, controlling low-level rank tuning parameters.

Normally, you do not need to change from the default rank model named "default".

AuthorityWeight

Optional

System.Nullable

This integer parameter sets the authority weight component of the rank profile.

Ranking on authority takes the connections between items into account when calculating a relevancy score. Items that are central and frequently linked to receive a higher authority relevancy score than items that are hard to reach and less frequently linked to.

The authority relevancy score is calculated regularly by the Webanalyzer component.

FreshnessManagedPropertyReference

Optional

Microsoft.SharePoint.Search.Extended.Administration.Schema.ManagedProperty

This parameter specifies which managed properties to use when calculating freshness.

The managed property must be represented with a ManagedPropertyImpl object (e.g., as returned from Get-FASTSearchMetadataManagedProperty).

The managed property must be of the type "datetime" and have sorting enabled.

The value of the managed property is used when calculating how fresh the document is. Items with an older datetime value in the specified managed property get a lower freshness relevancy score.

FreshnessResolution

Optional

System.Nullable

This integer parameter sets the size of the freshness boost tick.

The tick is the smallest amount of time the freshness relevancy considers to be a different value.

If set to an hour, all documents with a freshness managed property value within the same hour receive the same freshness relevancy score.

If set to a day, documents with the same day value in the freshness managed property are considered equally fresh.

Valid values are:

0 (Second)

1 (Minute)

2 (Hour)

3 (Day)

4 (Year)

FreshnessWeight

Optional

System.Nullable

This integer parameter sets the weight of the freshness relevancy component when calculating the relevance of a result.

The freshness rank value goes down the older the document is.

See the FreshnessManagedPropertyReference parameter to change the managed property used for calculating freshness.

Set this value to 0 to ignore item freshness when calculating relevancy.

PositionStopWordThreshold

Optional

System.Nullable

This integer parameter sets the position stop word threshold.

If a query term occurs more often than position-stop-word-threshold (independent of the number of items it occurs in), then proximity relevancy calculations are not done for that term.

If the query term count does not exceed the position stop word threshold, an extra rank score is added if query terms are positioned close to each other in the managed properties.

If you do not want to use proximity as part of the relevancy model, set this parameter to 0 to disable proximity calculation. This will decrease CPU use when searching.

QualityWeight

Optional

System.Nullable

This integer parameter sets the weight of the quality component of the relevancy model.

Quality denotes the assigned importance of a document. The quality metrics are assigned to individual documents without taking into account the query being evaluated.

An excessively high quality weight makes item ranking less dependent on the user query terms, which can lead to a poor search experience, since items with a high quality relevancy score could outweigh items that are more relevant to the query terms.

The quality relevancy score is calculated dynamically, based on one or more quality components. The GetQualityComponents() method available on the RankProfileImpl object lists the managed properties used for this.

QueryAuthorityWeight

Optional

System.Nullable

This integer parameter sets the weight of the query authority relevancy component.

The query authority increases for an item when a user clicks through to it from search results. The more popular the item is for a given query, the higher the query authority relevancy score.

RankModelName

Optional

System.String

The name of a rank profile to modify.

StopWordThreshold

Optional

System.Nullable

This integer parameter sets the stop word threshold of the rank profile.

A stop word is a search term that is so common in the result set that it is not counted as part of the relevancy calculation.

When a query term exceeds this threshold, FAST Search Server 2010 for SharePoint retries the query with a higher full text index importance level until it can find a level where the query term is not a stop word (see Set-FASTSearchMetadataFullTextIndexMapping for details about importance levels).

If this is not possible, the query term is not included in the result set's relevancy. A low StopWordThreshold value gives better search performance, but a lower result set relevancy (since there is a bigger chance that a query term does not influence which items are in the result set).

Detailed Description

This cmdlet adjusts the relevancy components of a rank profile to improve search result relevance. This cmdlet contains more than one parameter set. You may only use parameters from one parameter set, and you may not combine parameters from different parameter sets. When doing simple operations, there is no difference between using the RankProfile versus the Name parameter. The only difference is when using the Name parameter you look up the rank profile each time, while when using the RankProfile parameter you use the object you already have a reference to. However, when dealing with operations involving a large set of objects (for example iterating over all categories), avoiding extra lookups by name can be slightly more efficient. For more information about how to use parameter sets, see Cmdlet Parameter Sets.

All weights are relative to each other. If QualityWeight is 400, and AuthorityWeight is 100, this means that QualityWeight is 4 times more important than AuthorityWeight.

After being multiplied by the weights determined in the rank profile, all relevance components are combined into a single rank score, which decides the sorting of the result set (as long as non-relevancy sorting is not specified). All changes are effective immediately, and the content does not need to be re-fed for this to take effect.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchMetadataRankProfile -Name ExtraRankProfile -QualityWeight 100 -AuthorityWeight 200 -QueryAuthorityWeight 10 -FreshnessWeight 0

This example changes the weights of the relevancy components on the rank profile named "ExtraRankProfile". You can specify one or more weights to modify at a time.

This example turns off the freshness relevancy component by setting the weight to 0, while adjusting the other components.

---------------EXAMPLE 2-----------------

Copy Code
$processingtime = Get-FASTSearchMetadataManagedProperty -name Processingtime
Set-FASTSearchMetadataRankProfile -Name ExtraRankProfile -FreshnessManagedPropertyReference $processingtime

This example changes the "ExtraRankProfile" rank profile so that it no longer determines document freshness based on the "Write" managed property.

Instead, freshness is based on when the item was last processed (fed) into the FAST Search Server 2010 for SharePoint system. The time the item was processed is stored in the "processingtime" managed property by default.

---------------EXAMPLE 3-----------------

Copy Code
Set-FASTSearchMetadataRankProfile -Name ExtraRankProfile -StopWordThreshold 100000

This example sets the stop word threshold of the rank profile named "ExtraRankProfile" to 100000. The stop word threshold determines if the query term is too common to be taken into account when calculating the relevancy of an item.

Lowering the stop word threshold means that more terms may be ignored when calculating relevancy, which can have a positive impact on performance.

---------------EXAMPLE 4-----------------

Copy Code
$rankprofile = Get-FASTSearchMetadataRankProfile -Name default
$rankprofile.GetQualityComponents()

The quality rank metric is an importance score assigned to a document, independent of query terms. This example lists the managed properties involved in calculating the quality rank by calling the GetQualityComponents() method call on the rank profile.

---------------EXAMPLE 5-----------------

Copy Code
$new_rank_component = New-FASTSearchMetadataManagedProperty -Name sitecredibility
$rankprofile = Get-FASTSearchMetadataRankProfile -Name default
$rankprofile.CreateQualityComponent($new_rank_component,100)
$rankprofile.GetQualityComponents()

The quality rank metric is an importance score assigned to a document, independent of query terms. This example adds another managed property, named "sitecredibility", to the list of quality components which form the quality rank score. The "sitecredibility" managed property is weighted the same as the other quality components (100).

The next step would be to populate "sitecredibility" with a suitable crawled property.

See Also

Reference

New-FASTSearchMetadataRankProfile
Remove-FASTSearchMetadataRankProfile
Get-FASTSearchMetadataRankProfile
Get-FASTSearchMetadataManagedProperty


Installation cmdlets (FAST Search Server 2010 for SharePoint)

Installation cmdlets provide specialized functionality for your FAST Search Server 2010 for SharePoint configuration. Use them to reinstall an administration database that was corrupted, to create firewall rules, or to configure the FAST Search Server 2010 for SharePoint local instance. You will rarely have to use these cmdlets.

Installation cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Task | Cmdlet
Install the database used by the FAST Search Server 2010 for SharePoint administration services. Reinstall (destructively) the database schema. Upgrade the database schema from an earlier version. | Install-FASTSearchAdminDatabase
Uninstall the administration services database and all the data in it. | Uninstall-FASTSearchAdminDatabase
Create or delete firewall rules to configure Internet Protocol security (IPsec) for the local machine. | Set-FASTSearchIPSec
Configure the local instance of FAST Search Server 2010 for SharePoint. | Set-FASTSearchConfiguration


Install-FASTSearchAdminDatabase
See Also Example
[image]

Published: May 12, 2010

Installs the database used by the Microsoft FAST Search Server 2010 for SharePoint administration services.

Install-FASTSearchAdminDatabase [-DbConnection <String>] [-DbName <String>] [-DbServer <String>] [-Force <SwitchParameter>] [-Upgrade <$true | $false>]

Parameters

Parameter Required Type Description

DbConnection

Optional

System.String

Specifies the database connection string for the FAST Search Server 2010 for SharePoint administration database, in the same format as the ConnectionString property of the System.Data.SqlClient.SqlConnection .NET Framework type.

DbName

Optional

System.String

Specifies the name of the FAST Search Server 2010 for SharePoint administration database.

The name only takes effect when used in conjunction with the DbServer parameter, not the DbConnection parameter.

DbServer

Optional

System.String

Specifies the database server instance that contains the FAST Search Server 2010 for SharePoint administration database.

Only one of the DbServer and DbConnection parameters must be specified. Using DbServer with a value of sqlserver.contoso.com is equivalent to using DbConnection with the value "Data Source=sqlserver.contoso.com;Database=FASTSearchAdminDatabase;integrated Security=True"

Force

Optional

System.Management.Automation.SwitchParameter

The Force parameter specifies that the FAST Search Server 2010 for SharePoint administration database already exists, but should be reinstalled.

This will delete all data that already exists in the database.

Upgrade

Optional

System.Boolean

Specifies that the FAST Search Server 2010 for SharePoint administration database already exists and that it should be upgraded, because it uses an older version of the database schema.

The upgrade process will attempt to migrate all data to the new database schema.

Important:

It is strongly recommended that you back up the database before performing this operation.

Detailed Description

This cmdlet installs the database used by the FAST Search Server 2010 for SharePoint administration services. This cmdlet is primarily called during installation, by the FAST Search Server 2010 for SharePoint installer.

The administration database default name is "FASTSearchAdminDatabase" and should be located in the Microsoft SQL Server instance specified by the DbServer or DbConnection parameters.

In addition to performing an initial installation of the database schema, this cmdlet can also:

-- Reinstall (destructively) the database schema if it already exists.

-- Upgrade the database schema from an earlier version.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Install-FASTSearchAdminDatabase -DbConnection "Data Source=sqlserver.contoso.com;Database=FASTSearchAdminDatabase;integrated Security=True; "

This example installs the database schema in the database server specified by the connection string.

---------------EXAMPLE 2-----------------

Copy Code
Install-FASTSearchAdminDatabase -Upgrade -DbServer "sqlserver.contoso.com"

This example upgrades the database schema already installed in the FAST Search Server 2010 for SharePoint administration database on the given SQL Server instance.

---------------EXAMPLE 3-----------------

Copy Code
Install-FASTSearchAdminDatabase -Force -DbServer "sqlserver.contoso.com"

This example deletes and reinstalls the database schema already installed in the FAST Search Server 2010 for SharePoint administration database on the given SQL Server instance.

See Also

Reference

Uninstall-FASTSearchAdminDatabase


Set-FASTSearchConfiguration
Example
[image]

Published: May 15, 2010

Configures the local instance of Microsoft FAST Search Server 2010 for SharePoint.

Set-FASTSearchConfiguration [-LogFile <String>]

Parameters

Parameter | Required | Type | Description
LogFile | Optional | System.String | Specifies the full path of a file where all log messages will be written.

Detailed Description

This cmdlet configures the local instance of FAST Search Server 2010 for SharePoint.

FAST Search Server 2010 for SharePoint must be installed, but not running, on the local machine before you run this cmdlet.

If you are not running this cmdlet on the admin node in the FAST Search Server 2010 for SharePoint installation, make sure that the admin node is started before you run this cmdlet.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FastSearchConfiguration -LogFile C:\log.txt

This example configures the local machine and writes the log output to C:\log.txt.


Set-FASTSearchIPSec
Example
[image]

Published: May 15, 2010

Configures local Internet Protocol security (IPsec) for Microsoft FAST Search Server 2010 for SharePoint.

Set-FASTSearchIPSec -Create <SwitchParameter> -Password <SecureString> -UserName <String> [-LogFile <String>]
Set-FASTSearchIPSec -Remove <SwitchParameter> [-LogFile <String>]

Parameters

1
Parameter | Required | Type | Description
Create | Required | System.Management.Automation.SwitchParameter | Specifies that IPsec rules will be created.
Password | Required | System.Security.SecureString | Specifies the password of the domain user that can look up machines in the domain controller.
UserName | Required | System.String | Specifies the username of the domain user that can look up machines in the domain controller.
LogFile | Optional | System.String | Specifies the full path of a file where all log messages will be written.

2
Parameter | Required | Type | Description
Remove | Required | System.Management.Automation.SwitchParameter | Specifies that IPsec rules will be deleted.
LogFile | Optional | System.String | Specifies the full path of a file where all log messages will be written.

Detailed Description

This cmdlet deletes or creates firewall rules to configure IP security for the local machine.

FAST Search Server 2010 for SharePoint must be installed on the local machine before you run this cmdlet.

If you are not running this cmdlet on the admin node in the FAST Search Server 2010 for SharePoint installation, make sure that the admin node is started before you run this cmdlet.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchIPSec -Create -UserName user1 -Password xxx

This example sets up Internet Protocol security on the local machine.

---------------EXAMPLE 2-----------------

Copy Code
Set-FASTSearchIPSec -Remove

This example removes Internet Protocol security rules from the local machine.


Uninstall-FASTSearchAdminDatabase
See Also Example
[image]

Published: May 12, 2010

Uninstalls the database used by the Microsoft FAST Search Server 2010 for SharePoint administration services. This cmdlet is primarily called during uninstallation by the FAST Search Server 2010 for SharePoint installer.

Uninstall-FASTSearchAdminDatabase [-DbConnection <String>] [-DbName <String>] [-DbServer <String>]

Parameters

Parameter Required Type Description

DbConnection

Optional

System.String

Specifies the database connection string for the FAST Search Server 2010 for SharePoint administration database, in the same format as the ConnectionString property of the System.Data.SqlClient.SqlConnection .NET Framework type.

DbName

Optional

System.String

Specifies the name of the FAST Search Server 2010 for SharePoint administration database.

The name only takes effect when used in conjunction with the DbServer parameter, not the DbConnection parameter.

DbServer

Optional

System.String

Specifies the database server instance that contains the FAST Search Server 2010 for SharePoint administration database

Only one of the DbServer and DbConnection parameters must be specified. Using DbServer with a value of sqlserver.contoso.com is equivalent to using DbConnection with the value "Data Source=sqlserver.contoso.com;Database=FASTSearchAdminDatabase;integrated Security=True""

Detailed Description

This cmdlet uninstalls the database used by the FAST Search Server 2010 for SharePoint administration services. All data is deleted. This cmdlet is primarily called during uninstallation, by the FAST Search Server 2010 for SharePoint installer.

The administration database default name is "FASTSearchAdminDatabase" and is located in the Microsoft SQL Server instance specified by the DbServer or DbConnection parameters.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Uninstall-FASTSearchAdminDatabase -DbServer=sqlserver.contoso.com

This example uninstalls the FAST Search Server 2010 for SharePoint administration database on the given SQL Server instance. All data will be deleted.

See Also

Reference

Install-FASTSearchAdminDatabase


Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)
See Also

Microsoft FAST Search Server 2010 for SharePoint provides a set of cmdlets to control spell tuning, used to fine-tune your spell checking dictionaries against the content of the processed documents.

Manage spell tuning

Spell tuning is the process that fine-tunes the spell checking dictionaries in a FAST Search Server 2010 for SharePoint system. Spell checking is a process that looks for misspelled query terms based on the spelling dictionaries and suggests alternative spellings to end users through the Did You Mean search feature, whereas spell tuning is a process that you enable to automatically update the spelling dictionaries.

When spell tuning is enabled, spell checking dictionaries are aligned with the frequency of words in the processed documents. This ensures that searchers only get spelling check suggestions that are relevant within the processed content. Spell checking is implemented by the Did You Mean feature in a FAST Search Server 2010 for SharePoint search application. Spell checking is dictionary based; when spell tuning is enabled, the Did You Mean feature only suggests terms that are actually in the processed documents, and prioritizes terms that occur more frequently.

For performance reasons, deletions of documents or of an entire document collection are not tracked. If a substantial amount of documents or an entire collection has been deleted, you should reset the spell checking dictionaries to their original state by using the Reset-FASTSearchSpelltuning cmdlet.

When you disable spell tuning, new dictionaries are no longer created, and the Did You Mean feature draws suggestions from the standard dictionaries which were delivered in the original installation.

Note:

You can also override dictionary-based spell checking for specific words using the Central Administration > Search Service Application: FAST Search Administration > Spell Checking page. The Central Administration Spell Checking feature lets you add to the spell checking exception list (words that are never offered as suggestions to searchers).

Spell tuning cmdlets

Use these cmdlets to manage spell tuning:

Task | Cmdlet
Enable the spell tuning processor, to align the spell checking dictionaries with the processed documents. | Add-FASTSearchSpelltuning
Retrieve the status of spell tuning, either active or inactive | Get-FASTSearchSpelltuningStatus
Replace the Did You Mean spell checking dictionaries with the standard dictionaries | Reset-FASTSearchSpelltuning
Disable spell tuning and stop the generation of new dictionaries | Remove-FASTSearchSpelltuning
Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

See Also

Concepts

Identify unmapped crawled properties by using Windows PowerShell (FAST Search Server 2010 for SharePoint)


Add-FASTSearchSpelltuning
See Also Example
[image]

Published: May 12, 2010

Enables the spell tuning processor, used to fine-tune a spell check dictionary.

Add-FASTSearchSpelltuning

Parameters

Detailed Description

This cmdlet activates alignment of the spell checking dictionary with the index, so that spell checking dictionaries are built from content that is fed into the index (not from standardized spelling dictionaries).

The Did You Mean feature is dictionary based; when spell tuning is activated, Did You Mean only suggests terms that are actually in the index.

Input Types

Return Types

Example

Copy Code

  See Also
Reference
Get-FASTSearchSpelltuningStatus
Remove-FASTSearchSpelltuning
Reset-FASTSearchSpelltuning


Get-FASTSearchSpelltuningStatus
See Also Example
[image]

Published: May 12, 2010

Retrieves the status of spell tuning.

Get-FASTSearchSpelltuningStatus

Parameters

Detailed Description

This cmdlet returns the status of the automated alignment of the spell checking dictionary with the index; spell tuning is either active or inactive.

Input Types

Return Types

Example

Copy Code

  See Also
Reference
Reset-FASTSearchSpelltuning
Add-FASTSearchSpelltuning
Remove-FASTSearchSpelltuning


Remove-FASTSearchSpelltuning
See Also Example
[image]

Published: May 12, 2010

Disables spell tuning.

Remove-FASTSearchSpelltuning

Parameters

Detailed Description

This cmdlet disables the alignment of the spell check dictionaries with indexed content. This cmdlet stops the generation of new dictionaries, and the Did You Mean feature will only use content that was fed into the index before this cmdlet was run.

After disabling the dictionary-index alignment, use the Reset-FASTSearchSpelltuning cmdlet to return to the out-of-the-box spelling dictionaries. Did You Mean will then make suggestions based on standard spell checking dictionaries, not on indexed content.

Input Types

Return Types

Example

Copy Code

  See Also
Reference
Get-FASTSearchSpelltuningStatus
Reset-FASTSearchSpelltuning
Add-FASTSearchSpelltuning


Reset-FASTSearchSpelltuning
See Also Example
[image]

Published: May 12, 2010

Resets spell tuning.

Reset-FASTSearchSpelltuning

Parameters

Detailed Description

This cmdlet replaces the Did You Mean spell checking dictionaries with the default dictionaries which were delivered in the original installation. All changes made by the automated alignment of the spell check dictionary with the index are deleted.

The Did You Mean feature will now suggest terms generated from standardized dictionaries, not indexed content.

After a reset, to re-align the dictionaries with indexed content once again, you must re-crawl your content sources.

Input Types

Return Types

Example

Copy Code

  See Also
Reference
Get-FASTSearchSpelltuningStatus
Remove-FASTSearchSpelltuning
Add-FASTSearchSpelltuning


Security cmdlets (FAST Search Server 2010 for SharePoint)

Microsoft FAST Search Server 2010 for SharePoint provides a full set of cmdlets to manage and control item level security through the FAST Search Authorization (FSA) component. By using item level security, users only gain access to search results that they are entitled to see.

FAST Search Authorization

Item level security is enforced in two phases:

  • Phase 1: Indexing - Content repositories are traversed and indexes are created. Authorization information is added to each item's authorization managed properties (the item's ACL, or access control list), identifying users and groups that are granted or denied access to the item.

  • Phase 2: Searching - A user submits a query and the indexes are used to determine search results. In this phase, the query processing service rewrites the user's query so that the user only sees items that he is authorized to see. This security trimming is performed with the help of a user search security filter, which is created by FSA based on the user's specific content permissions. By checking item managed properties (item ACLs which define who has permissions to view each item) against that user's search security filter, inappropriate search results are filtered out.

FSA has two components: the FSA manager (one per FAST Search Server 2010 for SharePoint system) and the FSA worker (one on each server that processes queries).

  • The FSA manager service receives security changes from indexing connectors and pushes the updates to all the query processing nodes in the system. The FSA manager also keeps the security-related configuration of these nodes consistent by administering the FSA workers and synchronizing changes across the nodes.

  • The FSA worker is part of the Query and Result Service (query processing node). FSA workers generate user search security filters based on user credentials. To do this, the FSA worker obtains group membership information for users from the FSA user stores by using principal aliasing to map users/groups from one store to another.

Note:

Note that FSA does not authenticate users. Authentication is performed by the SharePoint Server search front-end. See Plan authentication methods (SharePoint Server 2010).

In this section:

  • Manage general FAST Search Authorization settings

  • Manage claims-based security user stores

  • Manage Lotus Notes security user stores

  • Manage principal alias mapping

Manage general FAST Search Authorization settings

There are many times when you must check or change general configuration settings for FAST Search Authorization (FSA). Several cmdlets address general settings such as log file configurations and system defaults.

Cmdlets for general FAST Search Authorization settings

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to help manage item level security in FAST Search Server 2010 for SharePoint:

Task Cmdlet

Retrieve and view general security settings, such as the following:

  • Default user store ID

  • CCTK server port number

  • Default log level

  • Log level namespaces

Note that not all settings apply to every kind of user store.

Get-FASTSearchSecurityGeneralSettings

Check on or set the security configuration status of the current FSA worker node, which indicates if the node is configured correctly for secure searching. The query node will only accept search requests when this status is true.

Get-FASTSearchSecurityConfigurationStatus

Set-FASTSearchSecurityConfigurationStatus

View and change the default user store used when a user store is not specified at query time.

Get-FASTSearchSecurityDefaultUserStore

Set-FASTSearchSecurityDefaultUserStore

Retrieve one security user store or a list of all user stores.

Get-FASTSearchSecurityUserStore

Review and change the filter for publicly viewed items.

Get-FASTSearchSecurityPublicFilter

Set-FASTSearchSecurityPublicFilter

Retrieve the user search security filter for a specified user.

Get-FASTSearchSecurityUserFilter

Retrieve a list of all groups that a user belongs to.

Get-FASTSearchSecurityUserStoreGroupExpansion retrieves all groups in one user store that the user is a member of. Get-FASTSearchSecurityCompleteGroupExpansion provides the user, expanded groups that contain the user, all groups that contain other expanded groups, and aliaser-mapped users and groups. Both cmdlets are used for troubleshooting a user search security filter.

Get-FASTSearchSecurityUserStoreGroupExpansion

Get-FASTSearchSecurityCompleteGroupExpansion

Retrieve a user or group identity by decoding or encoding a Windows security identifier (SID).

Get-FASTSearchSecurityEncodedSid returns a Base64 encoded security identifier for a user, group, or Windows SID. Get-FASTSearchSecurityDecodedSid decodes an encoded security identifier (SID) and returns the user/group identifier (the common name) and Windows SID. Both cmdlets are used for troubleshooting.

Get-FASTSearchSecurityEncodedSid

Get-FASTSearchSecurityDecodedSid

View and change the log settings that control how much information is logged in all the FSA manager and FSA worker logs.

Get-FASTSearchSecurityLogLevel

Set-FASTSearchSecurityLogLevel

View the URI and status of one or more FSA workers (the Windows service that generates user search security filters).

Get-FASTSearchSecurityWorkerNode

Retrieve information about the caching of certain user search security filters.

Get-FASTSearchSecurityCacheState

Manage claims-based security user stores

FAST Search Server 2010 for SharePoint supports claims-based authentication, which is a set of operations that establishes trust relationships between claims providers and applications. Claims authentication foregoes the need to connect to a particular enterprise directory for looking up user identities. Instead, a user's request arrives with all the identity details that the application needs. These identity attributes may include name, e-mail address, membership, etc., and is called a claim. When a claim arrives, the user has already been authenticated, and FSA makes an access control decision based on that claim which is provided by the SharePoint front-end.

FSA works with security user stores, which are logical groupings of users, groups, and content permissions that serve as security gateways to third-party content repositories to help protect their content from unauthorized access. A claims user store is created as the default user store when you decide to work with claims authentication.

Claims user store cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to manage your claims user stores:

Task | Cmdlet
Add new claims content to the FSA configuration | New-FASTSearchSecurityClaimsUserStore
Edit the claims content security configuration | Set-FASTSearchSecurityClaimsUserStore
Retrieve and view the configuration of one or more claims user stores | Get-FASTSearchSecurityClaimsUserStore
Delete a claims content user store | Remove-FASTSearchSecurityUserStore

Manage Lotus Notes security user stores

A user store in FAST Search Authorization (FSA) is a logical grouping of users, groups, and content permissions that serves as a security gateway to a third-party content repository to help protect its content from unauthorized access. A Lotus Notes user store caches user credentials for Lotus Notes database collections.

A Lotus Notes user store is used by FSA to generate a user search security filter. This filter is attached to the user's query to enforce access control that is based on the user's credentials. Before you run the FAST Search Lotus Notes user directory connector, you must create a new Lotus Notes user store that will be populated with user and group information downloaded from your Lotus Domino server.

Lotus Notes user store cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets to help manage Lotus Notes user stores:

Task | Cmdlet
Create a user store in FSA to provide item level security for Lotus Notes content | New-FASTSearchSecurityLotusNotesUserStore
Edit a Lotus Notes user store configuration | Set-FASTSearchSecurityLotusNotesUserStore
Retrieve and view one or all previously defined Lotus Notes user stores | Get-FASTSearchSecurityLotusNotesUserStore
View the CCTK port number that uploads Lotus Notes credentials to FSA (CCTK=Content Connector Toolkit) | Get-FASTSearchSecurityCCTKServer Set-FASTSearchSecurityCCTKServer
Delete a Lotus Notes security user store | Remove-FASTSearchSecurityUserStore

Manage principal alias mapping

Principal aliasing defines equivalences between user stores.

When group expansion information is requested during user security filter generation, FAST Search Authorization (FSA) works through each user store to gather all the security IDs that apply to the user performing the query. Many users exist in multiple user stores. For example, a Windows user may have a Lotus Notes account that is a different ID. To narrow the query results, FSA must know the user's IDs and groups in all stores. FSA includes a process that is known as principal aliasing that maps users/groups from one user store to their equivalent IDs in another user store. Principal aliasing facilitates the gathering of security IDs, regardless of which user store the user or group is authenticated to. For example, you can map the user jbrown in a claims user store to be the same as the user brownj in the Lotus Notes user store.

There are two kinds of principal aliasing:

  • XML: Uses an XML file to map each specific security ID to its equivalent security ID in another store (recommended).

  • Regular expression (regex): Uses regular expression patterns to define maps from one user store to another. This kind of aliasing is used to pattern match a set of users/groups from one security store to another when their user ids are constructed following a specific pattern.

Principal aliasing cmdlets

Note:

To use the cmdlets, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use these cmdlets for your principal aliasing tasks:

Task | Cmdlet
View, troubleshoot, and manage any mapping | Get-FASTSearchSecurityAliaser
Map users or groups from one user store to another by using an XML file format | New-FASTSearchSecurityXMLAliaser
Map users or groups from one user store to another by using a regular expression map | New-FASTSearchSecurityRegexAliaser
Change an XML alias mapping | Set-FASTSearchSecurityXMLAliaser
Change a regular expression alias mapping | Set-FASTSearchSecurityRegexAliaser
View XML principal alias mappings | Get-FASTSearchSecurityXMLAliaser
View regular expression principal alias mappings | Get-FASTSearchSecurityRegexAliaser
Create a regular expression pattern to use when you create or change a regular expression alias mapping (input to New-FASTSearchSecurityRegexAliaser and Set-FASTSearchSecurityRegexAliaser) | New-FASTSearchSecurityRegexAliaserPattern
Remove a mapping | Remove-FASTSearchSecurityAliaser


Get-FASTSearchSecurityAliaser
See Also Example
[image]

Published: May 12, 2010

Gets a security principal aliaser.

Get-FASTSearchSecurityAliaser [[-Identity] <String>]

Parameters

Parameter | Required | Type | Description
Identity | Optional | System.String | The identity of a regular expression or XML aliaser to get. If not specified, the cmdlet gets a list of all aliasers.

Detailed Description

This cmdlet gets a regular expression or XML principal aliaser. If the Identity parameter is not specified, it gets a list of all aliasers.

A principal aliaser maps users from one user store into users in other user stores.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityAliaser

This example gets a list of all XML and regular expression principal aliasers.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityAliaser -Identity xmlalias4

This example gets the principal aliaser with the ID "xmlalias4".

See Also

Reference

Remove-FASTSearchSecurityAliaser


Get-FASTSearchSecurityCacheState
Example
[image]

Published: May 12, 2010

Gets information about, or controls the caching of user security search filters..

Get-FASTSearchSecurityCacheState [-Action] <String> [[-Identity] <String>] [[-User] <String>] [[-Worker] <WorkerPoolNode>]

Parameters

Parameter | Required | Type | Description
Action | Required | System.String | The Action parameter determines the type of task to perform. Valid values and actions include: -- GetCount - Counts the number of user search security filters in the cache. -- IsUserCached - Returns true if the search security filter cache contains a filter for the user. -- RemoveUserCache - Removes a user's search security filter from the cache. -- ClearCache - Removes all filters from the security search filter cache.
Identity | Optional | System.String | When the Action parameter is set to IsUserCached or RemoveUsercache, this parameter is used to identify the user store that contains the user.
User | Optional | System.String | When the Action parameter is set to IsUserCached or RemoveUsercache, this parameter identifies the user associated with the cached search filter.
Worker | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.Troubleshooting.WorkerPoolNode | The optional Worker parameter specifies the WorkerPoolNode object for the security worker that is to handle the request.

Detailed Description

This cmdlet displays information about the caching of user search security filters and controls the caching of the filters. The cmdlet's results depend on the value of the Action parameter:

-- GetCount - Counts the number of user search security filters in the cache.

-- IsUserCached - Returns true if the search security filter cache contains a filter for the user.

-- RemoveUserCache - Removes a user's search security filter from the cache.

-- ClearCache - Removes all filters from the security search filter cache.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityCacheState  -action GetCount

This example gets the count of user search filters in the security cache.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityWorkerNode workerserver | Get-FASTSearchSecurityCacheState -action IsUserCached -SecurityContentId tst -User usera

This example returns true if the search security filter for "usera" is cached, and false if the user's search filter is not cached. The request is handled by the worker with the server name "workerserver", specified by the piped in WorkerPoolNode parameter.


Get-FASTSearchSecurityCCTKServer
See Also Example
[image]

Published: May 12, 2010

Gets the CCTK server general setting.

Get-FASTSearchSecurityCCTKServer

Parameters

Detailed Description

This cmdlet gets the configuration setting for the server that handles CCTK requests.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityCCTKServer

This example retrieves the configuration setting for the server that handles CCTK requests.

See Also

Reference

Set-FASTSearchSecurityCCTKServer


Get-FASTSearchSecurityClaimsUserStore
See Also Example
[image]

Published: May 12, 2010

Get the claims security user store.

Get-FASTSearchSecurityClaimsUserStore [[-Identity] <String>]

Parameters

Parameter | Required | Type | Description
Identity | Optional | System.String | The identity of a claims user store to display. If an Identity is not specified, all claims user stores are displayed.

Detailed Description

This cmdlet retrieves a specific claims user store.

If the Identity parameter is not specified, the cmdlet gets a list of all claims user stores.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityClaimsUserStore -Identity clm

This example retrieves the claims user store with the identity "clm".

See Also

Reference

Set-FASTSearchSecurityClaimsUserStore
New-FASTSearchSecurityClaimsUserStore


Get-FASTSearchSecurityCompleteGroupExpansion
Example
[image]

Published: May 12, 2010

Gets complete group expansion information for a user.

Get-FASTSearchSecurityCompleteGroupExpansion [-Identity] <String> [-User] <String> [[-ExtraProperties] <SwitchParameter>] [[-XmlFile] <String>] [[-Worker] <WorkerPoolNode>]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The identity of the user store containing the user to expand.
User | Required | System.String | The user or group identifier to expand.
ExtraProperties | Optional | System.Management.Automation.SwitchParameter | If specified, the command gets the extra properties for all principals (user and groups) returned by the resolve.
XmlFile | Optional | System.String | The absolute path to an XML file which is accessible and writable from the security manager. If specified, complete expansion information is written as XML to this file.
Worker | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.Troubleshooting.WorkerPoolNode | The optional Worker parameter specifies the WorkerPoolNode object for the security worker that is to handle the request.

Detailed Description

This cmdlet gets troubleshooting information for group expansions and aliaser processing performed to create the user search security filter.

The group expansion information contains the user, expanded groups that contain the user (i.e., the user is a member of the group), and all groups that contain other expanded groups.

The aliaser procession information contains all mapped to users and groups based on the regular expression and XML aliaser mapping definitions.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityCompleteGroupExpansion -Identity ln1 -User user1

This example retrieves complete group expansion information for "user1" in the user store "ln1", showing all equivalent users and groups in all user stores, by finding all parents and processing all aliaser mappings.


Get-FASTSearchSecurityConfigurationStatus
See Also Example
[image]

Published: May 12, 2010

Retrieves the Boolean value specifying if the current query node has the necessary security configuration to enforce security on query requests.

Get-FASTSearchSecurityConfigurationStatus

Detailed Description

This cmdlet retrieves a Boolean value (true or false) indicating if this query node has the necessary security configuration to enforce security on query requests. If this query node does not have the necessary security configuration, query requests will not be processed by this query node.

Input Types

Return Types

Example

Copy Code

  See Also
Reference
Set-FASTSearchSecurityConfigurationStatus


Get-FASTSearchSecurityDecodedSid
See Also Example
[image]

Published: May 12, 2010

Gets the user identifier and SID from the encoded security identifier.

Get-FASTSearchSecurityDecodedSid [-EncodedSID] <String>

Parameters

Parameter | Required | Type | Description
EncodedSID | Required | System.String | The encoded extended security identifier that will be decoded.

Detailed Description

This cmdlet decodes an encoded security identifier (SID) and returns the user/group identifier (the common name) and Windows SID.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityDecodedSid -EncodedSID aebaaaaaaaaakiaaaaaccaqaaa

This example retrieves the Windows SID and user or group identifier for an encoded security identifier.

See Also

Reference

Get-FASTSearchSecurityEncodedSid


Get-FASTSearchSecurityDefaultUserStore
See Also Example
[image]

Published: May 12, 2010

Gets the default user store general setting.

Get-FASTSearchSecurityDefaultUserStore [[-Worker] <WorkerPoolNode>]

Parameters

Parameter | Required | Type | Description
Worker | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.Troubleshooting.WorkerPoolNode | The optional Worker parameter specifies the WorkerPoolNode object for the security worker that is to handle the request.

Detailed Description

This cmdlet gets the identity of the default user store that users and groups are looked up in. The purpose of the command is to determine the user's or group's document access if the user store identity is not specified when looking up the user or group.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityDefaultUserStore

This example gets the identity of the default user store.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityWorkerNode workerserver | Get-FASTSearchSecurityDefaultUserStore

This example gets the default user store's general setting information for the security worker with a server name of "workerserver".

See Also

Reference

Set-FASTSearchSecurityDefaultUserStore


Get-FASTSearchSecurityEncodedSid
See Also Example
[image]

Published: May 12, 2010

Gets the encoded extended security identifier.

Get-FASTSearchSecurityEncodedSid [[-SID] <String>] [[-User] <String>]

Parameters

Parameter | Required | Type | Description
SID | Optional | System.String | The Windows security identifier to encode.
User | Optional | System.String | The user or group identifier to encode.

Detailed Description

This cmdlet returns a Base64 encoded security identifier for a user, group, or Windows SID (Windows security identifier).

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityEncodedSid -SID S-1-5-7

This example returns an encoded security identifier and user name for the Windows SID, "5-1-5-7".

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityEncodedSid -User users

This example returns the encoded security identifier for the Windows built-in "Users" group.

See Also

Reference

Get-FASTSearchSecurityDecodedSid


Get-FASTSearchSecurityGeneralSettings
Example
[image]

Published: May 12, 2010

Gets the security general settings.

Get-FASTSearchSecurityGeneralSettings [[-Worker] <WorkerPoolNode>]

Parameters

Parameter | Required | Type | Description
Worker | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.Troubleshooting.WorkerPoolNode | The optional Worker parameter specifies the WorkerPoolNode object for the security worker that is to handle the request.

Detailed Description

This cmdlet retrieves configuration information for general security settings, such as:

-- Default user store ID

-- CCTK server port number

-- CCTK server enabled

-- Default log level

-- Log level namespaces

-- Public Filter FQL

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityGeneralSettings | fl

This example retrieves the configuration setting for all the general security settings and formats the information as a list.


Get-FASTSearchSecurityLogLevel
See Also Example
[image]

Published: May 12, 2010

Gets the log level general setting.

Get-FASTSearchSecurityLogLevel [[-Worker] <WorkerPoolNode>]

Parameters

Parameter | Required | Type | Description
Worker | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.Troubleshooting.WorkerPoolNode | The optional Worker parameter specifies the WorkerPoolNode object for the security worker that is to handle the request.

Detailed Description

This cmdlet retrieves the configuration information for the log level general setting. The log level controls the type of information that is logged by the security system.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityLogLevel

This example gets the configuration information for the log level's general setting, specifying the data logged by the security system.

See Also

Reference

Set-FASTSearchSecurityLogLevel


Get-FASTSearchSecurityLotusNotesUserStore
See Also Example
[image]

Published: May 12, 2010

Gets a Lotus Notes user store.

Get-FASTSearchSecurityLotusNotesUserStore [[-Identity] <String>]

Parameters

Parameter | Required | Type | Description
Identity | Optional | System.String | The identity of the Lotus Notes user store to get. If not specified, a list of all Lotus Notes user stores is returned.

Detailed Description

This cmdlet retrieves a specific Lotus Notes user store. If the Identity parameter is not specified, the cmdlet gets a list of all Lotus Notes user stores.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityLotusNotesUserStore -Identity ln1

This example gets the Lotus Notes user store with the identity "ln1".

See Also

Reference

New-FASTSearchSecurityLotusNotesUserStore
Set-FASTSearchSecurityLotusNotesUserStore


Get-FASTSearchSecurityPublicFilter
See Also Example
[image]

Published: May 12, 2010

Gets the public filter general setting.

Get-FASTSearchSecurityPublicFilter [[-Worker] <WorkerPoolNode>]

Parameters

Parameter | Required | Type | Description
Worker | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.Troubleshooting.WorkerPoolNode | The optional Worker parameter specifies the WorkerPoolNode object for the security worker that is to handle the request.

Detailed Description

This cmdlet retrieves the public filter general setting configuration information. The public filter is the search security filter that is used to find documents that all users have access to.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityPublicFilter

This example gets the security public filter's general setting configuration information.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityWorkerNode workerserver | Get-FASTSearchSecurityPublicFilter

This example gets the security public filter's general setting configuration information for a security worker with the server name of "workerserver".

See Also

Reference

Set-FASTSearchSecurityPublicFilter


Get-FASTSearchSecurityRegexAliaser
See Also Example
[image]

Published: May 12, 2010

Gets a regular expression principal aliaser.

Get-FASTSearchSecurityRegexAliaser [[-Identity] <String>]

Parameters

Parameter | Required | Type | Description
Identity | Optional | System.String |

Detailed Description

This cmdlet retrieves a regular expression aliaser. A regular expression aliaser maps users/groups from one user store to users/groups in other user stores, based on a regular expression pattern.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityRegexAliaser -Identity regex1

This example retrieves a regular expression aliaser with an identity of "regex1".

See Also

Reference

New-FASTSearchSecurityRegexAliaser
Set-FASTSearchSecurityRegexAliaser
New-FASTSearchSecurityRegexAliaserPattern


Get-FASTSearchSecurityUserFilter
Example
[image]

Published: May 12, 2010

Gets a user's search security filter.

Get-FASTSearchSecurityUserFilter [-Identity] <String> [-User] <String> [[-Worker] <WorkerPoolNode>]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The identity of the user store containing the user.
User | Required | System.String | The identifier of the user or group.
Worker | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.Troubleshooting.WorkerPoolNode | The optional Worker parameter specifies the WorkerPoolNode object for the security worker that is to handle the request.

Detailed Description

This cmdlet retrieves the user search security filter containing all groups that the user has access to.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityUserFilter -Identity ln1 -User user1

This example gets the search security filter for user "user1" in user store "ln1".


Get-FASTSearchSecurityUserStore
See Also Example
[image]

Published: May 12, 2010

Gets a user store.

Get-FASTSearchSecurityUserStore [[-Identity] <String>]

Parameters

Parameter | Required | Type | Description
Identity | Optional | System.String | The identity of the user store to get. If not specified, the cmdlet gets a list of all user stores.

Detailed Description

This cmdlet retrieves a security user store. If the Identity parameter is not specified, it gets a list of all user stores.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityUserStore -Identity ln1

This example gets the user store with an identity of "ln1".

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityUserStore

This example gets a list of all user stores.

See Also

Reference

Remove-FASTSearchSecurityUserStore


Get-FASTSearchSecurityUserStoreGroupExpansion
Example
[image]

Published: May 12, 2010

Gets all groups in a user store that the specified user is a member of.

Get-FASTSearchSecurityUserStoreGroupExpansion [-Identity] <String> [-User] <String> [[-Worker] <WorkerPoolNode>]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The identity of the user store to look in.
User | Required | System.String | The user or group identifier to look up.
Worker | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.Troubleshooting.WorkerPoolNode | The optional Worker parameter specifies the WorkerPoolNode object for the security worker that is to handle the request.

Detailed Description

This cmdlet gets all the groups in a user store that the user is a member of, and all the groups that any owning group is a member of.

Input Types

Return Types

Errors

Error | Description

Exceptions

Exceptions | Description

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityUserStoreGroupExpansion -Identity ln1 -User user1

This example gets all the groups in the "ln1" user store that are parents, grand-parents, etc. of the "user1" user.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityWorkerNode workerserver | Get-FASTSearchSecurityUserStoreGroupExpansion -Identity ln2 -User user1

This example gets all the groups in the "ln2" user store from the worker with a server name of "workerserver" that are parents, grand-parents, etc. of the "user1" user.


Get-FASTSearchSecurityWorkerNode
Example
[image]

Published: May 12, 2010

Gets status information about a security worker or workers.

Get-FASTSearchSecurityWorkerNode [[-Identity] <String>] [[-Active] <SwitchParameter>] [[-Dead] <SwitchParameter>]

Parameters

Parameter | Required | Type | Description
Identity | Optional | System.String | The identity of the worker to get. This is a URI or a partial URI, the server name for example.
Active | Optional | System.Management.Automation.SwitchParameter | If specified, get a list of all active workers.
Dead | Optional | System.Management.Automation.SwitchParameter | If specified, get a list of all inactive workers. A worker is "dead" if it does not respond to requests from the security manager. This may occur if a worker machine goes down, or if there are network problems.

Detailed Description

This cmdlet retrieves the URI and status for a security worker or a group of workers. A security worker is the Windows service that generates user search security filters. The cmdlet's function changes depending upon the parameter values:

-- If Identity is specified, the command gets the worker associated with the given URI.

-- If Active is specified, the command gets a list of all active workers.

-- If Dead is specified, the command gets a list of all inactive workers.

-- If none of these parameters are specified, the command returns a list of all workers.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityWorkerNode -Identity localhost

This example gets the status and complete URI for the worker with "localhost" in its URI.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityWorkerNode

This example returns a list of all workers, both active and inactive.


Get-FASTSearchSecurityXMLAliaser
See Also Example
[image]

Published: May 12, 2010

Gets a security principal aliaser.

Get-FASTSearchSecurityXMLAliaser [[-Identity] <String>]

Parameters

Parameter | Required | Type | Description
Identity | Optional | System.String | The identity of a regular expression or XML aliaser to get. If not specified, the cmdlet gets a list of all aliasers.

Detailed Description

This cmdlet gets a regular expression or XML principal aliaser. If the Identity parameter is not specified, it gets a list of all aliasers.

A principal aliaser maps users from one user store into users in other user stores.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Get-FASTSearchSecurityAliaser

This example gets a list of all XML and regular expression principal aliasers.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityAliaser -Identity xmlalias4

This example gets the principal aliaser with the ID xmlalias4.

See Also

Reference

New-FASTSearchSecurityXMLAliaser
Set-FASTSearchSecurityXMLAliaser


New-FASTSearchSecurityClaimsUserStore
See Also Example
[image]

Published: May 12, 2010

Creates a FAST Search Server 2010 for SharePoint security user store to provide security for claims content.

New-FASTSearchSecurityClaimsUserStore [-Identity] <String> [[-Description] <String>] [[-Issuer] <String>] [-ObjectToClone <ClaimsUserStore>]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The user store identity.
Description | Optional | System.String | A brief explanation of the user store.
Issuer | Optional | System.String | The Microsoft SharePoint Server Security Token Service (STS) claims provider that issued the claims.
ObjectToClone | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.ClaimsContent.ClaimsUserStore | A claims user store object whose property values are used as a baseline to create another claims user store.

Detailed Description

This cmdlet configures extended security for claims content.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchSecurityClaimsUserStore -Identity clm

This example creates a new security claims user store with a user store Identity of "clm".

This example uses default values for all parameters except the required Identity parameter.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityClaimsUserStore -Identity clm | New-FASTSearchSecurityClaimsUserStore -Identity cl2

This example creates a new security claims user store with an identity of "cl2" using all properties from the claims user store "clm" as a baseline. The New-FASTSearchSecurityClaimsUserStore cmdlet uses the ObjectToClone value on the pipeline as input to the New cmdlet.

See Also

Reference

Get-FASTSearchSecurityClaimsUserStore
Set-FASTSearchSecurityClaimsUserStore


New-FASTSearchSecurityLotusNotesUserStore
See Also Example
[image]

Published: May 12, 2010

Creates a FAST Search Server 2010 for SharePoint security user store to provide security for Lotus Notes content.

New-FASTSearchSecurityLotusNotesUserStore [-Identity] <String> [[-Description] <String>] [[-UseDenyOnDefault] <Nullable>] [[-UseDocumentSecurity] <Nullable>] [[-UseWildcardSecurity] <Nullable>] [[-UseViewSecurity] <Nullable>] [[-AllowPrefix] <String>] [[-DenyPrefix] <String>] [[-NonIndexableCharacterHandling] <String>] [[-NonIndexableCharacterEncodingPrefix] <String>] [[-NonIndexableCharacterPattern] <String>] [-ObjectToClone <LotusNotesUserStore>]

Parameters

Parameter Required Type Description

Identity

Required

System.String

User store identity.

Description

Optional

System.String

A brief explanation of the user store.

UseDenyOnDefault

Optional

System.Nullable

If true, support for default access with deny on users/groups is enabled on database level security.

You must enable this filter for an inverse security model: first grant access to everyone, and then explicitly deny access to certain groups or users.

UseDocumentSecurity

Optional

System.Nullable

If true, item security is enabled (i.e., the filter contains an item level security condition).

If set to false, item level security is ignored; only database level security (and possibly view security) is used.

If items always inherit the ACLs of the database, you can turn off this option.

UseWildcardSecurity

Optional

System.Nullable

If true, database access through wildcard group entry (starting with */) is enabled.

If set to false, the wildcard check is ignored. Only normal groups are checked against the database ACL.

UseViewSecurity

Optional

System.Nullable

If true, view security is enabled (i.e., the filter contains a view level security condition).

If set to false, view level security is ignored; only database level security (and possibly item security) is used.

If views always inherit the ACLs of the database, or if views are not indexed, you can turn off this option.

AllowPrefix

Optional

System.String

A string value prepended to the document ACL to control which items the user is allowed access to and which items will be returned in the query.

DenyPrefix

Optional

System.String

A string value prepended to the ACL to control which items the user is denied access to and which items are not returned from the query.

NonIndexableCharacterHandling

Optional

System.String

Specifies what to do with ACL values that match the NonIndexableCharacterRegExPattern regular expression.

Valid values:

-- encode - The ACL is encoded into a single searchable term.

-- encodeWithPrefix - The value of NonIndexableEncodedPrefix is concatenated with the ACL to be encoded, producing a single searchable term.

-- remove - The ACL is removed from both allowing and denying access. Specify "remove" if the ACL is never used to deny access.

NonIndexableCharacterEncodingPrefix

Optional

System.String

If NonIndexableCharacterHandling is set to encodeWithPrefix, this parameter specifies the prefix added to the encoded ACL.

NonIndexableCharacterPattern

Optional

System.String

Specifies a regular expression pattern used to detect non-indexable characters in an ACL.

Identities matching this regular expression pattern will be processed based on NonIndexableCharacterHandling.

ObjectToClone

Optional

Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.LotusNotesContent.LotusNotesUserStore

A Lotus Notes user store object whose property values are used as a baseline for the Lotus Notes user store being created.

Detailed Description

This cmdlet configures FAST Search Server 2010 for SharePoint security for Lotus Notes content.

A Lotus Notes security user store uses a user-dependent query filter that is applied to each query to enforce the defined access control policies based on indexed values. The Lotus Notes security user store search filter requires that the user has read access to both the item and the database to view a specific item.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
new-FASTSearchSecurityLotusNotesUserStore ln1

This example creates a new security Lotus Notes user store with a user store identity of "ln1".

This example uses default values for all parameters except the required Identity parameter.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityLotusNotesUserStore -Identity ln1 | New-FASTSearchSecurityLotusNotesUserStore -Identity ln3 -UseViewSecurity $true

This example creates a new Lotus Notes user store with an identity of "ln3" and enables view level security. The New-FASTSearchSecurityLotusNotesUserStore command uses the ObjectToClone object on the pipeline as input to the New command for default parameter values.

See Also

Reference

Set-FASTSearchSecurityLotusNotesUserStore
Get-FASTSearchSecurityLotusNotesUserStore


New-FASTSearchSecurityRegexAliaser
See Also Example
[image]

Published: May 12, 2010

Creates a regular expression aliaser.

New-FASTSearchSecurityRegexAliaser [-InputUserStoreId] <String> [-OutputUserStoreIds] <String[]> [-Patterns] <List> [[-Identity] <String>] [[-CaseSensitive] <$true | $false>] [[-UnicodeCaseSensitive] <$true | $false>] [-ObjectToClone <SecurityRegexAliaser>]

Parameters

Parameter | Required | Type | Description
InputUserStoreId | Required | System.String | The identity of the mapped-from user store.
OutputUserStoreIds | Required | System.String[] | A comma-separated list of mapped-to user store identities.
Patterns | Required | System.Collections.Generic.List | A list of RegExAliasPatternMapConfig instances containing the regular expression patterns used for the mapping.
Identity | Optional | System.String | The identity of the aliaser.
CaseSensitive | Optional | System.Boolean | Specifies whether the regular expression patterns defined in the RegExAliasPatternMapConfig list are matched based on case sensitivity matching rules. The default is true.
UnicodeCaseSensitive | Optional | System.Boolean | Specifies whether the regular expressions patterns defined in the RegExAliasPatternMapConfig list are matched based on Unicode case sensitivity matching rules.
ObjectToClone | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.RegexAliaser.SecurityRegexAliaser | A SecurityRegexAliaser object whose property values are used for the SecurityRegexAliaser aliaser being created.

Detailed Description

This cmdlet creates a regular expression aliaser to map users/groups from one user store to users/groups in other user stores, based on a regular expression pattern.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchSecurityRegexAliaserPattern -Regex .* -MapToUsername \1 | New-FASTSearchSecurityRegexAliaser -Identity regex1 -InputUserStoreId ln1 -OutputUserStoreIds ln2

This example creates a regular expression aliaser pattern object and pipes it into a New-FASTSearchSecurityRegexAliaser cmdlet to create a regular expression aliaser.

---------------EXAMPLE 2-----------------

Copy Code
New-FASTSearchSecurityRegexAliaserPattern -Regex .* -MapToUsername \1 | New-FASTSearchSecurityRegexAliaser -InputUserStoreId ln1 -OutputUserStoreIds ln2,ln3 -CaseSensitive $False -UnicodeCaseSensitive $False

This example creates a regular expression aliaser pattern object and pipes it into a New-FASTSearchSecurityRegexAliaser cmdlet to create a regular expression aliaser. The example sets the case-sensitive flags so pattern matching is performed in a case-insensitive manner. The OutputUserStoreIds parameter maps users/groups to two user stores: "ln2" and "ln3". The example generates the aliaser identity.

See Also

Reference

Set-FASTSearchSecurityRegexAliaser
Get-FASTSearchSecurityRegexAliaser
New-FASTSearchSecurityRegexAliaserPattern


New-FASTSearchSecurityRegexAliaserPattern
See Also Example
[image]

Published: May 12, 2010

Creates a new regular expression pattern for a regex principal aliaser.

New-FASTSearchSecurityRegexAliaserPattern [-Regex] <String> [-MapToUsername] <String> [[-InputPropertyName] <String>] [-Patterns <List>]

Parameters

Parameter | Required | Type | Description
Regex | Required | System.String | The regular expression pattern that determines if the user or group should be mapped to another user store. This regular expression pattern may contain regular expression groups that may be used in the generation of the MapToUsername value.
MapToUsername | Required | System.String | The user or group name to look to. This can also contain regular expression group replacements using the syntax: \# where # is the group capture index.
InputPropertyName | Optional | System.String | The user/group property name this regular expression pattern is matched against. The InputPropertyName is one of the following: -- $PRINCIPAL_REFERENCE_ID - The identifier of the user or group. This is the default. -- $PRINCIPAL_REFERENCE_ALIAS - The alias of the user or group. Any other property of the user or group. Refer to your specific user store for other properties that the user store exposes.
Patterns | Optional | System.Collections.Generic.List | A list of RegExAliasPatternMapConfig objects. The RegExAliasPatternMapConfig being created is added to this list.

Detailed Description

This cmdlet creates a new regular expression pattern that can be added to the list of patterns for a regular expression based principal aliaser. This cmdlet creates a regular expression pattern that is used by and input to the New-FASTSearchSecurityRegexAliaser and Set-FASTSearchSecurityRegexAliaser cmdlets.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchSecurityRegexAliaserPattern -Regex .* -InputPropertyName mail -MapToUsername "\1"

This example creates a regular expression pattern that can be used to map the "mail" property of a Microsoft user store to identities in other user stores.

Use the regex pattern configuration object output from this command for the value of a pattern's parameter in a New-FASTSearchSecurityRegexAliaser or a Set-FASTSearchSecurityRegexAliaser cmdlet.

See Also

Reference

Get-FASTSearchSecurityRegexAliaser
New-FASTSearchSecurityRegexAliaser
Set-FASTSearchSecurityRegexAliaser


New-FASTSearchSecurityXMLAliaser
See Also Example
[image]

Published: May 12, 2010

Creates an XML principal aliaser.

New-FASTSearchSecurityXMLAliaser [-InputUserStoreId] <String> [-OutputUserStoreIds] <String[]> [[-Identity] <String>] [[-InputPropertyName] <String>] [-ObjectToClone <SecurityXMLAliaser>]

Parameters

Parameter | Required | Type | Description
InputUserStoreId | Required | System.String | The identity of the mapped-from user store.
OutputUserStoreIds | Required | System.String[] | A comma-separated list of mapped-to user store identities.
Identity | Optional | System.String | The identity of the aliaser. An identity is generated if the parameter is not specified.
InputPropertyName | Optional | System.String | The user/group property name this regular expression pattern is matched against. The InputPropertyName is one of the following: -- $PRINCIPAL_REFERENCE_ID - The identifier of the user or group. This is the default. Be sure to enclose in single quotes: '$PRINCIPAL_REFERENCE_ID' -- $PRINCIPAL_REFERENCE_ALIAS - The alias of the user or group. Be sure to enclose in single quotes: '$PRINCIPAL_REFERENCE_NAME' Any other property of the user or group. Refer to your specific user store for other properties that the user store exposes.
ObjectToClone | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.XmlAliaser.SecurityXMLAliaser | A SecurityXMLAliaserobject whose property values are copied as a baseline in the creation of another SecurityXMLAliaser aliaser.

Detailed Description

This cmdlet creates an XML aliaser to map users and groups from one user store to users and groups in other user stores, based on an XML mapping file.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
New-FASTSearchSecurityXMLAliaser -InputUserStoreId win -OutputUserStoreIds ln3 -InputPropertyName mail

This example creates an XML principal aliaser to map the mail property of a Microsoft user store to users and groups in another user store.

Use the Set-FASTSearchSecurityXMLAliaser cmdlet to upload the XML mapping file.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityAliaser xmlalias2 | New-FASTSearchSecurityXMLAliaser -Identity myxmlalias -OutputUserStoreIds ln2

This example creates an XML principal aliaser based on an existing XML aliaser but sets the output user store IDs to 'ln2'.

Use the Set-FASTSearchSecurityXMLAliaser cmdlet to upload an XML mapping file with desired mappings for the new aliaser.

See Also

Reference

Set-FASTSearchSecurityXMLAliaser
Get-FASTSearchSecurityXMLAliaser


Remove-FASTSearchSecurityAliaser
See Also Example
[image]

Published: May 12, 2010

Removes a security aliaser.

Remove-FASTSearchSecurityAliaser [-Identity] <String> [[-SecurityAliaserToRemove] <SecurityAliaser>] [-Confirm [<SwitchParameter>]] [-WhatIf [<SwitchParameter>]]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The identity of the regular expression or XML aliaser to remove.
SecurityAliaserToRemove | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.SecurityAliaser | A SecurityXMLAliaser or SecurityRegexAliaser object whose identity is used as the aliaser identity to remove.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated and you must confirm the remove request. To suppress user prompting, set the confirm parameter to false. For example: -confirm:$false
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet deletes a regular expression or XML aliaser.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchSecurityAliaser -Identity xmlalias2

This example deletes an XML aliaser. A prompt is displayed to confirm the request.

---------------EXAMPLE 2-----------------

Copy Code
Remove-FASTSearchSecurityAliaser -Identity xmlalias4 -Confirm:$False

This example deletes an XML aliaser without displaying a prompt.

See Also

Reference

Get-FASTSearchSecurityAliaser


Remove-FASTSearchSecurityUserStore
See Also Example
[image]

Published: May 12, 2010

Removes a user store.

Remove-FASTSearchSecurityUserStore [-Identity] <String> [[-SecurityUserStoreToRemove] <SecurityUserStore>] [-Confirm [<SwitchParameter>]] [-WhatIf [<SwitchParameter>]]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The identity of the user store to remove.
SecurityUserStoreToRemove | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.SecurityUserStore | A MicrosoftUserStore, LotusNotesUserStore, or ClaimsUserStore instance whose identity is used to remove the user store.
Confirm | Optional | System.Management.Automation.SwitchParameter | Activates user prompting to confirm the operation. If set, prompting is activated and you must confirm the remove request. To suppress user prompting, set the confirm parameter to false. For example: -confirm:$false
WhatIf | Optional | System.Management.Automation.SwitchParameter | Displays a message that describes the effect of the command instead of executing the command. For more information, type the following command: get-help about_commonparameters

Detailed Description

This cmdlet deletes a security user store. The user store may be a Lotus Notes, Microsoft, or claims user store.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Remove-FASTSearchSecurityUserStore -Identity wn1

This example deletes the "wn1" user store. A prompt is displayed to confirm the request.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityLotusNotesUserStore ln2 | remove-FASTSearchSecurityUserStore

This example deletes the user store specified by a piped SecurityUserStoreToRemove instance. A prompt is displayed to confirm the request.

See Also

Reference

Get-FASTSearchSecurityUserStore


Set-FASTSearchSecurityCCTKServer
See Also Example
[image]

Published: May 12, 2010

Updates the CCTK server general setting.

Set-FASTSearchSecurityCCTKServer [[-Port] <Int32>] [[-Enable] <Nullable>] [[-GeneralSetting] <CCTKServerSetting>]

Parameters

Parameter | Required | Type | Description
Port | Optional | System.Int32 | Do not use.
Enable | Optional | System.Nullable | A Boolean flag to determine if the CCTK service will accept requests.
GeneralSetting | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.GeneralSettings.CCTKServerSetting | A CCTKServerSetting instance whose property values are used for the CCTKServerSetting being updated.

Detailed Description

This cmdlet updates the configuration information for the Content Connector Toolkit (CCTK) server general settings.

Input Types

Return Types

Example

---------------EXAMPLE 1----------------

Copy Code
Set-FASTSearchSecurityCCTKServer -Enable $true

This example enables the CCTK server to accept requests.

See Also

Reference

Get-FASTSearchSecurityCCTKServer


Set-FASTSearchSecurityClaimsUserStore
See Also Example
[image]

Published: May 12, 2010

Updates a claims user store.

Set-FASTSearchSecurityClaimsUserStore [-Identity] <String> [[-SecurityContent] <ClaimsUserStore>] [[-Description] <String>] [[-Issuer] <String>]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The user store identity.
SecurityContent | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.ClaimsContent.ClaimsUserStore | A claims user store object whose property values are used to update another claims user store.
Description | Optional | System.String | A brief explanation of the user store.
Issuer | Optional | System.String | The Microsoft SharePoint Server Security Token Service (STS) claims provider that issued the claims.

Detailed Description

This cmdlet updates the configuration settings for a claims user store.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchSecurityClaimsUserStore -Identity clm -Description "Sharepoint Users"

This example updates the description of the "clm" claims user store.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityClaimsUserStore -Identity cl2 | Set-FASTSearchSecurityClaimsUserStore -Issuer Windows

This example sets the issuer of the "cl2" claims user store to "Windows". The Set-FASTSearchSecurityClaimsUserStore cmdlet updates the SecurityContent value piped from the Get-FASTSearchSecurityClaimsUserStore.

See Also

Reference

Get-FASTSearchSecurityClaimsUserStore
New-FASTSearchSecurityClaimsUserStore


Set-FASTSearchSecurityConfigurationStatus
See Also Example
[image]

Published: May 12, 2010

Sets a Boolean value specifying if the current query node has the necessary security configuration to enforce security on query requests.

Set-FASTSearchSecurityConfigurationStatus [-status] <$true | $false>

Parameters

Parameter | Required | Type | Description
status | Required | System.Boolean | Specifies whether sufficient security configuration is present on this query node to enforce security on query requests.

Detailed Description

This cmdlet sets the status specifying if the necessary security configuration to enforce security on query requests is available on the current query node. True means that the necessary security configuration is on this query node to enforce security on query requests, false means that the security configuration is missing or insufficient to enforce security on this query node.

The Microsoft FAST Search Server 2010 for SharePoint query node will not accept search requests until this status is set to true.

You should only set the status manually if the administration node is down and you want to allow search requests with the current configuration. Manually setting this value to true may result in query responses containing search requests for documents that users may not have rights to access.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchConfigurationStatus $true

This example sets the status of the current query node to true so that the node can provide query responses that enforce security based on the current security configuration.

See Also

Reference

Get-FASTSearchSecurityConfigurationStatus


Set-FASTSearchSecurityDefaultUserStore
See Also Example
[image]

Published: May 12, 2010

Updates the default user store general setting.

Set-FASTSearchSecurityDefaultUserStore [[-DefaultUserStoreId] <String>]
Set-FASTSearchSecurityDefaultUserStore [[-GeneralSetting] <DefaultUserStoreSetting>]

Parameters

1
Parameter | Required | Type | Description
DefaultUserStoreId | Optional | System.String | The identity of the user store to set as the default user store.

2
Parameter | Required | Type | Description
GeneralSetting | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.GeneralSettings.DefaultUserStoreSetting | A DefaultUserStoreSettinginstance whose property values are used for the DefaultUserStoreSetting being updated.

Detailed Description

This cmdlet updates the identity of the user store that users and groups are looked up in, to determine the appropriate document access if the user store identity is not specified when looking up the user or group.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchSecurityDefaultUserStore -DefaultUserStoreId win

This example sets the default user store to "win".

---------------EXAMPLE 2-----------------

Copy Code
$defuserstore | Set-FASTSearchSecurityDefaultUserStore

This example updates the default user store using values from a pipeline GeneralSetting parameter.

In this example, the $defuserstore GeneralSetting was created with this command: $defuserstore = Get-FASTSearchSecurityDefaultUserStore.

See Also

Reference

Get-FASTSearchSecurityDefaultUserStore


Set-FASTSearchSecurityLogLevel
See Also Example
[image]

Published: May 12, 2010

Updates the log level general setting.

Set-FASTSearchSecurityLogLevel [[-GeneralSetting] <LogLevelSetting>] [-DebugNameSpaceLogLevel <String[]>] [-DefaultLogLevel <String>] [-ErrorNameSpaceLogLevel <String[]>] [-IncludeExceptionStack <Nullable>] [-InfoNameSpaceLogLevel <String[]>] [-WarningNameSpaceLogLevel <String[]>]

Parameters

Parameter Required Type Description

GeneralSetting

Optional

Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.GeneralSettings.LogLevelSetting

A LogLevelSetting whose property values are used for the LogLevelSetting being updated.

DebugNameSpaceLogLevel

Optional

System.String[]

A list of C# namespaces. Any class contained in that namespace logs all messages to the log.

A class can occur in only one namespace. If you specify the same class in more than one name space log level, the log level that generates the most messages will be set.

DefaultLogLevel

Optional

System.String

Specifies the type of log messages that each class writes to the log, unless the class is contained in a namespace explicitly specified in ErrorLogLevelNameSpaces, WarningLogLevelNameSpaces, InfoLogLevelNameSpaces, and DebugLogLevelNameSpaces.

Valid values are:

-- Error

-- Warning

-- Info

-- Debug

ErrorNameSpaceLogLevel

Optional

System.String[]

A list of C# namespaces. Any class included in the namespace only logs error messages.

A class can occur in only one namespace. If you specify the same class in more than one namespace log level, the log level that generates the most messages will be set.

IncludeExceptionStack

Optional

System.Nullable

Whether or not to include the exception stack in the log. Set to $True to include the exception stack.

InfoNameSpaceLogLevel

Optional

System.String[]

A list of C# namespaces. Any class contained in that namespace only logs error, warning, and info messages.

A class can occur in only one namespace. If you specify the same class in more than one namespace log level, the log level that generates the most messages will be set.

WarningNameSpaceLogLevel

Optional

System.String[]

A list of C# namespaces. Any class included in the namespace only logs error and warning messages.

A class can occur in only one namespace. If you specify the same class in more one than namespace log level, the log level that generates the most messages will be set.

Detailed Description

This cmdlet updates the configuration information for the log level's general setting. The log level controls the type of information that is logged by the security system.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchSecurityLogLevel -DefaultLogLevel debug -IncludeExceptionStack $True

This example sets the default log level to the "Debug" level and enables the setting to include the exception stack in the log.

---------------EXAMPLE 2-----------------

Copy Code
Set-FASTSearchSecurityLogLevel -WarningNameSpaceLogLevel Microsoft

This example sets the log level setting for the "Microsoft" namespace to the "Warning" level.

See Also

Reference

Get-FASTSearchSecurityLogLevel


Set-FASTSearchSecurityLotusNotesUserStore
See Also Example
[image]

Published: May 12, 2010

Updates a Lotus Notes user store.

Set-FASTSearchSecurityLotusNotesUserStore [-Identity] <String> [[-SecurityContent] <LotusNotesUserStore>] [[-Description] <String>] [[-UseDenyOnDefault] <Nullable>] [[-UseDocumentSecurity] <Nullable>] [[-UseWildcardSecurity] <Nullable>] [[-UseViewSecurity] <Nullable>] [[-AllowPrefix] <String>] [[-DenyPrefix] <String>] [[-NonIndexableCharacterHandling] <String>] [[-NonIndexableCharacterEncodingPrefix] <String>] [[-NonIndexableCharacterPattern] <String>]

Parameters

Parameter Required Type Description

Identity

Required

System.String

Identity of the Lotus Notes user store to update.

SecurityContent

Optional

Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.LotusNotesContent.LotusNotesUserStore

A Lotus Notes user store object whose property values are used for the Lotus Notes user store being updated.

Description

Optional

System.String

A brief explanation of the user store.

UseDenyOnDefault

Optional

System.Nullable

If true, support for default access with deny on users/groups is enabled on database level security.

You must enable this filter for an inverse security model: first grant access to everyone, and then explicitly deny access to certain groups or users.

UseDocumentSecurity

Optional

System.Nullable

If true, item security is enabled (i.e., the filter contains a item level security condition).

If set to false, item level security is ignored; only database level security (and possibly view security) is used.

If items always inherit the ACLs of the database, you can turn off this option.

UseWildcardSecurity

Optional

System.Nullable

If true, database access through wildcard group entry (starting with */) is enabled.

If set to false, the wildcard check is ignored. Only normal groups are checked against the database ACL.

UseViewSecurity

Optional

System.Nullable

If true, view security is enabled (i.e., the filter contains a view level security condition).

If set to false, view level security is ignored; only database level security (and possibly item security) is used.

If views always inherit the ACLs of the database, or if views are not indexed, you can turn off this option.

AllowPrefix

Optional

System.String

A string value prepended to the document ACL to control which items the user is allowed access to and which items will be returned in the query.

DenyPrefix

Optional

System.String

A string value prepended to the ACL to control which items the user is denied access to and which items are not returned from the query.

NonIndexableCharacterHandling

Optional

System.String

Specifies what to do with ACL values that match the NonIndexableCharacterRegExPattern regular expression.

Valid values:

-- encode - The ACL is encoded into a single searchable term.

-- encodeWithPrefix - The value of NonIndexableEncodedPrefix is concatenated with the ACL to be encoded, producing a single searchable term.

-- remove - The ACL is removed from both allowing and denying access. Specify "remove" if the ACL is never used to deny access.

NonIndexableCharacterEncodingPrefix

Optional

System.String

If NonIndexableCharacterHandling is set to encodeWithPrefix, this parameter specifies the prefix added to the encoded ACL.

NonIndexableCharacterPattern

Optional

System.String

Specifies a regular expression pattern used to detect non-indexable characters in an ACL. Identities matching this regular expression pattern will be processed based on NonIndexableCharacterHandling.

Detailed Description

This cmdlet updates the configuration settings for a Lotus Notes user store.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchSecurityLotusNotesUserStore -Identity ln4 -Description "My Lotus user store"

This example updates the description of the "ln4" Lotus Notes user store.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityLotusNotesUserStore -Identity ln3 | Set-FASTSearchSecurityLotusNotesUserStore -UseDocumentSecurity $false

This example sets the UseDocumentSecurity of the "ln3" Lotus Notes user store to false to skip item level security.

In this case, only database level security is used. The Set-FASTSearchSecurityLotusNotesUserStore command updates the SecurityContent object piped from the Get-FASTSearchSecurityLotusNotesUserStore.

See Also

Reference

Get-FASTSearchSecurityLotusNotesUserStore
New-FASTSearchSecurityLotusNotesUserStore


Set-FASTSearchSecurityPublicFilter
See Also Example
[image]

Published: May 12, 2010

Updates the public filter general setting.

Set-FASTSearchSecurityPublicFilter [[-Filter] <String>]
Set-FASTSearchSecurityPublicFilter [[-GeneralSetting] <PublicFilterSetting>]

Parameters

1
Parameter | Required | Type | Description
Filter | Optional | System.String | The search security filter that is used to find documents that all users have access to.

2
Parameter | Required | Type | Description
GeneralSetting | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.GeneralSettings.PublicFilterSetting | A PublicFilterSetting whose property values are used for the PublicFilterSetting being updated.

Detailed Description

This cmdlet updates the public filter general setting configuration. The public filter is the search security filter that is used to find documents that all users have access to.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchSecurityPublicFilter -Filter docacl:unknown

This example updates the search security filter that is used to find documents that all users have access to.

---------------EXAMPLE 2-----------------

Copy Code
$pubfilter | Set-FASTSearchSecurityPublicFilter

This example updates the public filter setting using values from a pipeline GeneralSetting parameter.

In this example, the $pubfilter GeneralSetting was created with this command: $pubfilter = Get-FASTSearchSecurityPublicFilter.

See Also

Reference

Get-FASTSearchSecurityPublicFilter


Set-FASTSearchSecurityRegexAliaser
See Also Example
[image]

Published: May 12, 2010

Updates a regular expression aliaser.

Set-FASTSearchSecurityRegexAliaser [-Identity] <String> [-CaseSensitive <$true | $false>] [-InputUserStoreId <String>] [-OutputUserStoreIds <String[]>] [-Patterns <List>] [-RegexAliaser <SecurityRegexAliaser>] [-UnicodeCaseSensitive <$true | $false>]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The identity of the aliaser to update.
CaseSensitive | Optional | System.Boolean | Specifies whether the regular expressions patterns defined in the RegExAliasPatternMapConfig list are matched based on case sensitivity matching rules. The default is true.
InputUserStoreId | Optional | System.String | The identity of the mapped-from user store.
OutputUserStoreIds | Optional | System.String[] | A comma-separated list of mapped-to user store identities.
Patterns | Optional | System.Collections.Generic.List | List of RegExAliasPatternMapConfig instances containing the regular expression patterns using for the mapping. Use the New-RegExAliasPatternMapConfig cmdlet to create patterns.
RegexAliaser | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.RegexAliaser.SecurityRegexAliaser | A SecurityRegexAliaser object whose property values are used for the SecurityRegexAliaser aliaser being created.
UnicodeCaseSensitive | Optional | System.Boolean | Specifies whether the regular expressions patterns defined in the RegExAliasPatternMapConfig list are matched based on Unicode case sensitivity matching rules.

Detailed Description

This cmdlet updates the configuration settings for a regular expression aliaser to map users/groups from one user store to users/groups in other user stores, based on a regular expression pattern.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchSecurityRegexAliaser -Identity regex1 -OutputUserStoreIds ln2, ln3

This example modifies the output user store ids for the "regex1" regular expression aliaser.

---------------EXAMPLE 2-----------------

Copy Code
New-FASTSearchSecurityRegexAliaserPattern -Regex .* -InputPropertyName mail -MapToUsername "\1" | Set-FASTSearchSecurityRegexAliaser -Identity regexalias1

This example sets the regular expression pattern for the Regex Aliaser "regexalias1". The regular expression pattern can be used to map the "mail" property of a Microsoft user store to identities in other user stores.

See Also

Reference

New-FASTSearchSecurityRegexAliaser
Get-FASTSearchSecurityRegexAliaser
New-FASTSearchSecurityRegexAliaserPattern


Set-FASTSearchSecurityXMLAliaser
See Also Example
[image]

Published: May 12, 2010

Updates an XML aliaser.

Set-FASTSearchSecurityXMLAliaser -Identity <String> [-InputPropertyName <String>] [-InputUserStoreId <String>] [-OutputUserStoreIds <String[]>] [-PathToXMLFile <String>] [-XMLAliaser <SecurityXMLAliaser>]

Parameters

Parameter | Required | Type | Description
Identity | Required | System.String | The identity of the XML aliaser to update.
InputPropertyName | Optional | System.String | The user/group property name this regular expression pattern is matched again. The inputPropertyName is one of the following: -- $PRINCIPAL_REFERENCE_ID - The identifier of the user or group. This is the default. Be sure to enclose in single quotes: '$PRINCIPAL_REFERENCE_ID' -- $PRINCIPAL_REFERENCE_ALIAS - The alias of the user or group. Be sure to enclose in single quotes: '$PRINCIPAL_REFERENCE_NAME' Any other property of the user or group. Refer to your specific user store for other properties that the user store exposes.
InputUserStoreId | Optional | System.String | The identity of the mapped-from user store.
OutputUserStoreIds | Optional | System.String[] | A comma-separated list of mapped-to user store identities.
PathToXMLFile | Optional | System.String | The absolute path to the XML mapping file to upload to the XML aliaser. The path must specify a file accessible to the security manager.
XMLAliaser | Optional | Microsoft.SharePoint.Search.Extended.Security.PowerShell.Commands.XmlAliaser.SecurityXMLAliaser | A SecurityXMLAliaserobject whose property values are being updated.

Detailed Description

This cmdlet updates the configuration settings for an XML principal aliaser.

An XML aliaser maps users/groups from one user store to users/groups in other user stores, based on an XML mapping file.

Input Types

Return Types

Example

---------------EXAMPLE 1-----------------

Copy Code
Set-FASTSearchSecurityXMLAliaser -Identity xmlalias1 -PathToXMLFile "C:\dev\temp\mapping1.xml"

This example uploads and updates the XML mapping for an XML aliaser.

---------------EXAMPLE 2-----------------

Copy Code
Get-FASTSearchSecurityAliaser myxmlalias | Set-FASTSearchSecurityXMLAliaser -OutputUserStoreIds ln2 -InputPropertyName '$PRINCIPAL_REFERENCE_ALIAS'

This example changes the OutputUserStoreIds and InputPropertyName for the XML aliaser piped from a Get-FASTSearchSecurityAliaser cmdlet.

See Also

Reference

New-FASTSearchSecurityXMLAliaser
Get-FASTSearchSecurityXMLAliaser


Valid value ranges for data types (FAST Search Server 2010 for SharePoint)
See Also

Managed properties in Microsoft FAST Search Server 2010 for SharePoint can be one of the following data types:

Data type Valid values Handling of items with invalid values

text

UTF-8 encoded text string.

Text from crawled properties mapped to managed properties in other character encoding is converted to UTF-8 as part of input processing. The text string can consist of words with lengths from 1 byte to a maximum of 1000 byte of length.

  • Words longer than 1000 bytes are truncated.

  • If there are more than 4294967296 words in one full text index field, or across all managed properties, the item will not be indexed, and a warning will be logged by the indexer producer.

    Note:

    By default, the maximum size of an input document is 20MB. Any documents longer than this are truncated to this length.

  • Illegal XML characters: Item indexed, illegal XML characters filtered out during item processing.

  • Illegal UTF-8 characters: Item not indexed, and warning logged during processing.

integer

Signed 64 bit integer.

  • Larger number than can fit in 64 bits: Item is not indexed, and warning logged to the index producer logs.

  • Illegal characters (non-numeric): Item not indexed, and warning logged to the index producer logs.

boolean

true|false

  • Integers are converted to Boolean values during processing (0 is treated as false, while any other positive or negative number is treated as true).

  • Other illegal characters (not true or false): Item not indexed, and warning logged to the indexer producer logs.

float

Double-precision 64-bit floating point, as specified in IEEE-754.

  • Larger than can fit in 64 bits double format: Item not indexed, and warning logged to the indexer producer logs.

decimal

Integer with up to 27 decimals

The number of decimals is set through the DecimalPlaces parameter of the managed property (see Set-FASTSearchMetadataManagedProperty).

  • Integer part larger than can fit in the available integer range with the configured decimal precision: Item not indexed, and warning logged to the index producer logs.

  • Fractional part longer than configured decimal precision: Item not indexed, and warning logged to the indexer producer logs.

  • Illegal characters (not numeric or decimal separator: Item not indexed, and warning logged to the index producer logs.

datetime

Date range in ISO8601 format.

Valid date range is -29000-01-01T00:00:00,000 to 29000-12-31T23:59:59,999.

Managed datetime properties always have sorting enabled.

  • Datetime value not compliant with ISO-8601 format: Item not indexed, and warning logged to the index producer logs.

  • Datetime value is compliant with ISO-8601 format, but outside the allowed range: Item not indexed, and warning logged to the index producer logs.

See Also

Reference

New-FASTSearchMetadataManagedProperty


Supported variant type mapping (FAST Search Server 2010 for SharePoint)
See Also

When you map a crawled property into a managed property, there has to exist a supported conversion from the variant type of the crawled property into the data type specified for the managed property in Microsoft FAST Search Server 2010 for SharePoint.

Supported variant type conversions

The following variant types can be converted into the native FAST Search Server 2010 for SharePoint types. Variant types that have no mapping will be dropped during processing. For more information about variants, see PROPVARIANT Structure.

Variant Id | Propvariant type | FAST Search Server 2010 for SharePoint type
0 | VT_EMPTY |
1 | VT_NULL |
2 | VT_I2 | int,bool
3 | VT_I4 | int,bool
4 | VT_R4 | Float
5 | VT_R8 | decimal
6 | VT_CY | int,bool
7 | VT_DATE | datetime
8 | VT_BSTR | string
9 | VT_DISPATCH |
10 | VT_ERROR |
11 | VT_BOOL | bool
12 | VT_VARIANT |
13 | VT_UNKNOWN |
14 | VT_DECIMAL | decimal
16 | VT_I1 | int,bool
17 | VT_UI1 | int,bool
18 | VT_UI2 | int,bool
19 | VT_UI4 | int,bool
20 | VT_I8 | int,bool
21 | VT_UI8 | int,bool
22 | VT_INT | int,bool
23 | VT_UINT | int,bool
30 | VT_LPSTR | string
31 | VT_LPWSTR | string
64 | VT_FILETIME | datetime
72 | VT_CLSID | string
Note:

All variant types are masked by VT_TYPEMASK = 0xFFF prior to mapping.

See Also

Reference

New-FASTSearchMetadataCrawledPropertyMapping


About command-line tools (FAST Search Server 2010 for SharePoint)

Microsoft FAST Search Server 2010 for SharePoint provides command-line tools for many system administration tasks.

Using FAST Search for SharePoint command-line tools

Many FAST Search Server 2010 for SharePoint operations are run and controlled with command-line tools, instead of from a Central Administration page. Typical tasks include monitoring system performance, running and controlling the FAST Search Web crawler, using connectors to extract documents, and controlling the FAST Search Server 2010 for SharePoint indexers.

Note:

To use the command-line tools, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Categories of command-line tools

FAST Search Server 2010 for SharePoint command-line tools cover these operational areas.

Operational Area | Command-Line Tool | Task
General administration | infotool.exe | Create a zipped system report that includes log files, configuration files, and performance data. Use for system diagnostic work.
 | rc.exe | Collect monitoring data from FAST Search Server 2010 for SharePoint components, such as the content distributor, document processors, indexing dispatcher, and indexers.
 | nctrl.exe | Manage the node controller; perform operations such as start, remove, and resume for all FAST Search Server 2010 for SharePoint components.
 | psctrl.exe | Change the settings, retrieve statistics, and view the status of a processor server.
 | nameserver_client.exe | Retrieve a set of entries from the FAST Search Server 2010 for SharePoint nameserver, representing middleware server objects. Use for debugging only.
Monitoring service | monitoringserviceconfig.exe | Install, reconfigure, or uninstall WMI classes and performance counters published by the FAST Search Server 2010 for SharePoint monitoring service.
FAST Search Web crawler | crawler.exe | Start a single node or multiple node FAST Search Web crawler. This is a server process that will rarely be run from a command line.
 | crawleradmin.exe | Configure, control, and monitor crawl collections. Add, update, or delete crawl collections. Suspend or resume content feeding.
 | crawlerdbtool.exe | View entries in the FAST Search Web crawler databases. View information about URIs that were crawled.
 | crawlerconsistency.exe | Verify and repair the consistency of the FAST Search Web crawler document and metadata structures on disk.
 | beadmin.exe | Control and monitor the browser engine. View processing statistics. Process individual URIs.
 | postprocess.exe | Manually refeed items to content indexing for one or more crawl collections.
 | ppdup.exe | Start a duplicate server to provide a centralized duplicate detection function for each node scheduler/postprocess host.
Connectors | jdbcconnector.bat | Extract items from JDBC sources that will be indexed in FAST Search Server 2010 for SharePoint
 | lotusnotesconnector.bat | Extract items from Lotus Notes sources that will be indexed in FAST Search Server 2010 for SharePoint.
 | lotusnotessecurity.bat | Extract user and group information from Lotus Domino directories that will be used for security in search results.
 | connectorpwgen.bat | Encrypt passwords used by the FAST Search database connector and the FAST Search Lotus Notes connectors.
Indexing and item processing | doclog.exe | Retrieve status and logs of recently processed items.
 | indexeradmin.exe | Remotely control one or more indexers. Suspend new items from being indexed. Enable debug logging. Defragment the intermediate FIXML files.
 | indexerinfo.exe | Retrieve information about the indexer (the number of documents indexed, the status of the indexing process).
 | docpush.exe | Feed one item into a FAST Search Server 2010 for SharePoint installation.
 | pm.exe | Diagnose problems related to item feeding and indexing. pm processes logs and reports problems from components such as the FAST Search connector Content Search Service Application, the content distributor, the indexing dispatcher, and command-line clients such as docpush.
Search | searchinfo.exe | Retrieve information about the search nodes.
Analytical tools | waadmin.exe | The WebAnalyzer analyzes data to improve the relevance of search results by counting and analyzing links. Use waadmin.exe to configure the WebAnalyzer, schedule the hyperlink analysis, and retrieve status information.
 | spreladmin.exe | SPRel is a clickthrough log analysis engine that improves search results relevancy by analyzing the entries that users click on in search result sets. Use spreladmin.exe to configure SPRel, to schedule the clickthrough log analyses, and to retrieve status information.


beadmin.exe reference
[image]

Published: May 12, 2010

Use the beadmin tool to control and monitor the browser engine. Typical tasks include viewing processing statistics from the browser engine, and processing individual URIs.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\beadmin.exe [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

beadmin options

Option (and short name) Value Description

--help (-h)

Displays help information.

--version (-v)

Displays version information.

--browserengine

<host:port>

Specifies the host name and port of the browser engine to use. Can be used with:

  • --uri

  • --urifile

  • --status

  • --statistics

Default: Queries the Microsoft FAST Search Server 2010 for SharePoint name service for the browser engine.

--loglevel (-l)

off|all|critical|error|warning|info|verbose

Specifies the log level threshold to use.

Default: info

--outfile (-o)

<path>

When using --uri or --urifile, writes the HTML body to the specified file instead of displaying it in the command line output.

--status (-a)

Displays the status of the browser engine. The output lists the currently processing URIs.

--statistics (-s)

Displays the processing statistics from the browser engine.

--uri (-u)

<uri>

Specifies a URI to be processed by the browser engine, and displays the HTTP cookies and hyperlinks found. If log level is set to verbose the item's HTML body is also displayed.

--urifile (-f)

<path>

Same as --uri, but specifies that the browser engine process all the URIs listed in the specified text file.

Use for debugging only.

Examples

This example processes a URI:

<FASTSearchFolder>\bin\beadmin.exe -u http://www.contoso.com/

The following example processes a URI and displays the HTML body:

<FASTSearchFolder>\bin\beadmin.exe -u http://www.contoso.com/ -l verbose

To display browser engine statistics, follow this example:

<FASTSearchFolder>\bin\beadmin.exe --statistics

The following example displays browser engine status:

<FASTSearchFolder>\bin\beadmin.exe --status


connectorpwgen.bat reference
[image]

Published: May 12, 2010

Use the connectorpwgen.bat command-line tool to encrypt passwords that are used by the FAST Search Lotus Notes connector and the FAST Search database connector.

When you run connectorpwgen.bat, you are prompted to type the password two times. The tool encrypts the password and prints the encrypted password on the console.

Copy the password (excluding any new lines or white spaces) and paste it into the appropriate configuration file, for example in the Domino server password parameter when you configure a FAST Search Lotus Notes connector.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\connectorpwgen.bat

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

The connectorpwgen.bat tool takes no command-line options or parameters.

Examples

Copy Code
C:\FASTSearch\bin>connectorpwgen.bat

(c) Microsoft Corporation. All rights reserved.
Enter password:
Please confirm:
Encrypted password = AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAPF3E/uqo8kuy05Vfl6hBWAAAAA
BSAAAATQBpAGMAcgBvAHMAbwBmAHQAIABFAG4AdABlAHIAcAByAGkAcwBlACAAUwBlAGEAcgBjAGgAIAA
tACAAQwBvAG4AbgBlAGMAdABvAHIAcwAAAANmAACoAAAAEAAAALG84sTQsg91I5PQX9azUCcAAAAABIAAA
KAAAAAQAAAAFYCsN/QZphncvAAveH7CjRAAAAAF6eq3gS5zFv5wP8ye68ChFAAAAHBysRCUfxLotig5I
viTQwWz9+0P

The encrypted password starts with "AQAA", ends with "9+0P", and has no newline characters. When copying more than one line from a console, the copied string contains new lines; remove the new lines when pasting the encrypted password into a configuration file.

Remarks

Important:

The encryption/decryption key depends on the user who performs the encryption/decryption. The user who will run the FAST Search Lotus Notes connector or the FAST Search database connector) must run the connectorpwgen.bat tool.

You must run the connectorpwgen.bat tool from the <FASTSearchFolder>\bin folder and have JAVA_HOME set to a 32-bit JRE (Java 1.6 or newer).


crawler.exe reference
See Also
[image]

Published: May 12, 2010

Use the crawler binary to start a FAST Search Web crawler. You can specify options to start the crawler as a stand-alone single node crawler or as part of a distributed multiple node crawler (as a multi-node scheduler or a node scheduler).

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\crawler [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

No option is required; all are optional.

crawler basic options

Option Value Description

-h

Displays help information.

-v

Displays version information.

-P

[<hostname>:]<crawlerbaseport>

Specifies the crawler base port (XML-RPC interface).

Use when you run several instances of the crawler on the same node.

  • <hostname> sets the bind address for XML-RPC interfaces (optional); can be a host name or an IP address (some hosts have multiple IP addresses).

  • <crawlerbaseport> sets the start of a port number range available to the crawler.

Default: 13000

-d

<path>

Specifies the data storage directory.

Use this option to store crawl data, runtime configuration, and logs in subdirectories in the specified directory.

Default: If the <FASTSearchFolder> environment variable is set, the default path is <FASTSearchFolder>\data\crawler; otherwise the default path is data.

-f

<XML config file>

Specifies the crawl collection configuration(s).

Use this option to specify the location of an XML file that contains one or more crawl collections.

The crawler will parse the contents of this file, add or update the specified crawl collections, and start crawling.

-c

<number>

Specifies the number of site manager processes to start, equal to or less than the number of clusters defined in the crawl collection specification.

For larger crawls, specify a process count that is equal to or greater than the number of your CPUs.

A maximum of 8 processes is supported.

Default: 2

crawler advanced options

Option Value Description

-D

<number>

Specifies the maximum DNS requests per second.

The FAST Search Web crawler has a built-in DNS lookup facility that can communicate with one or more DNS servers to perform DNS lookups. Use this option to limit the number of DNS requests per second that the crawler sends to the DNS server(s).

The DNS resolver will automatically decrease the lookup rate if it determines that the DNS server is unable to handle the current rate. The actual rates are reported in the collection statistics output.

Default: 100 requests

-F

<file>

Specifies the crawler global configuration file.

Use this option to specify the location of an XML file that contains the crawler's global configuration, which may contain default values for all command-line options.

Many options can be specified both in the XML configuration file and the crawler command-line tool. Options specified in the command-line tool take precedence.

At startup, the crawler will look for a default CrawlerGlobalDefaults.xml configuration file, located in the current directory or the <FASTSearchFolder>\etc folder.

-n

Shuts down the crawler when idle.

The refresh setting in a crawl collection must be higher than the time that is required to crawl the complete collection.

Default: disabled

-T

Enables profiling.

Use for debugging only.

-t

Enables profiling using the hotshot module.

Use for debugging only.

crawler logging options

Option Value Description

-L

<path>

Specifies the log storage directory.

Use this option to store crawler specific logs in subdirectories of the specified directory.

Default: If the %FASTSEARCH% environment variable is set, the default path is %FASTSEARCH%\var\log\crawler; otherwise the default path is data\log.

-q

Disables verbose logging.

Use this option to log only CRITICAL, ERROR, and WARNING log messages.

-l

<log level>

Specifies the kind of information to log:

  • debug

  • verbose

  • info

  • warning

  • error

crawler integration options

Option | Value | Description
-o |  | Enables FAST Search Server 2010 for SharePoint mode. Use this option when you run the crawler in a FAST Search Server 2010 for SharePoint setting.
-i |  | Ignores the configuration server. The crawler continues to run even if the configuration server is unreachable.

crawler multi-node options

Option Value Description

-U

Runs the FAST Search Web crawler as a multi-node scheduler in a multiple node setup.

Use the -S option to connect node schedulers to the XML-RPC port.

-S

<multinode scheduler host>:<multinode scheduler port>

Starts the crawler as a node scheduler to a multi-node scheduler.

Specifies the host name and port number of the multi-node scheduler.

Example: crawler1.contoso.com:13000

-s

Enables survival mode in a distributed setup, keeping the node scheduler alive and trying to reconnect to the multi-node scheduler until a successful connection is made.

This option only applies to the node scheduler.

-I

<node identifier>

Specifies a symbolic name for the crawler node.

Use of this option is rare. In a multiple node crawler setup, each crawler node must have a unique symbolic name, used by crawl collection configurations to specify the crawler nodes included in a crawl.

This option only applies to the node scheduler. The default value is auto generated and stored in the configuration database. If you use this option, you only have to specify an alternative value once: the first time that the crawler is started.

Examples

To start a stand-alone single node crawler at port 13000, follow this example:

<FASTSearchFolder>\bin\crawler -o -P 13000

This example starts a multi-node scheduler in a multiple node setup at port 13000:

<FASTSearchFolder>\bin\crawler -o -P 13000 -U

The following example starts a node scheduler in a multiple node setup on a different node connecting to the multinode scheduler:

<FASTSearchFolder>\bin\crawler -o -P 13000 -S crawler1.contoso.com:13000

See Also

Concepts

crawlerglobaldefaults.xml reference


crawleradmin.exe reference
[image]

Published: May 12, 2010

Use the crawleradmin tool to configure, control, and monitor crawl collections. For example, use crawleradmin to add, update, or delete crawl collections; to suspend or resume content feeding; or to monitor a crawl in progress (using the FAST Search Web crawler). A crawl collection is a set of Web sites that are crawled using the same crawl configuration.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\crawleradmin [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

crawleradmin general options

Option (and short name) Value Required Description

--crawlernode (-C)

<hostname>[:<port>]

No

Manages the FAST Search Web crawler at the specified host name and port (optional).

Default: localhost:13000

--offline (-o)

<configuration_directory>

No

Accesses databases that are locked by a running crawler so that crawleradmin can work directly on the databases (offline mode), not through the crawler administrative API.

The crawler must be stopped before issuing this command.

If you do not specify a configuration_directory for the crawler, the command uses the default configuration directory: FASTSEARCH\data\crawler\config

Or, if the FASTSEARCH environment variable is not set: data\config

You can use this option with:

  • -a

  • -d

  • -c

  • -q

  • -G

  • -f

  • -d

  • --getdata

  • --verifyuri

-l

<log-level>

No

Specifies the kind of information to log:

  • debug

  • verbose

  • info

  • warning

  • error

-h

No

Displays help information.

-v

No

Displays version information.

crawler configuration options

Option (and short name) | Value | Required | Description
--addconfig (-f) | <Path_to_XML_file> | No | Adds or updates a crawl collection configuration(s) from the specified XML file.
--getcollection (-G) | <crawl_collection> | No | Outputs the XML configuration for the specified crawl collection name (defined in the crawler XML configuration file) to stdout and saves stdout to a file.
--delcollection (-d) | <crawl_collection> | No | Deletes a crawl collection and its stored content from the crawler.
--encrypt (-e) | <cachesize> | No | Encrypts a password for security in a crawl collection configuration file.

crawler control options

Option (and short name) | Value | Required | Description
--suspendcollection (-s) | <crawl_collection> | No | Suspends crawling of a specified crawl collection. Feeding of items in the feed queue continues.
--resumecollection (-r) | <crawl_collection> | No | Resumes crawling of the specified crawl collection.
--suspendfeed | <crawl_collection>[:targets] | No | Suspends content feeding of a crawl collection. Optionally specify a comma-separated list of feeding targets (symbolic names in the crawl collection configuration).
--resumefeed | <crawl_collection>[:targets] | No | Resumes content feeding of a crawl collection. Optionally specify a comma-separated list of feeding targets (symbolic names in the crawl collection configuration).
--enable-refreshing-crawlmode | <crawl_collection> | No | Enables the refresh crawl mode for a crawl collection. If enabled, the crawler only crawls/refreshes URIs that were previously crawled.
--disable-refreshing-crawlmode | <crawl_collection> | No | Disables the refresh crawl mode for a crawl collection, and resumes regular crawl mode.

URI submission, refetching, and refeeding options

Option (and short name) Value Required? Description

--adduri (-u)

<crawl_collection>:<URI>

No

Appends the specified URI to the crawl collection work queue.

Combine with the --force flag to prepend the URIs and crawl them immediately.

--addurifile

<crawl_collection>:<file>

No

Appends all URIs from the specified file to the crawl collection work queue.

Combine with the --force flag to prepend the URIs and crawl them immediately.

--refetch (-F)

<crawl_collection>

No

Forces a re-fetch of a crawl collection.

The crawler will delete all existing work queues, clear all caches, start a new crawl cycle, and put all known start URIs on the work queue. Does not increment the counter used for orphan detection (dbswitch).

--refetchuri (-F)

<crawl_collection>:<URI>

No

Forces a re-fetch of a URI in the specified crawl collection.

The URI needn't be previously crawled. However, it must fall within the include/exclude rules for the crawl collection.

This also triggers crawling of the site to which the URI belongs (unless the site was already crawled in this refresh period).

--refetchsite

<crawl_collection>:<URI>

No

Forces a re-fetch of the crawl site from a URI in the specified crawl collection.

--force

No

Assures that a URI is crawled immediately (potentially preempting active sites) when it is used with:

  • --adduri

  • --addurifile

  • --refetchuri

  • --refetchsite

--feed

No

Refeeds URIs to the content indexing process (regardless of item change status) when it is used with:

  • --refetchuri

  • --refetchsite

--refeedsite

<crawl_collection>:<crawl_site>

No

Refeeds all items in the crawler store to the content indexing process for a crawl site.

A <crawl_site> is a combination of a host name (may be fully qualified) and port. Together they uniquely identify a server which may host a Web server.

This is the same as running the command-line tool and parameter postprocess -R on a single site, but does not require stopping the crawler. You should limit the number of concurrent re-feeds at run time to prevent overloading the crawler.

The refed URIs are put in a high priority queue. Feeding will occur from both the high priority and regular priority queues at the same time. You may notice a delay before an item is searchable.

--refeeduri

<crawl_collection>:<URI>

No

Refeeds the specified URI from the crawler store to the indexing process.

See --refeedsite for more information.

--refeedprefix

<prefix>

No

Specifies a URI prefix (including scheme) that URIs must match for refeeding.

Use with --refeedsite.

--refeedtarget

<destination>:<crawl_collection>

No

Specifies a feeding destination and content collection for the --refeed option.

Preempting, quarantine, and deletion options

Option (and short name) | Value | Required? | Description
--preemptsite (-p) | <crawl_collection>:<crawl_site> | No | Preempts crawling of a site for a specified crawl collection.
--quarantine | <crawl_collection>:<crawl_site>:<time> | No | Blocks a site from crawling for a specified number of seconds.
--unquarantine | <crawl_collection>:<crawl_site> | No | Removes a crawl site from quarantine for a specified crawl collection.
--deletesite | <crawl_collection>:<crawl_site> | No | Deletes a crawl site from the crawler for a specified crawl collection.
--deluri | <crawl_collection>:<URI> | No | Deletes a URI from a crawl collection.
--delurifile | <crawl_collection>:<file> | No | Deletes URIs from a crawl collection. <file> is a newline-separated file of URIs that should be removed from the crawl collection.

Statistics options

Option (and short name) Value Required? Description

--collstats (-q)

<crawl_collection>

No

Displays crawl statistics for a crawl collection.

--collstatsquiet (-Q)

<crawl_collection>

No

Displays a summary of crawl statistics for a crawl collection.

--statistics (-c)

No

Displays crawl statistics for all collections.

--sitestats

<crawl_collection>:<crawl_site>

No

Displays crawl statistics for a specified crawl site in a crawl collection.

--cycle

A number from 1 to n, or "all"

No

Displays statistics from a specific crawl refresh cycle when it is used with:

  • --collstats

  • --collstatsquiet

  • --statistics

  • --sitestats

Specify "all" to merge statistics from all refresh cycles.

Default: the current cycle

Monitoring options

Option (and short name) Value Required? Description

--status

No

Displays the status for all crawl collections, including the following:

  • Crawl collection name

  • Status (e.g., Idle)

  • Feeding status (e.g., Feeding)

  • Number of active sites

  • Number of stored items

  • Item Rate

--nodestatus

No

Displays the status (per node) for all crawl collections (information reported resembles the items in the --status option

--active (-a)

No

Displays all active crawl collection names.

--nummanagers (-n)

No

Displays the number of crawl sites currently being crawled.

--sitemanagerstatus (-S)

<id>

No

Displays the status for the specified site manager <id>.

Status includes a list of sites that are currently crawling at the specified site manager, and additional information such as the work queue size, pending URIs, and global cache statistics.

--numworkers (-N)

<id>

No

Displays the number of active sites for a site manager.

--sites (-t)

<id>

No

Lists sites currently being crawled by a site manager.

--starturistat

No

Displays the feeding status of start URI files.

Debugging options (for advanced users)

Option (and short name) Value Required? Description

--getlogin

<form URI>

No

Downloads a <form URI>, extracts logon information from the form, and produces an XML configuration from the form document.

--extractlinks

<URI>

No

Downloads a URI and extracts forward links from it.

--addheader

<http header string>

No

Adds additional HTTP headers to the HTTP requests when it is used with:

  • --getlogin

  • --extractlinks

--verifyuri

<crawl_collection>:<URI>

No

Verifies if a URI can be crawled for a crawl collection (based on the crawl collection configuration's include and exclude rules).

--getdata

<crawl_collection>:<URI>

No

Retrieves the downloaded content from the crawler store for a URI in one crawl collection.

--idn

<URI>

No

Gets an IDNA encoded version of a URI.

Examples

To add or update the crawler with a collection configuration:

<FASTSearchFolder>\bin\crawleradmin -f MyCrawlCollectionConfig.xml

To remove a collection from the crawler:

<FASTSearchFolder>\bin\crawleradmin -d MyCollection

To suspend crawling of a collection:

<FASTSearchFolder>\bin\crawleradmin --suspendcollection MyCollection

To add a URI to a collection for crawling:

<FASTSearchFolder>\bin\crawleradmin --adduri MyCollection:http://www.contoso.com/

To show statistics for a crawl collection:

<FASTSearchFolder>\bin\crawleradmin -q MyCollection

To show statistics for a specific site in a crawl collection:

<FASTSearchFolder>\bin\crawleradmin --sitestats MyCollection:www.contoso.com

To temporarily block the www.contoso.com site from being crawled for 1 hour (even if the site is currently being crawled):

<FASTSearchFolder>\bin\crawleradmin --quarantine MyCollection:www.contoso.com:3600

To show the status for all collections in the crawler:

<FASTSearchFolder>\bin\crawleradmin --status


crawlerconsistency.exe reference
See Also
[image]

Published: May 12, 2010

Use the crawlerconsistency tool to verify and repair the consistency of the crawler item and metadata structures on disk. You can also use the tool to verify and maintain internal crawler store consistency, or when recovering a damaged crawl store.

By default, the tool detects and attempts to repair the following inconsistencies:

  • Items referenced in metadatabases, but not found in the item store.

  • Invalid items in the item store.

  • Unreferenced items in the item store (requires the docrebuild mode.)

  • Duplicate database checksums not found in metadatabases.

  • Multiple checksums assigned to the same URI in the duplicate database.

These inconsistencies are automatically corrected in the doccheck or docrebuild mode, followed by the metacheck mode. Any non-consistent URIs are logged, and a delete operation is issued to the indexer (which can be disabled) to ensure synchronization.

In a multi-node crawler environment, you can also use the tool to rebuild a duplicate server from the contents of per-node scheduler post-process checksum databases by using the ppduprebuild mode. Since this mode builds the duplicate server from scratch, you can also use it to change the number of duplicate servers that are used, by first changing the configuration and then rebuilding.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\crawlerconsistency [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

crawlerconsistency options

Option Value Required Description

-M

<mode>[<mode,..., <mode>]

Yes

Specifies one or more modes to run the tool in:

  • doccheck - Verifies that all items referenced in the metadatabases also exist on disk.

  • docrebuild - Same as doccheck, but rewrites all referenced items to a fresh item store, removing orphans in the item store.

    Note:

    This can take a long time.

  • metacheck - Verifies that all checksums referenced in the postprocess databases also exist in the metadatabases.

  • metarebuild - Attempts to recovery a damaged metastore. Supports rebuilding a bad or lost site database based on segment databases.

  • duprebuild - Rebuilds the contents of the duplicate server(s) from the local post process database. Run this mode alone (without other modes).

Additionally, you can add the following modifiers:

  • updatestat - Updates the statistics item store counter.

    Note:

    Use with the doccheck and docrebuild modes only

  • routecheck - Verifies that sites/URIs are routed to the correct node scheduler.

Include additional modifiers in a comma-separated <mode> list. For example:

-M doccheck,docrebuild,updatestat

Applies to multi-node crawlers only.

-O

<path>

Yes

The folder for all output logs.

The tool creates a subfolder named with the current date: <year><month><date>.

If the subdirectory already exists, a number is appended (e.g. ".1") to the name.

-d

<path>

No

Location of crawl data, run-time configuration, and logs in subdirectories in the specified directory.

Default: data

-U

No

Indicates that you are running the tool on the multi-node scheduler and that data is in a different folder: data\crawler\config\multinode vs. data\crawler\config\node.

Applies to routecheck mode.

-C

<crawl_collection>[,<crawl_collection>,...,<crawl_collection>]

No

A comma-separated list of collections to check.

Default: all collections

-c

<cluster>[,<cluster>,...,<cluster>]

No

A comma-separated list of clusters to check.

Applies to doccheck and docrebuild modes.

Default: all clusters

-S

<crawl_site>[,<crawl_site>,...,<crawl_site>]

No

Only process the specified site(s).

Applies to doccheck mode.

Default: All sites

-z

No

Compresses items in the item store when you run the docrebuild mode. This overrides the collection level option to compress items (if you specify it).

Default: off

-i

No

Skips free disk space checks. Normally the tool checks the free disk space periodically; if it drops under 1GB, it will stop the operation and exit.

Caution:

Use this option with caution.

-n

No

Indicates that delete operations are not submitted to the indexer; they are only logged to files.

To ensure that deleted items are not left in the index, manually delete those items, or refeed the collection into an empty index.

-F

<file>

No

Loads the crawler global configuration from <file>. Any conflicting options on the command line override values in the file.

-T

No

Runs the tool in test mode. The tool does not delete anything from disk or issue any deletes to the indexer.

-h

No

Displays help.

-v

No

Displays version information.

-l

<log-level>

No

Specifies the kind of information to log:

  • debug

  • verbose

  • info

  • warning

  • error

Examples

The following example verifies and repairs item store and metastore consistency, and updates statistics counters.

<FASTSearchFolder>\bin\crawlerconsistency -M doccheck,metacheck,updatestat -O <FASTSearchFolder>\var\log\crawler\consistency\ -C MyCollection

This will verify each metadatabase entry, verify any corresponding item content in the crawler store, and log inconsistencies in the specified log files.

Remarks

The tool generates the following log files. Log files are only created when the first URI is written to a file.

Log file name Description

<mode>_ok.txt

Lists URIs found that were not removed as inconsistencies.

The output from the metacheck mode lists every URI with a unique checksum, useful for comparing against the index.

Note:

Items may have been dropped by the pipeline, leaving URIs in this file that are not in the index. You can safely remove URIs in the index that are not in this file.

<mode>_deleted.txt

Lists URIs deleted by the tool. Unless you disabled indexer deletes with the -n option, the URIs were removed from the index. Since these URIs were deleted as crawler inconsistencies, they may still exist on the Web servers and should be indexed. Recrawl this list of URIs with the crawleradmin tool by using the --addurifile option (also use the --force option to speed up crawling).

<mode>_deleted_reasons.txt

This log file is identical to the <mode>_deleted.txt file but also includes an "error code" to identify each URI delete reason. Definitions for each error code include the following:

  • 101 - Item not found in item store

  • 102 - Item found, but unreadable, in item store

  • 103 - Item found, but length does not match meta information

  • 201 - Metadata for item not found

  • 202 - Metadata found, but unreadable

  • 203 - Metadata found, but does not match checksum in duplicate database

  • 204 - Metadata found, but has no checksum

  • 206 - URI's host name not found in routing database

<mode>_wrongnode.txt

Used for multi-node crawls, this file outputs all URIs removed from a node because of incorrect routing. These URIs should be crawled by a different master node. The URIs are logged, but not deleted from the index.

<mode>_refeed.txt

Lists URIs that had their URI equivalence class updated by running the tool. To bring the index in sync, use postprocess refeed with the -i option to refeed the contents of this file. Or, perform a full refeed.

Note:

Always redirect the stdout and stderr output to a log file on disk.

See Also

Reference

crawleradmin.exe reference

Concepts

Web Crawler XML configuration reference


crawlerdbtool.exe reference
[image]

Published: May 12, 2010

Use the crawlerdbtool tool to view entries from the crawler databases. This includes information that is stored about the URIs that were crawled with the FAST Search Web crawler.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where Microsoft FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\crawlerdbtool [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

crawlerdbtool general options

Option | Value | Required | Description
-h |  | No | Displays help information.
-v |  | No | Displays version information.

crawlerdbtool mode options

Option Value Required Description

-m

<mode>

No

Specifies the mode for running the crawlerdbtool:

  • check - Reports corrupted databases only.

  • repair - Attempts to repair corrupted databases by copying elements to new databases. New databases are verified before replacing corrupted databases.

  • delete - Deletes corrupted databases.

  • compact - Compacts a database (specify file name or directory) or item store cluster (specify cluster directory).

  • list - Outputs all keys in a database.

  • count - Counts the number of keys in a database.

  • view - Displays an entry in a database based on the key that is specified with -k. If no key is specified, all keys are output.

  • viewraw - Displays the same data as the view mode, but without any formatting.

  • export - Exports a database to marshalled data.

  • import - Imports a database from marshalled data.

  • align - Aligns a routing database with a duplicate server database. Only the routing database is altered. It must be manually copied to all node schedulers afterward.

  • reroute - Creates a new routing database based on the contents in the metastore databases. Must be run on all nodes with the same routetab.hashdb and then distributed to all nodes.

  • analyze - Analyzes a meta database.

  • bloomfix - Recreates the bloom filter for the specified database. The -b option must be specified.

  • pphl2gb - Convert a postprocess checksum database from hashlog to gigabase format.

Default mode: check

-d

<directory|file>

Yes

Specifies a directory or file to process.

Required for all modes except align mode.

The -f option is ignored if a file is specified.

-f

<filemask>

No

Specifies the filemask/wildcard to work on. To specify multiple filemasks, use multiple parameters, such as the following:

-f *.metadb -f *.hashdb

Default: *

-c

<cachesize>

No

Specifies the cache size (in bytes) to be used when you open databases.

Default: 8388608

-s

<frequency>

No

Specifies the frequency of database synchronizations during repair. Enter the number of operations between synchronizations. A value of 1 will synchronize after each operation.

Default: 10

-t

<time-out>

No

Specifies a time-out in seconds after which a database check/repair process is ended. The database will be considered corrupted and beyond repair, and will be deleted.

Caution:

Use with caution.

Default: none

-k

<key>

Specifies the database key to view.

Applies to view mode only.

-K

<key>

Same as -k, but assumes that the key is repaired and will call eval() before using it.

Use for viewing entries in postprocess or duplicate server databases.

-S

<site>

No

Specifies a crawl site to apply the current mode to. Use for inspecting meta databases.

If <site> is "all", all sites will be traversed.

If <site> is "list", all sites will be listed.

-p

<duplicate server db directory>

or

<metastore cluster directory>

No

Applies to align and reroute modes only.

In align mode, specifies the duplicate server database directory.

In reroute mode, specifies the metastore database cluster directory.

-r

<routing database file>

No

Applies to align and reroute modes only.Specifies the routing database file. For example:

<FASTSearchFolder>\data\crawler\config\MyCollection\routetab.hashdb

-M

<node1, node2,...,nodeN>

or

<current node scheduler identifier>

No

Applies to align and reroute modes only.

In align mode, specifies the node schedulers active in the routing scheme in a comma-separated list (no spaces allowed). For example:

-M node1,node2,node3

In reroute mode, specifies the node scheduler identifier of the current node.

-D

No

Applies to align and reroute modes only.

Enables domain based routing (instead of host name based).

-i

<intermediate format>

No

Applies to import and export modes only.

Exports to or imports from the specified format.

Valid formats:

  • marshal - fast space-efficient format

  • pickle - version and platform independent format

Default: marshal

-e

<maximum entries>

No

Applies to export mode only.

Specifies an upper limit to the number of elements stored per exported file.

Default: 10000

-b

<bits>

No

Applies to bloomfix mode only.

Specifies the number of bits to use in the bloom filter.

Examples

The following example lists all URIs crawled from the site www.contoso.com for the crawl collection MyCollection.

<FASTSearchFolder>\bin\crawlerdbtool -m list -d <FASTSearchFolder>\data\crawler\store\MyCollection\db -S www.contoso.com

The following example lists all known sites crawled for the collection MyCollection.

<FASTSearchFolder>\bin\crawlerdbtool -m list -d <FASTSearchFolder>\data\crawler\store\MyCollection\db -S all

The following example shows statistics for all URIs crawled for the collection MyCollection.

<FASTSearchFolder>\bin\crawlerdbtool -m analyze -d <FASTSearchFolder>\data\crawler\store\MyCollection\db -S all

The following example lists all URIs (keys) in a database for the crawl collection MyCollection.

<FASTSearchFolder>\bin\crawlerdbtool -m list <FASTSearchFolder>\data\crawler\store\MyCollection\db\1\0.metadb2

To view the record of a specific URI (database key) for the collection MyCollection, follow this example.

<FASTSearchFolder>\bin\crawlerdbtool -m view <FASTSearchFolder>\data\crawler\store\MyCollection\db\1\0.metadb2 -k "http://www.contoso.com/example.html"


doclog.exe reference
[image]

Published: May 12, 2010

Use the doclog tool to retrieve the status and the logs of recently processed items (documents).

By default, the logs from all item processor servers that are registered with the configuration component are aggregated.

Note:

An identifier can be processed on different item processor servers, and logs from previous runs can still be present in the system. To show the most recent log entry for each identifier, use the -a command option.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\doclog [options] <command> [ID ...]

Parameters

Parameter | Description
[ID …] | A list of one or more item identifiers separated by a space
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

doclog options

Option | Value | Description
-C | <host:port> | Specifies the host name and port number of the configuration component. The default values are loaded from <FASTSearchFolder>\etc\CSLocation.xml.
-P | <host:port> | Specifies the host name and port number to poll one specific item processor server.

doclog commands

Command | Description
-a | Shows all available item logs with the most recent log entry per identifier.
-b | Shows all available batch logs.
-e | Shows all item logs that contain errors.
-h | Displays help.
-l | Lists identifiers per server. Only identifiers with log entries are listed. An identifier can appear on several servers.
-v | Shows version information.
-w | Shows all item logs that contain warnings.

Examples

To list all item identifiers per server:

<FASTSearchFolder>\bin\doclog -l

To retrieve the log entries for one specific document:

<FASTSearchFolder>\bin\doclog documentID_1


docpush reference
[image]

Published: May 12, 2010

docpush is a test and diagnostics tool that you can use to feed a single item into a Microsoft FAST Search Server 2010 for SharePoint farm. You can also use the tool to delete single items from the index.

docpush has two modes of operation:

  • Redirect mode, enabled with the -r option

  • Direct mode, which is enabled by default

Use redirect mode (-r) to specify one or more relative file system references to have the DocumentRetriever item processing stage in the processor server download the items. Redirect mode assumes that all running processor servers have access to the specified file system reference, e.g. through a mounted share if the processor servers are running on a different network host.

You must specify one or more file references or URIs after the global options.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\docpush [global options] <file or URI> ... <file or URI>

Parameters

Parameter | Description
<file or URI> ... <file or URI> | Specifies the files or URIs to feed.
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

docpush global options

Option Value Required? Description

-h

No

Displays help information.

-v

No

Displays version information.

-c

<collection>

Yes

Specifies the content collection name. A collection called sp is created when FAST Search Server 2010 for SharePoint is deployed.

-r

No

Uses redirection instead of reading and submitting the file content directly. docpush will send a reference to the file or item as a URI instead of reading the content locally and submitting it directly.

Note:

This option will not work if a firewall or proxy server does not allow the item processing servers within the FAST Search Server 2010 for SharePoint farm to access to the file or item. The item processing server must have read access rights to the file share and the file itself.

-g

No

Redirection prefix. This specifies a common file prefix that is typically used when several file references are specified for submission.

Default: file:/<current directory>/

-u

<URI prefix>

No

Specifies a URI prefix that is prepended to a file name reference and used as the item ID when the file is submitted. Used with direct mode only.

Default: http://cohowinery.com/

Note:

This option is normally not applicable, as the firewall or proxy settings will not enable the item processing servers to access external URIs.

-U

No

Pushes URIs on the command line. Redirect mode (-r) is automatically enabled with this option.

The URI specifies the item ID and the getpath item attribute for the operation that is fed to the system. No local content/files are required when this option is used.

-d

No

Deletes an item, instead of pushing it.

-b

<number of documents>

No

When several items are pushed at the same time, sends them as a batch with the specified number of items in each batch.

-C

<callback>

No

Specifies the last callback in the feeding chain to wait for. The callbacks available in the default feeding chain are as follows:

  • PROCESSED

  • PERSISTED

  • LIVE

-V

No

Enables verbose output.

-l

<debug level>

No

Sets the debug level. Valid options are as follows:

  • debug

  • error

  • info

  • verbose

  • warning

Examples

The following example submits the URI http://cohowinery.com to the sp collection (the DocumentRetriever item processing stage will download the item referenced by the URI):

<FASTSearchFolder>\bin\docpush -c sp -U http://cohowinery.com

This example feeds the file c:\test.txt to the sp collection:

<FASTSearchFolder>\bin\docpush -c sp c:\test.txt

The following example reads the file test.txt in the current directory and feeds it to the sp collection:

<FASTSearchFolder>\bin\docpush -c sp -r test.txt

To read the file c:\test.txt and feed it to the sp collection with an absolute prefix of file:///, follow this example:

<FASTSearchFolder>\bin\docpush -c sp -r -g file:/// C:\test.txt


indexeradmin.exe reference (FAST Search Server 2010 for SharePoint)
See Also
[image]

Published: May 12, 2010

Use the indexer administration tool to remotely control one or more indexing components. Use indexeradmin to suspend new items from being indexed, to disable/enable debug logging, or to defragment the intermediate FIXML files. The indexer must be running when you use the indexeradmin tool.

Note:

The indexeradmin commands are asynchronous; under heavy content load, indexeradmin commands may not take effect immediately.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\indexeradmin [options] <command>

Where:

  • <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

indexeradmin options

Option (and short name) Default Value Description

--help (-h)

<none>

Displays help.

--version (-v)

<none>

Displays version information.

-a

<none>

Runs the commands for all indexers (index columns and indexer rows) in the FAST Search Server 2010 for SharePoint farm.

--clustername

webcluster

This option is not used. Do not set to another value than the default value.

--subsystem

indexing

This option is not used. Do not set to another value than the default value.

--column=

0

Request information for a specific index column.

Set this option without specifying a row option to run the command on all indexer servers (primary and backup) in the specified index column.

To find a column identifier, use the command indexerinfo -a status or look up the column number for the indexer servers in the deployment.xml configuration file.

--row=

0

Request information for a specific indexer row. Row 0 is the primary indexer.

Set this option without specifying a column option to run the command on all indexer servers of the same type (primary or backup) in the specified indexer row.

To find a row identifier, use the command indexerinfo -a status or look up the row number for the indexer servers in the deployment.xml configuration file.

indexeradmin commands

Command Description

disabledebug

Disables debug logging.

Disabling debug logging is not persisted to the configuration file. This change will be lost when the indexer restarts.

enabledebug

Enables debug logging for troubleshooting.

Enabling debug logging is not persisted to the configuration file. This change will be lost when the indexer restarts.

resetindex

Re-indexes all content from the pre-index content store (FiXML files).

If indexing is suspended, resetindex will resume it.

For more information, see Remarks.

resumeindexing

Resumes indexing on a suspended indexer.

selfcheck

Performs an indexer status and content consistency check. This operation can run a long time and cannot be canceled.

suspendindexing

Suspends indexing.

When indexing is suspended, the indexer will not start any new indexing jobs. However, the indexer continues to receive items for indexing. These items are stored in FiXML files.

Use this command when backing up the data\data_index folder and for conserving system resources with large document submissions.

enabletracemode

Enables debug tracing of indexed items. This produces a very detailed log of all indexing activities.

disabletracemode

Disables debug tracing of indexed items.

Examples

To enable debug logging on all indexers (primary and backup) in column 2:

Copy Code
indexeradmin --column=2 enabledebug

To re-build the content index from the FiXML files in column 1:

Copy Code
indexeradmin --column=1 resetindex

Remarks

After item processing the crawled items are distributed to individual index columns based on a statistical routing algorithm. The items are stored in an XML-based pre-index format (FiXML files). The indexer uses the FiXML files as input to the indexing process.

The indexeradmin resetindex command re-builds the content index from the FiXML files. This is normally needed only if the content index files are corrupted or damaged.

Search is available when the resetindex command runs, but new content will not be indexed before the command is finished. Depending on the amount of items, the resetindex command may take many hours.

Note:

The resetindex command is different from the SharePoint Server 2010 index reset procedure described in Reset the content index (FAST Search Server 2010 for SharePoint).

The indexeradmin resetindex command performs a local re-build of the content index within one or more indexer server.

The SharePoint Server 2010 index reset procedure will erase the content index and re-crawl all content sources.

See Also

Concepts

Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
deployment.xml reference


indexerinfo.exe reference (FAST Search Server 2010 for SharePoint)
See Also
[image]

Published: May 12, 2010

Use the indexerinfo tool to retrieve information about the Microsoft FAST Search Server 2010 for SharePoint indexer, such as the number of items indexed or the status of the indexing process.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\indexerinfo [options] <command>

Where:

  • <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

indexerinfo options

Option (and short name) Default Value Description

--help (-h)

<none>

Displays help.

--version (-v)

<none>

Displays version information.

-a

<none>

Runs the commands for all indexers (index columns and indexer rows) in the FAST Search Server 2010 for SharePoint farm.

--clustername

webcluster

This option is not used. Do not set to another value than the default value.

--subsystem

indexing

This option is not used. Do not set to another value than the default value.

--column=

0

Request information for a specific index column.

Set this option without specifying a row option to run the command on all indexer servers (primary and backup) in the specified index column.

To find a column identifier, use the command indexerinfo -a status or look up the column number for the indexer servers in the deployment.xml configuration file.

--row=

0

Request information for a specific indexer row. Row 0 is the primary indexer.

Set this option without specifying a column option to run the command on all indexer servers of the same type (primary or backup) in the specified indexer row.

To find a row identifier, use the command indexerinfo -a status or look up the row number for the indexer servers in the deployment.xml configuration file.

indexerinfo commands

Command Value Description

doccount

<collection>

Returns the total number of items in the index column. Specify a content collection to retrieve the item count for one content collection within the index column.

If indexing is ongoing, this number may include some unindexed items which have arrived in the indexer but which have not been indexed yet. To determine how many items have actually been indexed, use the status command.

reportcontents

<collection>

Returns a list of all items for the indexer or for a specific collection. Items are reported as internal identifiers in the format: <internal_doc_id>_<collection_id>. Used for debugging.

Note:

This command will impact indexer performance when running.

fixmlfillrate

Identifies the total number of valid (active) items vs. invalidated (deleted/updated) items. When an item is deleted or updated, the old version is not removed immediately. Very low fill rates can affect indexing performance.

Note:

This command is related to the pre-index item storage (FiXML files).

listcontrollers

Lists all query matching servers connected to the indexer.

hasdocument

< item>

Checks an indexer for one item, specified as an internal identifier in the format: <internal_doc_id>_<collection_id>. Used for debugging.

activeindexset

Reports the current active index set on the indexer. The index set determines which version of the index is used for each indexing partition. Used for debugging.

status

Retrieves a status report about the indexer(s) in XML format. For details on the XML status report syntax, see indexerinfo XML status report syntax (FAST Search Server 2010 for SharePoint).

The report provides the following main information:

  • Total number of items indexed in each partition.

  • Total number of items indexed in this indexer, and total number not yet indexed.

  • Status of the indexing process on each partition (idle, waiting, indexing or copying).

  • The last operation received by the indexer, and when it occurred.

  • The column role of the indexer, for fault tolerance (primary or backup)

Tip:

You can get a similar XML status report about search status from the searchinfo tool.

Examples

To report the number of items in each indexer, run the following command in a FAST Search Server 2010 for SharePoint shell:

Copy Code
indexerinfo -a doccount

To retrieve an XML status report for the primary indexer in index column 1, run the following command in a FAST Search Server 2010 for SharePoint shell:

Copy Code
indexerinfo --row=0 --column=1 status

See Also

Concepts

Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
deployment.xml reference
indexerinfo XML status report syntax (FAST Search Server 2010 for SharePoint)


indexerinfo XML status report syntax (FAST Search Server 2010 for SharePoint)
See Also

Published: December 2, 2010

The indexerinfo status command retrieves an XML-formatted report for the Microsoft FAST Search Server 2010 for SharePoint indexer status. This article describes the syntax of the report.

Element quick reference

The report uses the following elements:

Element | Description
<indexer> | This top-level element specifies the overall status of the indexer.
<documents> | This element shows information about indexed items.
<column_role> | This element specifies the role of an indexer in an index column.
<index_frequence> | This element shows the number of items indexed per second.
<partition> | This element shows information for one index partition.
<partition/documents> | This element shows the number of indexed items in an index partition.
<document_api> | This element shows status information related to the input API queues for the indexer.
<queue_size> | This element shows the size of the input API queues for the indexer.
<operations_processed> | This element shows the number of item batch operations processed by the indexer.

indexerinfo status XML format

The status output follows all conventions and syntax rules that are defined for the XML file format.

The basic XML structure is as follows:

Copy Code
<?xml version="1.0"?>
<indexer >
  <documents />
  <column_role />
  <index_frequence />
  <partition>
    <documents />
  </partition>
  <document_api>
    <queue_size />
    <operations_processed />
  </document_api>
</indexer>

indexer

This is the top-level element. The element specifies the overall status of the indexer.

Attributes

Attribute Description

hostname

The fully qualified domain name of the server that runs the indexer.

port

The lowest port in the range of ports that are used by the indexer.

cluster

This attribute contains information that is mainly used for debugging.

column

The index column identifier.

You can look up the column identifiers for the indexers in the deployment.xml configuration file.

row

The indexer row identifier.

You can look up the row identifiers for the indexers in the deployment.xml configuration file.

factory_type

This attribute contains information that is mainly used for debugging.

preferred_master

This attribute contains information that is mainly used for debugging.

connected_feeds

The number of active feeding (crawling) sessions that are associated with the indexer.

status

Shows the status of the indexer:

  • running ok: The indexer is running in regular operation mode.

  • reset index in progress: The indexer is re-building the content index.

    Note:

    This is a local re-build of the content index, and does not include a re-crawl of the content.

  • indexing suspended: The indexer is suspended.

heartbeat

Shows the status of the indexer heartbeat:

  • Running: The indexer heartbeat is running.

  • Suspended: The indexer heartbeat is suspended. This may occur when the backup indexer is out of sync with the primary indexer.

The backup indexer uses the heartbeat to verify that a primary indexer is available.

exclusion_interval

This attribute contains information that is mainly used for debugging.

time_since_last_index

Shows the number of seconds since a new index set was created.

Remarks

If a local re-build of the index (reset index) is in progress, you can determine the progress by looking at the status for each index partition (the status attribute for the partition sub-elements).

documents

This element contains information about indexed items related to the indexer.

Attributes

Attribute | Description
size | Shows the total size of the pre-index item store (FiXML files), in bytes.
total | Shows the total number of items received by the indexer. This includes items that are not yet available in the search index.
indexed | Shows the total number of items that are available in the search index.
not_indexed | Shows the number of items that are received by the indexer but are not yet available in the search index.

column_role

This element specifies the role of an indexer in an index column.

Attributes

Attribute Description

state

Shows the role of the indexer:

  • Master: The indexer is the primary indexer.

  • Backup: The indexer is the backup indexer.

  • Standalone: There is no backup indexer defined for this index column.

  • N/A: The indexer role is unknown. This state can occur immediately after a restart of the server where the indexer is running.

For more information about indexer roles, see Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint).

backups

Shows the number of backup indexers that are associated with the indexer. Typically, there will only be one backup indexer.

index_frequence

This element shows the number of items indexed per second.

Attributes

Attribute | Description
min | Shows the lowest measurement of indexed items per second for the indexer. The value is reset when the indexer is restarted.
max | Shows the highest measurement of indexed items per second for the indexer. The value is reset when the indexer is restarted.

partition

This element contains information for one index partition. The report contains one partition element for each index partition.

Attributes

Attribute Description

id

Shows the internal index partition number.

index_id

Shows a unique string identifying the content index associated with this index partition.

status

Shows the status of an index partition:

  • idle: The index partition is in an idle state.

  • waiting: The index partition is waiting for a lower partition to finish indexing.

  • copying (x%): The index that is associated with the index partition is being copied to another server. The value in parentheses specifies the percentage of the copying that is finished.

  • indexing (x%): The index that is associated with the index partition is being re-indexed. The value in parentheses specifies the percentage of the re-indexing that is finished.

type

This attribute contains information that is mainly used for debugging.

timestamp_indexed

Shows the point in time when the index partition was last updated, specified as the number of seconds since midnight January 1, 1970 (UTC), not counting leap seconds.

indexed_per_second

Shows the average number of items indexed per second for this index partition.

pending_exclusionlist

This attribute contains information that is mainly used for debugging.

range

The indexed items are stored in a pre-index item store (FiXML files). The FIXML structures are numbered by using increasing sequence numbers. The range attribute specifies the range of FIXML structures included in the index that is associated with the index partition. If no items are indexed, the range is specified as "0-0".

Remarks

Within each index column, the indexer uses a dynamic partitioning of the index to handle large number of indexed items with low indexing and query latency. This partitioning is dynamic and is handled internally on each indexer.

partition/documents

This element shows the number of indexed items in an index partition.

Attributes

Attribute | Description
active | Shows the number of searchable items in the index that is associated with the index partition.
total | Shows the total number of items in the index that is associated with the index partition. This includes the items in the exclusion list for the partition.

document_api

This element shows status information related to the internal input API queues for the indexer.

Attributes

Attribute | Description
number_of_elements | Shows the number of item batches stored on the internal input API queue for the indexer.
last_sequence | Shows the identifier of the last sequence operation that was processed.
frequence | Shows the number of content batch operations processed per second (average value).
load | Shows the amount of memory that is used by the input queue, as a percentage of the maximum allowed size.
status_updates | Shows the number of failed item operations registered during the uptime of the indexer server.

queue_size

This element shows the size of the internal input API queues for the indexer.

Attributes

Attribute | Description
current | Shows the size of the input queue in bytes.

operations_processed

This element shows the number of item batch operations processed by the indexer.

Attributes

Attribute | Description
api | Shows the number of item batch operations processed during the uptime of the indexer server.

Example

The following example shows the XML output for a single-server FAST Search Server 2010 for SharePoint deployment when you run the command indexerinfo status.

Copy Code
<indexer hostname="host3.site1.contoso.com"
         port="13050" cluster="webcluster"
         column="0" row="0" factory_type="filehash"
         preferred_master="no" connected_feeds="1"
         status="running ok" heartbeat="Running"
         exclusion_interval="60" time_since_last_index="5043">
  <documents size="670099.000000" total="5" indexed="5" not_indexed="0"/>
  <column_role state="standalone" backups="0"/>
  <index_frequence min="0.125000" max="0.833333"/>
  <partition id="0" index_id="1284101760870988000" status="idle" type="dynamic"
             timestamp_indexed="1284101766" indexed_per_second="1.000000"
             pending_exclusionlist="0" range="1-19">
    <documents active="5" total="5"/>
  </partition>
  <partition id="1" index_id="1283161117300848000" status="idle" type="dynamic"
             timestamp_indexed="1283161145" indexed_per_second="0.000000"
             pending_exclusionlist="0" range="0-0">
    <documents active="0" total="0"/>
  </partition>
  <partition id="2" index_id="1283161073250105000" status="idle" type="dynamic"
             timestamp_indexed="1283161117" indexed_per_second="0.000000"
             pending_exclusionlist="0" range="0-0">
    <documents active="0" total="0"/>
  </partition>
  <partition id="3" index_id="1283161051967838000" status="idle" type="dynamic"
             timestamp_indexed="1283161073" indexed_per_second="0.000000"
             pending_exclusionlist="0" range="0-0">
    <documents active="0" total="0"/>
  </partition>
  <partition id="4" index_id="1283160986359326000" status="idle" type="dynamic"
             timestamp_indexed="1283161051" indexed_per_second="0.000000"
             pending_exclusionlist="0" range="0-0">
    <documents active="0" total="0"/>
  </partition>
  <document_api number_of_elements="0" last_sequence="77" frequence="0.000000"
                load="0.000000" status_updates="0">
    <queue_size current="0"/>
    <operations_processed api="41"/>
  </document_api>
</indexer>

The XML output shows that there are five items (documents) in the index, and that the indexer is active (not suspended).

See Also

Reference

indexerinfo.exe reference (FAST Search Server 2010 for SharePoint)

Concepts

Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
deployment.xml reference


infotool.exe reference
[image]

Published: May 12, 2010

Use infotool to create a zipped system report that includes system and performance information in addition to log and configuration files.

The infotool report is used for:

  • Troubleshooting a problem with Microsoft Support - Generate a report before contacting support staff.

  • Backing up a system for reproduction purposes - Microsoft FAST Search Server 2010 for SharePoint includes functionality to back up configuration files and database content that can be restored to a new system. When backing up a scenario, run infotool with the backup pipeline option to include a backup in the report file

Run infotool without any parameters to create a default report stored in <FASTSearchFolder>\var\log\infotool\infotool_date-time.

If your installation includes sensitive content, run infotool with the declassify pipeline option to remove secure content from the report.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\infotool [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

infotool options

Option | Value | Description
-h |  | Displays help information.
-v |  | Displays version information.
-t |  | Tests a pipeline configuration (specified with the -p option) for semantic errors. Used for debugging.
-p | default|declassify|backup | Specifies the pipeline to run from InfotoolConfig.xml. Default: default.
-c |  | Clocks the running time of each collector.
-d |  | Enables debug logging to provide more detailed information.
-r |  | Routes the output to a specified file. If the path is not absolute, the base directory is <FASTSearchFolder>\var\log\infotool.

Examples

Default run

To run infotool with the default pipeline:

<FASTSearchFolder>\bin\infotool

This command routes output to <FASTSearchFolder>\var\log\infotool\infotool_date-time\infotool_default.zip.

Declassify option

To remove sensitive or secure information from infotool's report, use the declassify pipeline.

First, record the strings that should be removed from log and configuration files that are included in the infotool report in the file <FASTSearchFolder>\etc\infotool\Declassifier.xml as follows:

Copy Code
<declassifier>
    <strip>hostname_to_be_removed</strip>
</declassifier>

Replace hostname_to_be_removed with the host reference to delete from the infotool output. Add multiple <strip> elements to remove additional secure content.

Then run:

<FASTSearchFolder>\bin\infotool -p declassify

to compile the output in <FASTSearchFolder>\var\log\infotool\infotool_date-time\infotool_declassify.zip. This .zip file references all deleted strings (defined in Declassifier.xml) as: REMOVED_[number]].

Note:

The declassify pipeline does not remove sensitive content from the infotool.log file, which is copied into the infotool_declassify.zip file. Manually check the log and configuration files in the .zip file for sensitive content before sharing the file in a non-secure environment.

Backup option

The infotool backup option includes a backup in the report file that provides support when reproducing an environment. Infotool does not provide restore functionality.

To run infotool with the backup pipeline:

<FASTSearchFolder>\bin\infotool -p backup

which routes output to <FASTSearchFolder>\var\log\infotool\infotool_date-time\infotool_backup.zip. This .zip file includes a backup folder that contains output from the backup Windows PowerShell script in <FASTSearchFolder>\bin\backup.ps1. It is not possible to declassify the contents of the backup pipeline without causing problems with content restoration.

Remarks

The default and backup pipelines collect firewall rules from both the localhost and nodes in the deployment.xml configuration file, if available by netsh. To include these rules in your report, you must run infotool as an administrator.

The backup pipeline only includes backup contents for single-node installations of FAST Search Server 2010 for SharePoint. For farm installations, you must run the backup.ps1 script manually to include backup contents.


jdbcconnector.bat reference
See Also
[image]

Published: May 12, 2010

Use the FAST Search database connector tool, jdbcconnector.bat, to extract items from JDBC sources that will be indexed in Microsoft FAST Search Server 2010 for SharePoint.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

%FASTSEARCH%\bin\jdbcconnector.bat <command> [options]

Parameters

Parameter | Description
%FASTSEARCH% | The Microsoft FAST Search Server 2010 for SharePoint installation folder

jdbcconnector.bat commands

Command | Description
start | Starts the JDBC content extraction with the configuration specified in the -f option. Do not start more than one JDBC content extraction using the same configuration file at the same time.
stop | Shuts down the JDBC content extraction for the configuration specified in the -f option.
testconnections | Checks the availability of the JDBC sources and FAST Search Server 2010 for SharePoint resources specified in the configuration file (-f option).

jdbcconnector.bat options

Option (and short name) | Value | Required? | Description
--help (-h) | <none> | No | Displays help.
-v | <none> | No | Displays version information and exits.
-f | File path of configuration file (absolute or relative reference) | Yes | Specifies the configuration file that is used to extract JDBC content.

Examples

To start JDBC content extraction:

C:\%FASTSEARCH%\bin>jdbcconnector.bat start -f ..\etc\my_test_db_configuration.xml

To stop JDBC content extraction:

C:\%FASTSEARCH%\bin>jdbcconnector.bat stop -f ..\etc\my_test_db_configuration.xml

Remarks

You must run the jdbcconnector.bat tool from the %FASTSEARCH%\bin folder and have JAVA_HOME set to a 32-bit JRE (Java 1.6 or newer).

See Also

Concepts

jdbctemplate.xml reference


lotusnotesconnector.bat reference
See Also
[image]

Published: May 12, 2010

Use the FAST Search Lotus Notes content connector tool, lotusnotesconnector.bat, to extract items from Lotus Notes sources that will be indexed in Microsoft FAST Search Server 2010 for SharePoint.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\lotusnotesconnector.bat <command> [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

lotusnotesconnector.bat commands

Command | Description
start | Starts the Lotus Notes content extraction with the configuration specified in the -f option. Do not start more than one Lotus Notes content extraction using the same configuration file at the same time.
stop | Shuts down the Lotus Notes content extraction for the configuration specified in the -f option. After setting the stop flag, the running connector stops gracefully in several minutes.
testconnections | Checks the availability of the Lotus Notes sources and FAST Search Server 2010 for SharePoint resources specified in the configuration file (-f option).

lotusnotesconnector.bat options

Option | Value | Required? | Description
-h | <none> | No | Displays help.
-v | <none> | No | Displays version information and exits.
-f | File path of configuration file (absolute or relative reference) | Yes | Specifies the configuration file that is used to extract Lotus Notes content.

Examples

To start Lotus Notes content extraction:

<FASTSearchFolder>\bin>lotusnotesconnector.bat start -f ..\etc\my_test_ln_configuration.xml

To stop Lotus Notes content extraction:

<FASTSearchFolder>\bin>lotusnotesconnector.bat stop -f ..\etc\my_test_ln_configuration.xml

Remarks

You must run the lotusnotesconnector.bat tool from the <FASTSearchFolder>\bin folder and have JAVA_HOME set to a 32-bit JRE (Java 1.6 or newer). You must also point Notes_ExecDirectory to the installation path of the Lotus Notes Client, and NOTES_JAR_LIB to the folder where the file Notes.jar is located.

See Also

Reference

lotusnotessecurity.bat reference

Concepts

lotusnotestemplate.xml reference
lotusnotessecuritytemplate.xml reference


lotusnotessecurity.bat reference
See Also
[image]

Published: May 12, 2010

Use the FAST Search Lotus Notes user directory connector tool, lotusnotessecurity.bat, to extract user and group information from Lotus Domino directories that will be used for security in Microsoft FAST Search Server 2010 for SharePoint search results.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\lotusnotessecurity.bat <command> [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

lotusnotessecurity.bat commands

Command | Description
start | Starts the Lotus Notes user directory extraction with the configuration specified in the -f option. Do not start more than one Lotus Notes user directory extraction using the same configuration file at the same time.
stop | Shuts down the Lotus Notes user directory extraction for the configuration specified in the -f option. After setting the stop flag, the running connector stops gracefully in several minutes.

lotusnotessecurity.bat options

Option (and short name) | Value | Required? | Description
-h | <none> | No | Displays help.
-v | <none> | No | Displays version information and exits.
-f | File path of configuration file (absolute or relative reference) | Yes | Specifies the configuration file that is used to extract Lotus Notes content.

Examples

To start Lotus Notes user directory extraction:

<FASTSearchFolder>\bin>lotusnotessecurity.bat start -f ..\etc\my_ln_sec_configuration.xml

To stop Lotus Notes user directory extraction:

<FASTSearchFolder>\bin>lotusnotessecurity.bat stop -f ..\etc\my_ln_sec_configuration.xml

Remarks

You must run the lotusnotessecurity.bat tool from the <FASTSearchFolder>\bin folder and have JAVA_HOME set to a 32-bit JRE (Java 1.6 or newer). You must also point Notes_ExecDirectory to the installation path of the Lotus Notes Client, and NOTES_JAR_LIB to the folder where the file Notes.jar is located.

See Also

Reference

lotusnotesconnector.bat reference

Concepts

lotusnotessecuritytemplate.xml reference


monitoringserviceconfig.exe reference
[image]

Published: May 12, 2010

Use the monitoringserviceconfig tool to install, reconfigure or uninstall Windows Management Instrumentation (WMI) classes and performance counters published by the Microsoft FAST Search Server 2010 for SharePoint monitoring service. This tool is primarily used by the installer; advanced administrators can use it to reconfigure published performance metrics for an installed instance of FAST Search Server 2010 for SharePoint.

monitoringserviceconfig reads configuration data from MonitoringService.xml, located in the <FASTSearchFolder>\etc folder.

Changing the configuration requires administrative permissions and temporarily stopping the monitoring service.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\monitoringserviceconfig.exe [options] <command>

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

monitoringserviceconfig options

Option | Description
-h | Prints help information.
-v | Prints version information.
-d | Enables debug logging to provide more detailed information.
-wmi | Specifies that the following command only updates WMI classes.
-perfctr | Specifies that the following command only updates performance counters.

monitoringserviceconfig commands

Command Description

/checkconfig

Validates configuration files.

In a production environment, use this command first as a dry run to verify that the configuration is valid. When all configuration checks pass, you must stop the monitoring service briefly while the configuration is updated, minimizing downtime of the monitoring and logging services.

This command does not require administrative permissions and can run independently of the monitoring service (FASTSearchMonitoring).

/install

Registers WMI classes and performance counters in the operating system.

/uninstall is automatically executed first.

This command generates DLLs and other metadata files in the <FASTSearchFolder>\var\monitoringservice folder.

/uninstall

Removes WMI classes and performance counters from the operating system.

Examples

The following example validates all configuration files. If the command passes without any errors, the configuration is valid for use with both the monitoring service and this configuration tool.

<FASTSearchFolder>\bin\MonitoringServiceConfig.exe /checkconfig

The following example registers WMI classes and performance counters in the operating system. You must stop the monitoring service while the configuration is updated, and restart it after monitoringserviceconfig /install has finished.

net stop FASTSearchMonitoring

<FASTSearchFolder>\bin\MonitoringServiceConfig.exe /install

net start FASTSearchMonitoring

The generated files use a .gen<x> suffix to avoid file locking issues with previous generations of the same files.

Remarks

Many of the performance counters and WMI classes that are defined in MonitoringService.xml are used by the FAST Search Server 2010 for SharePoint management pack for System Center Operations Manager (SCOM) 2007. If your environment is monitored by SCOM, removing or changing the configuration for these metrics could cause failures in the management pack.

You must run the monitoringserviceconfig /install command if the MonitoringService.xml configuration file is changed, to update the respective WMI and performance counter objects in the Windows operating system. The FAST Search Server 2010 for SharePoint monitoring service reads its configuration when it starts, and failures will occur if the configuration is not in sync with the registered WMI and performance counter operating system objects.


nameserver_client.exe reference
[image]

Published: May 12, 2010

Use nameserver_client to retrieve entries from the name server in a FAST Search Server 2010 for SharePoint installation.

Note:

The nameserver_client tool is provided for debugging only.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\nameserver_client [global options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

The nameserver_client command-line tool retrieves and prints a set of entries from the name server. The following options determine which entries to retrieve from the name server:

  • --name-prefix

  • --interface

  • --version

  • --host

Matches against the --interface, --version, and --host options must be exact on the full strings. A match against the --name-prefix option must be exact on a prefix.

Output entries are Abstract Object References (AORs), representing middleware server objects in a running deployment. The output contains a single AOR for each line in the format:

Copy Code
aor[host:port interface_type interface_version object_id name]

The host is matched against the --host option, interface_type against the --interface option, interface_version against the --version option, and name against the --name-prefix option.

If no options are specified, all AOR entries registered in the name server are retrieved.

nameserver_client global options

-h |  | Displays help information.
-v |  | Displays version information.
--name-prefix | <name_prefix> | Matches a prefix of the AOR registered name against the specified <name prefix>.
--interface | <interface type> | Matches the interface_type of an AOR against the specified <interface type>.
--version | <interface version> | Matches the interface_version of an AOR against the specified <interface version>.
--host | <host name> | Matches the host of an AOR against the specified <host name>.

Example

In the following example, the AOR host is cohowinery.com, the port is 8080, the interface_type is mymodule::myinterface, the interface_version is 1.2, the object_id is 1225101520000000010, and the name is mysystem/my_name.

aor[cohowinery.com:8080 mymodule::myinterface v1.2 1225101520000000010

'mysystem/my_name']


nctrl.exe reference
[image]

Published: May 12, 2010

Use nctrl to manage the node controller that holds the Microsoft FAST Search Server 2010 for SharePoint processes and to perform general operations on the components (e.g., start, add, remove, resume, etc.).

There is one node controller on each FAST Search Server 2010 for SharePoint node. The node controller runs as a Windows service and can also be managed via the Windows Services controller.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\nctrl [options] <command>

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

All options and commands are optional.

nctrl options

Option Value Description

-h

Displays help.

-v

Displays version information.

-l

<log_level>

Specifies what information to record in the log entries:

  • debug

  • verbose

  • info

  • warning

  • error

-c

<directory>

Specifies the directory location of the configuration file, NodeConf.xml.

Default: <FASTSearchFolder>\etc

nctrl commands

Command | Value | Description
start | [process 1]...[process n] | Starts all or specified processes. When the node controller service is running, the start command also resumes suspended processes. When the node controller service is not running, the start command without arguments starts the service and all non-suspended processes.
stop | [process 1]...[process n] | Stops all (including the node controller service) or specified processes. When the node controller service is running, the stop command behaves identically to the suspend command. The processes will remain stopped (suspended) even if the node controller service is restarted.
kill | [process 1]...[process n] | Terminates selected processes.
restart | [process 1]...[process n] | Restarts selected processes (i.e., the same as issuing stop followed by start).
status |  | Displays services that are running on the node and their status.
add | [process] | Adds another instance of type process. The process must be configured for dynamic multiplication, that is, marked with multi=yes in the node controller configuration file.
remove | [process] | Removes a process from configuration. The process must support dynamic multiplication, that is, marked with multi=yes in the node controller configuration file.
suspend | [process 1]...[process n] | Stops the specified processes. The processes must be resumed to be restarted.
resume | [process 1]...[process n] | Starts the previously suspended processes to continue to process.
reloadcfg |  | Reloads the configuration for the running node controller service from <FASTSearchFolder>\etc\NodeConf.xml.

Examples

This example stops search on all indexer and search nodes:

<FASTSearchFolder>\bin\nctrl stop search-1

The following example restarts the configserver (which is the same as stop followed by start):

<FASTSearchFolder>\bin\nctrl restart configserver

Remarks

Remember to run nctrl reloadcfg after you change the node controller configuration file <FASTSearchFolder>\etc\NodeConf.xml for your changes to take effect.

The list of valid nctrl processes and the component they belong to are:

Process name | Associated with
browserengine | A component of the FAST Search Web Crawler
configserver | Administration (configuration)
contentdistributor | Content distributor
crawler | The main component of the FAST Search Web Crawler
fdmworker | A component of the Web Analyzer
indexer | Ilyzernandexing
indexingdispatcher | Indexing dispatcher
nameservice | Administration (name service)
nctrl | Administration (node controller)
procserver_n | Instances of item processing
qrproxy | Query processing component, interface with the Query SSA
qrserver | Main query processing component
samadmin | Administration (item level security)
samworker | Query processing component, handling item level security
search_n | An instance of the query matching process for one index partition
spelltuner | Administration (spell check dictionary improvement)
sprel | A component of the Web Analyzer, handing search clickthrough analysis
topfdispatch | Top level query dispatcher running on the query processing server
walinkstorerreceiver | A component of the Web Analyzer
walookupdb0 | A component of the Web Analyzer
webanalyzer | The main component of the FAST Search Web Analyzer
Note:

The processes associated with the FAST Search Web Crawler will run on a FAST Search Server 2010 for SharePoint single-server deployment even if the FAST Search Web Crawler is not used.

An operation like nctrl start can also be performed using net start fastsearchservice. The list of processes that are Windows Services and the name to use with Windows Service Controller are:

Process name | Name associated with Windows Service Controller
browserengine | FASTSearchBrowserEngine
nctrl | FASTSearchService
qrproxy | QRProxyService
samadmin | FASTSearchSAMAdmin
samworker | FASTSearchSAMWorker


pm.exe reference
[image]

Published: May 12, 2010

Use pm to debug problems related to item feeding and indexing in a Microsoft FAST Search Server 2010 for SharePoint installation.

You can configure FAST Search Server 2010 for SharePoint to consume log file output from components such as the FAST Search connector Content Search Service Application, the content distributor, the indexing dispatcher, and command-line clients such as docpush, can be configured to output file logs with processing information. pm processes these logs and report problems. The most common problems are missing callbacks related to operation processing. Use this information for debugging; for example, to determine why particular items are missing from the search index.

Before you can consume the log files with pm, you should enable operational logging, as explained in the remarks section of this article.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\pm [global options] <log file> ... <log file>

Parameters

Property value/return value

Parameter | Description
<log file> ... <log file> | Specifies which log files to process. If you use the -d global option, you do not have to specify log files.
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

pm global options

Option Value Description

-h

Displays help.

-v

Displays version information.

-d

Processes all client feeder log files that are located in the <FASTSearchFolder>\var\log\dsfeeder directory. Client feeders such as the FAST Search Web Crawler and the docpush command-line utility create log files in this directory.

-s

Prints operation related statistics based on processing the log files. Statistics include the number of errors and the total number of callbacks.

-t

<seconds>

The maximum allowed time to complete processing an operation set (i.e., a batch of operations). If an operation set uses more than the specified number of seconds to complete processing (that is, all expected callbacks are sent and received) pm will report this.

Default: 600 seconds

-b

<session id>:<batch id>

Tracks the progress of the specified operation sets for the given session. Separate multiple instances with a comma. Operation sets are specified with the format:

[start:stop]

where start and stop specify the first and last operations in the set.

-c

<log>=<subsystem>:<secured>:<completed>

Overrides the expected callbacks for a given log file.

  • <log> must be either Feeder, CD.in, CD.out, ID.in, or ID.out

  • <subsystem> must be either processing or indexing

  • <completed> and <secured> must be either 0 or 1 depending on whether the completed or secured callback is expected or not

Examples

The following example processes all files in the <FASTSearchFolder>\var\log\dsfeeder directory and outputs the status:

<FASTSearchFolder>\bin\pm -d

The following example processes contentdistributor operations logs, tracking specific batches (batch [2,43] in session 4, and batch [300,500] in session 3):

<FASTSearchFolder>\bin\pm -b 4[2:43],3[300:500]

<FASTSearchFolder>\var\log\contentdistributor.input

<FASTSearchFolder>\var\log\contentdistributor.output

Remarks

To enable operational logging:

  1. Stop FAST Search Server 2010 for SharePoint.

  2. Add the --debug option as a parameter to the <indexingdispatcher> process.

  3. Start FAST Search Server 2010 for SharePoint.

  4. After FAST Search Server 2010 for SharePoint has started, run this command:

    indexeradmin -a enabletracemode

  5. Create a string value called ContentAPILogFile under the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Office Server\14.0\Search\Global\Gathering Manager that contains the full path and the file name to use for logging. The file name must start with the string ContentApi.

  6. Restart the mssearch.exe process by restarting the osearch14 service.

The contentdistributor will now log operation input and output files in <FASTSearchFolder>\var\log\contentdistributor and indexingdispatcher logs to <FASTSearchFolder>\var\log\indexingdispatcher. Input log files have the input file suffix, and output log files have the output file suffix.


postprocess.exe reference
[image]

Published: May 12, 2010

Use the postprocess tool to manually refeed items to content indexing for one or more crawl collections. Before submission, each item's unique checksum fingerprint is checked against the duplicate database (unless duplicate detection is turned off).

By default, the logs from all item processor servers that are registered with the configuration component are aggregated.

Note:

An identifier can be processed on different item processor servers, and logs from previous runs can still be present in the system. To show the most recent log entry for each identifier, use the -a command option.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\postprocess [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

No option is required; all are optional.

postprocess general options

Option Value Description

-h

Displays help information.

-v

Displays version information.

-l

<log level>

Specifies the kind of information to log:

  • debug

  • verbose

  • info

  • warning

  • error

-I

<node identifier>

Specifies the node that postprocess is working with.

Do not specify unless you have deleted the node_id.dat file and you are sure that you are passing in the correct identifier.

-d

<path>

Specifies the data storage directory.

Use this option to store crawl data, runtime configuration, and logs in subdirectories within the specified directory.

Default: <FASTSearchFolder>\data\crawler

-R

<crawl_collections>

Refeeds collections. Refeeds all items to content indexing even if items were previously added.

Specify <crawl_collections> as a single collection or a comma-separated list of collections (without any white spaces).

Specify '*' to refeed all collections. Be sure to use quotation marks around the asterisk.

-P

[<address>:]<port number>

Specifies the postprocess port.

Use <port number> to specify the beginning of a port number range used by postprocess (default: crawler baseport + 6).

Optionally, specify an IP address (by host name or value).

Default: 13000

-U

<config file>

Uses the crawler global default configuration file.

This option first tries to find CrawlerGlobalDefaults.xml in the current directory. If not found, it looks in <FASTSearchFolder>\etc\.

Conflicting options that are specified on the command line override values in the configuration file.

-D

Enables direct I/O.

Enable only if supported by the OS.

Default: off

postprocess refeed (-R) mode options

Option Value Description

-r

<crawl_site>

Resumes refeeding after the specified crawl site. Do not use this option with -s.

Note:

Use the special keyword @auto for <crawl_site> to have postprocess auto-resume the crawl where your last refeed left off.

-s

<crawl_site>

Processes only the specified site name (hostname).

Do not use this option with -r.

-i

<file>

Processes URIs and sites listed in the specified file.

The -r, -s and -i options are mutually exclusive.

<file> is a newline-separated file that contains URIs, crawl sites, or both.

-x

Processes all permitted URIs. Includes all URIs matching the current collection include/exclude rules; ignores URIs that do not match.

Use with the -u option to specify an updated collection specification XML file.

-X

Deletes excluded URIs.Deletes URIs that do not match the collection specification include/exclude rules.

All other URIs are ignored, unless combined with -x (to process all permitted URIs).

This option is useful with -u.

-b

Applies robots.txt rules when the program is checking for included/excluded items. Robots.txt rules are also applied to the -x and -X options.

-u

<XML config file>

Updates the include/exclude regular expressions loaded from the configuration database with rules from the < XML config file>. This update is only in effect during the postprocess refeed.

-b

Resumes feeding content from the content indexing queues.

If the crawler is processing faster than content indexing, postprocess queues items to disk, and works off those queues. This option processess (empties) any existing queues.

-k

<content_collection>|<feeding_target>

Overrides the content collection specified as the crawl's feed destination.

Or, you can specify the symbolic name of a feeding target (as defined in the collection configuration).

Default: <content_collection>

postprocess internal options

Use these options for testing and debugging only, or internally when postprocess is started by the crawler.

Option | Value | Description
-p | <port> | Connects to the node scheduler process at <port>.
-S | <port> | Listens at <port>. Used internally only.
-o |  | Enables FAST Search output mode.
-F | <fileserver port> | Tells postprocess to use an external file server at the given port.
-T |  | Enables profiling. Use the profile_methods.py tool to review the profile.
-t |  | Enables profiling using the hotshot module. Use the profile_lines.py tool to review the profile.
-n |  | Uses a null feeder (\dev\null equivalent) (items will not be fed). Use for testing.

Examples

This example refeeds the crawl collection MyCollection to content-indexing:

<FASTSearchFolder>\bin\postprocess -R MyCollection

The following example refeeds all crawl collections to content indexing:

<FASTSearchFolder>\bin\postprocess -R '*'

The following example refeeds the crawl site www.contoso.com in the crawl collection MyCollection to content indexing:

<FASTSearchFolder>\bin\postprocess -R MyCollection -s www.contoso.com

To refeed the collection MyCollection to content indexing while removing items that no longer match the crawl collection include/exclude rules, follow this example:

<FASTSearchFolder>\bin\postprocess -R MyCollection -x -X

This example updates the crawl collection MyCollection with custom include/exclude rules and refeeds items matching the rules:

<FASTSearchFolder>\bin\postprocess -R MyCollection -x -u MyCustomConfig.xml

MyCustomConfig.xml typically contains finer granularity of includes or excludes for items that you want to reprocess than an option such as -s provides.

Remarks

The crawler uses postprocess to perform duplicate detection and item submission to content indexing. Like the site manager processes, postprocess starts automatically with the crawler. You can also run postprocess on its own when the crawler is not running, to manually refeed items in one or more crawl collections.

Postprocess submits new, modified, and deleted items as the crawler encounters them. Before submission, each item is checked against the duplicate database, unless duplicate detection is turned off.


ppdup.exe reference
[image]

Published: May 12, 2010

Use the ppdup tool to start a duplicate server that provides a centralized duplicate detection function for each node scheduler/postprocess host.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\ppdup [options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

No option is required; all are optional.

ppdup options

Option Value Description

-h

Displays help information.

-v

Displays version information.

-I

<identifier>

Specifies a symbolic duplicate server identifier.

Use this option to assign a symbolic name to the duplicate server. This name is used when the state of the duplicate server is copied to a replica.

By default, this is read from the crawler generated node identifier in <FASTSearchFolder>\data\crawler\node_id.dat.

When running multiple duplicate servers on the same node, specify a different identifier for at least one of the duplicate servers.

-P

[<address>:]<port number>

Specifies the port number and an optional interface that a postprocess communicates to the duplicate server in a multiple node setup.

-r

<port number>

Specifies a replication service port number to enable replica mode for the duplicate server.

The duplicate server listens for incoming replication requests on the specified port number.

-R

<host name>:<port number>

Specifies the address of a duplicate server that replicates the server state.

The host name must correspond to a server that is running the duplicate server that uses the -r option and the specified port number.

-d

<path>

Specifies the current working data directory for the duplicate server.

Default: If the FASTSEARCH environment variable is set, the default path is <FASTSearchFolder>\data\crawler\ppdup; otherwise the default path is data.

-c

<cache size>

Specifies the database cache size or hash size, in megabytes.

With a storage format of hashlog (see -S option), this value determines the size of the allocated memory hash. If the number of items that are stored in the hash exceeds the available capacity, the hash will be automatically converted into a disk hash and resized (2x increments).

With a storage format of diskhashlog, <cache size> determines the initial size of the hash on disk. For each overflow (when capacity is exceeded) the hash is resized, as described above.

When the storage format is gigabase, <cache size> specifies how much memory to reserve for database caches.

Note:

This value is per crawl collection. If you use multiple crawl collections, each crawl collection will allocate the specified amount of cache/memory/disk. If the duplicate server is run as both a primary and a replica, double the resources will be consumed.

Default: 64

-s

<stripes>

Specifies the number of stripes (separate files) used by the duplicate server databases.

Default: 1

-D

Enables direct I/O for the duplicate server's databases.

Use only if supported by your operating system.

-S

<database format>

Specifies a database storage format:

  • hashlog - This format first allocates a memory based hash structure with a data log on disk. The size of the memory hash is specified by the -c option. If the hash overflows, it automatically converts into a diskhashlog.

  • diskhashlog - This format resembles hashlog, but with a disk based hash structure.

  • gigabase - The gigabase format is a database structure on disk.

Default: hashlog

-N

Disables nightly compression of duplicate server databases.

-F

<file>

Specifies an XML file that contains the crawler global configuration. The file may contain default values for all command-line options.

Default: <FASTSearchFolder>\etc\CrawlerGlobalDefaults.xml

-T

Enables profiling.

Use for debugging only.

-t

Enables profiling using the hotshot module.

Use for debugging only.

-l

<log level>

Specifies the kind of information to log:

  • debug

  • verbose

  • info

  • warning

  • error

Examples

The following example starts a duplicate server at port 14000:

<FASTSearchFolder>\bin\ppdup -P 14000

The following example starts a duplicate server acting as a replica at port 14100:

<FASTSearchFolder>\bin\ppdup -r 14100

To start a duplicate server at port 14000 which copies its databases to a replica at port 14100, follow this example:

<FASTSearchFolder>\bin\ppdup -P 14000 -R 14100


psctrl.exe reference
[image]

Published: May 12, 2010

Use psctrl to alter settings, to retrieve statistics, and to view the status of a processor server.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\psctrl [options] <command>

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

Unless a specific processor server is specified with the -P option, the given command will be sent to all running processor servers in the system. For commands that alter a setting in the processor server (doctrace, memprofile, debug), the setting will only be effective until the setting is altered again or the processor server process is restarted.

All options and commands are optional.

psctrl options

Option | Value | Description
-h |  | Displays help information.
-v |  | Displays version information.
-C | <host:port> | Specifies the configuration server host name and port number. The default host name and port are loaded from <FASTSearchFolder>\etc\CSLocation.xml.
-P | <host:port> | Runs the command only for the processor server specified by the host name and port.

psctrl commands

Command Value Description

doctrace on

on|off

On: Enables item tracing to the internal item log. The processor server logs all changes that are made to attributes in an item to an internal item log.

Use the doclog command to retrieve the internal item log from a processor server.

Note:

Item tracing is only enabled until the processor is restarted.

Off: Disables item tracing.

debug on

on|off

On: Sets the processor server to log at the debug level.

Note:

Enabling debug logging generates many log messages and may affect the performance of the processor server.

Off: Logs at the regular level.

ping

Sends a ping message to the processor server to check whether it is responding. Returns an error message if a processor server does not respond.

status

Retrieves the status from the processor server.

statistics

Gets statistics from the processor server.

Statistics include timing information and the number of processing errors for each item processing stage within the processor server. If the memprofile command was previously run to enable memory profiling, the statistics also include the memory usage for each stage.

reset

Resets the processor server and re-initializes item processing stages. Also resets internal item logs and statistics.

stop

Terminates the processor server process.

memprofile on

on|off

On: Enables memory profiling for processors and pipelines. Memory profiling information also appears with the statistics command.

Note:

Enabling memory profiling affects the performance of the processor server.

Off: Disables memory profiling.

leakdetect

Writes a memory leak report to stderr. Output from stderr is stored in <FASTSearchFolder>\var\log\procserver\*.scrap files.

flushstate

Flushes the internal state of item processing stages. Also resets all internal item logs and statistics.

Examples

The following example enables item tracing on all processor servers:

<FASTSearchFolder>\bin\psctrl doctrace on

The following example resets all processor servers:

<FASTSearchFolder>\bin\psctrl reset

The following example checks which processor servers are responding:

<FASTSearchFolder>\bin\psctrl ping


rc.exe reference
[image]

Published: May 12, 2010

Use the rc tool to collect monitoring data from different parts of a Microsoft FAST Search Server 2010 for SharePoint installation.

The rc tool uses the name service to find all registered component interfaces that satisfy your specified search criteria (data filters).

The rc tool only collects data from running processes.

All FAST Search Server 2010 for SharePoint back-end components can be monitored by using rc. They include the content distributor, document processors, the indexing dispatcher, and indexers.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\rc [global options] [command options] [<path1> ... <pathN>]

Parameters

Parameter | Description
<path1> ... <pathN> | These parameters specify where (within the name server) the component interfaces (for data collection) are registered. The rc tool will search within these paths.
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

rc global options

Global option | Value | Description
-n | <host:port> | Connects to the name server at <host:port>.

rc command options

Command option Value Description

-h

<none>

Displays help information.

-v

<none>

Displays version information.

-r

<none>

Traverses the name service for paths to components in a FAST Search Server 2010 for SharePoint back-end that supports rc monitoring.

When using this option, <path1> ... <pathN> parameters are interpreted as substrings which are matched against the paths inferred from the name servers. Further commands are only applied to matching paths. Use to filter commands by specific components or subsystems.

-f

<type>

Filters the output by a given type. Valid types are as follows:

  • scope

  • alloc

  • long

  • int

  • float

  • bool

  • string

  • version

Use this option multiple times on the same command line when more than one type is desired.

Example: -f int -f float

If this option is not supplied, all types are given as output.

-a

<none>

Lists entries of type scope or alloc where the current value is greater than 0.

-t

<seconds>

Specifies a time-out period (in seconds) for rc to wait for a response from the components.

rc uses the <path1> ... <pathN> parameters and the name server to look up the physical address of the components.

Use 0 (zero) to disable the time-out.

-s

<number>

Specifies the number of times the command should be run. Use the -i option to specify the time to wait between command repetitions.

Default: 1.

-i

<seconds>

Specifies the number of seconds to wait before the command is repeated (as set in the -s option).

-d

<module:num>

Sets the debug level on a <module> at the component interface to <num>.

Note:

This changes the log output from the components.

-V

<none>

Prints distinct version strings for paths and then exits.

Examples

To list all component interfaces with active scopes:

<FASTSearchFolder>\bin\rc -r -a -f scope

To list all processing component interfaces:

<FASTSearchFolder>\bin\rc -r esp/subsystems/processing

To list all floating-point numbers and integers for the content distributor:

<FASTSearchFolder>\bin\rc -f float -f int esp/subsystems/processing/dispatcher/0

To list all unique versions for all component interfaces:

<FASTSearchFolder>\bin\rc -r -V

To list all component interfaces and allocations with active scopes on the remote name server at host <host> and port <port>:

<FASTSearchFolder>\bin\rc -r -a -n <host>:<port>


searchinfo.exe reference (FAST Search Server 2010 for SharePoint)
See Also
[image]

Published: May 12, 2010

Use the searchinfo command-line tool to retrieve information for the query matching and query dispatching components in a Microsoft FAST Search Server 2010 for SharePoint farm.

The command-line tool retrieves the status information from the search controller component on the servers that has query matching and query dispatching components.

You also find a query dispatching component on the servers that has the query role in the deployment file (deployment.xml).

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\searchinfo [options] <command>

Where:

  • <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

searchinfo options

Option (and short name) Default Value Description

-a

<none>

Runs the commands for all search controllers in the FAST Search Server 2010 for SharePoint farm.

Note:

If you do not use this option, and do not specify a row or column, the command will return status information for the search controller on index column 0 and search row 0.

--help (-h)

<none>

Displays help.

--version (-v)

<none>

Displays version information.

--clustername

webcluster

This option is not used. Do not set to another value than the default value.

--subsystem

indexing

This option is not used. Do not set to another value than the default value.

--column=

0

Request information for a specific index column.

Set this option without specifying a row option to run the command on all query matching servers in the specified index column.

To find a column identifier, use the command indexerinfo -a status or look up the column number for the indexer servers in the deployment.xml configuration file.

--row=

0

Request information for a specific indexer row. Row 0 is the primary indexer.

Set this option without specifying a column option to run the command on all query matching servers in the specified indexer row.

To find a row identifier, use the command indexerinfo -a status or look up the row number for the indexer servers in the deployment.xml configuration file.

searchinfo commands

Command Description

status

Retrieves a status report from the search node(s) in XML format.

The XML status report displays the status for all servers that have the query matching or query dispatching role. For details on the XML status report syntax, see searchinfo XML status report syntax (FAST Search Server 2010 for SharePoint)

Tip:

You can get a similar XML status report about indexing status from the indexerinfo tool.

Examples

To retrieve an XML status report from all search controllers in the system, run the following command in a FAST Search Server 2010 for SharePoint shell:

Copy Code
searchinfo -a status

See Also

Concepts

Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
deployment.xml reference
searchinfo XML status report syntax (FAST Search Server 2010 for SharePoint)


searchinfo XML status report syntax (FAST Search Server 2010 for SharePoint)
See Also

Published: December 2, 2010

The searchinfo status command retrieves an XML-formatted report for the Microsoft FAST Search Server 2010 for SharePoint query matching status. This article describes the syntax of the report.

The command returns one report for each server that is requested in the command.

Element quick reference

The report uses the following elements:

Element | Description
<search-stats> | This is the top-level element.
<error> | This element describes an error condition related to the query matching component.
<fdispatch> | This element shows information related to the fdispatch process, which is responsible for dispatching queries to query matching components.
<hostname> | This element shows the host name of the server that runs this component.
<searchhandler> | This element shows information about search handlers, which are instances that process incoming search queries.
<datasets> | This element contains a single dataset sub-element.
<dataset> | This element shows information about each query matching component (engine), related to queries that are submitted by the dispatcher.
<engine> | This element shows information about the fsearch process that is associated with one index partition.
<fsearch> | This element shows information related to the fsearch process, which is responsible for query matching.
<search-statistics> | This element shows information about query matching statistics.
<attributevectors> | This element shows information related to memory structures for deep query refinement and sorting.

The following elements contain information that is mainly used for debugging:

<version>, <filedesc-resource>, <httpdconn>, <file-statistics>, <transportd>, <result-cache>, <resultattr-cache>, <docinfo-cache>, <boolean-occ-cache>, <pos-occ-cache>, <phrase-occidx-cache>, <phrase-occ-cache2>, <bitvector-cache>, <int-range-bitvector-cache>, <int-occ-cache>, <filter-occ-cache>, <dict-cache>, <subquery-cache>, <attributevector>

searchinfo status XML format

The status output follows all conventions and syntax rules that are defined for the XML file format.

The basic XML structure is as follows:

Copy Code
<search-stats>
 <error />
 <fdispatch>
   <error />
   <version />
   <hostname />
   <searchhandler />
   <datasets>
    <dataset>
     <engine />
    </dataset>
   </datasets>
   <filedesc-resource />
 </fdispatch>
 <fsearch>
   <version />
   <search-statistics />
   <httpdconn />
   <file-statistics />
   <transportd />
   <result-cache />
   <resultattr-cache />
   <docinfo-cache />
   <boolean-occ-cache />
   <pos-occ-cache />
   <phrase-occidx-cache />
   <phrase-occ-cache2 />
   <bitvector-cache />
   <int-range-bitvector-cache />
   <int-occ-cache />
   <filter-occ-cache />
   <dict-cache />
   <subquery-cache />
   <attributevectors >
     <attributevector />
   </attributevectors>
   <filedesc-resource />
 </fsearch>
</search-stats>

fdispatch

This element shows information related to the fdispatch process, which is responsible for dispatching queries to query matching components.

There is one instance of the fdispatch process on the query processing server, which dispatches queries to individual query matching servers.

There is one instance of the fdispatch process on each query matching server, which dispatches queries to the individual query matching partitions on the query matching servers.

Note:

In a multiple server deployment, the query processing server usually runs on a different computer than the query matching server.

engine elements of type search represent the underlying fsearch processes that manage query matching on different index partitions. In a multiple server deployment with more than one index column or search row, the top-level fdispatch is also listed as type dispatch, because it is a dispatcher to underlying query matching servers.

hostname

This element shows the host name of the server that runs this component.

Attributes

Attribute | Description
name | The host name of the server that runs this component. For more information about the server roles in the deployment, see the deployment.xml configuration file.

Remarks

For the top-level query dispatcher (on the query processing server), the host name may be displayed as "localhost" instead of the actual host name. You can see that this is the top-level query dispatcher by looking at the engine sub-elements, where type has the value dispatch.

searchhandler

This element shows information about search handlers, which process incoming search queries.

Attributes

Attribute | Description
instantiate-count | Shows the total number of search handler threads.
active-count | Shows the number of queries that are currently being evaluated.

dataset

This element contains information about each query matching component, related to queries that are submitted by the dispatcher.

Attributes

Attribute | Description
active-nodes | Shows the number of active query matching (fsearch) processes.
total-search-time | Shows the total time used for query matching.
up-time | Shows the total uptime for this component since the last restart.
avg-seconds-per-search | Shows the average number of seconds used per query evaluation.
partitions | Shows the number of index partitions as seen from this query dispatcher. If this is the top-level dispatcher, the number will reflect the number of index columns.
timed-out-percentage | Shows the percentage of queries that have timed out.
max-active-partitions | Shows the maximum number of active index partitions that were created.
avg-searches-per-sec | Shows the average number of queries per second.
first-partition | Shows the first index partition number.
samples | Shows the number of queries that were used to calculate the avg-searches-per-sec value
id | This attribute contains information that is mainly used for debugging.
unit-selection-cost | This attribute contains information that is mainly used for debugging.
timed-out | Shows the number of query timeouts that occurred.
total-searches | Shows the total number of queries since the last index set change.
max-active-nodes | This attribute contains information that is mainly used for debugging.
total-selection-cost | This attribute contains information that is mainly used for debugging.
active-partitions | Shows the total number of active index partitions.
avg-uncached-search-time | Shows the average number of seconds per query, when the query cache is not used.

Remarks

Within each index column, the indexer uses a dynamic partitioning of the index in order to handle large numbers of indexed items with low indexing and query latency. This partitioning is dynamic and is handled internally on each indexer.

engine

This element shows information about the underlying query matching engines as seen from this fdispatch processes.

If this is the fdispatch process on the query matching server, then each engine element represents an fsearch process that is associated with one index partition on this server.

If this is the fdispatch process on the query processing server, then each engine element represents an fdispatch process on a query matching server.

Attributes

Attribute Description

socket

Shows the connection identifier for the query matching component that is associated with the index partition. The identifier is specified by using the following format:

<hostname>:<port>

samples

Shows the number of queries that were issued.

avg-search-time

Shows the average number of seconds per query.

Note:

This attribute is not returned if the value for samples is 0.

partition

Shows the index partition identifier.

type

Shows the type for this engine element.

  • dispatch: The engine element represents an fdispatch process on a query matching server.

  • search: The engine element represents an fsearch process.

state

Shows the state of the fsearch or fdispatch process associated with this engine element.

  • up: Active and able to serve queries.

  • down: Not active. This usually indicates that there are two fsearch processes that run in parallel on the same partition, but only one of them is running and serving queries.

docstamp

Shows the date/time when the index files were created, measured in seconds since January 1, 1970.

row

Shows the associated search row.

refcost

This attribute contains information that is mainly used for debugging.

active

Shows the number of searchable items in the index that is associated with the index partition.

Remarks

You may experience high avg-search-time values (evaluation time for queries) for newly created indexes where the number of queries (samples) is very low. This is related to the initial creation of query caches.

fsearch

This element contains statistics and resource usage information related to the fsearch process, which is responsible for query matching.

There is one fsearch process for each index partition in an index column.

Attributes

Attribute | Description
name | Specified as "localhost:<port number>", where <port number> is the port number for the fsearch process. You can find the associated partition number from the fdispatch/engine element that has the same value for the socket attribute.

search-statistics

This element shows information about query matching statistics.

Attributes

Attribute | Description
up-time | Shows the number of seconds since this process was started (restarted).
total-search-time | Shows the total number of seconds used for query matching.
total-searches | Shows the number of queries processed.
avg-searches-per-second | Shows the average number of queries per second.
avg-search-time | Shows the average number of seconds per query.

attributevectors

This element contains information related to memory structures for deep query refinement and sorting.

Attributes

Attribute | Description
total-memory-usage | The total amount of memory that is allocated for deep query refinement and sorting of data structures for this index partition.

Remarks

The name of attribute vectors for sorting starts with the prefix "batv". The name of attribute vectors for deep query refinement starts with the prefix "bavn".

Examples

The following examples show a subset of the XML output when you run the command searchinfo -a status.

Example 1

Example 1 shows a subset of the XML output for a query matching server associated with index column 0 and search row 0.

Copy Code
<search-stats>
 <fdispatch>
   <version value="fs14.301"/>
   <hostname name="host1.site1.contoso.com"/>
   <searchhandler active-count="0" instantiate-count="0"/>
   <datasets>
    <dataset id="0" partitions="5" first-partition="0" max-active-nodes="5"
             active-nodes="5" max-active-partitions="5" active-partitions="5"
             unit-selection-cost="1" total-selection-cost="0"
             up-time="168163.798" total-search-time="0.000" total-searches="0"
             avg-searches-per-sec="0.000" samples="0" timed-out="0">
     <engine socket="localhost:13056" state="up" type="search" partition="0"
             row="0" refcost="1" docstamp="1286267246" samples="0"/>
     ...
     <engine socket="localhost:13096" state="up" type="search" partition="4"
             row="0" refcost="1" docstamp="1286254268" samples="0"/>
    </dataset>
   </datasets>
   <filedesc-resource wait-queue-len="0" resource-usage="18"
                      total-resource-count="1014"/>
 </fdispatch>
 <fsearch name="localhost:13056">
   <version value="fs14.9999"/>
   <search-statistics total-search-time="0.000" total-searches="0"
                      up-time="0.000"/>
   ...
   <attributevectors total-memory-usage="48">
     <attributevector name="batvprocessingtime" type="int64" multi-value="no"
                      elements="0" memory-bytes="0"
basefile="D:\FASTSearch\data\data_index\0\index_1286267244861467000\index_data\merged\batvprocessingtime"
                      status="OK" custom="format=enum"/>
     ...
   </attributevectors>
   ...
 </fsearch>
 <fsearch name="localhost:13096">
   <version value="fs14.9999"/>
   <search-statistics total-search-time="0.000" total-searches="0"
                      up-time="0.000"/>
   ...
   <attributevectors total-memory-usage="1077786037">
     <attributevector name="batvprocessingtime" type="int64" multi-value="no"
                      elements="5999523" memory-bytes="25799404"
basefile="D:\FASTSearch\data\data_index\4\index_1286242512297821000\index_data\merged\batvprocessingtime"
                      status="OK" custom="format=enum"/>
     ...
   </attributevectors>
   ...
 </fsearch>
 ...
</search-stats>

The XML output shows data for index partition 0 and 4. Index partition 0 is the partition that holds recently updated items.

Example 2

Example 2 shows a subset of the XML output for a query processing server. The status information that is shown is related to the query dispatcher component on this server. There are three index columns and one search row in this deployment. Each query matching server is represented as an engine entry in the XML.

Copy Code
<search-stats>
 <fdispatch>
   <version value="fs14.301"/>
   <hostname name="localhost"/>
   <searchhandler active-count="0" instantiate-count="0"/>
   <datasets>
    <dataset id="0" partitions="3" first-partition="0" max-active-nodes="15"
                    active-nodes="15" max-active-partitions="15"
                    active-partitions="15" unit-selection-cost="1"
                    total-selection-cost="0" up-time="169002.845"
                    total-search-time="0.000" total-searches="0"
                    avg-searches-per-sec="0.000" samples="0" timed-out="0">
     <engine socket="host1.site1.contoso.com:13052" state="up"
             type="dispatch" partition="0" row="0" refcost="11"
             docstamp="1287410432" samples="0"/>
     <engine socket="host2.site1.contoso.com:13052" state="up"
             type="dispatch" partition="1" row="0" refcost="11"
             docstamp="1287410430" samples="0"/>
     <engine socket="host3.site1.contoso.com:13052" state="up"
             type="dispatch" partition="2" row="0" refcost="11"
             docstamp="1287410432" samples="0"/>
    </dataset>
   </datasets>
   ...
 </fdispatch>
 ...
</search-stats>
Note:

partitions indicate the number of index columns as seen from this query dispatcher. active-partitions indicate the total number of active index partitions. There are five active partitions within three index columns in this example.

Example 3

Example 3 shows the query dispatcher view of one query matching server, represented as an engine element. Only one query was issued (samples=1), and you can see that the average query evaluation time is high, because of empty query cache.

Copy Code
<engine socket=" host21.site1.contoso.com:13052"
        state="up" type="dispatch" partition="9" row="1" refcost="11"
        docstamp="1276035428" avg-search-time="1.919" samples="1"/>

See Also

Reference

searchinfo.exe reference (FAST Search Server 2010 for SharePoint)

Concepts

Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)
Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)
deployment.xml reference


spreladmin.exe reference
[image]

Published: May 12, 2010

SPRel is a search clickthrough log analysis engine. It resembles the WebAnalyzer because it analyzes data to improve the relevance of search results. But instead of counting and analyzing links like WebAnalyzer, SPRel analyzes which entries users click on in search results. The most frequently clicked items are flagged as the best matches for those query terms. By analyzing search clickthrough logs in search results for the last 30 days, SPRel can improve the relevance of future queries.

Use the spreladmin tool to configure SPRel and to retrieve status information. This command-line tool also lets you manage the schedule for running search clickthrough log analyses.

This tool is only used infrequently. You may use it most often to check the SPRel status with the show commands.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\spreladmin [global options] <command> [command options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

spreladmin global options

Option | Description
-h | Displays help.
-v | Displays version information.

spreladmin command options

Option | Description
-b | Specifies a backup file for the BackupConfig and RestoreConfig commands.
-i | Interval (in seconds) between each run. Default: 86400 seconds (one day)
-k | Specifies which configuration parameter to change when modifying configurations.
-l | Specifies a new verbosity level for SPRel.
-q | Provides a URI for the ShowURIRelevance command.
-u | Provides a new value for the parameter specified by -k.
-w | Specifies the time to start the first run. Format: yyyy-mm-dd,HH:MM

spreladmin processing commands

Command Required options Description

AbortProcessing

None

Stops current processing. Data will be persisted in its current state, but you must restart processing later. Any new raw data obtained since the start of the last processing run will be incorporated into the new processing run.

Note:

After stopping a run, SPRel may have to persist its files first, which may take some time.

Enqueue

None

Puts SPRel on standby. Processing will begin either when new raw data is obtained, or at the user-scheduled time.

Dequeue

None

Stops processing and takes SPRel off standby. No additional processing will occur until SPRel is re-enqueued.

PauseProcessing

None

Pauses ongoing processing. All data will be persisted and processing can be resumed later from its paused state, without including raw data obtained after the start of the processing run.

Note:

When pausing, SPRel may have to persist its files first. This may take some time.

StartProcessing

None

Forces SPRel to start an analysis run if it is not already running.

spreladmin set commands

Command | Required options | Description
SetConfig | -k<keyword> -u<value> | Changes the configuration of SPRel. See the SPRel global configuration options for valid keywords and default configuration values.
SetFDMConfig | -k<keyword> -u<value> | Alters the runtime behavior of FDM. See the SPRel FDM configuration options for valid keywords and default configuration values.
SetLogLevel | -l<log level> | Sets the verbosity level of SPRel to one of five levels: error, warning, info, verbose, debug.
SetSchedule | -w<date,time> -i<interval> | Defines the processing schedule by specifying a start time for the first processing run, and a time interval between each run.

spreladmin show commands

Command | Required options | Description
ShowConfig | None | Displays the current configuration of SPRel.
ShowFDMConfig | None | Displays the current configuration of FDM (Fast Distributed Make, a framework used to generate the analysis).
ShowLogLevel | None | Displays the current verbosity setting for SPRel.
ShowSchedule | None | Displays the current processing schedule.
ShowStatus | None | Displays the status of SPRel.
ShowURIRelevance | -q<URI> | Queries the SPRel lookup databases to find relevance information for a given URI.

spreladmin miscellaneous commands

Command | Required options | Description
BackupConfig | -b<filename> | Backs up the SPRel configuration.
RestoreConfig | -b<filename> | Restores the SPRel configuration from a file that is created by BackupConfig.
ExtendedHelp | any command name (e.g. ExtendedHelp ShowURIRelevance) | Displays extended help for the given options.

SPRel global configuration options

Keyword (Parameter) Value Description

batch_size

<value>

The maximum number of items in one batch.

Default: 128

callback_timeout

<value>

Specifies how long to wait (in seconds) for callback after the last batch was submitted.

Default: 1800

concurrent_feeds

<value>

Specifies how many feeding clients can run at the same time.

This value is the total number of clients on all nodes in the installation.

Default: 4

cpus

A value equal to the "max-targets" value specified during installation

Specifies how many processes can run at the same time.

If this value multiplied by the number of nodes in the system is lower than the concurrent_feeds value, this value specifies the number of feeding clients.

force_collections

true|false

Specifies if SPRel should attempt to update all crawl collections even if the crawl collections have not received new data.

If false, SPRel will only update crawl collections that it knows have received new data. But it will still add all crawl collections to the analysis.

Default: false

keep_clicklogs

<value>

The number of days to keep search clickthrough logs.

Default: 30

pollwalsr_callback_interval

<value>

The interval (in seconds) between times when SPRel checks for new data sent through the pipeline.

Default: 300

run_partial_update

true|false

Specifies if SPRel should update the index with new relevance data after it completes an analysis run.

If false, SPRel will not update the index, and the updates will not be stored and applied later.

Default: true

sort_buffer

<value>

The amount of memory (in megabytes) to use when sorting data. This value is specified per task that is running in the system.

Default: 200

use_clicklogs

<value>

The number of search clickthrough logs to include in the analysis.

Default: 30

SPRel FDM configuration options

Keyword (Parameter) | Value | Description
disk_free | <value> | Minimum amount of free disk space (in MB) that must be available on a node for processing. If the free disk space goes below this limit, the node will not be used in the analysis. Default: 2000
verbose | true|false | Verbosity of the analysis. Default: false

Examples

The following example shows the overall status of SPRel:

<FASTSearchFolder>\bin\spreladmin showstatus

The following example lists the SPRel configuration:

<FASTSearchFolder>\bin\spreladmin showconfig

The following example instructs SPRel to use two CPUs per processing node:

<FASTSearchFolder>\bin\spreladmin setconfig -kcpus -u2


waadmin.exe reference
[image]

Published: May 12, 2010

Use the waadmin tool to configure the WebAnalyzer and to retrieve status information. This command-line tool also lets you manage the WebAnalyzer's schedule for running hyperlink analysis. This tool is rarely needed. You will typically use it to check the WebAnalyzer status with the show commands.

Note:

To use a command-line tool, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

<FASTSearchFolder>\bin\waadmin [global options] <command> [command options]

Parameters

Parameter | Description
<FASTSearchFolder> | The path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

waadmin global options

Option | Description
-h | Displays help.
-v | Displays version information.

waadmin command options

Option | Description
-c | Specifies a given collection.
-n | Specifies a given view.
-d | Provides a new description for a given view.
-k | Specifies which configuration parameter to change when altering configurations.
-u | Provides a new value for the parameter specified by -k.
-l | Specifies a new verbosity level for the WebAnalyzer.
-q | Provides a URI for the ShowURIRelevance command.
-f | When using ShowURIRelevance, specifies the maximum number of anchor texts included in the result. Default: 10
-b | Specifies a backup file name for the BackupConfig and RestoreConfig commands.

waadmin processing commands

Command Required options Optional options Description

AbortProcessing

None

-n<viewname>

Stops processing on the default view, or the view provided with -n.

If the view is not being processed, it will be set directly in stopped state. If it is running, link data will be persisted but you must restart processing on the view.

EnqueueView

None

-n<viewname>

Sets a view into a scheduled state.

When a collection that is part of the view receives new content, the view is moved into the end of the processing queue. If no view name is provided, the default view is used.

DequeueView

None

-n<viewname>

Removes a view from the processing queue.

Views are typically re-enqueued as soon as they come out of the processing pipeline, but DequeueView discontinues this cycle for the specified view. A view can be added later with EnqueueView. If the view is being processed, it will complete its run first.

If no view name is provided, the default view is used.

ExpediteProcessing

None

-n<viewname>

Puts a view first in the view processing queue, even if the view is already in the queue.

ForceProcessing

None

-n<viewname>

Pauses any ongoing processing and starts to process the specified view, if the view has at least one member collection.

After link processing, the paused view (if any) will resume processing.

Alternative use: To analyze a view that has no new data. ForceProcessing will run link processing again on the existing data even if the link data has not changed since the last link processing.

Note:

The paused view must first persist its files, which may take some time.

ProcessOnce

None

-n<viewname>

ProcessOnce resembles ForceProcessing except that after the view is processed, it is not re-inserted into the queue.

After processing has finished on the specified view, the paused view (if any) resumes processing.

Alternative use: To analyze a view that has no new data. ProcessOnce will run link processing again on the existing data even if the link data has not changed since the last link processing.

Note:

The paused view must first persist its files, which may take some time.

PauseProcessing

None

None

Pauses ongoing processing on the current view. All link data will be persisted and processing can be resumed later.

Note:

The paused view must first persist its files, which may take some time.

waadmin view commands

Command | Required options | Optional options | Description
AlterViewDescription | -d<description> | -n<viewname> | Changes the description of the view.
CreateView | -n<viewname> | -d<description> | Creates a new view.
DeleteView | -n<viewname> | None | Deletes a view. The default view cannot be deleted.

waadmin crawl collection commands

Command | Required options | Optional options | Description
UnsetPreferredView | -c<collection name> | None | Assigns the default WebAnalyzer view to the crawl collection as a source of relevance data.
SetPreferredView | -c<collection name> | -n<viewname> | Assigns a WebAnalyzer view to a crawl collection for getting relevance data.
AddCollections | -c<collection name> -n<viewname> |  | Adds crawl collections to a view. The view gets hyperlinks from these crawl collections. Separate multiple collections with a comma.
RemoveCollections | -c <collection name> | -n<viewname> | Removes collections from a view. Separate multiple collections with a comma.

waadmin set commands

Command | Required options | Optional options | Description
SetFDMConfig | -k<keyword> -u<value> | None | Advanced option: Changes the processing configuration (FDM is a framework used to generate the analysis).
SetConfig | -k<keyword> -u<value> | None | Changes the runtime configuration of WebAnalyzer.
SetViewConfig | -k<keyword> -u<value> | -n<viewname> | Changes configuration parameters for the specified view.
SetLogLevel | -l<log level> | None | Sets the verbosity level of the WebAnalyzer log to one of five levels: error, warning, info, verbose, debug.

waadmin show commands

Command | Required options | Optional options | Description
ShowConfig | None | None | Displays the current runtime configuration of the WebAnalyzer.
ShowFDMConfig | None | None | Displays the current runtime configuration of FDM (Fast Distributed Make, a framework used to generate the analysis).
ShowStatus | None | None | Displays the WebAnalyzer's status.
ShowLogLevel | None | None | Displays the WebAnalyzer's current log verbosity setting.
ShowAllViews | None | None | Displays status for all views.
ShowAllPreferred | None | None | Displays the Lookup view for all collections.
ShowViewConfig | None | -n<viewname> | Shows the current configuration for the specified view.
ShowViewStatus | None | -n<viewname> | Displays the current status for a specified view.
ShowURIRelevance | -q<URI> | -n<viewname> -f<max_anchortexts> | Queries the WebAnalyzer lookup databases. A query returns the docrank, siterank, and anchortexts (up to the number that is specified by the -f option) of a specified URI.

waadmin miscellaneous commands

Command | Required options | Optional options | Description
BackupConfig | -b<filename> | None | Backs up the WebAnalyzer configuration.
RestoreConfig | -b<filename> | None | Restores the WebAnalyzer configuration from a file that is created by BackupConfig.
ExtendedHelp | (any command name, e.g. 'ExtendedHelp ShowURIRelevance') |  | Displays extended help for the specified options.

WebAnalyzer configuration options

Keyword (Parameter) | Value | Description
batch_size | <value> | The maximum number of items in a batch. Default: 128
callback_timeout | <value> | The time (in seconds) to wait for callback after the last batch is submitted. Default: 1800
concurrent_feeds | <value> | Specifies how many feeding clients can run at the same time. This value is the total number of clients on all nodes in the installation. Default: 4
cpus | A value equal to the "max-targets" value specified during installation | Specifies how many processes can be run at the same time. If this value times the number of nodes in the system is lower than the concurrent_feeds value, this value specifies the number of feeding clients.
pollwalsr_callback_interval | <value> | The interval (in seconds) between the times that the WebAnalyzer checks for new link data. Default: 300

FDM configuration options

Keyword (Parameter) | Value | Description
disk_free | <value> | Minimum amount of free disk space (in MB) that must be available on a node for link processing. If free disk space goes below this limit, the node will not be used for link processing. Default: 2000
verbose | true|false | Verbosity of the link analysis. Default: false

WebAnalyzer view configuration options

Keyword (Parameter) Value Description

drop_intra

true|false

Specifies if intra-site links are discarded from the analysis.

If false, all links are included in the analysis.

If true, only links going between items on different sites are included in the analysis.

Default: false

force_collections

true|false

Specifies if the WebAnalyzer should attempt to update all crawl collections in the view, even if the crawl collections have not received new content.

If false, the WebAnalyzer will only update crawl collections that it knows have received new . But it will still add all crawl collections in the view to the analysis.

Default: false

run_partial_update

true|false

Specifies if the WebAnalyzer should update the index with new relevance data after it completes an analysis run.

If false, the WebAnalyzer will not update the index, and the updates will not be stored and applied later.

Default: true

sort_buffer

<value>

The amount of memory (in megabytes) to use when sorting data. This value is specified per task running in the system.

Default: 200

Examples

The following example forces the WebAnalyzer to process the default WebAnalyzer view immediately, if it is not running:

<FASTSearchFolder>\bin\waadmin forceprocessing

The following example creates a new WebAnalyzer view called view1:

<FASTSearchFolder>\bin\waadmin createview -n view1

The following example configures view1 to use links from crawl collections sp and intrasitecoll:

<FASTSearchFolder>\bin\waadmin addcollections -nview1 -c"sp,intrasitecoll"

or

<FASTSearchFolder>\bin\waadmin addcollections -nview1 -csp

<FASTSearchFolder>\bin\waadmin addcollections -nview1 -cintrasitecoll

The following example shows the status for view1:

<FASTSearchFolder>\bin\waadmin showviewstatus -nview1

The following example configures view1 to run when crawl collection sp gets new content:

<FASTSearchFolder>\bin\waadmin enqueueview -nview1

The following example shows relevance data for an item with the item id <URL>, assuming that it is in the lookup databases for the default WebAnalyzer view:

<FASTSearchFolder>\bin\waadmin -q <URL> -f 5 ShowURIRelevance


About configuration files (FAST Search Server 2010 for SharePoint)

Microsoft FAST Search Server 2010 for SharePoint uses XML configuration files to control operations in several functional areas.

Using FAST Search for SharePoint configuration files

Many FAST Search Server 2010 for SharePoint operations are controlled by XML configuration files which hold specifications for Web crawlers, content connectors, the monitoring service, and logging. Many configuration files are supplied with default values, serving as templates for your site-specific needs.

Note:

To modify the configuration files, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Categories of configuration files

FAST Search Server 2010 for SharePoint configuration files control these functional areas.

Functional Area | Configuration File/Item Processing Stage | Usage
General administration | deployment.xml | Outline the topology of your FAST Search Server 2010 for SharePoint installation that includes all the nodes/servers in a deployment, and the distribution of components.
 | beconfig.xml | Configure options for the browser engine component in FAST Search Server 2010 for SharePoint.
 | user_converter_rules.xml | Register a third-party IFilter with FAST Search Server 2010 for SharePoint.
Monitoring service | monitoringservice.xml | Configure WMI classes and performance counters published by the FAST Search Server 2010 for SharePoint monitoring service.
FAST Search Web crawler | crawlerglobaldefaults.xml | Use this template file to specify options for the FAST Search Web crawler that apply to all crawl collections (DNS, content submission, duplicate detection, etc.).
 | crawlercollectiondefaults.xml | Use the parameters in this file to create or update one specific crawl collection. This configuration file controls the FAST Search Web crawler behavior, content gathering, content processing, and storage.
Connectors | jdbctemplate.xml | Use this template file to configure and crawl JDBC content with the FAST Search database connector.
 | lotusnotestemplate.xml | Use this template file to configure the FAST Search Lotus Notes content connector.
 | lotusnotessecuritytemplate.xml | Use this template file to configure the FAST Search Lotus Notes user directory connector.
 | cctklog4j.xml | Control the logs (console logs, file logs, network logger) created by the FAST Search database connector and the FAST Search Lotus Notes connector.
Item processing pipeline | pipelineextensibility.xml | Run external commands from the item processing pipeline to extract additional information from crawled documents. Typical uses include sentiment analysis, taxonomy classification, or other kinds of text analysis.
 | optionalprocessing.xml | Enable and disable optional processing stages in the item processing pipeline.
 | xmlmapper.xml | Extract and transform specific parts of crawled XML items.
 | OffensiveContentFilter (item processing stage) reference | Filter out items from the item processing pipeline according to their offensive content scores. Enabled and disabled as an optional processing stage.
Logging | loggerconfig.xml | Configure routing and filtering of log messages sent from FAST Search Server 2010 for SharePoint components. Configure the first step for how FAST Search Server 2010 for SharePoint processes output log messages.
 | logserverconfig.xml | Configure routing and filtering of log messages sent from FAST Search Server 2010 for SharePoint components. Configure the last two steps for the local and main log servers.
Note:

Modifying other configuration files than those listed in this table is not supported and could lead to system inconsistencies during re-configuration and software updates.


beconfig.xml reference

Use beconfig.xml to configure options for the browser engine component in Microsoft FAST Search Server 2010 for SharePoint. For example, use beconfig.xml to alter browser engine cache sizes or time-out settings.

The browser engine reads the beconfig.xml file in <FASTSearchFolder>\etc on startup.

Customizing beconfig.xml

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use a text editor (e.g. Notepad), not a general purpose XML editor, to change beconfig.xml.

To edit this file:

  1. Edit beconfig.xml in a text editor to specify settings. Use the existing file in <FASTSearchFolder>\etc\ as a starting point. Do not remove any attribute sections from the file.

  2. Run nctrl.exe restart browserengine to restart the browser engine process, with the new options.

beconfig.xml quick reference

The following table contains a list of the elements in beconfig.xml. These elements can appear in any order, but must occur inside other elements as specified in this table.

Element Description

<browserengine element>

Identifies this as a browser engine configuration file.

<browser element>

Specifies options for the virtual Web browser window. Can only occur inside a browserengine element.

<proxy element>

Specifies options for the internal proxy server. Can only occur inside a browserengine element.

<process element>

Specifies options that affect the processing of individual items. Can only occur inside a browserengine element.

<excludes element>

Contains one or more regexp elements, which specify regular exception rules that are used to exclude particular URIs from processing. Can only occur inside a browserengine element.

<regexp element>

Specifies a regular exclude rule. Can only occur inside an excludes attribute.

<pipeline element>

Specifies the processing pipeline options, and the pipeline steps to be performed on each item that is processed. Contains one or more extractor elements. Can only occur inside a browserengine element.

<extractor element>

Specifies an extractor. Must contain both a type and an assembly element, and may contain a parameters element. Can only occur inside a pipeline element.

Note:

The list of extractors and their sub-elements, as provided in <FASTSearchFolder>\etc\beconfig.xml, must not be altered.

beconfig.xml file format

XML elements in beconfig.xml begin with < and end with />.

The basic element format is as follows:

<element_name [attribute_name="value"] [attribute_name="value"] … />

For example:

<process maxOperations="1000" maxMemoryMB="1024" timeout="300" />

Elements and attributes are case-sensitive. Attribute values must be enclosed in quotation marks (" ") and are not case-sensitive.

An element definition can span multiple lines. Spaces, carriage returns, line feeds, and tab characters are ignored in an element definition.For example:

Copy Code
<process
    maxOperations="1000"
    maxMemoryMB="1024"
    timeout="300"
/>

For long element definitions, position attributes on separate lines and use indentation to make the file easier to read.

The basic structure of the beconfig.xml file is as follows:

Copy Code
<?xml version="1.0"?>
<browserengine>
    <browser ... />
    <proxy ... />
    <process ... />
    <excludes>
        ...
    </excludes>
    <pipeline>
        ...
    </pipeline>
</browserengine>

Comments can be added anywhere and are delimited by <!-- and -->.

browserengine element

Top level element.

Attributes

None

browser element

This element specifies options to the embedded Web browser component within the browser engine. Use this element to adjust the Web page item loading time-out period. For example, increase the time-out value if Web pages frequently time out during item loading.

Attributes

Attribute Value Description

width

<pixels>

Web pages are rendered in an invisible Web browser window. This option specifies the width of this window in pixels.

Default: 1280

height

<pixels>

Specifies the height of the invisible Web browser window in pixels.

Default: 1024

visible

true|false

Makes the Web browser window visible during processing. Use for debugging only.

Makes the Web browser window invisible during processing.

Default: false

images

true|false

Specifies that the browser engine should load the images contained on Web pages. Use for debugging only.

Specifies that the browser engine should not load the images contained on Web pages.

Default: false

timeout

<seconds>

Specifies the time-out period, in seconds, for the browser engine to load the Web page being processed. If a Web page takes longer to load, it will be discarded.

This option does not account for the time taken to run the processing pipeline after loading is completed.

Default: 60

Example

Copy Code
<browser width="1280" height="1024" visible="false" images="false" timeout="60"/>

proxy element

This element specifies options for the internal Web proxy and memory cache used by the browser engine. Use this element to adjust the cache size and maximum age of JavaScripts in the cache.

Attributes

Attribute | Value | Description
maxsize | <bytes> | Specifies the maximum size of a single JavaScript that will be downloaded from the Web or the Web crawler. Items that exceed this threshold will be discarded. Default: 10485760
timeout | <timeout> | Specifies the time-out period, in seconds, for any JavaScript or Web page downloaded from the Web or the Web crawler. If a download exceeds this time-out, it will be discarded. Default: 60
cacheSize | <megabytes> | Specifies the maximum size of the JavaScript cache within the browser engine. It is used for keeping frequently used JavaScripts available without re-downloading them. Default: 25
cacheTTL | <seconds> | Specifies the maximum age, in seconds, of JavaScripts in the cache before they are evicted. A JavaScript may be evicted earlier if the cache fills up. Default: 3600

Example

Copy Code
<proxy maxsize="10485760" timeout="60" cacheSize="25" cacheTTL="3600"/>

process element

This element specifies options that relate to the processing of Web items in the browser engine. Use this element to adjust the maximum memory usage and the pipeline time-out period.

Attributes

Attribute | Value | Description
maxOperations | <operations> | Specifies the maximum number of Web pages to be processed before the browser engine automatically restarts. This is useful to handle potential memory leaks and stuck processing that may be caused by some Web pages. Default: 1000
maxMemoryMB | <megabytes> | Specifies the maximum memory usage, in MB, before the browser engine automatically restarts. This is useful to handle potential memory leaks and stuck processing that may be caused by Web pages. Default: 1024
timeout | <timeout> | Specifies the time-out period, in seconds, for extracting hyperlinks from any specific Web page. This time-out is required to handle cases in which, for example, a JavaScript prevents the processing pipeline from completing processing of a Web page. Default: 300

Example

Copy Code
<process maxOperations="1000" maxMemoryMB="1024" timeout="300"/>

excludes element

This element specifies one or more regular expression rules used to prevent the download of specific JavaScript and cascading style sheet URIs. A typical use excludes known advertising scripts to speed up Web page processing and to prevent the scripts from appearing in the content index.

Attributes

None

Example

Copy Code
<excludes>
    <regexp value="http://ads\."/>
</excludes>

regexp element

This element specifies a single regular expression exclude rule and can only occur inside an excludes element. This element can occur multiple times.

Attributes

Attribute | Value | Description
value | <regexp> | Specifies a regular expression that is matched against all external JavaScript and cascading style sheet URIs discovered during processing the Web item. URIs matching the regular expression are not downloaded or included during Web page processing. Default: See <FASTSearchFolder>\etc\beconfig.xml for the default value.

Example

See excludes element example.

pipeline element

This element specifies the set of extractors that are executed on each Web page during processing in the browser engine. An extractor performs a set of operations, such as extracting a certain kind of hyperlink or HTTP cookies, generating checksum and the final item HTML used for content indexing.

Attributes

Attribute Value Description

name

default

Specifies the name of the pipeline. Only a single pipeline is supported and the name must be "default".

maxFrameLevels

<levels>

Specifies the number of HTML frame levels to process. Normally this option is set to 1, which means that only the top level frame and its immediate child frames (the frameset) are processed.

Increasing this number will recursively process multiple frame sets.

Default: 1

timeout

<seconds>

Specifies the maximum time that the processing pipeline can run on a single Web page before it is stopped.

Increasing this value will decrease browser engine throughput, but can help reduce Web page processing timeouts. Decreasing the value may improve throughput at the expense of possibly more timeouts.

Default:300

iterations

1

Specifies the number of iterations to run the pipeline on each Web page. Only one iteration is supported.

abortOnFailure

true|false

Specifies that the processing of a Web page should be stopped if any single extractor fails.

Specifies that the processing of a Web page should continue even if some extractors fail. This may improve link extraction, but can (in the worst case) lead to partial items being sent to the content index.

default

true

Specifies that this pipeline is the default pipeline. Because only one pipeline is supported, this value must always be set to "true".

Example

Copy Code
<pipeline name="default" maxFrameLevels="1" timeout="180" iterations="1" abortOnFailure="true" default="true">
..
</pipeline>

extractor element

This element specifies a single extractor in the pipeline. The list of extractors as provided in <FASTSearchFolder>\etc\beconfig.xml must not be altered.


cctklog4j.xml reference

Use the cctklog4j.xml file to control the logs created by the FAST Search Lotus Notes connector and the FAST Search database connector. This includes console logs, file logs, or a Microsoft FAST Search Server 2010 for SharePoint network logger. You can also specify the level of detail in these logs.

Warning:

Any changes that you make to this file will be overwritten and lost if you:

  • Run the Set-FASTSearchConfiguration Windows PowerShell cmdlet.

  • Install a FAST Search Server 2010 for SharePoint update or service pack.

To avoid losing your changes, make sure that you back up this file after you have modified it.

Remember to reapply your changes after you run the Set-FASTSearchConfiguration Windows PowerShell cmdlet or install a FAST Search Server 2010 for SharePoint update or service pack.

The FAST Search connector reads the cctklog4j.xml file at startup for log configuration instructions. This configuration file sets a log level for each connector component, which provides a filter for the generated messages. For example, you can set INFO level logs for the main connector component, and ERROR level logs for the libraries used by the connector.

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Note:

Logging uses the log4j Java utility. The cctklog4j.xml file includes links to Apache log4j documentation for additional reference.

Elements quick reference

There are three kinds of elements in cctklog4j.xml.

Element Type | Description
cctklog4j.xml appender elements reference | log4j uses an "appender" to define an output destination, such as the console or a file. You can configure three kinds of log appenders for the FAST Search connectors: console, rollingfile, and net. Each appender has a set of configuration parameters to control a specific log.
cctklog4j.xml logger element reference | Each connector component has a separate log and log level; logger defines the destination and the level of verbosity for these logs.
cctklog4j.xml root element reference | This element enables the appenders, and specifies a log level for components that are not explicitly configured.

cctklog4j.xml file format

XML elements in cctklog4j.xml begin with < and end with />.

The basic format is as follows:

<appender name=["value"]>

<param name=["value"] value=["value"] />

<layout class=["value"] >

<param name=["value"] value=["value"] />

</layout>

</appender>

<logger name=["value"]>

<level value=["value"] />

</logger>

<root>

<level value=["value"] />

<appender-ref ref=["value"] />

</root>

For example:

Copy Code
<logger name="org.apache.axis.utils.JavaUtils">
     <level value="ERROR"/>
</logger>

Elements and attributes are case-sensitive, attribute values are not. Attribute values must be enclosed in quotation marks (" ").

An element definition can span multiple lines. Spaces, carriage returns, line feeds, and tab characters are ignored in an element definition.

For example:

Copy Code
<logger name="org.apache.axis.utils.JavaUtils">

    <level value="ERROR"/>

</logger>
Tip:

For long element definitions, position attributes on separate lines and use indentation to make the file easier to read.

The <log4j:configuration> element is a special case and is required. All other elements are contained within the <log4j:configuration> element, and the element is closed with </log4j:configuration>.

You can add comments anywhere, delimited by <!-- and -->.

For example:

Copy Code
<!-- Setting the log level to ERROR avoids warnings about MIME -->
<logger name="org.apache.axis.utils.JavaUtils">
    <level value="ERROR"/>
</logger>

cctklog4j.xml appender elements reference

Appender elements define the logging output format and physical destination of the logs.

The elements are additionally specified with sub-parameters and layout classes.

cctklog4j.xml appender console element reference

Use the console appender to specify console output and message contents.

Attribute | Value | Description
ConversionPattern | %d{ABSOLUTE} %-5p [%c{1}] %m%n | Specifies the layout of the log message. See the Apache log4j documentation for detailed information. Default: time, level, class name, and message

Example

The following example configures output with a specific message layout:

Copy Code
<appender name="console" class="org.apache.log4j.ConsoleAppender">
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d{ABSOLUTE} %-5p [%c{1}] %m%n"/>
    </layout>
</appender>

cctklog4j.xml appender rollingfile element reference

The rollingfile appender logs messages to a configured file until that file reaches a specified size. When the log file reaches the maximum size, the current file is renamed (a sequential number is added to the file name) and a new log file is created. This appender also controls the maximum number of files in a rolling set, and the contents of log messages.

Attribute | Value | Description
Append | true|false | If true, appends to existing files when the connector is started. If false, starts from an empty file.
File | log/${cctk.module}connector.log | Path where the file will be written. Do not change this path. ${cctk.module} will be changed to lotusnotesconnector or jdbcconnector automatically at run time.
MaxFileSize | value | Specifies the maximum file size. The next message will be written to a new file that uses the same name. The old file name is modified with an appended sequence number. You can also specify MB (megabytes) instead of KB. Default: 5000KB
MaxBackupIndex | value | Specifies the maximum number of log files for the rotating logging. If all files have reached the maximum size, the oldest file is overwritten. Default: 10
ConversionPattern | %d{ISO8601} %-5p [%c{1}] %m%n | See the Apache log4j documentation for detailed information. Default: datetime, level, class name, and message.

Example

The following example sets up a rolling file appender, rotating on 10 log files of 5000 KB each:

Copy Code
<appender name="rollingfile" class="org.apache.log4j.RollingFileAppender">
    <param name="Append" value="true"/>
    <param name="File" value="log/${cctk.module}connector.log"/>
    <param name="MaxFileSize" value="5000KB"/>
    <param name="MaxBackupIndex" value="10"/>
    <layout class="org.apache.log4j.PatternLayout">
        <param name="ConversionPattern" value="%d{ISO8601} %-5p [%c{1}] %m%n"/>
    </layout>
</appender>

The files are named lotusnotesconnector.log.1, lotusnoteconnector.log.2, and so on Messages include time stamp, level, class name, and message content.

cctklog4j.xml appender net element reference

The net appender specifies the FAST Search Server 2010 for SharePoint network logger file name and message contents.

Note:

The network log file is created automatically; do not change this element and attributes.

Attribute | Value | Description
LogHost | hostname | Automatically set by the installer; do not change.
LogPort | portnumber | Automatically set by the installer; do not change.
LogModuleName | ${cctk.module}connector | Automatically set to the connector sending the logs at run time. Do not change.

Example

The following example locates the network logger at localhost, port 13415:

Copy Code
<appender name="net" class="com.fastsearch.esp.cctk.logging.ESPLogServerAppender">
    <param name="LogHost" value="localhost"/>
    <param name="LogPort" value="13415"/>
    <param name="LogModuleName" value="${cctk.module}connector"/>
</appender>

The module name in the logs is lotusnotesconnector or jdbcconnector.

cctklog4j.xml logger element reference

Use the logger element to define the level of verbosity in the logs for different connector components, which are defined through the package names for classes. The packages should be left as is (e.g., do not remove a logger), but you can change the level of verbosity.

Attribute Value Description

logger name

logger name

Specifies the connector component:

  • com.fastsearch.esp.cctk - defines the log level of CCTK (Connector Controller Tool Kit) messages.

  • org.apache.axis.utils.JavaUtils - defines the log level of MIME messages in content feed.

  • org.apache.commons.httpclient - defines the log level of cookie rejection messages.

  • com.fastsearch.esp.connectors.lotus - defines the log level of the FAST Search Lotus Notes content connector messages.

  • com.fastsearch.esp.connectors.lotus.userdirectory - defines the log level of the FAST Search Lotus Notes user directory connector messages.

  • com.fastsearch.esp.connectors.jdbc - defines the log level of the FAST Search database connector messages.

level value

logging level

Specifies the logging level:

  • INFO - logs all messages at INFO, WARN, and ERROR level.

  • DEBUG - logs all messages at DEBUG, INFO, WARN, and ERROR level.

  • WARN - logs all messages at WARN, and ERROR level.

  • ERROR - logs all messages at ERROR level.

  • TRACE - logs all messages at TRACE, DEBUG, INFO, WARN, and ERROR level.

Example

Copy Code
<logger name="com.fastsearch.esp.cctk">
    <level value="INFO"/>
</logger>

<logger name="org.apache.axis.utils.JavaUtils">
    <level value="ERROR"/>
</logger>

<logger name="org.apache.commons.httpclient">
    <level value="ERROR"/>
</logger>

<logger name="com.fastsearch.esp.connectors.lotus">
    <level value="INFO"/>
</logger>

<logger name="com.fastsearch.esp.connectors.lotus.userdirectory">
    <level value="INFO"/>
</logger>

<logger name="com.fastsearch.esp.connectors.jdbc">
    <level value="INFO"/>
</logger>

cctklog4j.xml root element reference

Use the root element to define log appenders that should be used and the level of verbosity for components that are not individually specified in the cctklog4j.xml configuration.

Attribute Value Description

level value

logging level

Specifies the logging level:

  • INFO - logs all messages at the INFO, WARN, and ERROR levels.

  • DEBUG - logs all messages at the DEBUG, INFO, WARN, and ERROR levels.

  • WARN - logs all messages at the WARN, and ERROR levels.

appender-ref ref

log

Specifies the log to enable:

  • console - enables the console log.

  • rollingfile - enables the rolling file log.

  • net - enables the FAST Search Server 2010 for SharePoint network logging.

Example

Copy Code
<root>

    <level value="WARN"/>

    <appender-ref ref="console"/>

    <appender-ref ref="rollingfile"/>

    <appender-ref ref="net"/>

</root>


Web Crawler XML configuration reference
See Also

The FAST Search Web crawler automatically retrieves information from Web sites, and passes this information to the Microsoft FAST Search Server 2010 for SharePoint index. The FAST Search Web crawler is configured by creating an XML configuration file formatted as specified in this article, and submitting it to the Web crawler using the crawleradmin.exe command-line tool.

The format specified in this document is also used by the crawlercollectiondefaults.xml file, which contains all the default options/values for new crawl collections. When you modify it, you change the defaults for all new collections. The default values are used for any option not specified in the XML configuration created for a specific crawl collection.

These configuration files must be formatted in compliance with the XML schema. This document includes a Simple configuration and a Typical configuration example of a configuration file. For an overview of the elements and sections in the configuration file, refer to the table in Web Crawler XML configuration quick reference.

Key terminology

Web site refers not to a SharePoint site, but to the content on a Web site such as www.contoso.com.

Host name refers to either "contoso" in http://contoso/ or "download.contoso.com" in http://download.contoso.com/. It can be either fully qualified or not. In this document, the difference between a Web site and a host name is that a Web site describes the actual site and its content, whereas the host name is the network name that is used to reach a given Web server. A single site might have multiple host names.

Creating a new crawl configuration

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Follow these steps to create a new crawl configuration using this XML configuration format:

  1. Copy one of the three supplied crawl configuration templates found in <FASTSearchFolder>\etc (where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch) to a new file such as MyCollection.xml, or create a new file. Edit the file in a text editor to include the elements and settings that you must have.

    Note:

    Use a text editor (e.g., Notepad) to change crawlercollectiondefaults.xml. Do not use a general-purpose XML editor.

  2. Run crawleradmin.exe -f MyCollection.xml to add the crawl configuration to the crawler. Replace MyCollection.xml with the name that you gave the file in step 1.

See crawleradmin.exe reference for more information.

Customizing crawlercollectiondefaults.xml

Warning:

Any changes that you make to this file will be overwritten and lost if you:

  • Run the Set-FASTSearchConfiguration Windows PowerShell cmdlet.

  • Install a FAST Search Server 2010 for SharePoint update or service pack.

To avoid losing your changes, make sure that you back up this file after you have modified it.

Remember to reapply your changes after you run the Set-FASTSearchConfiguration Windows PowerShell cmdlet or install a FAST Search Server 2010 for SharePoint update or service pack.

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

To edit this file:

  1. Edit crawlercollectiondefaults.xml in a text editor to include the elements and settings that you must have. Use the existing file in <FASTSearchFolder>\etc\ as a starting point.

    Note:

    Use a text editor (for example Notepad) to change crawlercollectiondefaults.xml. Do not use a general-purpose XML editor.

  2. Run nctrl.exe restart crawler to restart the FAST Search Web crawler process with the options that you set in step 1.

Web Crawler XML configuration quick reference

This table lists the elements in the Web Crawler XML configuration format. The elements can appear in any order with the following exceptions. CrawlerConfig holds the DomainSpecification element. The primary elements of SubDomain, Login, and Node occur inside the DomainSpecification element. The section and attrib sub-elements can occur in any of the primary elements, in any order. The member sub-elements must appear inside an attrib element only.

Copy Code
<CrawlerConfig>
      <DomainSpecification>
             <SubDomain/>
             <Login/>
             <Node/>
             <attrib>
                    <member/>
             </attrib>
             <section/>
      </DomainSpecifcation>
</CrawlerConfig>

Typically, you will include both attrib and section sub-elements in SubDomain, Login, and section elements. The Node element may contain all these elements and sub-elements.

Element | Description
CrawlerConfig | This top-level element specifies that the XML following it is a Web crawler configuration object.
DomainSpecification | This element specifies a crawl collection.
SubDomain | This element specifies the configuration of crawl sub collections.
Login | This element is used for HTML forms-based authentication.
Node | This element overrides configuration parameters in a crawl collection or a crawl sub collection for a particular node scheduler.
attrib | This sub-element specifies a configuration setting, either by its value or by a set of member elements.
member | This sub-element specifies a configuration setting in a list.
section | This sub-element specifies a section that contains multiple settings grouped by type. A table listing all possible sections follows.

This table defines the section options in the Web Crawler XML configuration format. Sections cannot occur inside the CrawlerConfig element.

Section name | Description
include_domains | Defines a set of host name filters that specify which URIs to include in a crawl collection
exclude_domains | Defines a set of host name filters that specify which URIs to exclude from a crawl collection
include_uris | Defines a set of URI rules that specify which URIs to include in a crawl collection
exclude_uris | Defines a set of URI rules that specify which URIs to exclude from a crawl collection
log | Specifies logging behavior for the Web crawler process
storage | Specifies how the Web crawler stores content and metadata
pp | Specifies the post processing behavior for a node scheduler
ppdup | Specifies duplicate server settings
feeding | Consists of at least one section element that specifies how to send a representation of the crawl collection to the indexing engine
cachesize | Configures the cache sizes for the Web crawler process
http_errors | Specifies how to handle HTTP/HTTPS error response codes and conditions
ftp_errors | Specifies how to handle response codes and error conditions for FTP URIs
workqueue_priority | Specifies the priority levels for the crawl queues, and specifies the rules and modes used to insert URIs into and extract URIs from the queues
link_extraction | Specifies which kind of hyperlinks to follow
limits | Specifies fail-safe limits for a crawl collection
focused | Configures focused scheduling
passwd | Configures credentials for Web sites that require authentication
ftp_acct | Specifies FTP accounts for crawling FTP URIs
exclude_headers | Specifies items to exclude from the crawl, based on the contents of the HTTP header fields
variable_delay | Specifies time slots that use a different delay request rate
adaptive | Specifies the adaptive crawling options
weights | Each URI is given a score in the adaptive crawling process. The weights section must occur inside an adaptive section.
sitemap_weights | <URL> entries in a sitemap can contain a changefreq element, which specifies how frequently a URI can be modified. The string values are converted into a numeric weight for adaptive crawling. The sitemap_weights section must occur in an adaptive section.
site_clusters | Specifies configuration parameters that override the crawler's usual behavior of routing host names in a node scheduler
crawlmode | Limits the span of a crawl collection
post_payload | Submits content to HTTP POST requests
rss | Initializes and configures RSS feed support in a crawl collection
logins | This is a special case of a Login element; multiple Login elements are merged into a logins section. Either a logins section or one or more Login elements are required when you define HTML forms-based authentication. You must use logins to remove a login because of the way partial configurations work. Note that exporting a configuration from the crawler with crawleradmin returns the Login element.
parameters | Sets the authentication credentials that are used in an HTML form. Must occur in a Login element or a logins section.
subdomains | Specifies the configuration of crawl sub collections. This is a special case of a SubDomain element; multiple SubDomain elements are merged into a subdomains section. You must use subdomains to remove a subdomain because of the way partial configurations work. Note that exporting a configuration from the crawler with crawleradmin returns the SubDomain element.

Web Crawler XML configuration file format

XML elements in the configuration file begin with < and end with />.

The basic element format is as follows:

<attrib name="value" type="value">value</attrib>

For example:

Copy Code
<attrib name="accept_compression" type="boolean"> yes </attrib>

Elements, section names, attributes, and attribute values are case-sensitive. Attribute names and types must be enclosed in quotation marks (" ").An element definition can span multiple lines. Spaces, carriage returns, line feeds, and tab characters are ignored in an element definition.

For example:

Copy Code
<attrib
    name=" accept_compression "
    type="boolean"
> yes </attrib
>
Tip:

For long parameter definitions, position values on separate lines and use indentation to make the file easier to read.

The <CrawlerConfig> element is a special case and is required. All other elements are contained within the <CrawlerConfig> element, and the element is closed with </CrawlerConfig>.

The basic structure of the XML file is in the following example:

Copy Code
<?xml version="1.0"?>
<CrawlerConfig>
    <DomainSpecification>
        ...
    </DomainSpecification>
</CrawlerConfig>

You can add comments anywhere, delimited by <!-- and -->.

CrawlerConfig

This top-level element specifies that the XML following it is a Web crawler configuration object. A Web crawler configuration file can contain only one CrawlerConfig XML element.

DomainSpecification

This element specifies a crawl collection.

Example

Copy Code
<CrawlerConfig>
  <DomainSpecification name="sp">
  ...
  </DomainSpecification>
</CrawlerConfig>

Replace "sp" with the crawl collection name.

attrib

This element specifies a configuration option, either a single value or a list using the member element.

Attributes

Name Type Value Meaning

info

string

A text description of the crawl collection.

fetch_timeout

integer

<seconds>

Specifies the maximum downloading time, in seconds, for a Web item. Increase this value if you expect to download large Web items from slow Web servers.

Default: 300

allowed_types

list-string

Specifies valid Web item MIME types.

The Web crawler process discards other MIME types. This configuration parameter supports wildcard expansion of a whole field. Wildcards are represented by an asterisk character. For example: "text/*" or "*/*" but not "*/html" or "application/ms*".

Default:

  • text/html

  • text/plain

  • application/msword

  • application/msexcel

  • application/ppt

  • application/pdf

force_mimetype_detection

boolean

yes|no

Specifies that the Web crawler process uses its own MIME type detection on items. In most cases, Web servers return the MIME type of Web items when they are downloaded, as part of the HTTP header. If this option is enabled, Web items will get tagged with the MIME type that looks most accurate: either the one received from the Web server or the result of the crawler's detection.

Default: no

allowed_schemes

list-string

HTTP

HTTPS

FTP

Specifies the URI schemes that the Web crawler should process.

Default: HTTP

ftp_passive

boolean

yes|no

Specifies that the Web crawler uses passive FTP mode.

Default: yes

domain_clustering

boolean

yes|no

Specifies whether to route host names from the same domain to the same site manager process. Useful when you are dealing with host names that must share information such as cookies, because this information is not exchanged between site manager processes. If enabled in a multiple node configuration, host names on the same domain (for example, www.contoso.com and forums.contoso.com) will also be routed to the same node scheduler.

Default for single node: no

Default for multiple node: yes

max_inter_docs

integer

positive integer, or no value

Specifies the maximum number of items to crawl before interleaving Web sites. By default, the crawler will crawl a Web site to exhaustion, or until the maximum number of Web items per Web site is reached. However, the crawler can be configured to crawl "batches" of Web items from Web sites at a time, interleaving between Web sites. This attribute specifies how many Web items to consecutively crawl from a server before the crawler interleaves and starts crawling other servers. The crawler will return to crawling the former server when resources are freed up.

Default: empty (disabled)

max_redirects

integer

<value>

Specifies the maximum number of HTTP redirects to follow from a URI.

Default: 10

diffcheck

boolean

yes|no

Specifies that the Web crawler performs duplicate detection. Duplicate detection is performed by checking whether two or more Web items have the same content.

Default: yes

near_duplicate_detection

boolean

yes|no

Specifies that the Web crawler must use a less strict duplicate detection algorithm. In this case duplicate items are detected by identifying a unique pattern of words.

Default: no

max_uri_recursion

integer

<value>

Use this attribute to check for repeating patterns in URIs. The option specifies the maximum number of times a pattern can be repeated before the resulting URI is discarded. A value of 0 disables the test.

For example, http://www.contoso.com/widget linking to http://www.contoso.com/widget/widget is a repetition of 1 element.

Default: 5

ftp_searchlinks

boolean

yes|no

Specifies that the Web crawler should search for hyperlinks in items downloaded from FTP servers.

Default: yes

use_javascript

boolean

yes|no

Specifies if JavaScript support should be enabled in the Web crawler. If enabled, the Web crawler will download, parse/execute, and extract links from any external JavaScript.

Note:

JavaScript processing is resource intensive and should not be enabled for large crawls.

Note:

Processing JavaScript uses the Browser Engine component. For more information, see beconfig.xml reference.

Default: no

javascript_keep_html

boolean

yes|no

Specifies what to submit to the indexing engine. If this parameter is set to yes, the HTML that results from the JavaScript processing is used. Otherwise, the original HTML item is used.

Do not use this option if the use_javascript configuration parameter is not set to yes.

javascript_delay

real

<seconds>

An empty value means that the Web crawler uses the same value as the delay configuration parameter

Specifies the delay in seconds, to use when you are retrieving dependencies associated with an HTML item with JavaScript.

Default: 0 (no delay)

exclude_exts

list-string

<comma delimited list of file_extensions>

Specifies file name extensions that should be excluded by the crawl.

Default list: empty

use_http_1_1

boolean

yes|no

Specifies that the Web crawler should use HTTP/1.1. When set to no, HTTP/1.0 is used.

Default: yes

accept_compression

boolean

yes|no

Specifies that the Web crawler should accept compressed Web items from the Web server. This parameter has no effect if the use_http_1_1 configuration parameter is not enabled.

Default: yes

dbswitch

integer

<value>

Specifies the number of crawl cycles that a Web item can remain in the crawl store and index without being found by the Web crawler, before it is deleted. The dbswitch_delete parameter determines the action that should be taken for Web items that are not seen for this number of crawl cycles.

Note:

Setting this value very low to 1 or 2 may accidentally delete Web items.

Default: 5

dbswitch_delete

boolean

yes|no

The Web crawler tries to detect Web items that were removed from the Web servers. This parameter determines what to do with those Web items. They can be deleted immediately or put in the work queue for retrieval to make sure that they are no longer available.

When set to yes, Web items that are too old are deleted. When set to no, Web items are scheduled for re-retrieval and only deleted if they no longer exist on the Web server.

This check is performed independently for each Web site, at the start of each refresh cycle.

Note:

You should keep this option at the default value.

Default: no

html_redir_is_redir

boolean

yes|no

Use this parameter with html_redir_thresh to treat META Refresh tags inside HTML Web items as if they were HTTP redirects. When enabled, the Web item that contains the META refresh will not be indexed. When disabled, they are treated as regular Web items and are indexed.

Default: yes

hmtl_redir_threshold

integer

<value>

Specifies the maximum number of seconds that a META Refresh tag inside an HTML Web item can be treated as an HTTP redirect. This parameter is ignored if html_redir_is_redir is not set.

Consider the following example:

Copy Code
<META HTTP-EQUIV="Refresh" CONTENT="3;URL=http://www.some.org/some.html">

If the number that is specified in the CONTENT attribute (in this example, it is 3) is less than or equal to the value of html_redir_threshold, the META Refresh tag is treated as a redirect.

Default: 3

robots_ttl

integer

<seconds>

Specifies how frequently the Web crawler should retrieve the robots.txt file from a Web site. The frequency must be specified in seconds.

Default: 86400

use_sitemaps

boolean

yes|no

Enables the Web crawler to discover and parse sitemaps.

The Web crawler uses the lastmod attribute in a sitemap to determine whether a Web item was modified since the last time that the sitemap was retrieved. Web items that were not modified will not be re-crawled.

An exception is if the collection usesadaptive refresh mode. In adaptive refresh mode, the crawler uses a sitemap's priority and changefreq attributes to determine how often a Web item should be crawled. Other tags found in sitemaps are stored in the crawler's meta database and are submitted to indexing as crawled properties.

Note:

Most sitemaps are specified in robots.txt. Thus, the robots attribute should beenabled for the best results.

Default: no

max_pending

integer

<value>

Specifies the maximum number of concurrent HTTP requests to a single Web site at any time.

Default: 2

robots_auth_ignore

boolean

yes|no

Specifies whether the Web crawler should ignore robots.txt if an HTTP 40x authentication error is returned by the Web server. When set to no, the Web crawler will not crawl the Web site upon encountering the error.

The robots.txt standard lists this behavior as a hint for Web crawlers to ignore the Web site completely. However, incorrect configuration of a Web server can incorrectly exclude a site from the crawl. Enable this option to make sure that the Web site is crawled.

Default: yes

robots_tout_ignore

boolean

yes|no

Specifies whether the Web crawler should ignore the robots.txt rules if the request for robots.txt times out.

Before crawling a Web site, the Web crawler requests the robots.txt file from the Web server. By the robots.txt standard, if the request for this file times out, the Web site will not be crawled. Setting this parameter to yes ignores the robots.txt rules in this case, and the Web site is crawled.

Note:

You should keep this option set to no if you do not own the Web site being crawled.

Default: no

rewrite_rules

list-string

Specifies a set of rules that are used to rewrite URIs.

A rewrite rule has two components: an expression to match (match_pattern), and a replacement string (replacement_string) that will replace the first expression. The expression to match is a grouped match regular expression.

The format of the rewrite rule is as follows: @match_pattern@replacement_string@, where @ is any non-white-space separator character that is not included in the expression itself.

extract_links_from_dupes

boolean

yes|no

Specifies that the Web crawler should extract hyperlinks from duplicate Web items. Even when two Web items have duplicate content, they may have different hyperlinks, which could lead to more content being found by the Web crawler.

Default: no

use_meta_csum

boolean

yes|no

Specifies that the Web crawler includes META tags in the generated duplicate detection fingerprint.

Default: no

csum_cut_off

integer

<value>

Specifies the maximum number of bytes to use to generate the duplicate detection fingerprint. If this parameter is set to 0, the feature is disabled (i.e., unlimited/all bytes will be used).

Default: 0

if_modified_since

boolean

yes|no

Specifies whether the Web crawler should send HTTP headers that contain a value of If-Modified-Since.

Default: yes

use_cookies

boolean

yes|no

Specifies whether the Web crawler should send and store cookies. This feature is automatically enabled for Web sites that use a login, but can also be turned on for all Web sites.

Default: no

uri_search_mime

list-string

<values>

Specifies the MIME types from which the Web crawler extracts hyperlinks.

This configuration parameter supports wildcard expansion only at the whole field level. A wildcard is represented by the asterisk character; for example, text/* or */* but not */html or application/ms*.

Default:

  • text/html

  • text/vnd.wap.xml

  • text/wml

  • text/x-wap.wml

  • x-application/wml

  • text/x-hdml

max_backoff_counter

integer

<value>

Together with max_backoff_delay, this option controls the algorithm by which a Web site experiencing connection failures is contacted less frequently.

For each consecutive network error, the request delay for that Web site is increased by the original delay setting, up to a maximum of max_backoff_delay seconds. This delay is maintained until a request successfully is completed, but for no more than max_backoff_counter number of requests. If the maximum count is reached, crawling of the Web site is temporarily stopped.

Otherwise, when network issues affecting the Web site are resolved, the internal backoff counter starts decreasing, and the request delay is decreased by half on each successful Web item download until the original delay setting is reached.

Default: 50

max_backoff_delay

integer

<seconds>

See max_backoff_counter.

Default: 600

delay

real

<seconds>

Specifies how frequently (in seconds) the Web crawler can retrieve a Web item from a Web site.

Default: 60.0

refresh

real

<minutes>

Specifies how frequently (in minutes) the Web crawler should start a new crawl refresh cycle.

The action that is performed at the time of refresh is determined by the refresh_mode setting.

Default: 1500.0

robots

boolean

yes|no

Specifies that the Web crawler should obey the rules found in robot.txt files.

Default: yes

start_uris

list-string

Specifies start URIs for the Web crawler. The Web crawler needs either start_uris or start_uri_files to start crawling.

Note:

If the crawl includes any IDNA host names, enter them using UTF-8 characters, not in the DNS encoded format.

start_uri_files

list-string

Specifies a list of files that contain start URIs. These files are stored in plain text file format, with one start URI per line.

Note:

In a multiple node deployment, these files must only be available on the server that runs the multi-node scheduler.

max_sites

integer

<value>

Specifies the maximum number of Web sites that can be crawled at the same time. In a multi-node Web crawler deployment, this value applies per node scheduler, not to the whole Web crawler.

For example, if max_sites is set to 5 and you have 10 sites to crawl, 5 sites must finish crawling before the crawler can crawl the other 5.

Note:

A high max_sites value can adversely affect system resource usage.

Default: 128

mirror_site_files

list-string

Specifies a list of files that contain mirror sites for a specified host name. A mirror site is a replica of an already existing Web site. This file uses the following format: a plain text file that has a space-separated list of host names with the preferred name listed first.

Note:

In a multiple node Web crawler deployment, this file must be available on all servers where a node scheduler is deployed.

proxy

list-string

Specifies a set of HTTP proxies that the Web crawler uses to fetch Web items.

Each proxy is specified by using the following format:

(http://)((domain!)username:password@)hostname(:port), optional parts are enclosed in parentheses.

The password can be encrypted as specified in passwd.

proxy_max_pending

integer

<value>

Specifies a limit on the number of outstanding open connections per HTTP proxy.

Default: maximum value of INT32

headers

list-string

<header>

Specifies additional HTTP headers to add to the request sent to the Web servers.

The current default is as follows: User-Agent: FAST Search Web Crawler <version>

cut_off

integer

Specifies the maximum number of bytes in an item. A Web item larger than this size limit is discarded or truncated depending on the value of the truncate configuration parameter.

If no cut_off configuration parameter is specified, this option is disabled.

Default: no cut-off

truncate

boolean

yes|no

Specifies whether a Web item should be truncated when a Web item exceeds the specified cut_off threshold.

Default: yes

check_meta_robots

boolean

yes|no

Specifies that the Web crawler should follow the <NoIndex /> and <NoFollow /> directives given by the robots META tag.

For example, a typical META tag might be:

Copy Code
<meta name="robots" content="nofollow,noindex"/>

or

Copy Code
<meta http-equiv="robots" content="nofollow,noindex"/>

The special value none means both nofollow and noindex.

Default: yes

obey_robots_delay

boolean

yes|no

Specifies that the Web crawler should follow the crawl-delay directive (if present) in robots.txt files. Otherwise, the delay setting is used.

Default: no

key_file

string

Specifies the path of an SSL client certificate key file that is used for HTTPS connections.

This feature is used for Web sites that require the Web crawler to authenticate itself using a client certificate.

This option must be used with cert_file.

Note:

In a multi-node Web crawler deployment, the file must be on all node schedulers.

cert_file

string

Specifies the path of an X509 client certificate file that is used for HTTPS connections.

This option must be used with key_file.

max_doc

integer

<value>

Specifies the maximum number of Web items to download from a Web site.

Default: 100000

enforce_delay_per_ip

boolean

yes|no

Specifies that the Web crawler limits requests to Web servers whose names map to a shared IPv4 or IPv6 address. This parameter depends on the delay configuration parameter.

Default: yes

wqfilter

boolean

yes|no

Specifies whether the Web crawler should use a bloom filter that removes duplicate URIs from the crawl queues.

Default: yes

smfilter

integer

<value>

Specifies the maximum number of bits in the bloom filter that removes duplicate URIs from the queue associated with the node scheduler.

A bloom filter is a space-efficient probabilistic data structure (a bit array) which is used to test whether an element is a member of a given set. The test may yield a false positive but never a false negative.

Default: 0

mufilter

integer

<value>

Specifies the maximum number of bits used in the bloom filter, which removes duplicate URIs, which are sent from a node scheduler to a multi-node scheduler.

We recommend that you turn on this filter for large crawls, with a value of 500000000 (500 megabit).

Default: 0

umlogs

boolean

yes|no

Specifies whether all logging is sent to the multi-node scheduler for storage.

If this parameter is not enabled, logs reside only on the node schedulers.

Default: yes

sort_query_params

boolean

yes|no

Specifies whether the Web crawler should sort the parameters in the query component of a URI.

Typically, query components are key-value pairs that are separated by semicolons or ampersands. When this configuration parameter is set, the query is sorted alphabetically by the key name.

Default: no

robots_timeout

integer

<seconds>

Specifies the maximum number of seconds that the Web crawler can use to download a robots.txt file.

Default: 300

login_timeout

integer

<seconds>

Specifies the maximum number of seconds that the Web crawler can use for a login request.

Default: 300

send_links_to

string

Specifies a crawl collection name to which all extracted hyperlinks are sent.

cookie_timeout

integer

<seconds>

Specifies the maximum number of seconds a session cookie is stored. A session cookie is a cookie that has no expiration date.

Default: 300

refresh_when_idle

boolean

yes|no

Specifies whether the Web crawler should trigger a new crawl refresh cycle when it becomes idle. This option should not be used in a multi-node installation.

Default: no

refresh_mode

string

append|prepend|scratch|soft|adaptive

Specifies the refresh mode of a crawl collection. Valid values are as follows:

  • append: Add the start URIs to the end of the crawl queue when a crawl refresh cycle begins. Existing queues are retained.

  • prepend: Add the start URIs to the beginning of the crawl queue when a crawl refresh cycle begins. Existing queues are retained.

  • scratch: Erase the crawl queue before appending the start URIs to the queue.

  • soft: If the crawl queue for a Web site is not empty at the end of a crawl refresh cycle, the Web crawler continues crawling into the next crawl refresh cycle. A crawl site is not refreshed until the crawl queue is empty.

  • adaptive: Build crawl queue according to the adaptive configuration.

Default: scratch

Examples

Copy Code
<attrib name="delay" type="real"> 60.0 </attrib>
Copy Code
<attrib name="max_doc" type="integer"> 10000 </attrib>
Copy Code
<attrib name="use_javascript" type="boolean"> no </attrib>
Copy Code
<attrib name="info" type="string">
My Web crawl collection crawling my intranet.
</attrib>
Copy Code
<attrib name="allowed_schemes" type="list-string">
    <member> http </member>
    <member> https </member>
</attrib>

member

This specifies an element in a list of option values.

The member element can only be used inside an attrib element.

Example

Copy Code
<attrib name="allowed_schemes" type="list-string">
    <member> http </member>
    <member> https </member>
</attrib>

section

This element groups a set of related options. A section element contains attrib elements.

Attributes

Attribute | Value | Description
name | <name> | Specifies the name of the section. Supported sections are described in this article.

Example

Copy Code
<section name="crawlmode">
    <attrib name="fwdlinks" type="boolean"> no </attrib>
    <attrib name="fwdredirects" type="boolean"> no </attrib>
    <attrib name="mode" type="string"> FULL </attrib>
    <attrib name="reset_level" type="boolean"> no </attrib>
</section>

include_domains

This section is a set of host name filters that specify which URIs to include in a crawl collection. An empty section matches any host name.

Attributes

The following table specifies attrib elements for this section.

Name Type Value Meaning

exact

list-string

Specifies a list of host names. If the host name of a URI exactly matches one of these host names, the URI is included by this rule.

prefix

list-string

Specifies a list of host names. If the host name of a URI begins with one of these host names, the URI is included by this rule.

suffix

list-string

Specifies a list of host names. If the host name of a URI ends with one of these host names, the URI is included by this rule.

regexp

list-string

Specifies a list of regular expressions. If the host name of a URI matches one of these regular expressions, the URI is included by this rule.

ipmask

list-string

Specifies a list of IPv4 address masks. If the IPv4 address of a retrieved URI matches one of these IPv4 address masks, the URI is include by this rule. An IPv4 address mask must follow one of the following formats:

  • A range of IPv4 address can be specified by writing an IPv4 address in string format and using a hyphen for the range. For example: 207.46.197.0-100 or 207.46.190-197.100

    If an IPv4 address is within this range, it is included by this mask.

  • An IPv4 mask can also be specified by examining the N most significant bits of an IPv4 address, where N is within the range of {0, 32}.

    The mask is an IPv4 address in string format followed by a forward slash and the number of most significant bits For example: 207.46.197.0 /24

    If an IPv4 address has the same N bits of the specified IPv4 address, it is included by this mask.

  • An IPv4 mask can also be specified by using a bitmask to mask out important bits of an IPv4 address.

    The format of this mask is IPv4 address in string format:ip-mask, where ip-mask is an IPv4 address in string format that is used for masking or a 32 bit hexadecimal digit. For example: 207.46.197.0:255.255.255.0 or 207.46.197.0:0xffffff00

    If an IPv4 address has the same bits set as specified by the ip-mask and the IPv4 address, it is included by this mask.

ip6mask

list-string

Specifies a list of IPv6 address masks. If the IPv6 address of a retrieved URI matches one of these IPv6 address masks, the URI is included by this rule.

An IPv6 address mask must follow one of the following formats:

  • A range of IPv6 address can be specified by writing an IPv6 address in string format and using a hyphen for the range. For example: 2002:CF2E:C500- C564:0:0:0:0:0 or ::ffff:207.46.197.0-100

    If an IPv6 address is within this range it is included by this mask.

  • An IPv6 mask can also be specified by looking at the N most significant bits of an IPv6 address, where N has the range of {0, 128}.

    This mask is an IPv6 address in string format followed by a forward slash and the number of most significant bits. For example: 2002:CF2E:C500:0:0:0:0:0/60

    If an IPv6 address has the same N bits of the specified IPv6 address, it is included by this mask.

Example

Copy Code
<section name="include_domains">
   <attrib name="exact" type="list-string">
      <member> www.contoso.com </member>
      <member> www2.contoso.com </member>
   </attrib>
   <attrib name="prefix" type="list-string">
      <member> www </member>
   </attrib>
   <attrib name="suffix" type="list-string">
      <member> .contoso.com</member>
      <member> .contoso2.com</member>
   </attrib>
   <attrib name="regexp" type="list-string">
      <member> .*\.contoso\.com </member>
   </attrib>
   <attrib name="file" type="list-string">
       <member> c:\myinclude_domains.txt </member>
   </attrib>
</section>

exclude_domains

This section is a set of host name filters that specify which URIs to exclude from a crawl collection. An empty section will not match any host name.

Attributes

See the table in include_domains for the attrib elements for this section.

Example

Copy Code
<section name="exclude_domains">
   <attrib name="exact" type="list-string">
      <member> www.contoso.com </member>
      <member> www2.contoso.com </member>
   </attrib>
   <attrib name="prefix" type="list-string">
      <member> www </member>
   </attrib>
   <attrib name="suffix" type="list-string">
      <member> .contoso.com</member>
      <member> .contoso2.com</member>
   </attrib>
   <attrib name="regexp" type="list-string">
      <member> .*\.contoso\.com </member>
   </attrib>
   <attrib name="file" type="list-string">
       <member> c:\myexclude_domains.txt </member>
   </attrib>
</section>

include_uris

This section is a set of URI-based rules that specify which URIs to include in a crawl collection. An empty section will match all URIs.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
exact | list-string |  | Specifies a list of URIs. If a URI exactly matches one of these URIs, the URI is included by this rule.
prefix | list-string |  | Specifies a list of strings. If a URI begins with one of these strings, the URI is included by this rule.
suffix | list-string |  | Specifies a list of strings. If a URI ends with one of these strings, the URI is included by this rule.
regexp | list-string |  | Specifies a list of regular expressions. If a URI matches one of these regular expressions, the URI is included by this rule.

Example

Copy Code
<section name="include_uris">
   <attrib name="exact" type="list-string">
      <member> http://www.contoso.com/documents/doc2.html </member>
   </attrib>
   <attrib name="prefix" type="list-string">
      <member> http://www.contoso.com/documents/ </member>
   </attrib>
   <attrib name="suffix" type="list-string">
      <member> /doc2.html </member>
   </attrib>
   <attrib name="regexp" type="list-string">
      <member> http://.*\.contoso\.com/documents.*</member>
   </attrib>
   <attrib name="file" type="list-string">
       <member> c:\myinclude_uris.txt </member>
   </attrib>
</section>

exclude_uris

This section is a set of URI-based rules that specify which URIs to exclude from a crawl collection. An empty section will not match any URIs.

Attributes

See the table in include_uris for the attrib elements for this section.

Example

Copy Code
<section name="exclude_uris">
   <attrib name="exact" type="list-string">
      <member> http://www.contoso.com/documents/doc2.html </member>
   </attrib>
   <attrib name="prefix" type="list-string">
      <member> http://www.contoso.com/documents/ </member>
   </attrib>
   <attrib name="suffix" type="list-string">
      <member> /doc2.html </member>
   </attrib>
   <attrib name="regexp" type="list-string">
      <member> http://.*\.contoso\.com/documents.*</member>
   </attrib>
   <attrib name="file" type="list-string">
       <member> c:\myexclude_uris.txt </member>
   </attrib>
</section>

log

This section specifies logging behavior for the Web crawler process.

Attributes

The following table specifies attrib elements for this section.

Name Type Value Meaning

fetch

string

text|none

Enable/disable logging of downloaded Web items. Valid values are as follows:

  • text: This creates a text-formatted log.

  • none: This disables logging.

Default: text

postprocess

string

text|xml|none

Enable/disable logging of node scheduler item post processing. Valid values are as follows:

  • text: This creates a text-formatted log.

  • xml: This creates an XML formatted log.

  • none: This disables logging.

Default: text

header

string

text|none

Enable/disable logging of HTTP headers. Valid values are as follows:

  • text: This creates a text-formatted log.

  • none: This disables logging.

screened

string

text|none

Enable/disable logging of all screened URIs. Valid values are as follows:

  • text: This creates a text-formatted log.

  • none: This disables logging.

scheduler

string

text|none

Enable/disable logging of adaptive crawling. Valid values are as follows:

  • text: This creates a text-formatted log.

  • none: This disables logging.

dsfeed

string

text|none

Enable/disable the logging of content submission to the indexing engine. Valid values are as follows:

  • text: This creates a text-formatted log.

  • none: This disables logging.

site

string

text|none

Enable/disable logging per crawl site. Valid values are as follows:

  • text: This creates a text-formatted log.

  • none: This disables logging.

Example

Copy Code
<section name="log">
   <attrib name="dsfeed" type="string"> text </attrib>
   <attrib name="fetch" type="string"> text </attrib>
   <attrib name="postprocess" type="string"> text </attrib>
   <attrib name="screened" type="string"> none </attrib>
   <attrib name="site" type="string"> text </attrib>
</section>

storage

This section specifies how the Web crawler stores data and metadata.

Attributes

The following table specifies attrib elements for this section.

Name Type Value Meaning

datastore

string

flatfile|bstore

Specifies the format for Web item content storage. Valid values are as follows:

  • flatfile: This format stores items directly into the file system.

  • bstore: This format partitions items into fixed sized blocks and distributes them across a set of files. An index maps the order of the blocks, and specifies which blocks belong to an item.

Default: bstore

store_http_header

boolean

yes|no

Specifies that the Web crawler should store the received HTTP header.

Default: yes

store_dupes

boolean

yes|no

Specifies that the Web crawler should store duplicate Web items.

Default: no

compress

boolean

yes|no

Specifies that downloaded items should be compressed before storing them.

Default: yes

compress_exclude_mime

list-string

Specifies a set of MIME types of Web items that should not be compressed when stored. Use for Web items that are already compressed, e.g. multimedia formats.

If the compress configuration parameter is not set, this parameter is not applicable.

remove_docs

boolean

yes|no

Specifies that the Web crawler should delete Web items from the Web crawler store as soon as they are submitted to the indexing engine. This will reduce disk space requirements for the Web crawler, but will make it impossible to refeed.

Default: no

clusters

integer

<value>

Specifies the number of clusters to use for storage in a crawl collection. Web items are distributed among these storage clusters.

Default: 8

defrag_threshold

integer

<percentage>

A non-zero value that specifies the threshold value (of used capacity) before defragmenting a data storage file. When the used space is less than thedefrag_threshold, the file is eligible for defragmentation to reclaim fragmented space caused by stored Web items. Database files are compacted regardless of fragmentation level.

The default of 85% means there must be 15% reclaimable space in the data storage file to trigger defragmentation.

A value of 0 disables defragmentation.

This setting is only applicable to the bstore value for the storage_format attribute.

Default: 85

uri_dir

string

<path>

Specifies a path for storing file lists of all hyperlinks that are extracted from Web items. Each site manager process uses a separate file. The name of a URI file is constructed by concatenating the process PID with .txt.

Example

Copy Code
<section name="storage">
   <attrib name="store_dupes" type="boolean"> no </attrib>
   <attrib name="datastore" type="string"> bstore </attrib>
   <attrib name="compress" type="boolean"> yes </attrib>
</section>

pp

This section specifies the post processing behavior for a node scheduler. Post processing consists of two primary tasks: feeding Web items to the index, and performing duplicate detection.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
use_dupservers | boolean | yes|no | Specifies that the Web crawler should use one or more duplicate servers. This option is applicable only in a multi-node installation. Default: no
max_dupes | integer | <value> | Specifies the maximum number of duplicates to record per Web item. Default: 10
stripe | integer | <value> | Specifies the number of data files to distribute the checksum data into. Increasing this value may improve the performance of post processing. Default: 1
ds_meta_info | list-string | duplicates|redirects|mirrors|metadata | Specifies the kind of metadata a node scheduler should report to the indexing engine. Valid values are as follows: duplicates: Reports URIs that are duplicates of this item redirects: Reports URIs that are redirected to this item metadata: Reports meta data of this item mirrors: Reports all mirror URIs of this Web item
ds_max_ecl | integer | <value> | Specifies the maximum number of duplicates or redirects to report to the indexing engine, as specified by the ds_meta_info configuration parameter. Default: 10
ecl_override | string |  | Specifies a regular expression that identifies redirect and duplicate URIs that should be stored and possibly submitted to the indexing engine, even though max_dupes is reached. For example: .*index\.html$
ds_send_links | boolean | yes|no | Specifies whether all extracted hyperlinks from a Web item should be sent to the indexing engine.
ds_paused | boolean | yes|no | Specifies whether a node scheduler should suspend the submission of content to the indexing engine.

Example

Copy Code
<section name="pp">
   <attrib name="max_dupes" type="integer"> 10 </attrib>
   <attrib name="use_dupservers" type="boolean"> yes </attrib>
   <attrib name="ds_paused" type="boolean"> no </attrib>
</section>

ppdup

This section specifies duplicate server settings.

Attributes

The following table specifies attrib elements for this section.

Name Type Value Meaning

format

string

gigabase|hashlog|diskhashlog

Specifies the duplicate server database format. Valid values are as follows:

  • gigabase: Gigabase is a simple key-value database.

  • hashlog: Hashlog is an in-memory data structure consisting of a hash table and a data log. The data log contains all stored keys and values, and can automatically rebuild the in-memory hash table if it is needed.

  • diskhashlog: Diskhashlog is the same as hashlog except that the data structure is accessed directly on disk.

cachesize

integer

<megabytes>

Specifies the duplicate server database cache size in megabytes. If the format configuration parameter is set to hashlog or diskhashlog this parameter specifies the initial size of the hash table.

stripes

integer

<value>

Specifies the number of data files to spread content to. By using multiple files, you can improve the performance of the duplicate server database.

compact

boolean

yes|no

Specifies whether the duplicate server database should perform compaction. For the hashlog and diskhashlog formats, compaction must be performed either manually with the crawlerdbtool or automatically by enabling this option. Otherwise, disk usage will increase for every record written or updated.

Default: yes

Example

Copy Code
<section name="ppdup">
   <attrib name="format" type="string"> hashlog </attrib>
   <attrib name="stripes" type="integer"> 1 </attrib>
      <!-- 1 GB memory hash -->
   <attrib name="cachesize" type="integer"> 1024 </attrib>
   <attrib name="compact" type="boolean"> yes </attrib>
</section>

feeding

The feeding section consists of at least one section XML element that specifies how to send a representation of the crawl collection to the indexing engine. Such a section defines a content destination. The name attribute specifies a unique name for the content destination.

Attributes

The following table specifies attrib elements for a content destination section.

Name | Type | Value | Meaning
collection | string | <name> | Specifies the name of the content collection for submitting Web items. This configuration parameter must be specified in a feeding section.
destination | string | default | Reserved. This configuration parameter must contain the value default.
paused | boolean | yes|no | Specifies whether the Web crawler should suspend the submission of content to the indexing engine. Default: no
primary | boolean | yes|no | Specifies whether this content destination is a primary or secondary content destination. A primary content destination can act on callback information during content submission to the indexing engine. If only one content destination is specified, it will be a primary destination.

Example

Copy Code
<section name="feeding">
    <section name="Global_News">
        <attrib name="collection" type="string"> collection_A </attrib>
        <attrib name="destination" type="string"> default </attrib>
        <attrib name="primary" type="boolean"> yes </attrib>
        <attrib name="paused" type="boolean"> no </attrib>
    </section>
    <section name="Local_News">
        <attrib name="collection" type="string"> collection_B </attrib>
        <attrib name="destination" type="string"> default </attrib>
        <attrib name="primary" type="boolean"> no </attrib>
        <attrib name="paused" type="boolean"> no </attrib>
     </section>
</section>

cachesize

This section configures the cache sizes for the Web crawler process.

Attributes

The following table specifies attrib elements for this section.

Note:

The default value for each attribute, if one is not specified in the table, is to have the Web crawler automatically determine the cache size at run time.

Name | Type | Value | Meaning
duplicates | integer | <value that represents a number of items> | Specifies the size of the duplicate checksum cache, per site manager process. This cache is used as a first level of duplicate detection at run time.
screened | integer | <value that represents a number of items> | Specifies the size of the screened URI cache, as the number of hyperlinks. The screened cache filters out duplicate hyperlinks that recently resulted in retrieval failures.
smcomm | integer | <value that represents a number of items> | Specifies the size of the bloom filter that is used by the cache filtering out duplicate hyperlinks flowing between the node scheduler and site managers.
mucomm | integer | <value that represents a number of items> | Specifies the size of the bloom filter that is used by the cache filtering out duplicate hyperlinks flowing between the multi-node scheduler and node scheduler.
wqcache | integer | <value that represents a number of items> | Specifies the size of the cache filtering out duplicate hyperlinks from the Web site crawl queues.
crosslinks | integer | <value that represents a number of items> | Specifies the size of the crosslink cache. The crosslink cache contains retrieved hyperlinks and referring hyperlinks. It filters out duplicate hyperlinks in the node scheduler if mufilter is not enabled.
routetab | integer | <value> | Specifies the crawl routing database cache size, in bytes. Default: 1048576
pp | integer | <value> | Specifies the post process database cache size, in bytes. Default: 1048576
pp_pending | integer | <value> | Specifies the post process pending cache size, in bytes. The pending cache contains entries that were not sent to the duplicate servers. Default: 131072
aliases | integer | <value> | Specifies the aliases data mapping database cache size, in bytes. A crawl site can be associated with one or more aliases (alternative host names). Default: 1048576

Example

Copy Code
<section name="cachesize">
      <!-- Specific cache size values (in number of items) for the following: -->
      <attrib name="duplicates" type="integer"> 128 </attrib>
      <attrib name="screened" type="integer"> 128 </attrib>
      <attrib name="smcomm" type="integer"> 128 </attrib>
      <attrib name="mucomm" type="integer"> 128 </attrib>
      <attrib name="wqcache" type="integer"> 4096 </attrib>
      <!-- Automatic cache size for crosslinks -->
      <attrib name="crosslinks" type="integer"> </attrib>
      <!-- Cache sizes in bytes for the following -->
      <attrib name="routetab" type="integer"> 1048576 </attrib>
      <attrib name="pp" type="integer"> 1048576 </attrib>
      <attrib name="pp_pending" type="integer"> 1048576 </attrib>
      <attrib name="aliases" type="integer"> 1048576 </attrib>
   </section>

http_errors

This section specifies how to handle HTTP/HTTPS error response codes and conditions.

Attributes

The following table specifies attrib elements for this section. Because there are multiple values for the name attribute, a description of each purpose is included in the name column.

Name

Type

Value

Meaning

The name attribute specifies the HTTP/HTTPS/FTP response code number to handle. The character "X" can be used as a wildcard. For example: 4XX

Other valid values are as follows:

  • net: used to handle network socket errors

  • int: used to handle internal errors in the Web crawler

  • ttl: used to handle HTTP/HTTPS/FTP connection time-outs

string

<value>

Specifies how the Web crawler handles HTTP/HTTPS/FTP and network errors. Valid options for handling individual response codes are as follows:

  • KEEP: Keep the Web item unchanged

  • DELETE[:X]: Delete the Web item if the error condition occurs for the Xth time. Deletion happens immediately if no X value is specified.

If RETRY[:X] is specified for either of these options, the Web crawler will re-download the Web item no more than X times in the same crawl refresh cycle period before failing the attempt. Otherwise, the crawler will not try to download the URI until the next crawl refresh cycle.

Default: See Default values for the http_errors section and Default values for the ftp_errors section.

Default values for the http_errors section

The following table specifies the default values for the http_errors section.

Name | Value | Meaning
4xx | DELETE:0 | Delete immediately.
5xx | DELETE:10 | Delete the tenth time this error is encountered for this URI, usually after 10 crawl cycles. The counter is reset if the URI is successfully retrieved.
int | KEEP:0 | Do not delete.
net | DELETE:3, RETRY:1 | Delete the third time. One retry is specified. This means that the URI will be deleted on the next refresh cycle if it still cannot be retrieved.
ttl | DELETE:3 | Delete the third time.

Example

Copy Code
<section name="http_errors">
    <attrib name="408" type="string"> KEEP </attrib>
    <attrib name="4xx" type="string"> DELETE </attrib>
    <attrib name="5xx" type="string"> DELETE:10, RETRY:3 </attrib>
    <attrib name="ttl" type="string"> DELETE:3 </attrib>
    <attrib name="net" type="string"> DELETE:3 </attrib>
    <attrib name="int" type="string"> KEEP </attrib>
</section>

ftp_errors

This section specifies how to handle response codes and error conditions for FTP URIs.

Attributes

See the table in http_errors for the attrib elements for this section.

Default values for the ftp_errors section

The following table specifies the default values for the ftp_errors section.

Name | Value | Meaning
4xx | DELETE:3 | Delete the third time that this error is encountered for this URI, usually after 3 crawl cycles. The counter is reset if the URI is successfully retrieved.
550 | DELETE:0 | Delete immediately.
5xx | DELETE:3 | Delete the third time, same as for 4xx.
int | KEEP:0 | Do not delete.
net | DELETE:3, RETRY:1 | Delete the third time. One retry is specified. This means that the URI will be deleted on the next refresh cycle if it still cannot be retrieved.

Example

Copy Code
<section name="ftp_errors">
    <attrib name="4xx" type="string"> DELETE:3 </attrib>
    <attrib name="550" type="string"> DELETE:0 </attrib>
    <attrib name="5xx" type="string"> DELETE:3 </attrib>
    <attrib name="int" type="string"> KEEP:0 </attrib>
    <attrib name="net" type="string"> DELETE:3, RETRY:1 </attrib>
    <attrib name="ttl" type="string"> DELETE:3 </attrib>
</section>

workqueue_priority

This section specifies the priority levels for the crawl queues, and specifies the rules and modes used to insert URIs into and extract URIs from the queues.

Attributes

The following table specifies attrib elements for this section.

Name Type Value Meaning

levels

integer

<value>

Specifies the number of priority levels used for the crawl queues.

Default: 1

default

integer

<value>

Specifies a default priority level that is assigned to URIs in a crawl queue.

Default: 1

start_uri_pri

integer

<value>

Specifies the priority level for start URIs. See the start_uris and the start_uri_files configuration parameters.

Default: 1

pop_scheme

string

default|rr|wrr|pri

Specifies the mode used by the Web crawler to extract URIs from the crawl queue. Valid values are as follows:

  • rr: This mode extracts URIs from the priority levels in round-robin order.

  • wrr: This mode extracts URIs from the priority levels in a weighted round-robin order. The weights are based on their respective share setting per priority level, as specified in Priority level section.

  • pri: This mode extracts URIs from the priority levels in priority order by when entries still remain in the crawl queue. 1 is the highest priority, as specified in Priority level section.

  • default: This mode is the same as wrr.

Default: default

put_scheme

string

default|include

Specifies which Web crawler mode to use when you insert URIs into the crawl queue. Valid values are as follows:

  • default: This mode always inserts URIs with the priority level specified in the default configuration parameter.

  • include: This mode inserts URIs with the priority level of include_domains or include_uris, as specified in Priority level section for every priority level. The Web crawler process assigns the default priority level when a URI does not match any of these sections.

Default: default

Priority level section

Within the workqueue_priority section, a set of sections, which specify priority levels and weights of the crawler queues, can be specified. These sections will only be used if the pop_scheme parameter is set to wrr or pri. The name attribute of these sections must be the priority level to be specified. The priority levels must begin at 1. (See <section name="1"> in the following example.)

The include_domains or include_uris section can be used within each priority level section, as specified in include_domains and include_uris. URIs that match these rules will be queued using the matching priority level. In addition, the following table specifies attrib elements for these sections.

Name | Type | Value | Meaning
share | integer |  | Specifies a weight to use for each crawl queue. This weight will only be used if the pop_scheme configuration parameter is set to wrr.

Example

Copy Code
<section name="workqueue_priority">
    <attrib name="levels" type="integer"> 2 </attrib>
    <attrib name="default" type="integer"> 2 </attrib>
    <attrib name="start_uri_pri" type="integer"> 1 </attrib>
    <attrib name="pop_scheme" type="string"> wrr </attrib>
    <attrib name="put_scheme" type="string"> include </attrib>
    <section name="1">
        <attrib name="share" type="integer"> 10 </attrib>
        <section name="include_domains">
            <attrib name="suffix" type="list-string">
                <member> web005.contoso.com  </member>
            </attrib>
        </section>
    </section>
    <section name="2">
        <attrib name="share" type="integer"> 5 </attrib>
        <section name="include_domains">
           <attrib name="suffix" type="list-string">
              <member> web002.contoso.com  </member>
           </attrib>
        </section>
    </section>
</section>

link_extraction

This section specifies which kind of hyperlinks to follow.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
a | boolean | yes|no | Extracts hyperlinks from <A/> HTML tags. Default: yes
action | boolean | yes|no | Extracts hyperlinks from action attributes in HTML tags. Default: yes
area | boolean | yes|no | Extracts hyperlinks from <AREA/> HTML tags. Default: yes
card | boolean | yes|no | Extracts hyperlinks from the <CARD/> Wireless Markup Language tags. Default: yes
comment | boolean | yes|no | Extracts hyperlinks from comments in a Web item. Default: yes
embed | boolean | yes|no | Extracts hyperlinks from <EMBED/> HTML tags. Default: yes
frame | boolean | yes|no | Extracts hyperlinks from <FRAME/> HTML tags. Default: yes
go | boolean | yes|no | Extracts hyperlinks from <GO/> Wireless Markup Language tags. Default: yes
img | boolean | yes|no | Extracts hyperlinks from <IMG/> HTML tags. Default: no
layer | boolean | yes|no | Extracts hyperlinks from <LAYER/> HTML tags. Default: yes
link | boolean | yes|no | Extracts hyperlinks from <LINK/> HTML tags. Default: yes
meta | boolean | yes|no | Extracts hyperlinks from <META/> HTML tags. Default: yes
meta_refresh | boolean | yes|no | Extracts hyperlinks from meta refresh HTML tags (<meta http-equiv="refresh" content="n" />). Default: yes
object | boolean | yes|no | Extracts hyperlinks from <OBJECT/> HTML tags. Default: yes
script | boolean | yes|no | Extracts hyperlinks from <SCRIPT/> HTML tags. Default: yes
script_java | boolean | yes|no | Extracts hyperlinks from <SCRIPT/> HTML tags that contain JavaScript. Default: yes
style | boolean | yes|no | Extracts hyperlinks from <STYLE/> HTML tags. Default: yes

Example

Copy Code
<section name="link_extraction">
   <attrib name="action" type="boolean"> yes </attrib>
   <attrib name="img" type="boolean"> no </attrib>
   <attrib name="link" type="boolean"> yes </attrib>
   <attrib name="meta" type="boolean"> yes </attrib>
   <attrib name="meta_refresh" type="boolean"> yes </attrib>
   <attrib name="object" type="boolean"> yes </attrib>
   <attrib name="script_java" type="boolean"> yes </attrib>
</section>

limits

The limits section specifies fail-safe limits for a crawl collection. When the collection exceeds the limit, it enters a "refresh only" crawl mode. This means that only previously-crawled URIs are crawled again.

Attributes

The following table specifies attrib elements for this section.

Name Type Value Meaning

disk_free

integer

<percentage>

Specifies the percentage of free disk space that must be available for the Web crawler to operate in normal crawl mode (specified in the crawlmode attribute). If the percentage becomes less than this limit, the Web crawler enters the "refresh only" crawl mode (when thresholds are reached).

If the parameter is set to 0, this feature is disabled.

Default: 0

disk_free_slack

integer

<percentage>

Specifies slack for the disk_free threshold, as a percentage.

This option creates a buffer zone around the disk_freethreshold. When the free disk space is within this buffer, the Web crawler will not change the crawl mode back to normal. This prevents the Web crawler from switching back and forth between crawl modes when the percentage of free disk space is close to the value specified by the disk_free parameter. When the freedisk space percentage exceeds disk_free + disk_free_slack, normal crawling resumes.

Default: 3

max_doc

integer

<value>

Specifies the number of stored Web items that will cause the crawler to enter "refresh" crawl mode.

Note:

The threshold is not an exact limit, because statistical reporting is somewhat delayed compared to crawling.

When set to 0, this feature is disabled.

Default: 0

max_doc_slack

integer

<value>

To avoid constant changes with the crawler entering and exiting "refresh only" crawl mode, you can specify athreshold range in addition to the absolute threshold value. The range is defined as: (threshold minus slack), to (threshold), where crawl mode behavior remains unchanged. The max_doc_slack attribute specifies the maximum number of items that can be contained in a slack, up to the max_doc configuration parameter threshold.

Default: 1000

Example

Copy Code
<section name="limits">
   <attrib name="disk_free" type="integer"> 0 </attrib>
   <attrib name="disk_free_slack" type="integer"> 3 </attrib>
   <attrib name="max_doc" type="integer"> 0 </attrib>
   <attrib name="max_doc_slack" type="integer"> 1000 </attrib>
</section>

focused

This section configures focused scheduling. An exclude_domains section can be used within the focused section to exclude host names from this focused scheduling. If no exclude_domains section is defined, all host names are included in the focused scheduling.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
languages | list-string |  | Lists the languages for items that can be stored by the Web crawler, as specified in ISO-639-1.
depth | integer | <value> | Specifies the number of page hops to follow for Web items that do not match the specified languages, as set by the languages configuration parameter.

Example

In the following example, the crawler will store all items with Norwegian, English, or unknown language content. For all non-specified languages, the crawler will only follow links to 2 levels. In addition, all content under contoso.com is excluded from the language checks and is automatically stored.

Copy Code
<section name="focused">
   <!-- Crawl Norwegian, English and content of unknown language -->
   <attrib name="languages" type="list-string">
      <member> norwegian </member>
      <member> unknown </member>
      <member> en </member>
   </attrib>
   <!--Follow hyperlinks containing other languages for 2 levels -->
   <attrib name="depth" type="integer"> 2 </attrib>
   <!-- Exclude anything under .contoso.com from language checks, -->
   <section name="exclude_domains">
      <attrib name="suffix" type="list-string">
         <member> .contoso.com </member>
      </attrib>
   </section>
</section>

passwd

This section configures credentials for Web sites that require authentication. The Web crawler supports basic authentication, digest authentication, and NTLM authentication.

Attributes

The following table specifies attrib elements for this section.

Name Type Value Meaning

name

string

The name attribute must contain a URI or realm. A valid URI behaves as a prefix value, because all hyperlinks extracted at its level or deeper use these authentication settings.

The credentials must be specified in one of the following formats: username:password or usename:password:realm:scheme.

The password component of the credential string can be encrypted; if it is not encrypted, it is given in plaintext.

An encrypted password is created by using the crawleradmin tool with the -e option. The encryption algorithm that is used is Advanced Encryption Standard AES-128 with the key found in <FASTSearchFolder>\etc\CrawlerEncryptionKey.dat.

If the credentials are given using the username:password format, the Web crawler automatically uses basic access authentication. Otherwise the configuration must specify an authentication scheme. Valid authentication schemes are as follows:

  • basic

  • digest

  • ntlmv1

  • ntlmv2

  • auto: Specifies that the Web crawler determines the authentication scheme by itself.

Example

Copy Code
<section name="passwd">
    <attrib name="http://www.contoso.com/confidential1/" type="string">
      user:password:contoso:auto
    </attrib>
</section>

ftp_acct

This section specifies FTP accounts for crawling FTP URIs.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
name | string | The value of the name XML attribute is the host name for which this FTP account is valid. | This is the user name and password for this FTP account. The string must be in the format: username:password

Example

Copy Code
<section name="ftp_acct">
   <attrib name="ftp.contoso.com" type="string"> user:pass </attrib>
</section>

exclude_headers

This section is used to exclude Web items from the crawl, based on the contents of the HTTP header fields.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
name The name attribute is used to set the name of the HTTP header to test. | list-string |  | Specifies a list of regular expressions. If the value of the specified HTTP header matches one of these regular expressions, the Web item is excluded from the crawl.

Example

Copy Code
<section name="exclude_headers">
   <attrib name="Header Name" type="list-string">
      <member> .*excluded.*value </member>
   </attrib>
</section>

variable_delay

This section specifies time slots that use a different request rate. When no time slot is specified, the crawler uses the delay configuration parameter as specified in attrib.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
name in the format: DDD:HH.MM-DDD:HH.MM | string | <value in seconds> suspend | Specifies the delay request rate for this time slot, in seconds. A value of suspend specifies that crawling of this crawl collection will be suspended.

Example

The following example shows how the Web crawler uses different delay intervals during the week. On Wednesday between 9:00 a.m. and 7:00 p.m. the Web crawler uses a delay of 20 seconds. On Monday between 9:00 a.m. and 5:00 p.m. the crawler suspends crawling, and any other time of the week the Web crawler uses a delay of 60 seconds.

Copy Code
<?xml version="1.0" encoding="utf-8"?>
<CrawlerConfig>
   <DomainSpecification name="variable_example">
      <section name="variable_delay">
         <attrib name="Wed:09-Wed:19" type="string">20 </attrib>
         <attrib name="Mon:09-Mon:17" type="string">suspend</attrib>
      </section>
   </DomainSpecification>
</CrawlerConfig>

adaptive

This section specifies the adaptive crawling options. The refresh_mode configuration parameter, specified in attrib, must be set to adaptive for this section to be used by the Web crawler.

The adaptive crawling behavior can be controlled with the weights and the sitemap_weights sections.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
refresh_count | integer | <value> | Specifies the number of minor refresh cycles. A refresh cycle can be divided into several fixed size time intervals that are called minor refresh cycles. Default: 4
refresh_quota | integer | <percentage> | Specifies the ratio of existing re-crawled URIs to new unseen URIs, expressed as a percentage. Setting the percentage low gives preference to new URIs. Default: 90
coverage_min | integer | <value> | Specifies a minimum number of URIs to crawl per Web site in a minor refresh cycle. Used to guarantee some coverage for small Web sites. Default: 25
coverage_max_pct | integer | <value> | Specifies a percentage of a Web site to re-crawl in a minor cycle. Ensures that small Web sites are not fully crawled each minor cycle, taking time away from larger Web sites. Default: 10

Example

Copy Code
        <section name="adaptive">
            <attrib name="refresh_count" type="integer"> 4 </attrib>
            <attrib name="refresh_quota" type="integer"> 98 </attrib>
            <attrib name="coverage_max_pct" type="integer"> 25 </attrib>
            <attrib name="coverage_min" type="integer"> 10 </attrib>

            <!-- Ranking weights. Each scoring criteria adds a score between -->
            <!-- 0.0 and 1.0 which is then multiplied with the associated    -->
            <!-- weight below. Use a weight of 0 to disable a scorer         -->

           <section name="weights">
                <attrib name="inverse_length" type="real"> 1.0 </attrib>
                <attrib name="inverse_depth" type="real"> 1.0 </attrib>
                <attrib name="is_landing_page" type="real"> 1.0 </attrib>
                <attrib name="is_mime_markup" type="real"> 1.0 </attrib>
                <attrib name="change_history" type="real"> 10.0 </attrib>
            </section>
        </section>

weights

In this section, each URI is given a score in the adaptive crawling process. The score prioritizes URIs and is based on a set of rules. Each rule is assigned a weight that determines its contribution towards the total score that is specified in the weights section.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
inverse_length | real | <value> | Specifies the weight for the inverse length rule. The inverse length rule gives URIs with few path segments (defined by the number of forward slashes) a higher score. URIs with 10 or more slashes receive a score of 0. Default: 1.0
inverse_depth | real | <value> | Specifies the weight for the inverse depth rule. The number of page hops from a start URI is computed; a high score is assigned to URIs that have less than 10 page hops. The rule gives a score of zero for URIs with 10 or more page hops. Default: 1.0
is_landing_page | real | <value> | Specifies the weight for the is_landing_page rule. This rule gives a URI that is considered a landing page a higher score. A landing page is a URI ending in one of /, /index.html, index.htm, index.php, index.jsp, index.asp, default.html, or default.htm. The rule gives no score to URIs that have query components. Default: 1.0
is_mime_markup | real | <value> | Specifies the weight for the is_mime_markup rule. This rule gives an additional score to pages whose MIME type is specified in the uri_search_mime configuration parameter in attrib. Default: 1.0
change_history | real | <value> | Specifies the weight for the change history rule. This rule scores based on the HTTP header "last-modified" value over time. Web items that frequently change have a higher score than items that change less frequently. Default: 10.0
sitemap | real | <value> | Specifies the weight for the sitemap rule. The score for the sitemap rule is specified in sitemap_weights. Default: 10.0

Example

Copy Code
<!-- Ranking weights. Each scoring criteria adds a score between -->
            <!-- 0.0 and 1.0 which is then multiplied with the associated    -->
            <!-- weight below. Use a weight of 0 to disable a scorer         -->
            <section name="weights">
                <!-- Score based on the number of /'es (segments) in the -->
                <!-- URI. Max score with one, no score with 10 or more   -->
                <attrib name="inverse_length" type="real"> 1.0 </attrib>

                <!-- Score based on the number of link "levels" down to -->
                <!-- this URI. Max score with none, no score with >= 10 -->
                <attrib name="inverse_depth" type="real"> 1.0 </attrib>

                <!-- Score added if URI is determined as a "landing page", -->
                <!-- defined as e.g. ending in "/" or "index.html". URIs   -->
                <!-- with query parameters are not given score             -->
                <attrib name="is_landing_page" type="real"> 1.0 </attrib>

                <!-- Score added if URI points to a markup document as    -->
                <!-- defined by the "uri_search_mime" option. Assumption  -->
                <!-- being that such content changes more often than e.g. -->
                <!-- "static" Word or PDF documents.                      -->
                <attrib name="is_mime_markup" type="real"> 1.0 </attrib>

                <!-- Score based on change history tracked over time by   -->
                <!-- using an estimator based on last modified date given -->
                <!-- by the web server. If no modified date returned then -->
                <!-- one is estimated (based on whether the document has  -->
                <!-- changed or not).                                     -->
                <attrib name="change_history" type="real"> 10.0 </attrib>
            </section>

sitemap_weights

In this section, <URL> entries in a sitemap can contain a changefreq element, which specifies how frequently a URI can be modified.

Valid string values for this element are as follows: always, hourly, daily, weekly, monthly, yearly, and never. The string values are converted into a numeric weight for adaptive crawling. The sitemap_weights section specifies a mapping of the string values to a numeric weight. This numeric weight is used to calculate the score to the sitemap score in the weights section.

The adaptive crawling score for a URI is calculated by multiplying the numeric weight by the sitemap configuration parameter weight.

Attributes

The following table specifies attrib elements for this section.

Important:

The range of these elements must be from 0.0 to 1.0.

Name | Type | Value | Meaning
always | real | <value> | Specifies the weight of the changefreq value always as a numeric value. Default: 1.0
hourly | real | <value> | Specifies the weight of the changefreq value hourly as a numeric value. Default: 0.64
daily | real | <value> | Specifies the weight of the changefreq value daily as a numeric value. Default: 0.32
weekly | real | <value> | Specifies the weight of the changefreq value weekly as a numeric value. Default: 0.16
monthly | real | <value> | Specifies the weight of the changefreq value monthly as a numeric value. Default: 0.08
yearly | real | <value> | Specifies the weight of the changefreq value yearly as a numeric value. Default: 0.04
never | real | <value> | Specifies the weight of the changefreq value never as a numeric value. Default: 0.0
default | real | <value> | Specifies the weight for all URIs that are not associated with a <changefreq> value. Default: 0.16

Example

Copy Code
<section name="sitemap_weights">
    <attrib name="always" type="real"> 1.0 </attrib>
    <attrib name="hourly" type="real"> 0.64 </attrib>
    <attrib name="daily" type="real"> 0.32 </attrib>
    <attrib name="weekly" type="real"> 0.16 </attrib>
    <attrib name="monthly" type="real"> 0.08 </attrib>
    <attrib name="yearly" type="real"> 0.04 </attrib>
    <attrib name="never" type="real"> 0.0 </attrib>
    <attrib name="default" type="real"> 0.16 </attrib>
</section>

site_clusters

This section specifies configuration parameters that override the crawler's behavior of routing host names in a node scheduler. This parameter ensures that a group of host names is routed to the same node scheduler and site manager. This is useful when the use_cookies setting is enabled, because cookies are global only throughout a site manager process. Also, if you know certain Web sites are closely interlinked, you can reduce internal communication by clustering their host names.

Attributes

The following table specifies attrib elements for this section.

Name | Type | value | Meaning
name | list-string |  | Specifies a list of host names that should be aggregated to a node scheduler.

Example

Copy Code
<section name="site_clusters">
    <attrib name="mycluster" type="list-string">
        <member> host1.constoso.com </member>
        <member> host2.constoso.com </member>
        <member> host3.constoso.com </member>
    </attrib>
</section>

crawlmode

This section limits the span of a crawl collection.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
mode | string |  | Specifies the depth of the crawling. Valid values are FULL or DEPTH:#, where # is the number of page hops from a start URI. Default: FULL
fwdlinks | boolean | yes|no | Specifies whether to follow hyperlinks that point to a different host name. Default: yes
fwdredirects | boolean | yes|no | Specifies whether to follow external HTTP redirects received from servers. External redirects are HTTP redirects that point from one host name to another host name. Default: no
reset_level | boolean | yes|no | Specifies whether to reset the page hop counter use by mode when you follow a hyperlink to another host name. Default: yes

Example

Copy Code
        <section name="crawlmode">
            <attrib name="mode" type="string"> DEPTH:1 </attrib>
            <attrib name="fwdlinks" type="boolean"> yes </attrib>
            <attrib name="fwdredirects" type="boolean"> yes </attrib>
            <attrib name="reset_level" type="boolean"> no </attrib>
        </section>

post_payload

This section is used to submit content to HTTP POST requests. The content is submitted to URIs that match an URI prefix or that exactly match a URI.

Attributes

The following table specifies attrib elements for this section.

Name

Type

Value

Meaning

name

string

Specifies the payload content string. This string is posted to URIs that matches a URI or prefix set by the name XML attribute.

The section requires a match if the name attribute specifies a URI.

To specify a URI prefix, the label prefix: must be used. Then the leading part of a URI specifies the rest of the match.

Example

Copy Code
<section name="post_payload">
    <attrib name="prefix:http://www.contoso.com/secure" type="string"> variable1=value1&amp;variableB=valueB </attrib>
</section>

rss

This section initializes and configures RSS feed support in a crawl collection.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
start_uris | list-string |  | Specifies a list of start URIs that point to RSS feed items.
start_uri_files | list-string |  | Specifies a list of paths to files that contain URIs that point to RSS feed items. The format of these files must be plain text files that have one URI per line.
auto_discover | boolean | yes|no | Specifies whether the Web crawler should discover new RSS feeds. If this option is not set, only feeds specified in the RSS start URIs and the RSS start URIs files sections will be treated as RSS feeds. Default: no
follow_links | boolean | yes|no | Specifies that the Web crawler should follow hyperlinks from Web items found in the RSS feed, which is the usual Web crawler behavior. If disabled, crawling happens only one hop away from a feed. Disable this option to only crawl feeds and Web items referenced by feeds. Default: yes
ignore_rules | boolean | yes|no | Specifies that the Web crawler should crawl all Web items referenced by the RSS feed, regardless of their inclusion in the include/exclude rules, as specified in include_domains, exclude_domains, include_uris, and exclude_uris. Default: no
index_feed | boolean | yes|no | Specifies whether the Web crawler should send RSS feeds themselves to the indexing engine, or only the Web items hyperlinked within the feeds. Default: no
del_expired_links | boolean | yes|no | Specifies whether the Web crawler should delete items from the RSS feed when they expire, as defined by max_link_age and max_link_count. Default: no
max_link_age | integer | <value> | Specifies the maximum age, in minutes, for a Web item found in an RSS feed. Only applies if the del_expired_links configuration parameter is set to yes. Default: 0
max_link_count | integer | <value> | Specifies the maximum number of hyperlinks the Web crawler saves for an RSS feed. If the Web crawler encounters more hyperlinks, they expire in a first-in-first-out order. Only applies if del_expired_links configuration parameter is set to yes. Default: 128

Example

Copy Code
        <section name="rss">
            <!-- Attempt to discover new rss feeds, yes/no                  -->
            <attrib name="auto_discover" type="boolean"> yes </attrib>
            <attrib name="del_expired_links" type="boolean"> yes </attrib>
            <attrib name="follow_links" type="boolean"> yes </attrib>
            <attrib name="ignore_rules" type="boolean"> no </attrib>
            <attrib name="index_feed" type="boolean"> no </attrib>
            <attrib name="max_link_age" type="integer"> 0 </attrib>
            <attrib name="max_link_count" type="integer"> 128 </attrib>
            <attrib name="start_uris" type="list-string">
                <member> http://www.startsiden.no/rss.rss </member>
            </attrib>
            <!-- Start uri files (optional)                                 -->
            <attrib name="start_uri_files" type="list-string">
                <member> /usr/fast/etc/rss_seedlist.txt </member>
            </attrib>
        </section>

logins

This section specifies at least one logins section element for HTML forms-based authentication. These are associated with specific Web site logins, each of which must contain a unique login name in the name attribute.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
preload | string | <value> | Specifies the full URI of the page to retrieve before processing the login form.
scheme | string | http|https | Specifies the URI scheme of the login Web site. Valid values: http or https
site | string | <value> | Specifies the host name of the login form page.
form | string | <value> | Specifies the path of the login form.
action | string | GET|POST | Specifies whether the form uses HTTP POST or HTTP GET. Valid values are as follows: GET or POST
sites | list-string | <value> | Specifies a list of Web sites or host names that the Web crawler should log on to before it begins the crawl process.
ttl | integer | <seconds> | Specifies the time, in seconds, that can elapse before requiring another login to continue the crawl.
html_form | string | <value> | Specifies the URI to the HTML page that contains the login form.
autofill | boolean | yes|no | Specifies whether the Web crawler should try to automatically fill out the HTML login form. The html_form configuration parameter must be specified if this attribute is set to yes.
relogin_if_failed | boolean | yes|no | Specifies whether the Web crawler can attempt to re-log on to the Web site after ttl seconds if the login failed.

Remarks

You can use Login elements as an alternative to the logins section.

Example

Copy Code
        <section name="logins">
            <section name="mytestlogin">
                <!-- Instructs the crawler to "preload" potential cookies by -->
                <!-- fetching this page and register any cookies before      -->
                <!-- proceeding with login                                   -->
                <attrib name="preload" type="string">http://preload.contoso.com/</attrib>
                <attrib name="scheme" type="string"> https </attrib>
                <attrib name="site" type="string"> login.contoso.com </attrib>
                <attrib name="form" type="string"> /path/to/some/form.cgi </attrib>
                <attrib name="action" type="string">POST</attrib>
                <section name="parameters">
                    <attrib name="user" type="string"> username </attrib>
                    <attrib name="password" type="string"> password </attrib>
                    <attrib name="target" type="string"> sometarget </attrib>
                </section>
                <!-- Host names of sites requiring this login to crawl -->
                <attrib name="sites" type="list-string">
                    <member> site1.contoso.com </member>
                    <member> site2.contoso.com </member>
                </attrib>
                <!-- Time to live for login cookie. Will re-log in when expires -->
                <attrib name="ttl" type="integer"> 7200 </attrib>
            </section>
        </section>

parameters

This section sets the authentication credentials that are used in a HTML form. It must be specified in a logins section, or in a Login element. The credential parameters are typically different for each HTML form.

If the autofill configuration parameter is enabled, only variables that are visible in the browser are specified. For example: username and password or equivalent. In this case, the Web crawler must retrieve the HTML page and read any "hidden" variables that are required to submit the form. A variable value that is specified in the configuration parameters will override any value that was stored in the form.

Attributes

The following table specifies attrib elements for this section.

Name | Type | Value | Meaning
name The name XML attribute contains the variable of the HTML form to set. | string |  | Specifies the values of the HTML form variable.

Example

Copy Code
<section name="parameters">
                <attrib name="user" type="string"> username </attrib>
                <attrib name="password" type="string"> password </attrib>
                <attrib name="target" type="string"> sometarget </attrib>
            </section>

subdomains

This section specifies the configuration of crawl sub collections. The subdomains section must contain at least one section XML element, each of which specifies a crawl sub collection. A crawl sub collection section must contain a unique name by setting the name attribute

Remarks

Instead of a subdomains section, you can use a SubDomain element.

You must specify include/exclude rules to limit the scope of a crawl sub collection. These include/exclude rules are as follows: include_domains, exclude_domains, include_uris and exclude_uris.

Only a sub-set of the configuration parameters specified in attrib can be used in a sub-section. These configuration parameters are as follows:

  • accept_compression

  • allowed_schemes

  • crawlmode

  • cut_off

  • delay

  • ftp_passive

  • headers

  • max_doc

  • proxy

  • refresh

  • refresh_mode

  • start_uris

  • start_uri_files

  • use_http_1_1

  • use_javascript

  • use_sitemaps

The refresh configuration parameters of a crawl sub collection must be set lower than the refresh rate of the main crawl collection. The use_javascript, use_sitemaps, and max_doc configuration parameters cannot be used if the include_uris or exclude_uris settings are used to specify the crawl sub collection.

In addition, you can use the rss and the variable_delay sections in a crawl sub collection.

Example

Copy Code
<?xml version="1.0" encoding="utf-8"?>
<CrawlerConfig>
   <DomainSpecification name="subcollection_example">
      <section name="subdomains">
         <section name="subdomain_1">
            <section name="include_uris">
               <attrib name="prefix" type="list-string">
                  <member> http://www.contoso.com/index </member>
               </attrib>
            </section>
            <attrib name="refresh" type="real"> 60.0 </attrib>
            <attrib name="delay" type="real"> 10.0 </attrib>
            <attrib name="start_uris" type="list-string">
               <member> http://www.contoso.com/ </member>
            </attrib>
         </section>
      </section>
</DomainSpecification>
</CrawlerConfig>

SubDomain

This element specifies the configuration of crawl sub collections. A crawl sub collection is an object that differentiates crawl collection members from one another by their definitions. A crawl collection can contain multiple SubDomain elements.

The configuration parameters for a SubDomain element are specified in subdomains.

A SubDomain element contains attrib elements and section elements.

Attributes

Attribute | Value | Meaning
name | <name> | A string specifying the name of the crawl sub collection.

Example

Copy Code
<?xml version="1.0" encoding="utf-8"?>
<CrawlerConfig>
   <DomainSpecification name="subcollection_example">
      <SubDomain name="subdomain_1">
         <section name="include_uris">
            <attrib name="prefix" type="list-string">
               <member> http://www.contoso.com/index </member>
            </attrib>
         </section>
         <attrib name="refresh" type="real"> 60.0 </attrib>
         <attrib name="delay" type="real"> 10.0 </attrib>
         <attrib name="start_uris" type="list-string">
            <member> http://www.contoso.com/ </member>
         </attrib>
      </SubDomain>
   </DomainSpecification>
</CrawlerConfig>

Login

This element is used for HTML forms-based authentication. The configuration parameters for a Login element are specified in logins. A crawl collection can contain multiple Login elements. A Login element contains attrib elements and section elements.

Attributes

Attribute | Value | Meaning
name | <value> | A string specifying the name of the login specification.

Example

Copy Code
<?xml version="1.0" encoding="utf-8"?>
<CrawlerConfig>
   <DomainSpecification name="login_example">
      <Login name="mytestlogin">
         <attrib name="preload" type="string">http://preload.contoso.com/
         </attrib>
         <attrib name="scheme" type="string"> https </attrib>
         <attrib name="site" type="string"> login.contoso.com  </attrib>
         <attrib name="form" type="string"> /path/to/some/form.cgi </attrib>
         <attrib name="action" type="string">POST</attrib>
         <section name="parameters">
            <attrib name="user" type="string"> username </attrib>
            <attrib name="password" type="string"> password </attrib>
         </section>
         <attrib name="sites" type="list-string">
            <member> site1.contoso.com  </member>
            <member> site2.contoso.com  </member>
         </attrib>
         <attrib name="ttl" type="integer"> 7200 </attrib>
         <attrib name="html_form" type="string">
            http://login.contoso.com/login.html
         </attrib>
         <attrib name="autofill" type="boolean"> yes </attrib>
         <attrib name="relogin_if_failed" type="boolean"> yes </attrib>
      </Login>
   </DomainSpecification>
</CrawlerConfig>

Node

This element is used to override configuration parameters in a crawl collection or a crawl sub collection for a particular node scheduler. The configuration parameters for a Node element are specified in SubDomain, Login, attrib and section.

A Node element contains attrib elements and section elements.

Attributes

Attribute | Value | Meaning
name | <value> | A string specifying the node scheduler for these configuration parameters.

Example

The following example uses a multi-node installation. One of the node schedulers is named "crawler_node1". This configures the "crawler_node1" with a different delay configuration parameter than the other nodes.

Copy Code
<?xml version="1.0" encoding="utf-8"?>
<CrawlerConfig>
   <DomainSpecification name="node_example ">
      <attrib name="delay" type="real"> 60.0 </attrib>
      <Node name="crawler_node1">
         <attrib name="delay" type="real"> 90.0 </attrib>
      </Node>
   </DomainSpecification>
</CrawlerConfig>

XML schema

A Web crawler configuration file must be formatted according to the following XML schema:

Copy Code
<?xml version="1.0" encoding="UTF-8" ?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema">

  <xs:element name="CrawlerConfig" type="CT_CrawlerConfig"/>

  <xs:complexType name="CT_CrawlerConfig >
    <xs:choice minOccurs="0" maxOccurs="unbounded">
      <xs:element name="DomainSpecification" type="CT_DomainSpecification"/>
    </xs:choice>
  </xs:complexType>

  <xs:complexType name="CT_DomainSpecification">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
      <xs:element name="attrib" type="CT_attrib" maxOccurs="unbounded"/>
      <xs:element name="section" type="CT_section"/>
      <xs:element name="SubDomain" type="CT_SubDomain"/>
      <xs:element name="Login" type="CT_Login"/>
      <xs:element name="Node" type="CT_Node"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required"/>
  </xs:complexType>

  <xs:complexType name="CT_attrib" mixed="true">
    <xs:sequence minOccurs="0" maxOccurs="unbounded">
      <xs:element name="member" type="ST_member"/>
    </xs:sequence>
    <xs:attribute name="name" type="xs:string" use="required"/>
    <xs:attribute name="type" type="ST_type" use="required"/>
  </xs:complexType>

  <xs:complexType name="CT_section">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element name="attrib" type="CT_attrib"/>
        <xs:element name="section" type="CT_section"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required"/>
  </xs:complexType>

  <xs:complexType name="CT_SubDomain">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
      <xs:element name="attrib" type="CT_attrib"/>
      <xs:element name="section" type="CT_section"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required"/>
  </xs:complexType>

  <xs:complexType name="CT_Login">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
      <xs:element name="attrib" type="CT_attrib"/>
      <xs:element name="section" type="CT_section"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required"/>
  </xs:complexType>

  <xs:complexType name="CT_Node">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
      <xs:element name="attrib" type="CT_attrib"/>
      <xs:element name="section" type="CT_section"/>
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required"/>
  </xs:complexType>

  <xs:simpleType name="ST_type">
    <xs:restriction base="xs:string">
      <xs:enumeration value="boolean"/>
      <xs:enumeration value="string"/>
      <xs:enumeration value="integer"/>
      <xs:enumeration value="list-string"/>
      <xs:enumeration value="real"/>
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="ST_member">
    <xs:restriction base="xs:string"></xs:restriction>
  </xs:simpleType>
</xs:schema>

Simple configuration

The following example configures a simple Web crawler configuration. It is configured to crawl only the contoso.com Web site.

Copy Code
<?xml version="1.0" encoding="utf-8"?>
<CrawlerConfig>
    <DomainSpecification name="default_example">
        <section name="crawlmode">
            <attrib name="fwdlinks" type="boolean"> no </attrib>
            <attrib name="fwdredirects" type="boolean"> no </attrib>
            <attrib name="mode" type="string"> FULL </attrib>
            <attrib name="reset_level" type="boolean"> no </attrib>
        </section>
        <attrib name="start_uris" type="list-string">
            <member> http://www.contoso.com </member>
        </attrib>
    </DomainSpecification>
</CrawlerConfig>

Typical configuration

The following example crawler configuration contains some common configuration parameters.

Copy Code
<?xml version="1.0" encoding="utf-8"?>
<CrawlerConfig>
    <DomainSpecification name="default_example">
        <attrib name="accept_compression" type="boolean"> yes </attrib>
        <attrib name="allowed_schemes" type="list-string">
            <member> http </member>
            <member> https </member>
        </attrib>
        <attrib name="allowed_types" type="list-string">
            <member> text/html </member>
            <member> text/plain </member>
        </attrib>
        <section name="cachesize">
            <attrib name="aliases" type="integer"> 1048576 </attrib>
            <attrib name="pp" type="integer"> 1048576 </attrib>
            <attrib name="pp_pending" type="integer"> 131072 </attrib>
            <attrib name="routetab" type="integer"> 1048576 </attrib>
        </section>
        <attrib name="check_meta_robots" type="boolean"> yes </attrib>
        <attrib name="cookie_timeout" type="integer"> 900 </attrib>
        <section name="crawlmode">
            <attrib name="fwdlinks" type="boolean"> yes </attrib>
            <attrib name="fwdredirects" type="boolean"> yes </attrib>
            <attrib name="mode" type="string"> FULL </attrib>
            <attrib name="reset_level" type="boolean"> no </attrib>
        </section>
        <attrib name="csum_cut_off" type="integer"> 0 </attrib>
        <attrib name="cut_off" type="integer"> 5000000 </attrib>
        <attrib name="dbswitch" type="integer"> 5 </attrib>
        <attrib name="dbswitch_delete" type="boolean"> no </attrib>
        <attrib name="delay" type="real"> 60.0 </attrib>
        <attrib name="domain_clustering" type="boolean"> no </attrib>
        <attrib name="enforce_delay_per_ip" type="boolean"> yes </attrib>
        <attrib name="exclude_exts" type="list-string">
            <member> .jpg </member>
            <member> .jpeg </member>
            <member> .ico </member>
            <member> .tif </member>
            <member> .png </member>
            <member> .bmp </member>
            <member> .gif </member>
            <member> .wmf </member>
            <member> .avi </member>
            <member> .mpg </member>
            <member> .wmv </member>
            <member> .wma </member>
            <member> .ram </member>
            <member> .asx </member>
            <member> .asf </member>
            <member> .mp3 </member>
            <member> .wav </member>
            <member> .ogg </member>
            <member> .ra </member>
            <member> .aac </member>
            <member> .m4a </member>
            <member> .zip </member>
            <member> .gz </member>
            <member> .vmarc </member>
            <member> .z </member>
            <member> .tar </member>
            <member> .iso </member>
            <member> .img </member>
            <member> .rpm </member>
            <member> .cab </member>
            <member> .rar </member>
            <member> .ace </member>
            <member> .hqx </member>
            <member> .swf </member>
            <member> .exe </member>
            <member> .java </member>
            <member> .jar </member>
            <member> .prz </member>
            <member> .wrl </member>
            <member> .midr </member>
            <member> .css </member>
            <member> .ps </member>
            <member> .ttf </member>
            <member> .mso </member>
            <member> .dvi </member>
        </attrib>
        <attrib name="extract_links_from_dupes" type="boolean"> no </attrib>
        <attrib name="fetch_timeout" type="integer"> 300 </attrib>
        <attrib name="force_mimetype_detection" type="boolean"> no </attrib>
        <section name="ftp_errors">
            <attrib name="4xx" type="string"> DELETE:3 </attrib>
            <attrib name="550" type="string"> DELETE:0 </attrib>
            <attrib name="5xx" type="string"> DELETE:3 </attrib>
            <attrib name="int" type="string"> KEEP:0 </attrib>
            <attrib name="net" type="string"> DELETE:3, RETRY:1 </attrib>
            <attrib name="ttl" type="string"> DELETE:3 </attrib>
        </section>
        <attrib name="headers" type="list-string">
            <member> User-Agent: FAST Enterprise Crawler 6 </member>
        </attrib>
        <attrib name="html_redir_is_redir" type="boolean"> yes </attrib>
        <attrib name="html_redir_thresh" type="integer"> 3 </attrib>
        <section name="http_errors">
            <attrib name="4xx" type="string"> DELETE:0 </attrib>
            <attrib name="5xx" type="string"> DELETE:10 </attrib>
            <attrib name="int" type="string"> KEEP:0 </attrib>
            <attrib name="net" type="string"> DELETE:3, RETRY:1 </attrib>
            <attrib name="ttl" type="string"> DELETE:3 </attrib>
        </section>
        <attrib name="if_modified_since" type="boolean"> yes </attrib>
        <attrib name="javascript_keep_html" type="boolean"> no </attrib>
        <section name="limits">
            <attrib name="disk_free" type="integer"> 0 </attrib>
            <attrib name="disk_free_slack" type="integer"> 3 </attrib>
            <attrib name="max_doc" type="integer"> 0 </attrib>
            <attrib name="max_doc_slack" type="integer"> 1000 </attrib>
        </section>
        <section name="link_extraction">
            <attrib name="a" type="boolean"> yes </attrib>
            <attrib name="action" type="boolean"> yes </attrib>
            <attrib name="area" type="boolean"> yes </attrib>
            <attrib name="card" type="boolean"> yes </attrib>
            <attrib name="comment" type="boolean"> no </attrib>
            <attrib name="embed" type="boolean"> no </attrib>
            <attrib name="frame" type="boolean"> yes </attrib>
            <attrib name="go" type="boolean"> yes </attrib>
            <attrib name="img" type="boolean"> no </attrib>
            <attrib name="layer" type="boolean"> yes </attrib>
            <attrib name="link" type="boolean"> yes </attrib>
            <attrib name="meta" type="boolean"> yes </attrib>
            <attrib name="meta_refresh" type="boolean"> yes </attrib>
        </section>
        <section name="log">
            <attrib name="dsfeed" type="string"> text </attrib>
            <attrib name="fetch" type="string"> text </attrib>
            <attrib name="postprocess" type="string"> text </attrib>
            <attrib name="site" type="string"> text </attrib>
        </section>
        <attrib name="login_failed_ignore" type="boolean"> no </attrib>
        <attrib name="login_timeout" type="integer"> 300 </attrib>
        <attrib name="max_backoff_counter" type="integer"> 50 </attrib>
        <attrib name="max_backoff_delay" type="integer"> 600 </attrib>
        <attrib name="max_doc" type="integer"> 1000000 </attrib>
        <attrib name="max_pending" type="integer"> 2 </attrib>
        <attrib name="max_redirects" type="integer"> 10 </attrib>
        <attrib name="max_reflinks" type="integer"> 0 </attrib>
        <attrib name="max_sites" type="integer"> 128 </attrib>
        <attrib name="max_uri_recursion" type="integer"> 5 </attrib>
        <attrib name="mufilter" type="integer"> 0 </attrib>
        <attrib name="near_duplicate_detection" type="boolean"> no </attrib>
        <attrib name="obey_robots_delay" type="boolean"> no </attrib>
        <section name="pp">
            <attrib name="ds_max_ecl" type="integer"> 10 </attrib>
            <attrib name="ds_meta_info" type="list-string">
                <member> duplicates </member>
                <member> redirects </member>
                <member> mirrors </member>
                <member> metadata </member>
            </attrib>
            <attrib name="ds_paused" type="boolean"> no </attrib>
            <attrib name="ds_send_links" type="boolean"> no </attrib>
            <attrib name="max_dupes" type="integer"> 10 </attrib>
            <attrib name="stripe" type="integer"> 1 </attrib>
        </section>
        <section name="ppdup">
            <attrib name="compact" type="boolean"> yes </attrib>
        </section>
        <attrib name="proxy_max_pending" type="integer"> 2147483647 </attrib>
        <attrib name="refresh" type="real"> 1440.0 </attrib>
        <attrib name="refresh_mode" type="string"> scratch </attrib>
        <attrib name="refresh_when_idle" type="boolean"> no </attrib>
        <attrib name="robots" type="boolean"> yes </attrib>
        <attrib name="robots_auth_ignore" type="boolean"> yes </attrib>
        <attrib name="robots_timeout" type="integer"> 300 </attrib>
        <attrib name="robots_tout_ignore" type="boolean"> no </attrib>
        <attrib name="robots_ttl" type="integer"> 86400 </attrib>
        <section name="rss">
            <attrib name="auto_discover" type="boolean"> no </attrib>
            <attrib name="del_expired_links" type="boolean"> no </attrib>
            <attrib name="follow_links" type="boolean"> no </attrib>
            <attrib name="ignore_rules" type="boolean"> no </attrib>
            <attrib name="index_feed" type="boolean"> no </attrib>
            <attrib name="max_link_age" type="integer"> 0 </attrib>
            <attrib name="max_link_count" type="integer"> 128 </attrib>
        </section>
        <attrib name="smfilter" type="integer"> 0 </attrib>
        <attrib name="sort_query_params" type="boolean"> no </attrib>
        <attrib name="start_uris" type="list-string">
            <member> http://www.contoso.com </member>
        </attrib>
        <section name="storage">
            <attrib name="clusters" type="integer"> 8 </attrib>
            <attrib name="compress" type="boolean"> yes </attrib>
            <attrib name="compress_exclude_mime" type="list-string">
                <member> application/x-shockwave-flash </member>
            </attrib>
            <attrib name="datastore" type="string"> bstore </attrib>
            <attrib name="defrag_threshold" type="integer"> 85 </attrib>
            <attrib name="remove_docs" type="boolean"> no </attrib>
            <attrib name="store_dupes" type="boolean"> no </attrib>
            <attrib name="store_http_header" type="boolean"> yes </attrib>
        </section>
        <attrib name="truncate" type="boolean"> no </attrib>
        <attrib name="umlogs" type="boolean"> yes </attrib>
        <attrib name="uri_search_mime" type="list-string">
            <member> text/html </member>
            <member> text/vnd.wap.wml </member>
            <member> text/wml </member>
            <member> text/x-wap.wml </member>
            <member> x-application/wml </member>
            <member> text/x-hdml </member>
        </attrib>
        <attrib name="use_cookies" type="boolean"> no </attrib>
        <attrib name="use_http_1_1" type="boolean"> yes </attrib>
        <attrib name="use_javascript" type="boolean"> no </attrib>
        <attrib name="use_meta_csum" type="boolean"> no </attrib>
        <attrib name="use_sitemaps" type="boolean"> no </attrib>
        <section name="workqueue_priority">
            <attrib name="default" type="integer"> 1 </attrib>
            <attrib name="levels" type="integer"> 1 </attrib>
            <attrib name="pop_scheme" type="string"> default </attrib>
            <attrib name="start_uri_pri" type="integer"> 1 </attrib>
        </section>
   </DomainSpecification>
</CrawlerConfig>

See Also

Reference

crawleradmin.exe reference

Concepts

crawlerglobaldefaults.xml reference


crawlerglobaldefaults.xml reference
See Also

Use crawlerglobaldefaults.xml to specify FAST Search Web crawler configuration options that apply to all crawl collections. Configuration options include DNS, content submission, duplicate detection, and other global settings. This is an advanced feature. You will rarely have to use it.

Warning:

Any changes that you make to this file will be overwritten and lost if you:

  • Run the Set-FASTSearchConfiguration Windows PowerShell cmdlet.

  • Install a FAST Search Server 2010 for SharePoint update or service pack.

To avoid losing your changes, make sure that you back up this file after you have modified it.

Remember to reapply your changes after you run the Set-FASTSearchConfiguration Windows PowerShell cmdlet or install a FAST Search Server 2010 for SharePoint update or service pack.

The FAST Search Web crawler looks for the crawlerglobaldefaults.xml file that is named in <FASTSearchFolder>\etc\ on startup (where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch). You can override this location by passing the -F <path> argument to the crawler.exe executable in NodeConf.xml (after you edit NodeConf.xml, restart nctrl.exe or run nctrl.exe reloadcfg).

If a crawlerglobaldefaults.xml file cannot be found, the FAST Search Web crawler reverts to defaults for the settings that can be specified in this file. Some settings can be overridden on the crawler.exe command line. For more information, see crawler.exe reference.

Customizing crawlerglobaldefaults.xml

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

To edit this file:

  1. Edit crawlerglobaldefaults.xml in a text editor, not a general-purpose XML editor. Use the existing file in <FASTSearchFolder>\etc\ as a starting point. Include the elements and settings that you must have.

  2. Run nctrl.exe restart crawler to restart the FAST Search Web crawler process with the options that you set in step 1.

If the FAST Search Web crawler is running as a multi-node crawler, this file must be edited on each server where a crawler is running. Each crawler must also be restarted, by running nctrl.exe restart multinodescheduler on the node running the multi-node scheduler and nctrl.exe restart nodescheduler on the servers that are running the node schedulers.

crawlerglobaldefaults.xml quick reference

This table lists the elements in crawlerglobaldefaults.xml. The elements can appear in any order, except for GlobalConfig, in which all sections and attributes must be contained, and member, which can only occur inside an attribute element.

Element Description

CrawlerConfig

This root element identifies the file as a FAST Search Web crawler configuration file.

GlobalConfig

This element identifies the file as a global configuration settings file for the FAST Search Web crawler.

attrib

This child element specifies a configuration setting, specified either by its value or a set of member elements. Formatted as:

<attrib name="name" type="string|integer|real|boolean"> value </attrib>

member

This child element can only occur in an attrib element. It specifies a configuration setting in a list, and is formatted as:

Copy Code
<attrib name="name" type="list-string">
  <member> first value </member>
  ..
  <member> last value </member>
</attrib>

section

This child element contains multiple settings grouped by type.

This table lists the options in crawlerglobaldefaults.xml.

Option | Description
GlobalConfig options | These options are valid inside the GlobalConfig element.
feeding options | These options are valid inside a section element that has the name "feeding". They configure characteristics of submitting Web items to content indexing.
dns options | These attributes specify settings related to the crawler's internal DNS resolver.
near_duplicate_detection options | These options configure the near duplicate detection algorithm for collections that have it enabled.
timeouts options | These options specify global crawler time-out settings.

crawlerglobaldefaults.xml file format

XML elements in crawlerglobaldefaults.xml begin with < and end with />.

The basic element format is as follows:

<attrib name="value" type="value">value</attrib>

For example:

<attrib name="sitemanager_numsites" type="integer" > 1024 </attrib>

Elements, section names, attributes, and attribute values are case-sensitive. Attribute names and types must be enclosed in quotation marks (" ").An element definition can span multiple lines. Spaces, carriage returns, line feeds, and tab characters are ignored in an element definition.

For example:

Copy Code
<attrib
    name="sitemanager_numsites"
    type="integer"
> 1024 </attrib
>
Tip:

For long parameter definitions, position values on separate lines and use indentation to make the file easier to read.

The <GlobalConfig> element is a special case and is required. All other elements are contained within the <GlobalConfig> element, and the element is closed with </GlobalConfig>.

The basic structure of the crawlerglobaldefaults.xml file is as follows:

Copy Code
<?xml version="1.0"?>
<CrawlerConfig>
    <GlobalConfig>
        ...
    </GlobalConfig>
</CrawlerConfig>

You can add comments anywhere, delimited by <!-- and -->.

CrawlerConfig

This is the top-level element. It has no attributes.

GlobalConfig

This element contains the global crawler configuration. It has no attributes.

attrib

This child element specifies a configuration option, either a single value or a list using the member element.

Attributes

Attribute Value Description

name

option name

Specifies the option to configure. See the valid options in the option sections later in this topic.

type

string|integer|real|boolean|list-string

Specifies the type for the option value:

  • string - specifies a string type for the option value.

  • integer - specifies an integer type for the option value. The integer range is 0-2^31 unless otherwise stated.

  • real - specifies a real type for the option value. The real range is 0-2^63 unless otherwise stated.

  • boolean - specifies a Boolean type for the option value. Valid Boolean values are "yes" and "no".

  • list-string - specifies that the option value is a list of values, specified by one or more member elements.

The value of the type attribute must match the type associated with the option that is specified for the name attribute. For example, the numprocs option must always be used with the integer type.

Example

The following example specifies the value 2 for the numprocs option:

Copy Code
<attrib name="numprocs" type="integer"> 2 </attrib>

member

This specifies an element in a list of option values. It has no attributes.

The member element can only be used inside an attrib element.

Example

The following example specifies two browser engines for the browser_engines option:

Copy Code
<attrib name="browser_engines" type="list-string">
    <member> hostname1:13045 </member>
    <member> hostname2:13045 </member>
</attrib>

section

This child element groups a set of related options. A section element contains attrib elements.

Attributes

Attribute | Value | Description
name | name | Specifies the name of the section. Supported sections are listed in the options tables later in this topic.

Example

The following example configures the DNS options, specifying only the timeout option:

Copy Code
<section name="dns">
    <attrib name="timeout" type="integer"> 30 </attrib>
</section>

GlobalConfig options

These options are valid inside the GlobalConfig element.

Option Type Value Description

browser_engines

list-string

hostname:port

List of browser engines. The crawler uses these to process Web pages that contain JavaScripts.

Default: Configured automatically by the installer

datadir

string

directory

The location of the crawler content store. Overridden by the -d option to crawler.exe.

dbtrace

boolean

yes|no

Enable/disable database operation tracing. For debugging only.

Default: no

directio

boolean

yes|no

Enable/disable direct I/O in postprocess and duplicate server. For debugging only.

Default: no

disk_resume_threshold

real

1-2^63

Threshold (in bytes) at which the crawler resumes crawling of all collections, if they have already been suspended by disk_suspend_threshold.

Default: 629145600

disk_suspend_threshold

real

1-2^63

Threshold (in bytes) when the crawler suspends crawling of all collections.

Default: 524288000

dns_resolver_threads

integer

1-64

Maximum number of DNS threads. Increasing this value may improve DNS resolve performance if you are crawling a large number of hostnames.

Default: 5

dns_use_platform_api

boolean

yes|no

Specifies whether to use the OS gethostbyname API for resolving DNS names and NetBIOS names, or the internal resolver.

The internal DNS resolver offers better performance and scalability, but does not support NetBIOS names.

Default: yes

duplicate_servers

list-string

hostname:port

List of duplicate servers.

Default: Configured automatically by the installer

logdir

string

directory

The location of the crawler log. Overridden by the -L option to crawler.exe

logfile_ttl

integer

1-2^31

How long (in days) to keep rotated log files before deleting them.

Default: 365

numprocs

integer

1-8

Number of site manager processes to start.

Default: 2

ppdup_dbformat

string

hastlog|diskhashlog|gigabase

Database format that is used by the duplicate server in a multi-node FAST Search Web crawler deployment.

Default: hashlog

rc_update_freq

integer

1-3600

Specifies the update frequency of crawl statistics (in seconds) to the monitoring service.

Default: 120

sitemanager_numsites

integer

1-1024

Maximum number of site workers per site manager.

Default: 1024

store_cleanup

string

hh:mm

Time of the daily storage cleanup that uses 24-hour clock time.

Default: 04:00

xmlrpcport

integer

port number

The crawler base port. Overridden by the -p option to crawler.exe

Example

The following example specifies options of different types:

Copy Code
<attrib name="ipv4" type="boolean"> yes </attrib>
<attrib name="numprocs" type="integer"> 2 </attrib>
<attrib name="disk_resume_threshold" type="real"> 629145600 </attrib>
<attrib name="browser_engines" type="list-string">
    <member> localhost:13045 </member>
</attrib>s

feeding options

The following options are valid inside a section element that has the name feeding. These options configure characteristics of submitting Web items to content indexing.

Option Type Value Description

feeder_threads

integer

1-8

Specifies the number of content feeder threads to start. For large-scale scenarios, increasing the number of threads can improve performance.

Note:

Must only be changed when the <FASTSearchFolder>\data\crawler\store\dsqueues directory is empty.

Default: 1

fs_threshold

integer

0-2^31

Specifies the maximum size of items sent in a batch to indexing. Any item larger than this value will be sent as a URL reference, which the item processor downloads individually from the crawler.

Default: 128

max_batch_datasize

integer

0-2^31

Specifies the maximum number of bytes per batch. Reducing the maximum batch data size may reduce item processor memory usage.

Default: 50MB

max_batch_size

integer

1-1024

The maximum number of items in each batch submission. Smaller batches may be sent if not enough items are available, or if the memory size of the batch grows too large.

Reducing the maximum batch size may reduce item processor memory usage, but may also decrease performance.

Default: 128

max_cb_timeout

integer

1-3600

The maximum number of seconds to wait for outstanding callbacks in content indexing during shutdown.

Default: 1800

Example

The following example specifies a typical feeding section:

Copy Code
<section name="feeding">
  <attrib name="feeder_threads" type="integer"> 1 </attrib>
  <attrib name="max_cb_timeout" type="integer"> 1800 </attrib>
  <attrib name="max_batch_size" type="integer"> 128 </attrib>
  <attrib name="max_batch_datasize" type="integer"> 52428800 </attrib>
  <attrib name="fs_threshold" type="integer"> 128 </attrib>
</section>

dns options

These attributes specify settings related to the crawler's internal DNS resolver. In single node installations, the node scheduler calls DNS to resolve host names. In a multiple node installation, this job is performed by the multi-node scheduler.

Option | Type | Value | Description
db_cachesize | integer | 1-2^31 | DNS database cache size in bytes. A multi-node scheduler will use 4 times this amount. Default: 10485760
ipv4 | Boolean | yes|no | Indicates if the crawler should resolve host names into IPv4 addresses. Default: yes
ipv6 | Boolean | yes|no | Specifies if the crawler should resolve host names into IPv6 addresses. Default: yes
max_rate | integer | 1-200 | Maximum number of DNS requests to issue per second. Default: 100
max_retries | integer | 1-10 | Maximum number of DNS retries to issue for a failed lookup before giving up. Default: 5
min_rate | integer | 1-10 | Minimum number of DNS requests to issue per second. Default: 5
min_ttl | integer | 1-2^31 | Minimum lifetime of resolved names (in seconds), before it tries to re-resolve. Default: 21600
timeout | integer | 1-300 | DNS request time-out (in seconds) before retrying. Default 30.

The min_rate, max_rate, max_retries and timeout settings only apply when the internal DNS resolver is used instead of the OS DNS resolver. Refer to the dns_use_platform_api option which controls this setting.You must specify either ip4 or ipv6 set to yes.

Example

The following example specifies a typical DNS section:

Copy Code
<section name="dns">
  <attrib name="min_rate" type="integer"> 5 </attrib>
  <attrib name="max_rate" type="integer"> 100 </attrib>
  <attrib name="max_retries" type="integer"> 5 </attrib>
  <attrib name="timeout" type="integer"> 30 </attrib>
  <attrib name="min_ttl" type="integer"> 21600 </attrib>
  <attrib name="db_cachesize" type="integer"> 10485760 </attrib>
  <attrib name="ipv4 " type="integer"> yes </attrib>
  <attrib name="ipv6 " type="integer"> yes </attrib>
</section>

near_duplicate_detection options

Near duplicate detection is enabled on a per-collection basis. Near duplicate detection only works for languages that use a white space word separator, e.g. western languages. These options configure the near duplicate detection algorithm for collections that have it enabled.

Option | Type | Value | Description
min_token_size | integer | 1-(max_token_size-1) | This option specifies the minimum number of characters a token must have to be included in the lexicon (the lexicon is a list of the words that occur in an item). Tokens that contain fewer characters are excluded from the lexicon. Default: 5
max_token_size | integer | 1-100 | Specifies the maximum character length for a token. Tokens that contain more characters are excluded from the lexicon (the lexicon is a list of the words that occur in an item). Default: 35
unique_tokens | integer | 1-10 | Specifies the minimum number of unique tokens a lexicon must contain to perform advanced duplicate detection. (A lexicon is the list of the words that occur in an item.) Below this level, the checksum is computed on the whole item. Default: 10
high_freq_cut | real | 0.0-1.0 | Specifies the percentage of tokens (as a decimal) with a high frequency to cut from the lexicon (a lexicon is the list of the words that occur in an item). Default: 0.1
low_freq_cut | real | 0.0-1.0 | Specifies the percentage of tokens (as a decimal) with a low frequency to cut from the lexicon (a lexicon is the list of the words that occur in an item). Default: 0.2

Example

The following example specifies a typical near_duplicate_detection section:

Copy Code
<section name='near_duplicate_detection'>
  <attrib name="min_token_size" type="integer"> 5 </attrib>
  <attrib name="max_token_size" type="integer"> 35 </attrib>
  <attrib name="unique_tokens" type="integer"> 10 </attrib>
  <attrib name="high_freq_cut" type="real"> 0.1 </attrib>
  <attrib name="low_freq_cut" type="real"> 0.2 </attrib>
</section>

timeouts options

These options specify various global crawler time-out settings.

Option | Type | Value | Description
compaction_idle | integer | 1-3600 | Specifies the time-out period (in seconds) for all ongoing crawl activity to stop, in preparation for the nightly content store defragmentation. Site managers that are not idle at this point must be stopped before defragmentation can start. Default: 600
compaction_kill | integer | 1-3600 | Specifies the time-out period (in seconds) for site managers to shut down before defragmentation. Site manager processes that are not stopped during this time will be killed. Default: 120
shutdown_fileserver | integer | 1-3600 | Specifies the shut-down time-out period (in seconds) for the file server. Processes that do not shut down within the time-out period are killed. Default: 10
shutdown_postprocess | integer | 1-3600 | Specifies the shut-down time-out period (in seconds) for postprocess. Processes that do not shut down within the time-out period are killed. Default: 300
shutdown_sitemanager | integer | 1-3600 | Specifies the shut-down time-out period (in seconds) for the site manager. Processes that do not shut down within the time-out period are killed. Default: 300

Example

The following example specifies a typical time-out section:

Copy Code
<section name="timeouts">
  <attrib name="compaction_idle" type="integer"> 600 </attrib>
  <attrib name="compaction_kill" type="integer"> 120 </attrib>
  <attrib name="shutdown_sitemanager" type="integer"> 300 </attrib>
  <attrib name="shutdown_postprocess" type="integer"> 300 </attrib>
  <attrib name="shutdown_fileserver" type="integer"> 10 </attrib>
</section>

See Also

Reference

crawler.exe reference

Concepts

Web Crawler XML configuration reference


deployment.xml reference
See Also

Use the deployment.xml file to outline the topology of your Microsoft FAST Search Server 2010 for SharePoint installation. The file describes all the servers in your FAST Search Server 2010 for SharePoint deployment and the distribution of services. Single server installations have a default deployment.xml file which may be modified.

This section specifies the deployment.xml file format. This file contains detailed structure information that specifies how FAST Search Server 2010 for SharePoint is distributed over multiple servers. One or more services will run on each node in the FAST Search Server 2010 for SharePoint application.

You can find example deployment.xml templates under <FASTSearchFolder>\etc\, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

The purpose of deployment.xml is to enable central control over FAST Search Server 2010 for SharePoint servers and services in an installation. You can reconfigure FAST Search Server 2010 for SharePoint by adding or removing servers or services in the deployment.xml file. See Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint).

The deployment.xml file must be created and modified on the FAST Search Server 2010 for SharePoint administration server only.

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Element quick reference

The following table lists the elements in deployment.xml. These elements can appear in any order.

Element | Description
<deployment> | Top level element describing the complete installation
<instanceid> | The unique name of the FAST Search Server 2010 for SharePoint instance. Used by SCOM.
<connector-databaseconnectionstring> | The JDBC URL used by indexing connectors for configuration data. Used by the FAST Search database connector and the FAST Search Lotus Notes connector.
<host> | Each server in the FAST Search Server 2010 for SharePoint deployment is defined within the host tag. Each host (server) holds a set of elements determining the components that will run on the server.
<searchcluster> | Defines the multiple server configuration for query matching and indexer nodes (search cluster). Describes which parts of the installation will serve queries and perform indexing. It also specifies if fault tolerance is enabled.
<admin> | Part of the host element. Specifies if a host (server) will have the administration component.
<document-processor> | Part of the host element. Specifies if a host (server) will have item processing components.
<content-distributor> | Part of the host element. Specifies if a host (server) will have the content distributor component.
<indexing-dispatcher> | Part of the host element. Specifies if a host (server) will have the indexing dispatcher component.
<crawler> | Part of the host element. Specifies if a host (server) will have a Web crawler component.
<webanalyzer> | Part of the host element. Specifies if a host (server) will have a Web analyzer component.
<searchengine> | Part of the host element. Specifies if a host (server) will have a search cluster component (indexing or query matching or both).
<query> | Part of the host element. Specifies if a host (server) will have the query processing component.
<row> | Part of the searchcluster element. Specifies the features of a single search row or indexer row in the search cluster (deployment).

How it works

The post-setup configuration wizard uses the deployment.xml file to configure the local server. On the administration server, the wizard retrieves a local copy, and on non-administration servers, the wizard gets a copy from the administration server.

Information in deployment.xml is used to generate the local configuration files that are needed to configure the instance of FAST Search Server 2010 for SharePoint.

The local configuration files are updated during post-setup configuration and when you call the cmdlet Set-FastSearchConfiguration.

deployment.xml file format

The deployment.xml file uses XML syntax. The XML schema section contains a formal specification of the file syntax.

The basic element format is as follows:

Copy Code
<element_name [attribute_name="value"] [attribute_name="value"] ... />

For example:

Copy Code
<document-processor processes="4" />

Elements and attributes are case-sensitive. Attribute values must be enclosed in quotation marks (" ") and are not case-sensitive.

Comments can be added anywhere and are delimited by <!-- and -->.

For example:

Copy Code
<!-- Add a useful comment here-->

deployment

Contains the layout of the FAST Search Server 2010 for SharePoint installation

Attributes

Attribute | Value | Description
version | <value> | Optional; version of deployment file
modifiedBy | user\domain | Optional; user who last modified the deployment file
modifiedTime | time | Optional; the time that the deployment file was last modified
comment |  | Optional notes
xmlns | http://www.microsoft.com/enterprisesearch | Required namespace definition
xmlns:xsi | http://www.w3.org/2001/XMLSchema-instance | Optional schema definition
xsi:schemaLocation | http://www.microsoft.com/enterprisesearch deployment.xsd | Optional location of schema

Example

Copy Code
<deployment
    version="1"
    modifiedBy="domain\user"
    modifiedTime="2009-03-14T14:39:17+01:00"
    comment="Comment"
    xmlns="http://www.microsoft.com/enterprisesearch"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.microsoft.com/enterprisesearch
    deployment.xsd">
    <!-- Hosts go here -->
</deployment>

instanceid

Identifies a unique logical name for the FAST Search Server 2010 for SharePoint instance

Attributes

None

Example

Copy Code
<instanceid>MyFastSearchServer</instanceid>

connector-databaseconnectionstring

The JDBC URL used by connectors to store its own content. This element can be left empty if the FAST Search database connector or the FAST Search Lotus Notes connector is not used.

Example

Copy Code
<connector-databaseconnectionstring>
    <![CDATA[jdbc:sqlserver://sqlservername\instancename:portnumber;
    DatabaseName=dbname]]>
  </connector-databaseconnectionstring>

host

This element specifies components that will run on a particular host (server). Add one host entry for each server in the FAST Search Server 2010 for SharePoint deployment.

Attributes

Attribute | Value | Description
name | <value> | Fully qualified domain name of this host (server)
useadminservicehttps | true|false | Set to true if components on this server must use HTTPS when communicating with the administration server. This attribute is optional, default is false.

Example

Copy Code
<host name="single01.search.contoso.com" useadminservicehttps="true">
    <!-- components go here -->
</host>

searchcluster

Contains information about how the index is configured.

Important:

The primary indexer must always be on row id="0".

Attributes

None

Example

Copy Code
<searchcluster>
    <row id="0" index="primary" search="true" />
</search-cluster>

admin

Add this element if a host (server) has the administration component. Only one host (server) can be configured to use the administration component.

The server that is selected as the Administration server in the post-configuration wizard, must be the same as the administration server in deployment.xml.

Attributes

None

Example

Copy Code
<admin />

document-processor

Add this element if a host (server) will run the item processing component.

Attributes

Attribute | Value | Description
processes | <value> | Number of item processors that will run on the host (server). Must be an integer value between 1 and 20. More processes can be added for better usage of servers that have many cores. Default: 4.

Example

Copy Code
<document-processor processes="4" />

content-distributor

Added if a host (server) will run the content distributor component

Attributes

Attribute | Value | Description
id | <value> | Must be an integer value 0 or larger. If several content distributors are added for redundancy, assign a unique id to each instance.

Example

Copy Code
<content-distributor id="0" />

indexing-dispatcher

Added if a host (server) has the indexing dispatcher component

Attributes

None

Example

Copy Code
<indexing-dispatcher />

crawler

Added if a host (server) has a Web crawler component.

Note:

Specify this element when you use the optional FAST Search Web crawler. You do not have to specify this element when you use the Web site indexing connector option in the FAST Search connector Content Search Service Application (Content SSA).

An installation can have several single server Web crawlers or a distributed Web crawler. Only one Web crawler element can be defined per host (server).

Attributes

Attribute | Value | Description
role | single|multi | Specifies if the crawler is a stand-alone crawler or part of a distributed crawler. Default: single
multi-node-scheduler | true|false | If part of a distributed crawler, this attribute specifies if the server will run the multi-node scheduler. A distributed crawler can have only one multi-node scheduler. Ignored if role is "single".
node-scheduler | true|false | This attribute specifies if the server will run a node scheduler. A distributed crawler can have several node-schedulers, but only one per server. Ignored if role is "single".
duplicate-server | true|false | This attribute specifies if the server will run a duplicate server. A distributed crawler can have several duplicate servers, but only one per server. Ignored if role is "single".
duplicate-server-replica | true|false | This attribute determines whether the server will run a duplicate server replica. A distributed crawler can have several duplicate server replicas, but only one per server. Ignored if role is "single".
duplicate-server-replica-host | <value> | This attribute specifies the location of the duplicate server replica that the local duplicate server will use. Ignored if role equals "single", or if no duplicate server is specified on the local server.

Remarks

Consult Crawling Web content with the FAST Search Web crawler for more information about the various Web crawler components.

Example

The following example describes a situation with three hosts, running a Web crawler with one Multi-node Scheduler, two Node-Schedulers and two Duplicate Servers, where one replicates the other.

Copy Code
<host name="node1">
      <!-- Multi Node Scheduler -->
      <crawler role="multi" multi-node-scheduler="true">
</host>
<host name="node2">
      <!-- Node Scheduler and Duplicate Server -->
      <crawler role="multi" node-scheduler="true" duplicate-server="true" duplicate-server-replica-host="node3">
</host>
<host name="node3">
      <!-- Node Scheduler and Duplicate Server replica -->
      <crawler role="multi" node-scheduler="true" duplicate-server-replica="true">
</host>

webanalyzer

Added if a server has the Web analyzer component. A cluster can have several single node crawlers or a distributed Web analyzer. You can have only one Web analyzer element per server.

Attributes

Attribute | Value | Description
server | true|false | If part of a distributed Web analyzer, this attribute specifies if the server will run the central administration of the Web analyzer. A distributed Web analyzer can have only one server. Must be "true" for a local instance of the Web analyzer.
max-targets | <value> | Specifies the number of processors that the Web analyzers will use. Default: 1
link-processing | true|false | If part of a distributed Web analyzer, this attribute specifies if the server will process links extracted in the item pipeline. A distributed Web analyzer can have several instances of link-processors in a cluster. Must be "true" for a single server Web analyzer.
lookup-db | true|false | If part of a distributed Web analyzer, this attribute specifies if the server will be used to look up data on links. A distributed Web analyzer can have several instances of lookup-db in a cluster. Must be "true" for a single server Web analyzer.
redundant-lookup | true|false | If part of a distributed Web analyzer, this attribute specifies if the server will have a redundant lookup database for failover. A distributed Web analyzer can have several instances of redundant lookup-databases in a cluster, but only on servers that have lookup-db enabled.

Remarks

For additional information about installing a distributed Web analyzer, see:

  • waadmin.exe reference

  • Plan FAST Search Server farm topology (FAST Search Server 2010 for SharePoint)

  • Recommendations: Redundancy and availability (FAST Search Server 2010 for SharePoint)

Example

Copy Code
<webanalyzer
      server="true" max-targets="1"
      link-processing="true" lookup-db="true"
      redundant-lookup="false"
    />

searchengine

Added if a host (server) has the core search components, which include indexer and query matching components

Attributes

Attribute | Value | Description
row | <value> | The search row or indexer row that the server belongs to.
column | <value> | The index column that the server belongs to.

Remarks

The attributes specify where the server belongs in the search grid, based on this and the information retrieved from the searchcluster tag.

Example

Copy Code
<searchengine row="0" column="0" />

query

Add this element to enable query and result processing on a host (server). You can configure the query component on more than one server (host) for fault-tolerance and performance.

Attributes

None

Example

Copy Code
<query />

row

Describes the role of search row or indexer row: if it indexes content, if it is a backup indexer, and if the row has search capabilities enabled

Attributes

Attribute Value Description

id

<value>

The row number; must start at 0 and increase sequentially by 1 for each row tag

index

primary|secondary|none

Specifies if the row will have indexing, if it is the primary indexer, the secondary indexer in a fault-tolerant setup, or if the row has no indexers (none).

Important:

The primary indexer must always be on row id="0".

search

true|false

Specifies if the row will have search dispatchers

Remarks

Consult the Search Cluster documentation for more information about how to configure a search cluster.

Example

Single node example:

Copy Code
<searchcluster>
    <row id="0" index="primary" search="true" />
</searchcluster>

Multi-node example:

Copy Code
<searchcluster>
    <row id="0" index="primary" search="false" />
    <row id="1" index="secondary" search="true" />
    <row id="2" index="none" search="true" />
</searchcluster>

Example deployment.xml for a single server deployment

The following is a sample configuration file for a single server deployment:

Copy Code
<?xml version="1.0" encoding="utf-8" ?>
<deployment
    version="14"
    modifiedBy="domain\user"
    modifiedTime="2009-03-14T14:39:17+01:00"
    comment="Comment"
    xmlns="http://www.microsoft.com/enterprisesearch"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.microsoft.com/enterprisesearch
    deployment.xsd">

  <instanceid>FAST Search Server Single Node</instanceid>

  <connector-databaseconnectionstring>
    <![CDATA[jdbc:sqlserver://sqlservername\instancename:portnuber;
    DatabaseName=dbname]]>
  </connector-databaseconnectionstring>

  <host name="single01.search.contoso.com">
    <admin />
    <document-processor processes="4"/>
    <content-distributor id="0" />
    <indexing-dispatcher />
    <crawler role="single" />
    <webanalyzer
      server="true" max-targets="1"
      link-processing="true" lookup-db="true"
      redundant-lookup="false"
    />
    <searchengine row="0" column="0" />
    <query />
  </host>

  <searchcluster>
    <row id="0" index="primary" search="true" />
  </searchcluster>

</deployment>

XML schema

The following XML schema specifies the component distribution configuration file format:

Copy Code
<?xml version="1.0" encoding="UTF-8"?>
<xs:schema targetNamespace="http://www.microsoft.com/enterprisesearch"
  elementFormDefault="qualified"
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  xmlns="http://www.microsoft.com/enterprisesearch">

  <!-- *********************** Global elements ********************* -->

  <xs:element name="deployment" type="CT_deployment" />

  <!-- ************************ Simple types ********************** -->

  <xs:simpleType name="ST_crawler-role">
    <xs:restriction base="xs:string">
      <xs:enumeration value="single"></xs:enumeration>
      <xs:enumeration value="multi"></xs:enumeration>
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="ST_index">
    <xs:restriction base="xs:string">
      <xs:enumeration value="primary"></xs:enumeration>
      <xs:enumeration value="secondary"></xs:enumeration>
      <xs:enumeration value="none"></xs:enumeration>
    </xs:restriction>
  </xs:simpleType>

  <!-- ************************ Complex types ********************** -->

  <xs:complexType name="CT_deployment">
    <xs:sequence minOccurs="1" maxOccurs="unbounded">
      <xs:element name="instanceid" minOccurs="1" maxOccurs="1" type="xs:string" />
      <xs:element name="connector-databaseconnectionstring" minOccurs="1"
        maxOccurs="1" type="xs:string" />
      <xs:element name="host" type="CT_host" minOccurs="1" maxOccurs="unbounded" />
      <xs:element name="searchcluster" type="CT_searchcluster" minOccurs="1"
        maxOccurs="1" />
    </xs:sequence>
    <xs:attribute name="version" use="optional" type="xs:int" />
    <xs:attribute name="modifiedBy" use="optional" type="xs:string" />
    <xs:attribute name="modifiedTime" use="optional" type="xs:dateTime" />
    <xs:attribute name="comment" use="optional" type="xs:string" />
  </xs:complexType>

  <xs:complexType name="CT_host">
    <xs:all>
      <xs:element name="admin" type="CT_admin" minOccurs="0" maxOccurs="1" />
      <xs:element name="content-distributor" type="CT_content-distributor"
        minOccurs="0" maxOccurs="1" />
      <xs:element name="indexing-dispatcher" type="CT_indexing-dispatcher"
        minOccurs="0" maxOccurs="1" />
      <xs:element name="document-processor" type="CT_document-processor"
        minOccurs="0" maxOccurs="1" />
      <xs:element name="crawler" type="CT_crawler" minOccurs="0" maxOccurs="1" />
      <xs:element name="webanalyzer" type="CT_webanalyzer" minOccurs="0"
        maxOccurs="1" />
      <xs:element name="searchengine" type="CT_searchengine" minOccurs="0"
        maxOccurs="1" />
      <xs:element name="query" type="CT_query" minOccurs="0" maxOccurs="1" />
      <xs:element name="custom" type="CT_custom" minOccurs="0" maxOccurs="1" />
    </xs:all>
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="useadminservicehttps" use="optional" type="xs:boolean" default="false"/>
  </xs:complexType>

  <xs:complexType name="CT_admin" />

  <xs:complexType name="CT_content-distributor">
    <xs:attribute name="id" use="optional" type="xs:int" default="1" />
  </xs:complexType>

  <xs:complexType name="CT_indexing-dispatcher" />

  <xs:complexType name="CT_document-processor">
    <xs:attribute name="processes" use="optional" type="xs:int" default="1" />
  </xs:complexType>

  <xs:complexType name="CT_crawler">
    <xs:attribute name="role" type="ST_crawler-role" />
    <xs:attribute name="multi-node-scheduler" use="optional" type="xs:boolean"
      default="false"/>
    <xs:attribute name="node-scheduler" use="optional" type="xs:boolean"
      default="false"/>
    <xs:attribute name="duplicate-server" use="optional" type="xs:boolean"
      default="false"/>
    <xs:attribute name="duplicate-server-replica-host" use="optional"
      type="xs:string"/>
    <xs:attribute name="duplicate-server-replica" use="optional" type="xs:boolean"
      default="false"/>
  </xs:complexType>

  <xs:complexType name="CT_webanalyzer">
    <xs:attribute name="server" use="required" type="xs:boolean" />
    <xs:attribute name="lookup-db" use="required" type="xs:boolean" />
    <xs:attribute name="link-processing" use="required" type="xs:boolean" />
    <xs:attribute name="max-targets" use="optional" type="xs:int" />
  </xs:complexType>

  <xs:complexType name="CT_searchengine">
    <xs:attribute name="row" type="xs:int" use="required" />
    <xs:attribute name="column" type="xs:int" use="required" />
  </xs:complexType>

  <xs:complexType name="CT_query" />

  <xs:complexType name="CT_custom">
    <xs:attribute name="ref" type="xs:string" use="required" />
  </xs:complexType>

  <xs:complexType name="CT_searchcluster">
    <xs:sequence>
      <xs:element name="row" type="CT_row" minOccurs="1" maxOccurs="unbounded" />
    </xs:sequence>
  </xs:complexType>

  <xs:complexType name="CT_row">
    <xs:attribute name="id" type="xs:int" use="required" />
    <xs:attribute name="index" type="ST_index" use="required" />
    <xs:attribute name="search" type="xs:boolean" use="required" />
  </xs:complexType>

</xs:schema>

See Also

Concepts

Reconfigure the farm deployment (FAST Search Server 2010 for SharePoint)


jdbctemplate.xml reference
See Also

Use a copy of jdbctemplate.xml to configure the settings for the FAST Search database connector.

The FAST Search database connector uses the configuration settings to control content extraction. You can run multiple instances of the database connector at the same time by using different configuration files for each instance.

Customizing jdbctemplate.xml

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

To extract content from a JDBC database and feed it to a FAST Search Server 2010 for SharePoint installation, first create a copy of the jdbctemplate.xml file. Edit the copy in a text or XML editor of your choice (except Notepad) to include the elements and settings for your environment.

To test your settings, run:

%FASTSEARCH%\bin\JDBCConnector.bat testconnections -f <configuration file>

where <configuration file> is the path and file name of your configuration, and %FASTSEARCH% is the FAST Search Server 2010 for SharePoint installation folder.

To run the connector and feed content to FAST Search Server 2010 for SharePoint, run:

%FASTSEARCH%\bin\JDBConnector.bat start -f <configuration file>

Parameter groups quick reference

The following table contains a list of the parameter groups in jdbctemplate.xml. These groups can appear in any order.

Parameter group | Description
Parameter group: Input | Defines the properties of the content sources.
Parameter group: Transformation | Describes which content is transformed into the FAST Search Server 2010 for SharePoint item model, and how it occurs.
Parameter group: FASTSearchSubmit | Describes the communication properties when submitting content to FAST Search Server 2010 for SharePoint.
Parameter group: XMLExport | Specifies the values for exporting to XML formatted file(s).
Parameter group: ChangeDetection | Specifies how to detect changes in the database since the last database feed by using row checksums.
Parameter group: ConnectorExecution | Specifies general values that are used when you run the connector.
Parameter group: General | Specifies the configuration name.

JDBC template file format

XML elements in jdbctemplate.xml begin with < and end with />. The XML is defined in FastConnectorConfig-1.0.dtd.

The basic format is as follows:

<group name="group_name">

<parameter name="parameter_name" type="parameter_type">

<value>value</value>

</parameter>

</group>

Parameter names are case-sensitive, types are not. Parameter names and types must be enclosed in quotation marks (" ").

A parameter definition can span multiple lines. Spaces, carriage returns, line feeds, and tab characters are ignored in an element definition.

For example:

Copy Code
<group name="ConnectorExecution">
    <parameter name="QueueSize" type="integer">
         <value>200</value>
    </parameter>
    <parameter name="WorkFolder" type="string">
         <value>../var/jdbcconnector</value>
    </parameter>
</group>
Tip:

For long parameter definitions, position values on separate lines and use indentation to make the file easier to read.

The FastConfig element is a special case and is required. All other elements are contained within the FastConfig element.

Here is a sample snippet from a configuration file:

Copy Code
<!DOCTYPE FastConfig SYSTEM "dtd/FastConnectorConfig-1.0.dtd">
<FastConfig>
    <group name="ConnectorExecution">
       <parameter name="QueueSize" type="integer">
             <value>200</value>
        </parameter>
       <parameter name="WorkFolder" type="string">
             <value>../var/jdbcconnector</value>
       </parameter>
    </group>
</FastConfig>

You can add comments anywhere, delimited by <!-- and -->.

For example:

Copy Code
<!DOCTYPE FastConfig SYSTEM "dtd/FastConnectorConfig-1.0.dtd"><FastConfig>
    <!-- The parameter group ConnectorExecution specifies general values
          used when running a connector. -->
    <group name="ConnectorExecution">
       <parameter name="QueueSize" type="integer">
             <value>200</value>
       </parameter>
       <parameter name="WorkFolder" type="string">
             <value>../var/jdbcconnector</value>
       </parameter>
    </group>
</FastConfig>

Parameter group: Input

The parameters in this group define the properties of the content sources.

Parameter Type Value Description

JDBCDriver

string

text

JDBC driver class name that is defined by the JDBC driver provider. Required.

Example value: com.microsoft.sqlserver.jdbc.SQLServerDriver. More examples can be found in the jdbctemplate.xml file.

JDBCURL

string

text

JDBC URL, specified by the JDBC driver provider. The URL may vary for different JDBC drivers. Required.

Example value: jdbc:sqlserver://<yourServerHere>:1433;database=<yourDataBaseNameHere>. More examples can be found in the jdbctemplate.xml file.

JDBCPreSQL

type="list" separator=";"

text

Specifies the SQL statements run before the JDBCSQL statement. Use this parameter for any pre-processing that is required on the database.

JDBCSQL

string

text

Specifies the SQL crawl query.You must specify either JDBCSQL or JDBCSQLFile.

If you use time stamp updates, this attribute requires one of the following:

  • %TIMESTAMP%: Last crawl time is specified as a datetime value.

  • %TIMESTAMPSEC%: Last crawl time is specified as the number of seconds since the epoch. (epoch is 1970-01-01T00:00:00Z - January 1, 1970)

Default: (none)

JDBCSQLFile

string

text

Specifies the name of the file, in an absolute path, that contains the SQL crawl query. You must specify either JDBCSQL or JDBCSQLFile.

JDBCDeleteSQL

string

text

Specifies the SQL delete query.

If you use time stamp updates, this attribute requires one of the following:

  • %TIMESTAMP%: Last crawl time is specified as a datetime value.

  • %TIMESTAMPSEC%: Last crawl time is specified as the number of seconds since the epoch. (epoch is 1970-01-01T00:00:00Z - January 1, 1970)

Default: (none)

JDBCDeleteSQLFile

string

text

Specifies the name of the file, in an absolute path, that contains the SQL delete query.

JDBCPostSQL

type="list" separator=";"

text

Specifies the SQL statements that run after the extract statement.

Default: (none)

JDBCTimeStampField

string

text

Specifies the time stamp field to use as the source for last crawl time (when you are using time stamp based update detection). Specify the name of the database time stamp column as the value. The highest value is set as the last crawl time.

Note:

Using this field requires that a time stamp is extracted by the SQL statement.

Default: (none)

JDBCTimeType

string

date|seconds

Choose date or seconds.

Default: date

JDBCDeleteTimeStampField

string

text

Time stamp field to use as the source for the last delete time (when you are using time stamp based deletion). Specify the name of the database time stamp column as the value. The highest value is set as the last crawl time.

Note:

Using this field requires that a time stamp is extracted by the SQL statement.

Default: (none)

JDBCDeleteTimeType

string

date|seconds

Time used in the JDBC delete time stamp file. Choose date or seconds.

Default: date

JDBCUpdateTimeStampOnFailure

boolean

true|false

Set to true if you want error callbacks from FAST Search Server 2010 for SharePoint or missing callbacks to update the time stamp file.

Default: false

JDBCUpdateTimeStampOnDeletion

boolean

true|false

If set to true, the timestamp file will be updated when documents are removed from the FAST Search Server 2010 for SharePoint index using the FAST Search database connector.

Default: false

JDBCUsername

string

text

Database user name.

May be omitted when using Windows integrated authentication to access SQL Server.

JDBCPassword

string

text

Database password.

May be omitted when using Windows integrated authentication to access SQL Server.

JDBCFetchSizeHint

integer

0|50 - 2000

Tells the JDBC driver how many rows to fetch from the database when more rows are needed. The number of rows specified only affects result sets created by using this statement. A value of zero is ignored. Range: 50 to 2000.

Default: 0

JDBCCharacterEncoding

string

text

Specifies the encoding used in the database, when the table contains character data (LONGVARCHAR type).

Default: UTF-8

Important:

Set only one of {JDBCSQL, JDBCSQLFile} and optionally only one of {JDBCDeleteSQL, JDBCDeleteSQLFile}. Setting more of these parameters in the same configuration file may result in unexpected results.

Example

The following example is for Microsoft SQL Server:

Copy Code
<group name="Input">

    <parameter name="JDBCDriver" type="string">
        <value>com.microsoft.sqlserver.jdbc.SQLServerDriver</value>
    </parameter>

    <parameter name="JDBCURL" type="string">
        <value>jdbc:sqlserver://192.168.1.100\SP:1433;DatabaseName=test</value>
    </parameter>

    <parameter name="JDBCSQL" type="string">
        <value>select * from employees</value>
    </parameter>

    <parameter name="JDBCUsername" type="string">
        <value>john</value>
    </parameter>

    <parameter name="JDBCPassword" type="string">
        <value>AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAmJRnYSadmEqSBy8q5KCI5QQAAAACAAAAAAADZgAAqAAAABAAAAClpb+U+s2eE9l7Nx34wd</value>
    </parameter>

</group>

Parameter group: Transformation

The parameters in this group specify content to transform into the FAST Search Server 2010 for SharePoint item model, and transformation details.

Parameter | Type | Value | Description
JDBCPrimaryKeyField | string | text | The database view/table primary key. Required. Used to create the item ID. Case-sensitive; must match your SELECT statement.
JDBCDeletePrimaryKeyField | string | text | The database view/table primary key. Used to delete the item ID.
JDBCUseBytesForString | boolean | true|false | If set to true, strings are treated as byte-arrays instead of strings. Default: false
UseMultiRowNormalizer | boolean | true|false | If set to true, values for columns of records with identical identifiers are merged. Default: false
JDBCAttributeColumn | string | text | For normalized tables, specifies the column that contains attribute names.
JDBCValueColumn | string | text | For normalized tables, specifies the column that contains attribute values.
JDBCMultiValueIncludeFields | string | text | Semicolon (;) separated list of column names that should be concatenated in the output. This field overrides any settings to JDBCMultiValueSkipFields.
JDBCMultiValueSkipFields | string | text | Semicolon (;) separated list of column names that have the same values in each row, and that should not be repeated in the output.
Note:

Do not use JDBCPrimaryKeyField and JDBCDeletePrimaryKeyField at the same time.

Example

Copy Code
<group name="Transformation">
    <parameter name="JDBCPrimaryKeyField" type="string">
        <value>ID</value>
    </parameter>
</group>

Parameter group: FASTSearchSubmit

The parameters in this group describe communication properties for submitting content to FAST Search Server 2010 for SharePoint.

Parameter | Type | Value | Description
Collection | string | text | Required name of the destination collection.
ActuallyPublish | boolean | true|false | If true, submits the item to FAST Search Server 2010 for SharePoint. Default: true

Example

Copy Code
<group name="FASTSearchSubmit">
    <parameter name="Collection" type="string">
        <value>collectionname</value>
    </parameter>
</group>

Parameter group: XMLExport

The parameters in this group specify an export to FASTXML formatted file(s).

Parameter | Type | Value | Description
ExportToXML | boolean | true|false | If true, exports all items as XMLExport.
BinaryFields | boolean | data | Semicolon (;) separated list of field names of binary fields. Uses BASE64 encoding in the XMLExport.

FASTXML is stored in the %FASTSEARCH%\var\jdbcconnector\<collection name>\fastxml folder.

%FASTSEARCH% is the FAST Search Server 2010 for SharePoint installation folder.

Example

The following example disables FASTXML export:

Copy Code
<group name="XMLExport">
    <parameter name="ExportToXML" type="boolean">
        <value>false</value>
    </parameter>
</group>

Parameter group: ChangeDetection

The parameters in this group specify how to detect database changes that were made since the last feed (from that database), using row checksums.

Parameter | Type | Value | Description
Enabled | boolean | true|false | Set to true to enable change detection. Default: false
ChangeDBPurge | boolean | true|false | Set to true to delete all entries in the checksum database at startup. Default: false
ChangeDBAbortPercent | integer | percentage | When using checksum-based change detection, this value defines the maximum percentage of accepted deleted records in the source. If a higher percentage of deletions is observed during a run, an error occurs and no changes are made to the collection. Default: 10
ChangeDBIncludeFields | string | text | A semicolon (;) separated list of field names that are included in the checksum computations. By default, all fields are included.
ChangeDBExcludeFields | string | text | A semicolon (;) separated list of field names that are excluded from the checksum computations. These fields take precedence over the include field list. For example, if only the metadata of an item may change (not the binary content), exclude the data element from the computations. By default, no fields are excluded.

Example

Copy Code
<group name="ChangeDetection">
    <parameter name="Enabled" type="boolean">
        <value>false</value>
    </parameter>
</group>

Parameter group: ConnectorExecution

The parameters in this group specify general values that are used during a connector run.

Parameter | Type | Value | Description
QueueSize | integer | 1 - 10000 | Size of the internal queue of items. Default: 200
EnableStatusTracker | boolean | true|false | The status tracker logs the item crawl status. To disable status tracking, set to false.
WorkFolder | string | text | The work folder for all file output that includes log files and XMLExport files. Default location: ..\var\jdbcconnector

Example

Copy Code
<group name="ConnectorExecution">
    <parameter name="QueueSize" type="integer">
        <value>200</value>
    </parameter>
</group>

Parameter group: General

This parameter specifies the configuration name by defining an optional prefix for the property set. If empty, the configuration uses a predefined propset.

Parameter | Type | Value | Description
Name | string | name | Specifies the configuration name, the prefix for the propset. The name must be unique, with a maximum length of 32 characters, in the range a-f and 0-9

Example

Copy Code
<group name="General">
    <parameter name="Name" type="string">
        <value>abcdef01</value>
    </parameter>
</group>

See Also

Reference

jdbcconnector.bat reference


loggerconfig.xml reference
See Also

Use LoggerConfig.xml to configure routing and filtering of log messages sent from the FAST Search Server 2010 for SharePoint components. The logging process consists of three steps. The first step is configured in LoggerConfig.xml. The last two steps, for the local and main log servers, are configured in LogServerConfig.xml.

Warning:

Any changes that you make to this file will be overwritten and lost if you:

  • Run the Set-FASTSearchConfiguration Windows PowerShell cmdlet.

  • Install a FAST Search Server 2010 for SharePoint update or service pack.

To avoid losing your changes, make sure that you back up this file after you have modified it.

Remember to reapply your changes after you run the Set-FASTSearchConfiguration Windows PowerShell cmdlet or install a FAST Search Server 2010 for SharePoint update or service pack.

The LoggerConfig.xml configuration file determines how FAST Search Server 2010 for SharePoint processes output log messages. Log messages are usually sent to the local log server for further routing and filtering, in addition to being sent to the log file and console output. For information about the configuration of the local and main log servers, see logserverconfig.xml.

LoggerConfig.xml includes a list of destinations which is used to set up the logging targets where log messages are sent. The programs list configures log level filtering for individual log sources. If an incoming log message does not match any of the defined programs, the default program entry (name=default) is used instead.

You can edit the installed version of LoggerConfig.xml to tune the existing values or to add more program elements to override filtering and routing for specific log sources. However, in most cases, the default values will be acceptable.

Customizing LoggerConfig.xml

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use a text or XML editor to change this file. Edits will not take effect until the FAST Search Server 2010 for SharePoint components related to the configuration change are restarted.

Element quick reference

LoggerConfig.xml uses the following elements:

Element Description

<config>

Top level element

<destinations>

<netdestination>

<filedestination>

<stdoutdestination>

A list of log targets

Configuration for network forwarding of log messages (to the local log server)

Configuration for logging to files

Configuration for output to the console or a pipe

<programs>

<program>

<output>

A list of log sources

Routing and filtering specifications for a log source

Specifications for routing log messages to a particular destination, with a log level threshold filter

LoggerConfig.xml file format

LoggerConfig.xml is an XML file and follows all conventions and syntax rules that are defined for the XML file format.

The basic file structure is as follows:

Copy Code
<?xml version="1.0" encoding="UTF-8"?>
<config version="2">

  <destinations>
    ... specification of where logs can be sent
  </destinations>

  <programs>
    <program name="default">
      ... routing to destination with log level filtering
    </program>
    ... add more program elements here to override defaults
  </programs>

</config>

config

This is the top level element of LoggerConfig.xml..

Attributes

Attribute | Value | Description
version | 2 | The version attribute provides future expansion of the file format. Currently only 2 is valid.

destinations

This element is a container for one or more destination elements.

Element | Description
netdestination | Configures network forwarding of log messages to the local log server
filedestination | Configures logging to files
stdoutdestination | Configures output to the console or a pipe

netdestination

This element defines a network log destination.

Attributes

Attribute | Value | Description
name | string | Symbol used to reference this log destination from a program element

Elements

Element | Value | Description
hostname | string | Network address of the local log server
port | integer | TCP port number of the local log server
timeout | integer | Timeout in seconds
retrycount | integer | Maximum number of attempts to write a log entry
retryperiod | integer | Delay between each retry attempt, in seconds

Remarks

This destination uses a TCP/IP network connection to forward logs to the local log server.

filedestination

This element defines a log destination which writes log files to a specified folder.

Attributes

Attribute | Value | Description
name | string | Symbol used to reference this log destination from a program element

Elements

Element | Value | Description
directory | string | Directory where the log files are created
backups | integer | Number of archived log files to keep
rotatesize | integer | Maximum size, in bytes, for each log file
rotatetime | HHMM | Time of day when log files are archived, regardless of size. For example, 0100 indicates that log files will be archived at 1 AM.

Remarks

You can specify either a rotatesize or a rotatetime element to define different procedures for archiving log files.

stdoutdestination

This element defines a log output to the console or to a command-line process pipe.

Attributes

Attribute | Value | Description
name | string | Symbol used to reference this log destination from a program element

Remarks

This destination logs messages to the standard output stream (stdout).

programs

This element specifies a list of log sources.

Elements

Element | Description
program | Defines routing and filtering for a log source

program

This element defines routing and filtering for a log source.

Attributes

Attribute | Value | Description
name | string | Process name

Elements

Element | Description
output | Defines routing of log messages to a specific destination, with a specified log level threshold filter

Example

The following example defines routing and filtering for log messages from the process named searchctrl-search. Only messages that have the log level set to INFO and higher are routed to the server destination. Messages with log level set to VERBOSE and higher are sent to the file destination. Log messages that have lower log levels are discarded.

Copy Code
<program name="searchctrl-search">
  <output destination="server" threshold="info"/>
  <output destination="file"   threshold="verbose"/>
</program>

output

Defines routing of log messages to a specific destination, with a specified log level threshold filter

Attributes

Attribute Value Description

destination

string

Symbol referencing a configured destination element

threshold

CRITICAL|ERROR|WARNING|INFO|VERBOSE|DEBUG

  • CRITICAL - route only CRITICAL messages.

  • ERROR - route messages that have log level ERROR and above.

  • WARNING - route messages that use log level WARNING and above.

  • INFO - route messages that use log level INFO and above.

  • VERBOSE - route messages that use log level VERBOSE and above.

  • DEBUG - route all log messages.

Example

The following example routes all messages to the destination named file:

Copy Code
<output destination="file" threshold="debug" />

See Also

Concepts

logserverconfig.xml reference


logserverconfig.xml reference
See Also

Use logserverconfig.xml to configure routing and filtering of log messages sent from Microsoft FAST Search Server 2010 for SharePoint components. The logging process consists of three steps. The first step is configured in loggerconfig.xml. logserverconfig.xml contains the configuration for the last two steps, the local and main log servers.

All FAST Search Server 2010 for SharePoint nodes have a local log server. This server performs logging locally and also forwards the log messages to the main (farm) log server which runs only on the administrative node.

Both log server subsystems run within the FAST Search Server 2010 for SharePoint Monitoring service. logserverconfig.xml is loaded when the Monitoring service is started up. The file must be located in the <FASTSearchFolder>\etc\ folder, where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

The list of destinations is used to set up the logging targets where incoming messages are sent.The programs list configures log level filtering for individual log sources. A default program entry (name = default) is defined for each log server. If an incoming log message does not match any of the defined programs, the default program entry is used instead.

You can edit the installed version of logserverconfig.xml to tune the existing values and to add program elements to override filtering and routing for specific log sources. However, in most cases, the default values will be acceptable.

Customizing logserverconfig.xml

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

Use a text or XML editor to change this file. Edits will not take effect until the FAST Search Server 2010 for SharePoint Monitoring service (FASTSearchMonitoring) is restarted.

Element quick reference

logserverconfig.xml uses the following elements:

Element Description

<logserverconfig element>

Top level element

<locallogserverconfig element>

Configuration for the local log server

<mainlogserverconfig element>

Configuration for the main log server

<flood_protect element>

Filter that blocks excessive logging

<destinations element>

A list of log targets:

  • <sharepointulslog element> - SharePoint Unified Logging Service (ULS) log destination for the local log server

  • <mainlogserver element> - Main log server forwarding destination for the local log server

  • <localeventlog element> - NT event log destination for the local log server

  • <farmeventlog element> - NT event log destination for the main (farm) log server

  • <filedestination element> - Log file destination for the main (farm) log server

<programs element>

A list of log sources:

  • <program element> - Routing and filtering specifications for a log source

  • <output element> - Specifications for routing log messages to a particular destination, with a log level threshold filter

logserverconfig.xml file format

logserverconfig.xml is an XML file and follows all conventions and syntax rules that are defined for the XML file format.

The basic file structure is as follows:

Copy Code
<?xml version="1.0" encoding="UTF-8"?>
<logserverconfig version="1.0">

  <locallogserverconfig>

    <destinations>
      ... specification of where logs can be sent
    </destinations>

    <programs>
      <program name="default">
        ... routing to destination with log level filtering
      </program>
      ... add more program elements here to override defaults
    </programs>

  </locallogserverconfig>

  <mainlogserverconfig>
    .. same basic structure as for locallogserverconfig
  </mainlogserverconfig>

</logserverconfig>

logserverconfig element

This is the top level element of logserverconfig.xml.

Attributes

Attribute | Value | Description
version | 1.0 | The version attribute provides future expansion of the file format. Currently only 1.0 is valid.

locallogserverconfig element

This element contains the configuration for the local log server.

mainlogserverconfig element

This element contains the configuration for the main (farm) log server.

flood_protect element

Filter that blocks log messages if flooding is detected.

Three time periods are available:

Period | Description
Measure | Blocking will be enabled if the number of log messages exceeds max_count during a single measurement period.
Block | If blocking is enabled, log messages will be filtered until the end of the current blocking period.
Trickle | Even if blocking is enabled, a single log message may still pass during each trickle period.

Log messages are filtered based on both log source and log level. For example, an active blocking of INFO messages will not affect ERROR messages from the same log source.

Attributes

Attribute | Value | Description
max_count | value | The number of log messages allowed for a single measurement period
measure_minutes | value | Length of measurement periods in minutes
block_minutes | value | Length of blocking periods in minutes
trickle_minutes | value | Length of trickle periods in minutes

Measure_minutes, block_minutes and trickle_minutes can be configured independently. However, the most logical results are achieved if the numbers are set to an integer multiple of the smallest value.

Block_minutes is usually set to a larger or equal value as measure_minutes. To disable the trickle feature, set trickle_minutes to the same value as block_minutes.

Example

In the following example, a maximum of 1000 messages is allowed for each 5 minute measurement period. If this threshold is exceeded, log messages will be blocked for the rest of the current 180 minute blocking period. However, 1 log message will be allowed to pass every 30 minutes (trickle period).

Copy Code
<flood_protect max_count       = "1000"
               measure_minutes = "5"
               block_minutes   = "180"
               trickle_minutes = "30" />

destinations element

This element is a container for one or more destination elements.

sharepointulslog element

This element defines a SharePoint Unified Logging Service (ULS) log destination for use with the local log server. The element can contain one flood_protect element to define an optional filter that blocks log messages if flooding is detected.

Attributes

Attribute | Value | Description
name | string | Symbol used to reference this log destination from a program element

This destination uses the trace logs feature of the SharePoint Unified Logging Service (ULS).

mainlogserver element

This element defines a log destination which forwards logs from the local log server to the main (farm) log server. The element can contain one flood_protect element to define an optional filter that blocks log messages if flooding is detected.

Attributes

Attribute | Value | Description
name | string | Symbol used to reference this log destination from a program element

The forwarding of log messages uses a WCF protocol. Edit the bin\MonitoringService.exe.config .NET file to configure WCF service parameters, bindings, and behavior.

localeventlog element

This element defines an NT event log destination used with the local log server. The element can contain one flood_protect element to define an optional filter that blocks log messages if flooding is detected.

Attributes

Attribute | Value | Description
name | string | Symbol used to reference this log destination from a program element

farmeventlog element

This element defines an NT event log destination for use with the main (farm) log server. The element can contain one flood_protect element to define an optional filter that blocks log messages if flooding is detected.

Attributes

Attribute | Value | Description
name | string | Symbol used to reference this log destination from a program element

filedestination element

This element defines a text file log destination for use with the main (farm) log server. The element can contain one flood_protect element to define an optional filter that blocks log messages if flooding is detected.

Attributes

Attribute | Value | Description
name | string | Symbol used to reference this log destination from a program element

This destination logs messages to files in the <FASTSearchFolder>\var\log\syslog\ directory.

programs element

This element is a container for one or more program elements, and contains a list of log sources.

program element

Defines routing and filtering for a log source. The element contains one or more output elements to define routing of log messages to a particular destination, with a specified log level threshold filter.

Attributes

Attribute | Value | Description
name | string | Log module name, matching the identifier that is used by the logging source. Inspect <FASTSearchFolder>\var\log\syslog\all.log on the main log server to retrieve the identifiers. Example: indexer

For the local log server, routing to the mainlogserver destination is implicit. This message forwarding cannot be disabled.

Example

In the following example, routing and filtering for log messages from the module indexer is defined. Only messages that use the log level of CRITICAL are routed to the destinations eventlog and ulslog. Log messages that use lesser log levels are discarded.

Copy Code
<program name="indexer">
  <output destination="eventlog" threshold="critical" />
  <output destination="ulslog"   threshold="critical" />
</program>

output element

Defines routing of log messages to a particular destination, with a specified log level threshold filter.

Attribute Value Description

destination

string

Symbol referencing a configured destination element

threshold

CRITICAL|ERROR|WARNING|INFO|VERBOSE|DEBUG

  • CRITICAL - route only CRITICAL messages.

  • ERROR - route messages that have log level ERROR and above.

  • WARNING - route messages that use log level WARNING and above.

  • INFO - route messages that use log level INFO and above.

  • VERBOSE - route messages that use log level VERBOSE and above.

  • DEBUG - route all log messages.

Example

In the following example, all messages are routed to the destination with the name file.

Copy Code
<output destination="file" threshold="debug" />

See Also

Concepts

loggerconfig.xml reference


lotusnotessecuritytemplate.xml reference
See Also

Use a copy of lotusnotessecuritytemplate.xml to configure the settings for the FAST Search Lotus Notes user directory connector.

The Lotus Notes user directory connector uses the configuration settings to control user and group extraction, and will always extract the whole user directory from Lotus Domino.

Customizing lotusnotessecuritytemplate.xml

To extract users and groups from Lotus Domino and submit them to FAST Search Server 2010 for SharePoint to enable Lotus Notes item level security, configure and run the FAST Search Lotus Notes user directory connector.

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

To configure the FAST Search Lotus Notes user directory connector, first create a copy of the lotusnotessecuritytemplate.xml file. Edit the copy in a text or XML editor of your choice (except Notepad) to include the elements and settings for your environment.

Parameter groups quick reference

The following table contains a list of the parameter groups in lotusnotestemplate.xml. These groups can appear in any order.

Parameter group | Description
Parameter group: NotesConnection | Defines the connection between the connector and the Lotus Domino server for content extraction.
Parameter group: ConnectorExecution | Specifies general behavior of the connector.
Parameter group: FSAOutput | Specifies where the connector should send its output. This includes log files and user store connection details.
Parameter group: SSOMapping | Specifies the properties for generating an XML aliasing file that details the mapping between Active Directory® users and Lotus Domino users.

lotusnotessecuritytemplate file format

XML elements in lotusnotessecuritytemplate.xml begin with < and end with />. The XML is defined in FastConnectorConfig-1.0.dtd.

The basic format is as follows:

<group name="group_name">

<parameter name="parameter_name" type="parameter_type">

<value>value</value>

</parameter>

</group>

Parameter names are case-sensitive, types are not. Parameter names and types must be enclosed in quotation marks (" ").

A parameter definition can span multiple lines. Spaces, carriage returns, line feeds, and tab characters are ignored in an element definition.

For example:

Copy Code
<group name="NotesConnection">
    <parameter name="NotesNameServer" type="string">
        <value>dominoserver</value>
    </parameter>
    <parameter name="NotesNameDatabase" type="string">
        <value>names.nsf</value>
    </parameter>
</group>
Tip:

For long parameter definitions, position values on separate lines and use indentation to make the file easier to read.

The FastConfig element is a special case and is required. All other elements are contained within the FastConfig element.

Here is a sample snippet from a configuration file:

Copy Code
<!DOCTYPE FastConfig SYSTEM "dtd/FastConnectorConfig-1.0.dtd">
<FastConfig>
    <group name="NotesConnection">
       <parameter name="NotesNameServer" type="string">
            <value>dominoserver</value>
       </parameter>
       <parameter name="NotesNameDatabase" type="string">
            <value>names.nsf</value>
       </parameter>
    </group>
</FastConfig>

You can add comments anywhere, delimited by <!-- and -->.

For example:

Copy Code
<!DOCTYPE FastConfig SYSTEM "dtd/FastConnectorConfig-1.0.dtd">
<FastConfig>
    <group name="NotesConnection">
       <!-- Trying out a different Domino server

       <parameter name="NotesNameServer" type="string">
            <value>dominoserver</value>
       </parameter>
      -->
       <parameter name="NotesNameServer" type="string">
            <value>dominoserver2</value>
       </parameter>
       <parameter name="NotesNameDatabase" type="string">
            <value>names.nsf</value>
       </parameter>
    </group>
</FastConfig>

Parameter group: NotesConnection

The parameters in this group specify how to connect to the Lotus Domino server for content extraction.

Parameter | Type | Value | Description
NotesNameServer | string | text | Specifies the host name of the Lotus Domino server.
NotesNameDatabase | string | text | Specifies the name of the database that has the .nsf extension. The database includes the users, groups, and certifiers to extract.
NotesPassword | string | text | The password that is used to authenticate to the Lotus Notes client. This must match the password that you last used to log on to the Lotus Notes client on this computer. The password must be encrypted. Encrypt it with the <FASTSearchFolder>\bin\connectorpwgen.bat utility.

The encrypted password cannot contain any new line or white-space characters; remove them when you copy the encrypted password from the console.

Note:

The password must be encrypted by the same user who will decrypt it (e.g., the user who will run the connector).

The name of the database that contains the Lotus Domino user directory is usually names.nsf.

Example

Copy Code
<group name="NotesConnection">
    <parameter name="NotesNameServer" type="string">
        <value>dominoserver.contoso.com</value>
    </parameter>
    <parameter name="NotesNameDatabase" type="string">
        <value>names.nsf</value>
    </parameter>
    <parameter name="NotesPassword" type="string">
        <value>AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAPF3E/uqo8kuy05Vfl6hBWAAAAABSAAAATQBpAGMAcgBvAHMAbwBmAHQAIABFAG4AdABlAHIAcAByAGkAcwBlACAAUwBlAGEAcgBjAGgAIAAtACAAQwBvAG4AbgBlAGMAdABvAHIAcwAAAANmAACoAAAAEAAAAN/ksvnBFepWFXzS9BFNY6AAAAAABIAAAKAAAAAQAAAAvbI2O5PXmb6wDzAeEfvNeRAAAABEqBeLrh8usMBfBhA7J8t1FAAAAD54sadCcKVwSRyxdU0TO6XjyCdH</value>
    </parameter>
</group>

Parameter group: ConnectorExecution

The parameter in this group specifies the logging behavior of the user directory connector.

Parameter Type Value Description

DocumentLoggingFrequency

integer

number of entities

The number of entities (users, groups and group memberships) processed before the connector logs a progress message.

If you have tens of thousands of entities, increase this value to avoid flooding the log with messages.

Default: 100

Example

Copy Code
<group name="ConnectorExecution">
    <parameter name="DocumentLoggingFrequency" type="integer">
        <value>100</value>
    </parameter>
</group>

Parameter group: FSAOutput

The parameters in this group specify where the connector should send its output. This includes log files and user store connection details.

Parameter | Type | Value | Description
OutputDirectory | string | text | The output folder for the connector. Only log files and the XML aliasing file are stored here. The folder is automatically created if it does not exist. The directory can be relative (to the bin folder) or absolute.
UserStoreHost | string | text | The host name of the server where the Lotus Notes user store is located. A user store contains group membership information about its users, serving as a security gateway to a third-party content repository.
UserStorePort | integer | portnumber | The port to the Lotus Notes user store.
SecurityDomainId | string | text | The ID of the Lotus Notes user store. This is a three-letter ID that distinguishes the Lotus Notes user store from other FAST Search Authorization (FSA) user stores. This ID must match the Normalization/SecurityDomainId value in the content connector configuration. Default: lnx

Normally, you do not need more than one user store for Lotus Notes. The default ID for Lotus Notes is lnx. Therefore, you rarely have to change the SecurityDomainId parameter.

Example

Copy Code
<group name="FSAOutput">
    <parameter name="OutputDirectory" type="string">
        <value>../var/lotusnotesconnector/security</value>
    </parameter>
    <parameter name="UserStoreHost" type="string">
        <value>localhost</value>
    </parameter>
    <parameter name="UserStorePort" type="integer">
        <value>13271</value>
    </parameter>
    <parameter name="SecurityDomainId" type="string">
        <value>lnx</value>
    </parameter>
</group>

Parameter group: SSOMapping

This group includes configuration parameters for generating an XML aliasing file for FAST Search Authorization (FSA) that lists the mapping between Active Directory users and Lotus Domino users. This allows an Active Directory user to perform a search that yields search results from Active Directory content and Lotus Domino databases that this user has access to.

To achieve these results, the user/group document in Lotus Domino must contain the Active Directory user name in one of its attributes.

After running the user directory connector and producing the XML aliasing file, you must manually import the file into FSA.

Parameter Type Value Description

UseSSOMapping

boolean

true|false

If set to true, an XML aliasing file is generated.

ADUserNameField

string

text

Specifies the name of the attribute in the user/group document (inside the Lotus Domino user directory) that contains the corresponding Active Directory user name.

Example: "ShortName"

This may also be on the form "FullName(N)" which means that the Active Directory user name will be one of the values in the FullName multi-value field. N specifies the index in FullName for retrieving the Active Directory user name. N=-1 means the last index.

XMLOutputFileName

string

text

Specifies the name of the generated XML aliasing file.

The file will be put in the folder that is specified in the FSAOutput/OutputDirectory parameter.

Example

The following example uses the default certificate generated by the installer:

Copy Code
<group name="SSOMapping">
    <parameter name="UseSSOMapping" type="boolean">
        <value>true</value>
    </parameter>
    <parameter name="ADUserNameField" type="string">
        <value>FullName(-1)</value>
    </parameter>
    <parameter name="XMLOutputFileName" type="string">
        <value>ssomapping.xml</value>
    </parameter>
</group>

See Also

Reference

lotusnotessecurity.bat reference

Concepts

Manage crawling with the FAST Search Lotus Notes user directory connector


lotusnotestemplate.xml reference
See Also

Use a copy of lotusnotestemplate.xml to change the settings for the FAST Search Lotus Notes connector.

The FAST Search Lotus Notes connector uses the configuration settings to control content extraction from Lotus Domino. You can run multiple instances of the FAST Search Lotus Notes connector at the same time if you use different configuration files for each instance.

Customizing lotusnotestemplate.xml

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

To extract content from a Lotus Notes database and feed it to a FAST Search Server 2010 for SharePoint installation, first create a copy of the lotusnotestemplate.xml file. Edit the copy in a text or XML editor of your choice (except Notepad) to include the elements and settings for your environment.

To test your settings, run:

<FASTSearchFolder>\bin\lotusnotesconnector.bat testconnections -f <configuration file>

where <configuration file> is the path and file name of your configuration, and <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

The FAST Search Lotus Notes connector connects to all external systems, such as the Lotus Domino server, the FAST Search Server 2010 for SharePoint installation, and the state database, to verify the connection parameters. The connector will not process any data.

To run the FAST Search Lotus Notes connector and feed content to FAST Search Server 2010 for SharePoint, run:

<FASTSearchFolder>\bin\ lotusnotesconnector.bat start -f <configuration file>

Parameter groups quick reference

The following table lists the parameter groups in lotusnotestemplate.xml. These groups can appear in any order.

Parameter group | Description
Parameter group: Connection | Specifies how to connect to the Domino server.
Parameter group: Filters | Specifies the content to crawl.
Parameter group: FASTSearchSubmit | Describes the properties used to communicate with FAST Search Server 2010 for SharePoint when submitting content.
Parameter group: XMLExport | Specifies the values for exporting to XML formatted file(s).
Parameter group: Normalization | Specifies how to transform/normalize items before the connector sends them to the content distributor.
Parameter group: ConnectorExecution | Specifies general values that were used when you run the connector.
Parameter group: StateTracker | Specifies the behavior of the state tracker database. It is used for incremental crawling.
Parameter group: General | Specifies the configuration name.

lotusnotestemplate file format

XML elements in lotusnotestemplate.xml begin with < and end with />. The XML is defined in FastConnectorConfig-1.0.dtd.

The basic format is as follows:

<group name="group_name">

<parameter name="parameter_name" type="parameter_type">

<value>value</value>

</parameter>

</group>

Parameter names are case-sensitive, types are not. Parameter names and types must be enclosed in quotation marks (" ").

A parameter definition can span multiple lines. Spaces, carriage returns, line feeds, and tab characters are ignored in an element definition.

For example:

Copy Code
<group name="ConnectorExecution">
    <parameter name="QueueSize" type="integer">
         <value>200</value>
    </parameter>
    <parameter name="WorkFolder" type="string">
         <value>../var/lotusnotesconnector</value>
    </parameter>
</group>
Tip:

For long parameter definitions, position values on separate lines and use indentation to make the file easier to read.

The <FastConfig> element is a special case and is required. All other elements are contained within the <FastConfig> element, and the element is closed with </FastConfig>.

Here is a sample snippet from a configuration file:

Copy Code
<!DOCTYPE FastConfig SYSTEM "dtd/FastConnectorConfig-1.0.dtd"><FastConfig>
    <group name="ConnectorExecution">
       <parameter name="QueueSize" type="integer">
             <value>200</value>
        </parameter>
       <parameter name="WorkFolder" type="string">
             <value>../var/lotusnotesconnector</value>
       </parameter>
    </group>
</FastConfig>

You can add comments anywhere, delimited by <!-- and -->.

For example:

Copy Code
<!DOCTYPE FastConfig SYSTEM "dtd/FastConnectorConfig-1.0.dtd"><FastConfig>
    <!-- The parameter group ConnectorExecution specifies general values
          used when running a connector. -->
    <group name="ConnectorExecution">
       <parameter name="QueueSize" type="integer">
             <value>200</value>
       </parameter>
       <parameter name="WorkFolder" type="string">
             <value>../var/lotusnotesconnector</value>
       </parameter>
    </group>
</FastConfig>

Parameter group: Connection

The parameters in this group specify the connection to the Lotus Domino server.

Parameter Type Value Description

Hostname

string

text

The host name of the Domino server.

This can be a host name, a fully qualified domain name, the Domino server name (in the form <host>/<domain>), or "localdisk".

The reserved keyword "localdisk" instructs the connector to look for databases on the local Lotus Notes client.

Password

string

text

The encrypted password that is used to authenticate to the Lotus Notes client.

This must match the password that you last used to log on to the Lotus Notes client on this computer. The password must be encrypted. Encrypt it with the <FASTSearchFolder>\bin\connectorpwgen.bat utility.

The encrypted password cannot contain any new line or white-space characters; remove them when you copy the encrypted password from the console.

Note:

The password must be encrypted by the same user who will decrypt it (e.g., the user who will run the connector).

Example

Copy Code
<group name="Connection">
    <parameter name="Hostname" type="string">
        <value>dominoserver.company.com</value>
    </parameter>
    <parameter name="Password" type="string">
        <value>AQAAANCMnd8BFdERjHoAwE/Cl+sBAAAAPF3E/uqo8kuy05Vfl6hBWAAAAABSAAAATQBpAGMAcgBvAHMAbwBmAHQAIABFAG4AdABlAHIAcAByAGkAcwBlACAAUwBlAGEAcgBjAGgAIAAtACAAQwBvAG4AbgBlAGMAdABvAHIAcwAAAANmAACoAAAAEAAAAN/ksvnBFepWFXzS9BFNY6AAAAAABIAAAKAAAAAQAAAAvbI2O5PXmb6wDzAeEfvNeRAAAABEqBeLrh8usMBfBhA7J8t1FAAAAD54sadCcKVwSRyxdU0TO6XjyCdH</value>
    </parameter>
</group>

Parameter group: Filters

The parameters in this group specify which content to crawl, through several include and exclude filters.

Parameter Type Value Description

DataBase

string

text

A list of databases to crawl. May contain view names.

Specify a view in the format: <database path>#viewname. Separate multiple databases with a semicolon (;).

When you specify a view, only the search formula of the view is used.

ViewName

string

text

Specifies a particular view on several databases at the same time.

Use if the view is not specified in the DataBase parameter.

When you specify a view, only the search formula of the view is used.

IncludeDataBases

string

text

Lists one or more include filters, regular expressions, for database names/paths. For example: Attach.*\.nsf;help\/.*\.nsf.

Separate multiple include filters with a semicolon (;).

Use when the DataBase parameter is empty, and in combination with the ExcludeDatabases filter.

ExcludeDataBases

string

text

Lists one or more exclude filters, regular expressions, for database names/paths. For example: mail\\.*\.nsf;schema.nsf;names.nsf;help.*\.nsf.

Separate multiple exclude filters with a semicolon (;).

Use when the DataBase parameter is empty, and in combination with the IncludeDatabases filter.

This parameter overrides the include filter; use it to narrow included content.

IncludeAttachmentExtensions

string

text

List of file name extensions for attachments that should be indexed. Separate multiple include filters with a semicolon (;).

If both IncludeAttachmentExtension and ExcludeAttachmentExtensions are specified, ExcludeAttachmentExtensions will be ignored.

ExcludeAttachmentExtensions

string

text

List of file name extensions for attachments that should not be indexed. Separate multiple exclude filters with a semicolon (;).

If both IncludeAttachmentExtension and ExcludeAttachmentExtensions are specified, ExcludeAttachmentExtensions will be ignored.

MaxAttachmentSize

integer

size

Specifies the maximum size of attachments, in kilobytes.

Default: 10000

SearchQuery

string

text

Specifies a query using the Notes Query Language.

For example: SELECT Form="Item"

The query is applied to any database that has no view specified.

If any of these parameters is changed so that items that were indexed previously no longer should be indexed, the corresponding items will not be removed from the index. In this case, you should clear the state tables by setting the StateTracker/PurgeAtStart parameter to true. You must also delete all the items in the FAST Search Server 2010 for SharePoint search index by deleting/purging the collection.

Example

Copy Code
<group name="Filters">
    <parameter name="DataBase" type="string" separator=";">
        <value></value>
    </parameter>
    <parameter name="ViewName" type="string">
        <value></value>
    </parameter>
    <parameter name="IncludeDataBases" type="string" separator=";">
        <value>help\\.*</value>
    </parameter>
    <parameter name="ExcludeDataBases" type="string" separator=";">
        <value>help\\.*api.*</value>
    </parameter>
    <parameter name="IncludeAttachmentExtensions" type="string" separator=";">
        <value>doc;xls;ppt;pdf</value>
    </parameter>
    <parameter name="ExcludeAttachmentExtensions" type="string" separator=";">
        <value></value>
    </parameter>
    <parameter name="MaxAttachmentSize" type="integer">
        <value>10000</value>
    </parameter>
    <parameter name="SearchQuery" type="string">
        <value>SELECT Form="Item"</value>
    </parameter>
</group>

Parameter group: FASTSearchSubmit

The parameters in this group describe communication properties for submitting content to FAST Search Server 2010 for SharePoint.

Parameter | Type | Value | Description
ActuallyPublish | boolean | true|false | If true, submits items to FAST Search Server 2010 for SharePoint. If false, the connector will crawl content from Lotus Domino. However, it will not submit it to the search index. Use this setting for test runs of other parameters (e.g., parameter settings in the Filters group). Default: true
Collection | string | text | Name of the FAST Search Server 2010 for SharePoint crawl collection to feed.

Example

Copy Code
<group name="FASTSearchSubmit">
    <parameter name="ActuallyPublish" type="boolean">
        <value>true</value>
    </parameter>
    <parameter name="Collection" type="string">
        <value>lotusnotes</value>
    </parameter>
</group>

Parameter group: XMLExport

The parameters in this group specify an export to XML formatted file(s). Export content to XML during testing, to see the contents.

Only text data is exported to XML; binary content is not exported.

Parameter | Type | Value | Description
ExportToXML | boolean | true|false | If true, exports all items to XML. Default: false

The exported XML is stored in <FASTSearchFolder>\var\lotusnotesconnector\<collection name>\xml.

Example

Copy Code
<group name="XMLExport">
    <parameter name="ExportToXML" type="boolean">
        <value>true</value>
   </parameter>
</group>

Parameter group: Normalization

The parameters in this group specify how to transform/normalize items before submitting them to the content index.

Parameter Type Value Description

FixedFields

string

text

Assigns fixed fields and field values to all crawled items. This is a list of semicolon separated field names/value pairs in the form: field1:value1;field2:value2.

The fields are added exactly as written. They are case-sensitive.

DominoWebServerPrefix

string

text

Specifies the Domino Web server that search results should link to.

If this Domino Web server differs from the server being crawled, this parameter will override the item URLs with those of the Domino Web server.

SecurityDomainId

string

text

Specifies the security domain ID for Lotus Notes (the ID provided when the Lotus Notes user store was created).

The domain ID should match the Identity property in the Lotus Notes user store cmdlet, and the FSAOutput/SecurityDomainId in the Lotus Notes user directory configuration file.

DateTimeFormat

string

text

Modifies the parsing of ACL change time stamps.

You should not have to change this parameter. The connector automatically parses time stamps the last time that database security changed. However, if the connector reprocesses databases that have not been modified since the previous run, provide a date-time format in this parameter.

Example date-time formats:

  • dd/MM/yyyy hh:mm:ss aa

  • MM/dd/yyyy hh:mm:ss aa

  • yyyy/MM/dd hh:mm:ss aa

Leave the Normalization/DateTimeFormat parameter empty, unless the connector is crawling a Lotus Notes database that has a different locale than the Domino server. For example: the Domino server has a locale with a time stamp in the form: "28/07/2009 14:56:34", but the database was created on a Domino server that has a locale where the same time stamp is formatted as: "07/28/2009 02:56:34 PM". In this case, the parameter should contain: "MM/dd/yyyy kk:mm:ss aa".

Example

Copy Code
<group name="Normalization">
    <parameter name="FixedFields" type="string">
        <value></value>
    </parameter>
    <parameter name="DominoWebServerPrefix" type="string">
        <value>http://dominowebserver.company.com</value>
    </parameter>
    <parameter name="SecurityDomainId" type="string">
        <value>lnx</value>
    </parameter>
    <parameter name="DateTimeFormat" type="string">
        <value></value>
    </parameter>
</group>

Parameter group: ConnectorExecution

The parameters in this group specify general values that were used during a connector run.

Parameter | Type | Value | Description
QueueSize | integer | 1 - 10000 | Specifies the size of the internal queue of items. Increasing this value may increase performance but it may also increase memory consumption of the connector. Default: 200
DocumentLoggingFrequency | integer | 10 - 50000 | Specifies the number of items processed before the connector logs a message about how many items it has processed. Default: 1000
WorkFolder | string | Folder name | Specifies the work folder (either a relative or absolute file path) for all file output. This includes log files and XML output files. The folder is automatically created if it does not exist. Multiple connector configurations should have different values for this parameter. Default: ../var/lotusnotesconnector
NumAdapters | integer | 1 - 32 | Specifies several adapter threads, or how many databases the connector crawls in parallel. Increasing this value will increase performance of the connector but it may also increase memory consumption. Default: 3
AdapterThrottleSleepMS | integer | 0 - 10000 | Specifies the number of milliseconds the adapter should wait between fetching each item from the source. Use to throttle the feeding rate of this connector instance. Default: 0
EnableStatustracker | boolean | true|false | If true, logs the item crawl status to the database. If false, disables status tracking. Disabling tracking will reduce the load on the state database. But you will be unable to check the status of individual items using SQL Server Reporting Services.

You do not have to change these settings unless you are running multiple connector configurations.

Example

Copy Code
<group name="ConnectorExecution">
    <parameter name="QueueSize" type="integer">
        <value>200</value>
    </parameter>
    <parameter name="DocumentLoggingFrequency" type="integer">
        <value>1000</value>
    </parameter>
    <parameter name="WorkFolder" type="string">
        <value>../var/lotusnotesconnector2</value>
    </parameter>
    <parameter name="NumAdapters" type="integer">
        <value>4</value>
    </parameter>
    <parameter name="AdapterThrottleSleepMS" type="integer">
        <value>0</value>
    </parameter>
    <parameter name="EnableStatustracker" type="boolean">
        <value>false</value>
    </parameter>
</group>

Parameter group: StateTracker

The parameters in this group specify the behavior of the state tracker database, used for incremental crawling.

Parameter | Type | Value | Description
PurgeAtStart | boolean | true|false | If true, all data in the state tracker tables is purged at startup, telling the connector to retrieve everything, even if it was crawled previously. Use when testing your configuration. Set to false when in production. Default: false

Example

Copy Code
<group name="StateTracker">
    <parameter name="PurgeAtStart" type="boolean">
        <value>false</value>
    </parameter>
</group>

Parameter group: General

Specifies the configuration name.

Parameter Type Value Description

Name

string

text

Specifies the configuration name.

Use only when you have multiple databases with the same fields that should be mapped to different managed properties.

The name must be unique, with a maximum length of 32 characters, in the range a-f and 0-9.

Names that are less than 32 characters will be padded to construct a GUID of 32 characters. This propset-ID GUID identifies the source of crawled properties as they enter the search engine, for mapping crawled properties to managed properties.

If Name is blank, a predefined Lotus Notes propset-ID and the predefined mapping will be used, which can be extended.

You do not have to enter a value for this parameter unless you have multiple databases with the same field that should be mapped to different managed properties via separate mappings.

For example: Database a.nsf and b.nsf both contain a field that is named "lastmodified". In a.nsf, this field is a string specifying the user who last modified the database. In b.nsf, this field is a datetime and contains the time stamp when the database was last modified. For a.nsf, "lastmodified" maps to the managed property "modifiedby". For b.nsf, "lastmodified" maps to the managed property "lastmodifiedtime". Two different connector configurations are required for crawling these two databases.

Example

Copy Code
<group name="General">
    <parameter name="Name" type="string">
        <value>abcdef01</value>
    </parameter>
</group>

See Also

Reference

lotusnotesconnector.bat reference

Concepts

Manage crawling with the FAST Search Lotus Notes content connector


monitoringservice.xml reference

Use monitoringservice.xml to configure WMI classes and performance counters published by the Microsoft FAST Search Server 2010 for SharePoint monitoring service.

Use the monitoringserviceconfig.exe utility to apply file changes and to synchronize the configuration, WMI classes, and performance counters registered in the Windows operating system.

You will rarely have to edit this file, because it requires advanced knowledge of FAST Search Server 2010 for SharePoint internals.

The FAST Search Server 2010 for SharePoint monitoring service reads this configuration file when it starts and uses the configuration to do the following tasks:

  • Retrieve data from the declared process sources

  • Publish the metrics data into the declared WMI class and performance counter target objects

monitoringserviceconfig.exe uses this file to register the WMI and performance counter objects in the Windows operating system. This step is a prerequisite for the monitoring service to function, and must be performed every time that monitoringservice.xml is changed.

Customizing monitoringservice.xml

Note:

To modify a configuration file, verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  1. Use a text or XML editor to change monitoringservice.xml.

  2. Run monitoringserviceconfig /checkconfig to validate the configuration.

  3. Run monitoringserviceconfig /install to apply the new configuration. You must temporarily stop the FAST Search Server 2010 for SharePoint Monitoring service (FASTSearchMonitoring) when you install the new configuration.

Element quick reference

monitoringservice.xml uses the following elements:

Element | Description
<configuration> | This is the top level element
<nameserver> | Settings that control how the monitoring service polls the middleware name server
<nodecontroller> | Settings that control how the monitoring service polls the node controller
<processes> | List of processes (metrics sources) with <process> metrics declarations
<wmi> | List of WMI classes (metrics targets) with WMI <class> and <value> (attribute) declarations
<perfctr> | List of performance counter categories (metrics targets) with <category> and <counter> declarations

monitoringservice.xml file format

monitoringservice.xml is an XML file and follows all conventions and syntax rules that are defined for the XML file format.

The basic structure of the file is as follows:

Copy Code
<?xml version="1.0" encoding="utf-8" ?>
<configuration .. >

  <nameserver     .. />
  <nodecontroller .. />

  <processes>
    <process .. />
    <process .. />
    ...
    ...
  </processes>

  <wmi>
    <class .. />
    <class .. />
    ...
    ...
  </wmi>

  <perfctr>
    <category .. />
    <category .. />
    ...
    ...
  </perfctr>

</configuration>

The process declarations serve as the sources for metrics data. The WMI class elements and performance counter category elements serve as the destinations.

The declarations in process elements are tied to FAST Search Server 2010 for SharePoint internals, and should not be edited.

Warning:

The default declarations in WMI class and performance counter category elements make up the official external monitoring interfaces of FAST Search Server 2010 for SharePoint; editing these elements can cause integration with other products to fail (e.g., the FAST Search Server 2010 for SharePoint management pack for System Center Operations Manager 2007).

configuration

This is the top level element.

Attributes

Attribute | Value | Description
version | 3 | The configuration file version for FAST Search Server 2010 for SharePoint

nameserver

Contains attributes that control how the middleware name server is polled

Attributes

Attribute | Value | Description
refresh_interval | <seconds> | Number of seconds to wait between each request to the middleware name server
timeout | <seconds> | Request timeout (default: 7 seconds)

nodecontroller

Contains attributes that control how the node controller is polled

Attributes

Attribute | Value | Description
refresh_interval | <seconds> | Number of seconds to wait between each request to the node controller
timeout | <seconds> | Request timeout (default: 10 seconds)

processes

Contains a list of process elements

process

Contains declarations of metrics sources, to control the collection of monitoring performance data and statistics

Attributes

Attribute | Value | Description
name | <string> | Process name

wmi

Contains a list of WMI class declarations

class

WMI class declaration; contains value elements that have class attribute declarations

Attributes

Attribute | Value | Description
name | <string> | WMI class name
description | <string> | WMI class description
instance_source | <string> | A string matching the symbol attribute of the corresponding process source

Example

Copy Code
<class name="Indexer" description="FAST Search Indexer" instance_source="indexer">
  <value symbol="rowId" name="RowId" type="int" />
  <value .. />
  ...
  ...
</class>

value

WMI class attribute declaration

Attributes

Attribute Value Description

symbol

<string>

This attribute is used to look up a corresponding value or attr element (the source for metrics data). Matching is performed against the symbol attribute of that value or attr element.

name

<string>

WMI attribute name

type

<data_type>

Valid values:

  • int - Integer data type (32-bit)

  • long - Long integer data type (64-bit)

  • bool - Boolean data type

  • string - String data type

  • float - Floating point data type

Example

Copy Code
<value symbol="rowId" name="RowId" type="int" />

perfctr

Contains a list of performance counter category declarations

category

Performance counter category declaration; contains counter elements that have class attribute declarations

Attributes

Attribute | Value | Description
name | <string> | Performance counter category name
description | <string> | Performance counter category description
instance_source | <string> | A string matching the symbol attribute of the corresponding process source

Example

Copy Code
<category name="FAST Search Indexer" description="FAST Search Indexer" instance_source="indexer">
  <counter symbol="rowId" name="Row id" type="NumberOfItems32" />
  <counter .. />
    ...
    ...
</category>

counter

Performance counter declaration

Attribute Value Description

symbol

<string>

This attribute is used to look up a corresponding value or attr element (the source for metrics data). Matching is performed against the symbol attribute of that value or attr element.

name

<string>

Performance counter name

type

<data_type>

  • NumberOfItems32 - Integer data type (32-bit)

  • NumberOfItems64 - Long integer data type (64-bit)

  • CounterDelta32 - Counter type displaying the difference between the current measurement and previous measurement (32-bit sources)

  • CounterDelta64 - 64-bit version

  • RateOfCountsPerSecond32 - Counter type displaying the difference between the current and previous measurements divided by the time difference between the two measurements (32-bit sources)

  • RateOfCountsPerSecond64 - 64-bit version

Example

Copy Code
<counter symbol="rowId" name="Row id" type="NumberOfItems32" />


OffensiveContentFilter (item processing stage) reference

The Microsoft FAST Search Server 2010 for SharePoint offensive content filtering is implemented as a separate item processing stage. Item content that is run through the filter is compared to predefined terms in dictionaries. The output of the filter is an overall score that indicates the likeliness that an item is pornographic. The item's offensive score is written to the crawled property OCF::Score. Any item that exceeds the score threshold of 30 will be dropped from indexing.

For more information, see Offensive Content Filtering on MSDN.


optionalprocessing.xml reference

Optional stages in item processing represent processing components that you can turn on or off in the Microsoft FAST Search Server 2010 for SharePoint item processing pipeline, and include property extraction, data mapping, linguistic processing, and content filtering.

For detailed information, see Configuring Optional Item Processing on MSDN.


pipelineextensibility.xml reference

The item processing pipeline prepares an item from a content source for indexing and searching. This preparation includes text extraction, language detection, and tokenization.

Some applications may require extensions to the item processing. You can generate additional searchable metadata from the content, or forward a piece of information from the item to a third-party application for statistical or monitoring purposes.

For more information, see Integrating an External Item Processing Component on MSDN.


user_converter_rules.xml reference

Microsoft FAST Search Server 2010 for SharePoint can crawl and index a large number of file types out of the box or by enabling the Advanced Filter Pack. For more information, consult the articles Include a file type in the content index (FAST Search Server 2010 for SharePoint) and IFilter and file type reference (FAST Search Server 2010 for SharePoint).

Some file types require the use of a third-party IFilter. If you are using a third-party IFilter, you have to register the IFilter with Windows Search, you have to configure the file user_converter_rules.xml to process items correctly, and you have to make sure that the file type is not excluded by the FAST Search Content Search Service Application.

The article Configure FAST Search Server for SharePoint to use a Third-Party IFilter on MSDN describes how to install and register the third-party IFilter, and contains information about how to configure the file user_converter_rules.xml to make sure that content in the file format of the third-party IFilter is processed correctly.


xmlmapper.xml reference

The custom XML item processing feature enables you to map XML-based items to managed properties. A typical scenario is to use the contents of certain XML elements or attributes as metadata for crawled items. For example, use text extracted from an XML element as the title of an item.

For detailed information, see Custom XML Item Processing on MSDN.


About WMI classes and performance counters (FAST Search Server 2010 for SharePoint)
See Also

Windows Management Instrumentation (WMI) provides a consistent model of the managed environment. For each manageable resource in FAST Search Server 2010 for SharePoint, there is a corresponding WMI class.

Another method to monitor FAST Search Server 2010 for SharePoint is to view information that is reported through performance counters.

In this section:

  • WMI classes (FAST Search Server 2010 for SharePoint)

  • Performance counters (FAST Search Server 2010 for SharePoint)

See Also

Concepts

Monitor FAST Search Server 2010 for SharePoint using WMI
Monitor FAST Search Server 2010 for SharePoint using performance counters


WMI classes (FAST Search Server 2010 for SharePoint)

This topic gives an overview of the available Windows Management Instrumentation (WMI) classes for Microsoft FAST Search Server 2010 for SharePoint.

FAST Search Server 2010 for SharePoint WMI classes

WMI Class - "Indexer"

WMI Class - "CrawlerCollection"

WMI Class - "QRServer"

WMI Class - "Collection"

WMI Class - "DocumentProcessor"

WMI Class - "ContentDistributor"

WMI Class - "IndexingDispatcher"

WMI Class - "WebAnalyzer"

WMI Class - "Crawler"

WMI Class - "Dataset"

WMI Class - "FDispatch"

WMI Class - "IndexerPartition"

WMI Class - "IndexerStatus"

WMI Class - "BEStatistics"

WMI Class - "NodeConfGlobal"

WMI Class - "NodeConfProcess"

WMI Class - "AccessControlCoordinator"

WMI Class - "AccessControlNonCachedRequestAverage"

WMI Class - "AccessControlNonCachedRequestCount"

WMI Class - "AccessControlRequestAverage"

WMI Class - "AccessControlRequestCount"

WMI Class - "DomainWMIProxy"

WMI Class - "PrincipalAliaserWMIProxy"

WMI Class - "Indexer"

Name | Description | Type
APIOperations | Indicates the total number of API operations (documents indexed and updated) that the indexer has processed. | int
APIOperPerSec | Indicates the average number of processed API operations per second by the indexer. | float
APIPartialUpdateOper | Indicates the number of API operations processed by the indexer that are partial update operations. | int
APIQueueLoad | Shows the current size (in bytes) of the API queue divided by the maximum allowed size of the API queue (in bytes). | float
APIQueueSize | Indicates the number of batches that are currently in the API queue. The API queue is used by the indexer to hold pending operations that are waiting to be persisted. | int
APIRemOperations | Indicates the number of API operations processed by the indexer that are remove operations. | int
APIStatusUpdateOper | Indicates the number of API operations processed by the indexer that are status updates (failed_operations). | int
CachePercentagePart0 | Indicates the percentage of the total cache given to partition 0 used by the search process. | int
CachePercentagePart1 | Indicates the percentage of the total cache given to partition 1 used by the search process. | int
CachePercentagePart2 | Indicates the percentage of the total cache given to partition 2 used by the search process. | int
CachePercentagePart3 | Indicates the percentage of the total cache given to partition 3 used by the search process. | int
CachePercentagePart4 | Indicates the percentage of the total cache given to partition 4 used by the search process. | int
ColectionsRemoved | Indicates the number of API operations processed by the indexer that are remove collection operations. | int
ColumnId | Indicates which column number the indexer is part of. | int
ColumnRole | Indicates if the indexer is a primary or a backup indexer (fault tolerance). | string
DiskLow | Indicates if the indexer has little disk to work with to persist documents and to build new indices. | int
DiskLowState | Indicates if the indexer is in a state where it has little disk to work with to persist documents and to build new indices. | bool
DocsIndexed | Indicates the number of API operations processed by the indexer that are add or update operations. | int
DocsInIndexer | Indicates the total number of documents in the indexer. This is the current total number of valid documents contained in the docindex of the indexer. The docindex is a mapping to the persisted documents on disk, so it translates the number of valid documents persisted on disk (fixml). | int
ExcludedDocsPart0 | Indicates the number of documents on the exclusion list in the currently active index on partition 0. This translates to documents that are in the index, but that should not be searchable (deleted, present in other partitions, etc). | int
ExcludedDocsPart1 | Indicates the number of documents on the exclusion list in the currently active index on partition 1. This translates to documents that are in the index but that should not be searchable (deleted, present in other partitions, etc). | int
ExcludedDocsPart2 | Indicates the number of documents on the exclusion list in the currently active index on partition 2. This translates to documents that are in the index but that should not be searchable (deleted, present in other partitions, etc). | int
ExcludedDocsPart3 | Indicates the number of documents on the exclusion list in the currently active index on partition 3. This translates to documents that are in the index but that should not be searchable (deleted, present in other partitions, etc). | int
ExcludedDocsPart4 | Indicates the number of documents on the exclusion list in the currently active index on partition 4. This translates to documents that are in the index but that should not be searchable (deleted, present in other partitions, etc). | int
FailedOperations | Indicates the number of API operations that could not be processed successfully. | int
FixmlDir | Location of the directory where the FiXML is stored. This directory is used by the indexer to persist the new documents to disk. The directory is also used by the indexer processes when building the index itself from disk (FiXML). | string
FixmlFillRate | Indicates an approximation of the ratio between the number of valid documents in a persisted file (FiXML file) and the total number of persisted documents. A value close to 100% means that the files on disk are well compacted together. | float
HeartbeatStatus | Indicates the status of the heartbeat thread: if it is running, suspended or in an unknown state. In a fault tolerant deployment the heartbeat thread periodically checks for the state of the indexer in the column. | string
IndexDir | Indicates where the binary indices are located on disk. | string
IndexecedDocsPerSecPart0 | Indicates the average number of documents that are indexed per second in partition 0. | float
IndexecedDocsPerSecPart1 | Indicates the average number of documents that are indexed per second in partition 1. | float
IndexecedDocsPerSecPart2 | Indicates the average number of documents that are indexed per second in partition 2. | float
IndexecedDocsPerSecPart3 | Indicates the average number of documents that are indexed per second in partition 3. | float
IndexecedDocsPerSecPart4 | Indicates the average number of documents that are indexed per second in partition 4. | float
IndexingSuspendedState | Indicates if the indexer has currently suspended indexing. This means that the indexer is not creating new indices, even though it might be persisting new documents to disk. | bool
LastBatchSize | Indicates the size (in bytes) of the previous batch processed by the indexer. | int
LastIndexSwitch | Indicates the time (in time_t) when the last index was finalized and activated on the indexer. | int
LastProcDate | Indicates the time when the latest operation was processed by the indexer in the sequence_builder. | string
LastProcOperation | Indicates the identifier of the last operation that was processed by the indexer in the sequence_builder. | string
ManualFTSyncNeeded | Indicates if a manual fault-tolerant sync is needed, and is always set to false. | bool
ManualResetState | Indicates when the indexer is in a reset index state manually performed by the user. | bool
MaxBatchSize | Indicates the size of the largest batch (in bytes) as seen during processing by the indexer. | int
MaxCacheKBPart0 | Indicates the maximum cache (in kilobytes) used by the search processes for partition 0. | int
MaxCacheKBPart1 | Indicates the maximum cache (in kilobytes) used by the search processes for partition 1. | int
MaxCacheKBPart2 | Indicates the maximum cache (in kilobytes) used by the search processes for partition 2. | int
MaxCacheKBPart3 | Indicates the maximum cache (in kilobytes) used by the search processes for partition 3. | int
MaxCacheKBPart4 | Indicates the maximum cache (in kilobytes) used by the search processes for partition 4. | int
MaxDocsPerSec | Indicates the highest achieved ratio between the number of indexed documents in an index job and the time it took to perform the job. | float
MinDocsPerSec | Indicates the lowest achieved ratio between the number of indexed documents in an index job and the time it took to perform the job. | float
NumDocsInPart0 | Indicates the number of searchable documents in the current active index on partition 0. | int
NumDocsInPart1 | Indicates the number of searchable documents in the current active index on partition 1. | int
NumDocsInPart2 | Indicates the number of searchable documents in the current active index on partition 2. | int
NumDocsInPart3 | Indicates the number of searchable documents in the current active index on partition 3. | int
NumDocsInPart4 | Indicates the number of searchable documents in the current active index on partition 4. | int
OngoingFTSequenceRecovery | Indicates if the ongoing fault-tolerant sequence recovery is used in order to find out if an indexer is currently recovering operations from other indexers in the column. | bool
PerformingResetIndexState | Indicates if the indexer currently is in a reset index state, meaning that it is indexing documents from disk (file_id), and not performing incremental indexing. | bool
RegFileReceivers | Indicates the number of file receivers currently registered on the indexer. This can apply to both backup indexers and search controllers. | int
RegSearchControllers | Indicates the number of search controllers currently registered on the indexer. These search controllers register to get new indices over from the indexer. | int
RowId | Indicates which row number the indexer is part of. | int
SCtlCurrAlloc | Indicates how many search_controller_stubs are currently allocated by the indexer, and translates how many search_controllers the indexer is communicating with. | int
SCtlTotalAlloc | Indicates how many times a search controller has been allocated on the indexer side. The search_controller_stub is used to communicate with a remote search_controller. | int
SizeOfDocuments | Indicates the total size of documents (in bytes) of all the valid documents persisted to disk (FiXML). | long
TotalIdxRunsPart0 | Indicates the total number of times that a new index has been built on partition 0. | int
TotalIdxRunsPart1 | Indicates the total number of times that a new index has been built on partition 1. | int
TotalIdxRunsPart2 | Indicates the total number of times that a new index has been built on partition 2. | int
TotalIdxRunsPart3 | Indicates the total number of times that a new index has been built on partition 3. | int
TotalIdxRunsPart4 | Indicates the total number of times that a new index has been built on partition 4. | int

WMI Class - "CrawlerCollection"

Applies to the FAST Search Web crawler only.

Name | Description | Type
ActiveSites | The number of sites that is being crawled right now. | long
ContentIndexingAdded | The number of ADD operations that is sent to content indexing. | long
ContentIndexingDeleted | The number of DELETE operations that is sent to content indexing. | long
ContentIndexingFailed | The total number of failed operations. | long
ContentIndexingModified | The number of MOD operations that is sent to content indexing. | long
ContentIndexingOperations | The total number of operations that is submitted to content indexing. | long
ContentIndexingSucceeded | The number of operations that is sent to content indexing that succeeded. | long
ContentIndexingUrlschanges | The number of urlschange operations sent to content indexing. | long
DataReadRate | Current network read rate (in bytes per second). | float
DataWriteRate | Current network write rate (in bytes per second). | float
DocRate | Current document download rate (in documents per second). | float
DocsInCrawlerStore | The number of documents (approximate) in the FAST Search Web crawler store. | long
DocSize | Total document size in bytes. | long
DocSizeAverage | Average document size in bytes. | long
DocSizeMaximum | Maximum document size in bytes. | long
DocsSkippedChecksumCache | The number of skipped documents due to the documents being screened by the run-time checksum cache (used for duplicate detection). | long
DocsSkippedChunkError | The number of skipped documents due to problems with chunked encoding. Failure to de-chunk content. | long
DocsSkippedConnectError | The number of skipped documents due to failure to connect to the remote Web server. | long
DocsSkippedConnectTimeout | The number of skipped documents due to a time out of the connection attempt to the remote Web server. | long
DocsSkippedEmptyDocument | The number of skipped documents because their size was 0 bytes. | long
DocsSkippedEncoding | The number of skipped documents due to problems with the document encoding. This typically indicates that there are problems with compressed content. | long
DocsSkippedHeaderExclude | The number of skipped document because they matched the 'exclude_headers' setting in the collection. | long
DocsSkippedHTTPHeader | The number of skipped documents due to errors with HTTP header. | long
DocsSkippedIncomplete | The number of skipped documents because they were incomplete. The Web server did not return the complete document as indicated by the HTTP header. | long
DocsSkippedMIMEType | The number of skipped documents because they did not match the 'allowed_types' setting in the collection. | long
DocsSkippedNetworkError | The number of skipped documents due to network errors. | long
DocsSkippedNoIndex | The number of skipped documents due to containing a META robots No Index tag. | long
DocsSkippedNoIndexRSS | The number of skipped documents because RSS feed indexing was disabled by the 'index_feed' setting. | long
DocsSkippedNoXXXxTarget | The number of skipped documents due to the documents not having a redirect target (but did have a redirect return code returned by the Web server). | long
DocsSkippedOtherError | The number of skipped documents due to other reasons. | long
DocsSkippedTimeout | The number of skipped documents because it took longer time to download them than the 'fetch_timeout' setting allowed. | long
DocsSkippedTooLarge | The number of skipped documents due to exceeding the collection 'cut_off' setting. | long
DocumentsDeleted | The number of documents deleted from the document store. | long
DocumentsDownloaded | The number of documents that were downloaded. | long
DocumentsModified | The number of stored documents that were modified. | long
DocumentsRequested | The number of URIs that the FAST Search Web crawler attempted to download. | long
DocumentsStored | The number of document that were written to the FAST Search Web crawler store. | long
DocumentsUnchanged | The number of documents that were downloaded that were unchanged. | long
DownloadAvgTime | Indicates the average download time. | float
DownloadMaxTime | Indicates the maximum download time. | float
DownloadTotTime | Indicates the total download time. | float
FeedingStatus | Specifies whether content submission to indexing is active (1) or paused (0). | long
FirstUpdateTime | Time of the initial statistics update (timestamp). | long
HTTPResponse200 | The number of times the FAST Search Web crawler received the response HTTP 200 (OK). | long
HTTPResponse226 | The number of times the FAST Search Web crawler received the response FTP 226 (Successful). | long
HTTPResponse300 | The number of times the FAST Search Web crawler received the response HTTP 300 (Multiple Choices). | long
HTTPResponse301 | The number of times the FAST Search Web crawler received the response HTTP 301 (Moved Permanently). | long
HTTPResponse302 | The number of times the FAST Search Web crawler received the response HTTP 302 (Found). | long
HTTPResponse303 | The number of times the FAST Search Web crawler received the response HTTP 303 (See Other). | long
HTTPResponse304 | The number of times the FAST Search Web crawler received the response HTTP 304 (Not Modified). | long
HTTPResponse307 | The number of times the FAST Search Web crawler received the response HTTP 307 (Temporary Redirect). | long
HTTPResponse400 | The number of times the FAST Search Web crawler received the response HTTP 400 (Bad Request). | long
HTTPResponse401 | The number of times the FAST Search Web crawler received the response HTTP 401 (Unauthorized). | long
HTTPResponse403 | The number of times the FAST Search Web crawler received the response HTTP 403 (Forbidden). | long
HTTPResponse404 | The number of times the FAST Search Web crawler received the response HTTP 404 (Not Found). | long
HTTPResponse406 | The number of times the FAST Search Web crawler received the response HTTP 406 (Not Acceptable). | long
HTTPResponse407 | The number of times the FAST Search Web crawler received the response HTTP 407 (Proxy Authentication Required). | long
HTTPResponse426 | The number of times the FAST Search Web crawler received the response FTP 426 (Transfer Aborted). | long
HTTPResponse500 | The number of times the FAST Search Web crawler received the response HTTP 500 (Internal Server Error). | long
HTTPResponse503 | The number of times the FAST Search Web crawler received the response HTTP 503 (Service Unavailable). | long
LastRefreshTime | Indicates the last time the collection was refreshed (timestamp). | long
LastUpdateTime | Indicates the last time the statistics were updated (timestamp). | long
LinksFwdNodeScheduler | The number of URIs/links forwarded to the Node Scheduler. | long
LinksFwdSiteManager | The number of URIs/links forwarded within a Site Manager. | long
LinksFwdSiteWorker | The numer of URIs/links forwarded within a Site Worker. | long
NetworkBytesReceived | The number of bytes (HTTP, HTTPS and FTP) that were read from the network. | long
NetworkBytesSent | The number of bytes (HTTP, HTTPS and FTP) that were written to the network. | long
NumMimeAtomXml | The number of documents with MIME type: application/atom+xml | long
NumMimeExcel | The number of documents with MIME type: application/msexcel | long
NumMimeHtml | The number of documents with MIME type: text/html | long
NumMimeJavascript | The number of documents with MIME type: application/x-javascript | long
NumMimeOctet | The number of documents with MIME type: application/octet-stream | long
NumMimePdf | The number of documents with MIME type: application/pdf | long
NumMimePlain | The number of documents with MIME type: text/plain | long
NumMimePostscript | The number of documents with MIME type: application/postscript | long
NumMimeRssXml | The number of documents with MIME type: application/rss+xml | long
NumMimeRtf | The number of documents with MIME type: application/rtf | long
NumMimeUnknown | The number of documents with MIME type: application/unknown | long
NumMimeVncPowerpoint | The number of documents with MIME type: application/vnd.ms-powerpoint | long
NumMimeVndExcel | The number of documents with MIME type: application/vnd.ms-excel | long
NumMimeVndWord | The number of documents with MIME type: application/vnd.ms-word | long
NumMimeWord | The number of documents with MIME type: application/msword | long
NumMimeZip | The number of documents with MIME type: application/zip | long
NumMineCss | The number of documents with MIME type: text/css | long
NumMineTextXml | The number of documents with MIME type: text/xml | long
NumMineXml | The number of documents with MIME type: application/xml | long
ProgressCycle | The current refresh cycle number. | long
ProgressNextRefresh | Time until next refresh. | string
RetriesURIs | The number of URIs that were retried. | long
StateUpdated | Time since the last statistics update. | string
Status | The crawl status of the current collection. | string
UniqueChecksumsStored | The number of unique checksums in the FAST Search Web crawler store | long
Uptime | Time since the FAST Search Web crawler was started or the collection was added (timestamp) | long
UrisSkipped | The number of skipped URIs due to the 'include_uris' or 'exclude_uris' settings. | long
UrisSkippedDomain | The number of skipped URIs due to the 'include_domains' or 'exclude_domains' settings. | long
UrisSkippedScheme | The number of skipped URIs due to the 'allowed_schemes' setting. | long

WMI Class - "QRServer"

Name | Description | Type
AverageQPS | Average number of QPS (Queries Per Second). | float
AvgDocsumLatency | Average Docsum latency. | float
AvgLatency | Average latency. | float
AvgLatencyLast | Average latency for the last x queries. | float
AvgSearchLatency | Average Search latency. | float
CurrentQPS | Current QPS (Queries Per Second) over the last x seconds. | float
NumFailSys | Number of failed queries (system). | int
NumFailTotal | Number of failed queries (total). | int
NumFailUser | Number of failed queries (user). | int
NumQueries | Number of queries. | long
NumRequests | Number of requests. | long
PeakQPS | Peak QPS (Queries Per Second). | float
SamplingPeriod | Sampling period. | float

WMI Class - "Collection"

Name | Description | Type
Cluster | Indicates the cluster of the collection. | string
Complete | Indicates whether or not the collection is complete. | bool
Description | Contains the description of the collection. | string
Name | Indicates the name of the collection. | string
Pipeline | Indicates the pipeline of the collection. | string

WMI Class - "DocumentProcessor"

Name | Description | Type
EndReassign | The number of times the reassignment of the content distributor dispatcher has completed. | int
LastProcessed | The last processed URI. | string
PipelineAttachments | Indicates whether or not the attachments pipeline is initialized. | bool
PipelineOffice14 | Indicates whether or not the Office14 pipeline is initialized. | bool
ProcessOperations | Indicates whether or not the document processor server is processing a set of operations. | bool
QueueFull | Indicates whether or not the indexing dispatcher input queue is full. | bool
ResolveDispatcher | Indicates the name server resolve of the content distributor dispatcher scope total. | bool
ResolveMaster | Indicates the name server resolve of the content distributor master scope total. | bool
StartReassign | The number of times the reassignment of the content distributor dispatcher has been started. | int
StartServant | Indicates whether or not the document processor server is initializing. | bool
StartServantTotal | The number of times the document processor server has been initialized. | int
SubmitOperations | Indicates whether or not the document processor server is submitting a set of operations to the indexing dispatcher. | bool

WMI Class - "ContentDistributor"

Name | Description | Type
AvgDispatchTime | The average dispatch time over the last period. | float
AvgProcessingTime | The average processing time over the last period. | float
DocumentProcessors | The number of document processors | long
DocumentProcessorsBusy | The number of document processors that are busy. | long
DocumentProcessorsInvalid | The number of invalid document processors. | long
DocumentProcessorsUnused | The number of unused document processors. | long
Mode | Indicates the dispatcher running mode. | string
OperationsLastPeriod | The number of operations over the last period. | int
OperationsPerSec | The number of operations per second over the last period. | float
OperationsReceived | The number of receive operations. | int
ProcessingLoad | The processing load. | float
ProcserverProcess | The number of operation sets that was submitted to document processors. | int
SessionCreate | The number of clients creating a feeding session. | int

WMI Class - "IndexingDispatcher"

Name | Description | Type
BatchesProcessed | The total number of batches processed by this indexing dispatcher instance. | int
Mode | The indexing dispatcher fault tolerance mode. | string
NumOfDispatchers | The number of indexing dispatchers currently registered in the name service. | int

WMI Class - "WebAnalyzer"

Name | Description | Type
AnalysisStatus | Indicates whether or not analysis is active. | string
ContributingLinks | The number of links involved in the analysis. | long
CurrentAnalysisRuntime | Current analysis runtime. | int
LastAnalysisEnded | The time that the last analysis ended. | long
PreviousAnalysisRuntime | The time it took to complete the last analysis. | int
SitesServed | The number of sites in the anchor database. | long
URIsServed | The number of URIs in the anchor database. | long

WMI Class - "Crawler"

Name | Description | Type
DNSNetworkReadBytes | The number of bytes read from the network with respect to DNS resolving. | long
DNSNetworkWriteBytes | The number of bytes written to the network with respect to DNS resolving. | long
DNSRateLimit | Current DNS rate threshold (requests/second) | long
DNSRequests | The number of issued DNS requests. | long
DNSResponses | The number of received DNS responses. | long
DNSRetries | The number of issued DNS request retries. | long
DNSTimeouts | The number of issued DNS requests that timed out. | long

WMI Class - "Dataset"

Name | Description | Type
ActiveNodes | The number of active nodes. | int
ActivePartitions | The number of active partitions. | int
AvgSearchesPerSec | Average number of searches per second. | float
AvgSecondsPerSearch | Average number of seconds per search. | float
AvgUncachedSearchTime | Average uncached search time. | float
FirstPartition | First partition number. | int
MaxActiveNodes | Maximum number of active nodes. | int
MaxActivePartitions | Maximum number of active partitions. | int
Partitions | The number of partitions. | int
Samples | Samples | long
TimedOut | The number of timeouts | long
TimeoutPercentage | Timeout percentage. | float
TotalSearches | The total number of searches. | long
TotalSearchTime | The total search time. | float
TotalSelectionCost | The total selection cost. | int
UnitSelectionCost | The unit selection cost. | int
UpTime | Dataset uptime. | float

WMI Class - "FDispatch"

Name | Description | Type
ActiveCount | Active count. | long
InstantiateCount | Instantiate count. | long
ResourceUsage | Resource usage. | long
TotalResourceCount | Total resource count. | long
Version | Indicates the software version. | string
WaitQueueLen | Indicates the length of the wait queue. | long

WMI Class - "IndexerPartition"

Name | Description | Type
ActiveDocuments | The number of active/live documents. | long
IndexedPerSecond | The number of documents indexed per second for this partition. | float
IndexId | Internal index id. | long
PendingExclusionlist | The number of documents yet to be excluded. | long
Range | The file id range represented by this index. | string
Status | The current state of the partition. | string
StatusCode | The current state code of the partition. | int
TimestampIndexed | The time stamp of the index. | long
TotalDocuments | The total number of documents, including the exclusion listed. | long
Type | The partition type. A static partition may not be indexed. | string
TypeCode | The partition type code. A static partition may not be indexed. | int

WMI Class - "IndexerStatus"

Name | Description | Type
APIOperationsProcessed | The total number of API operations processed. | long
Cluster | The Cluster name. | string
Column | The Column number. | int
ConnectedFeeds | The number of connected feeds. | int
CurrentQueueSize | The current indexer queue size in bytes. | long
DocApiFrequence | The average number of processed operations per second. | float
DocApiLoad | The current indexer queue load in percentage. | float
ExclusionInterval | The exclusion interval in seconds. Using the exclusion list is a mechanism to remove documents from the search results without rebuilding the indexes. | int
Heartbeat | The heart beat periodically checks all other servers, answering a broadcast for missing client IDs and missing packets. The heart beat may suspend while synchronizing the fault tolerance storage with another indexer. | string
HeartbeatCode | The heart beat periodically checks all other servers, answering a broadcast for missing client IDs and missing packets. The heart beat may suspend while synchronizing the fault tolerance storage with another indexer. | int
Hostname | The Hostname. | string
LastSequence | The last processed sequence id in the indexer's internal sequence processor. | long
NumberOfElements | The current number of unprocessed elements on the indexer queue. | long
NumStatusUpdates | The current number of status updates. | long
PreferredMaster | A preferred main indexer will force other main indexers in the column to unregister. | bool
Row | The row number. | int
Status | The status. | string
StatusCode | The status code. | int
TCPPort | The TCP port. | int
TimeSinceLastIndex | Time since the last index was built, in seconds. | long

WMI Class - "BEStatistics"

Name | Description | Type
ProcessedRequests | Total number of Web items processed by the Browser Engine | int64
FailedRequests | The number of failed Web items. | int64
TimedOutDocuments | The number of Web items that timed out during processing. | int64
DownloadedRequests | The number of Web items retrieved from the FAST Search Web crawler. | int64
ProcessTimeMax | The longest amount of time, in milliseconds, used to process a Web item. | int64
ProcessTimeMin | The shortest amount of time, in milliseconds, used to download a Web item. | int64
ProcessTimeCount | The number of processed Web items. | int64
ProcessTimeAverage | Average time, in milliseconds, used to process Web items. | float
DownloadTimeMax | The longest amount of time, in milliseconds, used to download a Web item. | int64
DownloadTimeMin | The shortest amount of time, in milliseconds, used to download a Web item. | int64
DownloadTimeTotal | Total time, in milliseconds, used to download Web items. | int64
DownloadTimeCount | The number of downloaded Web items. | int64
DownloadTimeAverage | Average time, in milliseconds, used to download Web items. | float
CacheHits | The number of cache hits. | int64
CacheMisses | The number of cache misses. | int64
CacheInserts | The number of Web items inserted to the cache. | int64
CacheSize | The current number of cache entries. | int64
CacheBytes | The current cache size, in bytes. | int64

WMI Class - "NodeConfGlobal"

Name | Description
BasePort | Base TCP/IP port number.
Column | Search column number or -1 if no column index is specified
Row | Search row number or -1 if no row index is specified
ConfigHostEntry | The configuration file etc\NodeConf.xml can contain the configuration for multiple nodes. This string matches the hostname attribute for this host's node element. Typically, NodeConf.xml only contains configuration for one node. The hostname attribute will then be set to 'localhost'.
MiddlewareHost | Hostname used for middleware name server registration. A host can have multiple host names, so to ensure proper host matching in the middleware this string should be used to reference a node.
InstanceID | Instance id for the search installation, specified when installing. This value is shared by all nodes and is used to group/cluster nodes in monitoring tools like System Center Operations Manager.

WMI Class - "NodeConfProcess"

Name | Description | Type
Id | Component instance identifier. | string
Name | Component name. | string
Description | Component description. | string
Executable | The name of the executable. | string
Param | Command line arguments. | string
Port | Process instance TCP/IP port base. | int
PortBase | TCP/IP port base for this component type. | int
PortIncrement | TCP/IP port increment for additional processes of this type. | int
PortCount | Number of ports used (from base). | int
Status | Process status. | string
StatusCode | Process status code. | int
Pid | Process identifier (PID). | int
Index | Not used (always 0). | int

WMI Class - "AccessControlCoordinator"

Name | Description | Type
RawCountAllRequests | Count of FSA create filter requests. | int64
RawCountNonCachedRequests | Count of FSA create filter requests not found in cache. | int64
RawNonCachedRequestsMS | The total number of elapsed milliseconds of NonCached FSA create filter requests. | int64
RawTotalAllRequestsMS | The total number of elapsed milliseconds of all FSA create filter requests. | int64

WMI Class - "AccessControlNonCachedRequestAverage"

Name | Description | Type
NonCachedRequestAverageMS | Average time (in milliseconds) to generate user security filter requests that are not in the cache. | int

WMI Class - "AccessControlNonCachedRequestCount"

Name | Description | Type
NonCachedRequestCount | The number of user security filter requests that are not in the cache. | int

WMI Class - "AccessControlRequestAverage"

Name | Description | Type
RequestAverageMS | Average time (in milliseconds) to generate user security filter requests. | int

WMI Class - "AccessControlRequestCount"

Name | Description | Type
RequestCount | The number of user security filter requests. | int

WMI Class - "DomainWMIProxy"

Name | Description | Type
RequestCount | Total Request Count to this FSA domain. | int

WMI Class - "PrincipalAliaserWMIProxy"

Name | Description | Type
RequestCount | Total Request Count to this principal aliaser. | int


Performance counters (FAST Search Server 2010 for SharePoint)

This topic gives an overview of the available performance counters for Microsoft FAST Search Server 2010 for SharePoint.

FAST Search Server 2010 for SharePoint Performance counters

Performance Counter Category - "FAST Search Indexing Dispatcher"

Performance Counter Category - "FAST Search QRServer"

Performance Counter Category - "FAST Search Indexer"

Performance Counter Category - "FAST Search Crawler"

Performance Counter Category - "FAST Search Crawler Collection"

Performance Counter Category - "FAST Search Document Processor"

Performance Counter Category - "FAST Search Web Analyzer"

Performance Counter Category - "FAST Search Content Distributor"

Performance Counter Category - "FAST Search Dataset"

Performance Counter Category - "FAST Search FDispatcher"

Performance Counter Category - "FAST Search Indexer Partition"

Performance Counter Category - "FAST Search Indexer Status"

Performance Counter Category - "OSS Search FAST Content Plugin"

Performance Counter Category - "FAST Search Indexing Dispatcher"

Name | Description | Type
Batches Processed | The total number of batches processed by this indexing dispatcher instance. | NumberOfItems32
Mode | The indexing dispatcher fault tolerance mode. | NumberOfItems32
Number of Dispatchers | The number of indexing dispatchers that are currently registered in the name service. | NumberOfItems32

Performance Counter Category - "FAST Search QRServer"

Name | Description | Type
# of Fail system queries | The number of failed system queries. | NumberOfItems32
# of Fail system queries/sec | The number of failed system queries per second. | RateOfCountsPerSecond32
# of Fail total queries | The total number of failed queries. | NumberOfItems32
# of Fail total queries/sec | The total number of failed queries per second. | RateOfCountsPerSecond32
# of Fail user queries | The number of failed user queries. | NumberOfItems32
# of Fail user queries/sec | The number of failed user queries per second. | RateOfCountsPerSecond32
# of Queries | The number of queries. | NumberOfItems64
# of Queries/sec | The number of queries per second. | RateOfCountsPerSecond64
# of Requests | The number of requests. | NumberOfItems64
# of Requests/sec | The number of requests per second. | RateOfCountsPerSecond64
Average document sum latency - ms | The average document sum latency (in milliseconds). | NumberOfItems32
Average latency - ms | The average latency (in milliseconds). | NumberOfItems32
Average latency last - ms | The average latency (in milliseconds) of the last queries. | NumberOfItems32
Average queries per minute | The average number of queries per minute. | NumberOfItems32
Average search latency - ms | The average search latency (in milliseconds). | NumberOfItems32
Peak queries per sec | Indicates the highest registered number of queries per second. | NumberOfItems32
Sampling period | The sampling period. | NumberOfItems32

Performance Counter Category - "FAST Search Indexer"

Name | Description | Type
# of documents - partition 0 | Indicates the number of searchable items in the current active index on partition 0. | NumberOfItems32
# of documents - partition 1 | Indicates the number of searchable items in the current active index on partition 1. | NumberOfItems32
# of documents - partition 2 | Indicates the number of searchable items in the current active index on partition 2. | NumberOfItems32
# of documents - partition 3 | Indicates the number of searchable items in the current active index on partition 3. | NumberOfItems32
# of documents - partition 4 | Indicates the number of searchable items in the current active index on partition 4. | NumberOfItems32
# of excluded documents - partition 0 | Indicates the number of items that are in the exclusion list of the currently active index on partition 0. This translates to items that are in the index but that should not be searchable (deleted, present in other partitions, etc). | NumberOfItems32
# of excluded documents - partition 1 | Indicates the number of items that are in the exclusion list of the currently active index on partition 1. This translates to items that are in the index but that should not be searchable (deleted, present in other partitions, etc). | NumberOfItems32
# of excluded documents - partition 2 | Indicates the number of items that are in the exclusion list of the currently active index on partition 2. This translates to items that are in the index but that should not be searchable (deleted, present in other partitions, etc). | NumberOfItems32
# of excluded documents - partition 3 | Indicates the number of items that are in the exclusion list of the currently active index on partition 3. This translates to items that are in the index but that should not be searchable (deleted, present in other partitions, etc). | NumberOfItems32
# of excluded documents - partition 4 | Indicates the number of items that are in the exclusion list of the currently active index on partition 4. This translates to items that are in the index but that should not be searchable (deleted, present in other partitions, etc). | NumberOfItems32
# of indexed documents/sec - partition 0 | Indicates the average number of items that are indexed per second in partition 0. | NumberOfItems32
# of indexed documents/sec - partition 1 | Indicates the average number of items that are indexed per second in partition 1. | NumberOfItems32
# of indexed documents/sec - partition 2 | Indicates the average number of items that are indexed per second in partition 2. | NumberOfItems32
# of indexed documents/sec - partition 3 | Indicates the average number of items that are indexed per second in partition 3. | NumberOfItems32
# of indexed documents/sec - partition 4 | Indicates the average number of items that are indexed per second in partition 4. | NumberOfItems32
API operations | The API operations (the number of items that are indexed and updated) counter indicates the total number of feed operations that the indexer has processed. | NumberOfItems32
API operations per sec | Indicates the average number of processed API operations per second by the indexer. | NumberOfItems32
API partial update operations | Indicates the number of API operations processed by the indexer that are partial update operations. | NumberOfItems32
API queue load | Indicates the current size of the API queue (in bytes) divided by the maximum allowed size of the API queue (in bytes). | NumberOfItems32
API queue size | Indicates the number of batches that are currently in the API queue. The API queue is used by the indexer to hold pending operations waiting to be persisted. | NumberOfItems32
API remove operations | Indicates the number of API operations processed by the indexer that are remove operations. | NumberOfItems32
API status update operations | Indicates the number of API operations processed by the indexer that are status updates (failed_operations). | NumberOfItems32
Cache percentage - partition 0 | Indicates the percentage of the total cache given to partition 0 that is used by the search process. | NumberOfItems32
Cache percentage - partition 1 | Indicates the percentage of the total cache given to partition 1 that is used by the search process. | NumberOfItems32
Cache percentage - partition 2 | Indicates the percentage of the total cache given to partition 2 that is used by the search process. | NumberOfItems32
Cache percentage - partition 3 | Indicates the percentage of the total cache given to partition 3 that is used by the search process. | NumberOfItems32
Cache percentage - partition 4 | Indicates the percentage of the total cache given to partition 4 that is used by the search process. | NumberOfItems32
Collections removed | Indicates the number of API operations processed by the indexer that are remove collection operations. | NumberOfItems32
Column id | Indicates which column number the indexer is part of. | NumberOfItems32
Column role code | Indicates if the indexer is a primary or a backup indexer (in the fault tolerance setup). | NumberOfItems32
Disk low state | Indicates if the indexer is in a state where it has little disk space to work with to persist items and to build new indices. | NumberOfItems32
Documents in indexer | Indicates the current total number of valid items that is contained in the docindex of the indexer. The docindex is a mapping to the persisted items on disk, so this translates to the number of valid items persisted on disk (FiXML). | NumberOfItems32
Documents indexed | Indicates the number of API operations processed by the indexer that are update (or add) operations. | NumberOfItems32
Failed operations | Indicates the number of API operations that could not be processed successfully. | NumberOfItems32
Fixml fill rate | An approximation of the ratio between the number of valid items in a persisted file (FiXML file) and the total number of persisted items. A value close to 100% means that the files on disk are well compacted together. | NumberOfItems32
Heartbeat status code | In a fault tolerant deployment, the heartbeat thread periodically checks for the state of the indexer in the column. The heartbeat status code shows if this thread is currently running, suspended or in an unknown state. | NumberOfItems32
Indexing suspended state | Indicates if the indexer currently has indexing suspended. This means that it is not creating new indices, although it is possibly persisting new items to disk. | NumberOfItems32
Last batch size | Indicates the size of the previous batch (in bytes) that was processed by the indexer. | NumberOfItems32
Manual reset state | Indicates when the indexer is in a reset index state. This state is entered when a user manually gives the command. | NumberOfItems32
ManualFTSyncNeeded | Manual fault-tolerant sync needed, is always set to false. | NumberOfItems32
MaxCacheKB - partition 0 | Indicates the maximum cache size (in kilobytes) that is used by the search processes for partition 0. | NumberOfItems32
MaxCacheKB - partition 1 | Indicates the maximum cache size (in kilobytes) that is used by the search processes for partition 1. | NumberOfItems32
MaxCacheKB - partition 2 | Indicates the maximum cache size (in kilobytes) that is used by the search processes for partition 2. | NumberOfItems32
MaxCacheKB - partition 3 | Indicates the maximum cache size (in kilobytes) that is used by the search processes for partition 3. | NumberOfItems32
MaxCacheKB - partition 4 | Indicates the maximum cache size (in kilobytes) that is used by the search processes for partition 4. | NumberOfItems32
Maximum batch size | Indicates the size of the largest batch (in bytes) that was seen during processing by the indexer. | NumberOfItems32
Maximum documents per sec | Indicates the highest achieved ratio between the number of indexed items in an index job and the time it took to perform the job. | NumberOfItems32
Minimum documents per sec | Indicates the lowest achieved ratio between the number of indexed items in an index job and the time it took to perform the job. | NumberOfItems32
OngoingFTSequenceRecovery | Indicates if an indexer is currently recovering operations from other indexers in the column in a fault tolerance setup. | NumberOfItems32
Performing reset index state | Indicates if the indexer is currently in a reset index state, meaning that it is indexing documents from disk (file_id), and not performing incremental indexing. | NumberOfItems32
Registered file receivers | Indicates the number of file receivers currently registered on the indexer. This can be both backup indexers and search controllers. These receivers can subscribe to different structures to get them copied over by the indexer. | NumberOfItems32
Registered search controllers | Indicates the number of search controllers currently registered on the indexer. These search controllers register to get new indices over from the indexer. | NumberOfItems32
Row id | Indicates which row number the indexer is part of. | NumberOfItems32
SCtl currently allocated | Indicates how many search controller stubs are currently allocated by the indexer. This translates to how many search controllers the indexer is communicating with. | NumberOfItems32
SCtl total allocated | Indicates how many times a search controller has been allocated on the indexer side. The search controller stub is used to communicate with a remote search controller. | NumberOfItems32
Size of documents | Indicates the size (in bytes) of all the valid items that are persisted on disk (FiXML). | NumberOfItems64
Total indexing runs - partition 0 | Indicates the number of times that a new index has been built on partition 0. | NumberOfItems32
Total indexing runs - partition 1 | Indicates the number of times that a new index has been built on partition 1. | NumberOfItems32
Total indexing runs - partition 2 | Indicates the number of times that a new index has been built on partition 2. | NumberOfItems32
Total indexing runs - partition 3 | Indicates the number of times that a new index has been built on partition 3. | NumberOfItems32
Total indexing runs - partition 4 | Indicates the number of times that a new index has been built on partition 4. | NumberOfItems32

Performance Counter Category - "FAST Search Crawler"

The performance counters in this category apply to the FAST Search Web crawler only.

Name | Description | Type
DNS Network Read Bytes | The number of bytes read from the network with respect to DNS resolving. | NumberOfItems64
DNS Network Write Bytes | The number of bytes written to the network with respect to DNS resolving. | NumberOfItems64
DNS Rate Limit | The current DNS rate threshold (requests per second). | NumberOfItems64
DNS Requests | The number of issued DNS requests. | NumberOfItems64
DNS Responses | The number of received DNS responses. | NumberOfItems64
DNS Retries | The number of issued DNS request retries. | NumberOfItems64
DNS Timeouts | The number of issued DNS requests that timed out. | NumberOfItems64

Performance Counter Category - "FAST Search Crawler Collection"

The performance counters in this category apply to the FAST Search Web crawler only.

Name | Description | Type
# of retries of URIs | The number of URIs that were retried. | NumberOfItems64
Active sites | The number of sites being crawled right now. | NumberOfItems64
Collection Status Code | The current collection crawl status. | NumberOfItems32
Content Indexing/Added | The number of ADD operations that were sent to content indexing. | NumberOfItems64
Content Indexing/Deleted | The number of DELETE operations that were sent to content indexing. | NumberOfItems64
Content Indexing/Failed | The number of failed operations. | NumberOfItems64
Content Indexing/Modified | The number of MOD operations that were sent to content indexing. | NumberOfItems64
Content Indexing/Operations | The number of operations that were submitted to content indexing. | NumberOfItems64
Content Indexing/Succeeded | The number of operations that were sent to content indexing that succeeded. | NumberOfItems64
Content Indexing/Urlschanges | The number of urlschange operations that were sent to content indexing. | NumberOfItems64
Document download rate per min | Indicates the current item download rate in items per minute. | NumberOfItems32
Document Size Average | The average item size (in bytes). | NumberOfItems64
Document Size Maximum | The maximum item size (in bytes). | NumberOfItems64
Document Size Total | The total size of the items (in bytes). | NumberOfItems64
Documents in CrawlerStore | The (approximate) number of items in the FAST Search Web crawler store. | NumberOfItems64
Documents Skipped/Checksum Cache | The number of items that were skipped because they were being screened by the run-time checksum cache that is used for duplicate detection. | NumberOfItems64
Documents Skipped/Chunk Error | The number of items that were skipped because there were problems with chunked encoding. Failure to de-chunk content. | NumberOfItems64
Documents Skipped/Connect Error | The number of items that were skipped because of failure to connect to the remote Web server. | NumberOfItems64
Documents Skipped/Connect Timeout | The number of items that were skipped because the connection attempt to the remote Web server timed out. | NumberOfItems64
Documents Skipped/Empty Document | The number of items that were skipped because their size was 0 bytes. | NumberOfItems64
Documents Skipped/Encoding | The number of items that were skipped because of problems with item encoding. This typically indicates problems with compressed content. | NumberOfItems64
Documents Skipped/Header Exclude | The number of items that were skipped because of matching the collection setting 'exclude_headers'. | NumberOfItems64
Documents Skipped/HTTP Header | The number of items that were skipped because of errors with the HTTP header. | NumberOfItems64
Documents Skipped/Incomplete | The number of items that were skipped because of being incomplete. The Web server did not return the complete document as indicated by the HTTP header. | NumberOfItems64
Documents Skipped/MIME Type | The number of items that were skipped because they did not match the collection setting 'allowed_types'. | NumberOfItems64
Documents Skipped/Network Error | The number of items that were skipped because of various network errors. | NumberOfItems64
Documents Skipped/No 30x Target | The number of items that were skipped because they did not have a redirect target, but a redirect return code was given by the Web server. | NumberOfItems64
Documents Skipped/NoIndex | The number of items that were skipped because they contained a META Robots No Index tag. | NumberOfItems64
Documents Skipped/NoIndex RSS | The number of items that were skipped because RSS feed indexing was disabled by the 'index_feed' setting. | NumberOfItems64
Documents Skipped/Other Error | The number of items that were skipped because of other reasons. | NumberOfItems64
Documents Skipped/Timeout | The number of items that were skipped because it took longer time to download them than the amount of time specified by the 'fetch_timeout' setting. | NumberOfItems64
Documents Skipped/Too Large | The number of items that were skipped because they exceeded the collection 'cut_off' setting. | NumberOfItems64
Documents/Deleted | The number of items deleted from the item store. | NumberOfItems64
Documents/Downloaded | The number of downloaded items. | NumberOfItems64
Documents/Modified | The number of stored items that were modified. | NumberOfItems64
Documents/Requested | The number of URIs that the FAST Search Web crawler attempted to download. | NumberOfItems64
Documents/Stored | The number of item writes to the FAST Search Web crawler store. | NumberOfItems64
Documents/Unchanged | The number of downloaded items that were unchanged. | NumberOfItems64
Download average time - ms | Indicates the average download time. | NumberOfItems32
Download maximum time - ms | Indicates the maximum download time. | NumberOfItems32
Download total time - ms | Indicates the total download time. | NumberOfItems32
Feeding Status | Specifies whether content submission to the index is active (1) or paused (0). | NumberOfItems64
HTTPResponse/200 | HTTP Response 200: OK. | NumberOfItems64
HTTPResponse/226 | FTP Response 226: Successful. | NumberOfItems64
HTTPResponse/300 | HTTP Response 300: Multiple Choices. | NumberOfItems64
HTTPResponse/301 | HTTP Response 301: Moved Permanently. | NumberOfItems64
HTTPResponse/302 | HTTP Response 302: Found. | NumberOfItems64
HTTPResponse/303 | HTTP Response 303: See Other. | NumberOfItems64
HTTPResponse/304 | HTTP Response 304: Not Modified. | NumberOfItems64
HTTPResponse/307 | HTTP Response 307: Temporary Redirect. | NumberOfItems64
HTTPResponse/400 | HTTP Response 400: Bad Request. | NumberOfItems64
HTTPResponse/401 | HTTP Response 401: Unauthorized. | NumberOfItems64
HTTPResponse/403 | HTTP Response 403: Forbidden. | NumberOfItems64
HTTPResponse/404 | HTTP Response 404: Not Found. | NumberOfItems64
HTTPResponse/406 | HTTP Response 406: Not Acceptable. | NumberOfItems64
HTTPResponse/407 | HTTP Response 407: Proxy Authentication Required. | NumberOfItems64
HTTPResponse/426 | FTP Response 426: Transfer Aborted. | NumberOfItems64
HTTPResponse/500 | HTTP Response 500: Internal Server Error. | NumberOfItems64
HTTPResponse/503 | HTTP Response 503: Service Unavailable. | NumberOfItems64
Links Fwd to node scheduler | Indicates the number of URIs/links that were forwarded to the FAST Search Web crawler node scheduler. | NumberOfItems64
Links Fwd with in site manager | Indicates the number of URIs/links that were forwarded within a FAST Search Web crawler site manager. | NumberOfItems64
Links Fwd with in site worker | Indicates the number of URIs/links that were forwarded within a FAST Search Web crawler site worker. | NumberOfItems64
MIME type application/atom xml | The number of items with MIME type: application/atom+xml. | NumberOfItems64
MIME type application/msexcel | The number of items with MIME type: application/msexcel. | NumberOfItems64
MIME type application/msword | The number of items with MIME type: application/msword. | NumberOfItems64
MIME type application/octet stream | The number of items with MIME type: application/octet-stream. | NumberOfItems64
MIME type application/pdf | The number of items with MIME type: application/pdf. | NumberOfItems64
MIME type application/postscript | The number of items with MIME type: application/postscript. | NumberOfItems64
MIME type application/rss xml | The number of items with MIME type: application/rss+xml. | NumberOfItems64
MIME type application/rtf | The number of items with MIME type: application/rtf. | NumberOfItems64
MIME type application/unknown | The number of items with MIME type: application/unknown. | NumberOfItems64
MIME type application/vnd ms excel | The number of items with MIME type: application/vnd.ms-excel. | NumberOfItems64
MIME type application/vnd ms powerpoint | The number of items with MIME type: application/vnd.ms-powerpoint. | NumberOfItems64
MIME type application/vnd ms word | The number of items with MIME type: application/vnd.ms-word. | NumberOfItems64
MIME type application/x javascript | The number of items with MIME type: application/x-javascript. | NumberOfItems64
MIME type application/xml | The number of items with MIME type: application/xml. | NumberOfItems64
MIME type application/zip | The number of items with MIME type: application/zip. | NumberOfItems64
MIME type text/css | The number of items with MIME type: text/css. | NumberOfItems64
MIME type text/html | The number of items with MIME type: text/html. | NumberOfItems64
MIME type text/plain | The number of items with MIME type: text/plain. | NumberOfItems64
MIME type text/xml | The number of items with MIME type: text/xml. | NumberOfItems64
Network/Bytes received | The number of bytes (HTTP, HTTPS and FTP) read from the network. | NumberOfItems64
Network/Bytes received per min | The current network read rate (in bytes per minute). | NumberOfItems32
Network/Bytes sent | The number of bytes (HTTP, HTTPS and FTP) written to the network. | NumberOfItems64
Network/Bytes sent per min | The current network write rate (in bytes per minute). | NumberOfItems32
Progress/Cycle | The current refresh cycle number. | NumberOfItems64
Unique checksums stored | The number of unique checksums in the FAST Search Web crawler store. | NumberOfItems64
URIs Skipped due to Domain rules | The number of URIs that were skipped because of the 'include_domains' or 'exclude_domains' settings. | NumberOfItems64
URIs Skipped due to Scheme rules | The number of URIs that were skipped because of the 'allowed_schemes' setting. | NumberOfItems64
URIs Skipped due to URI rules | The number of URIs that were skipped because of the 'include_uris' or 'exclude_uris' settings. | NumberOfItems64

Performance Counter Category - "FAST Search Document Processor"

Name | Description | Type
Pipeline Attachments initialized | Indicates whether the attachments pipeline is initialized (1) or not (0). | NumberOfItems32
Pipeline Office14 initialized | Indicates whether the FAST Search Server 2010 for SharePoint pipeline is initialized (1) or not (0). | NumberOfItems32
Queue Full State | Indicates whether the indexing dispatcher input queue is full (1) or not (0). | NumberOfItems32
Resolving dispatcher contentdistributor | Total number of invocations of the nameserver resolve of the contentdistributor dispatcher. | NumberOfItems32
Resolving master contentdistributor | Total number of invocations of the nameserver resolve of the primary contentdistributor. | NumberOfItems32
Server initializing | Indicates whether the document processor server is initializing (1) or not (0). | NumberOfItems32
Server processing | Indicates whether the document processor server is processing a set of operations (1) or not (0). | NumberOfItems32
Server submitting to indexing dispatcher | Indicates whether the document processor server is submitting a set of operations to the indexing dispatcher (1) or not (0). | NumberOfItems32
Total number of completed reassigns | The number of times the reassignment of the contentdistributor dispatcher has been completed. | NumberOfItems32
Total number of initializations | The number of times the document processor server has been initialized. | NumberOfItems32
Total number started reassigns | The number of times the reassignment of the contentdistributor dispatcher has been started. | NumberOfItems32

Performance Counter Category - "FAST Search Web Analyzer"

Name | Description | Type
Contributing Links | The number of links involved in the analysis. | NumberOfItems64
Current analysis runtime | Indicates the runtime of the current analysis. | NumberOfItems32
Last analysis ended | Indicates the time when the last analysis ended. | NumberOfItems64
Previous analysis runtime | Indicates how much time it took to complete the last analysis. | NumberOfItems32
Sites Served | The number of sites in the anchor database. | NumberOfItems64
URIs Served | The number of URIs in the anchor database. | NumberOfItems64

Performance Counter Category - "FAST Search Content Distributor"

Name | Description | Type
Average dispatch time - ms | Indicates the average dispatch time the last period (in milliseconds). | NumberOfItems32
Average processing time - ms | Indicates the average processing time last period (in milliseconds). | NumberOfItems32
Clients creating sessions | The number of clients creating a feeding session. | NumberOfItems32
Document processors | The number of document processors. | NumberOfItems64
Document processors busy | The number of document processors that are busy. | NumberOfItems64
Document processors invalid | The number of document processors that are not valid. | NumberOfItems64
Document processors unused | The number of document processors that are not used. | NumberOfItems64
Mode | Indicates the dispatcher running mode: primary or dispatcher. | Text
Operation sets submitted | The number of operation sets submitted to document processors. | NumberOfItems32
Operations last period | The number of operations over the last period. | NumberOfItems32
Operations received | The number of received operations. | NumberOfItems32
Operations/sec | The number of operations per second over the last period. | RateOfCountsPerSecond32
Processing load | Indicates the processing load. | NumberOfItems32

Performance Counter Category - "FAST Search Dataset"

Name | Description | Type
Active nodes | The number of active nodes. | NumberOfItems32
Active partitions | The number of active partitions. | NumberOfItems32
Average searches per second | The average number of searches per second. | RateOfCountsPerSecond64
Average time per search - ms | The average number of milliseconds per search. | NumberOfItems64
Average uncached search time - ms | The average uncached search time in milliseconds. | NumberOfItems32
FirstPartition | The first partition number. | NumberOfItems32
MaxActiveNodes | The maximum number of active nodes. | NumberOfItems32
MaxActivePartitions | The maximum number of active partitions. | NumberOfItems32
Partitions | The number of partitions. | NumberOfItems32
Samples | The number of queries used to calculate the average counters. | NumberOfItems64
Timed out | The number of timeouts. | NumberOfItems64
Timeout percentage | The timeout percentage. | NumberOfItems32
Total search time - ms | The total search time (in milliseconds). | NumberOfItems64
Total searches | The total number of searches. | NumberOfItems64
Total selection cost | The currently active reference cost for this FAST Search Dataset. The reference cost is used by fdispatch for load balancing queries. The lowest row reference cost will be used at any time. | NumberOfItems32
Unit selection cost | The reference cost that will be added to the total selection cost by selecting (in other words: running a query on) this FAST Search Dataset. | NumberOfItems32
Up time | The total uptime of the dataset. | NumberOfItems64

Performance Counter Category - "FAST Search FDispatcher"

Name | Description | Type
Active count | The number of actively running queries in the fdispatch process since the last index set change. | NumberOfItems64
Instantiate count | The total number of queries that have been dispatched since the last index set change. | NumberOfItems64
Resource usage | The total number of open file handles. | NumberOfItems64
Total resource count | The maximum number of open file handles. This is 1014 for fdispatch and 1004 for fsearch. | NumberOfItems64
Wait queue length | The number of queued queries that are temporarily put on hold. | NumberOfItems64

Performance Counter Category - "FAST Search Indexer Partition"

Name | Description | Type
Active documents | The number of active/live items. | NumberOfItems64
Index id | Indicates the internal index ID. | NumberOfItems64
Indexed per second | The number of items that are indexed per second for this partition. | NumberOfItems64
Pending exclusion list | The number of items that are yet to be excluded. | NumberOfItems64
Status code | The current state of the partition. | NumberOfItems32
Timestamp indexed | Indicates the index time stamp. | NumberOfItems64
Total documents | The total number of items, including the items that are on the exclusion list. | NumberOfItems64
Type code | Indicates the partition type. A static partition may not be indexed. | NumberOfItems32

Performance Counter Category - "FAST Search Indexer Status"

Name | Description | Type
# of connected feeds | The number of connected feeds. | NumberOfItems32
# status updates | The current number of status updates. | NumberOfItems64
API operations processed | The total number of operations processed. | NumberOfItems64
Column | Indicates the column number. | NumberOfItems32
Current queue size | The current indexer queue size in bytes. | NumberOfItems64
Doc API frequence | The average number of processed operations per second. | NumberOfItems64
Doc API load | The current indexer queue load in percentage. | NumberOfItems64
Exclusion interval | The exclusion interval in seconds. Exclusionlisting is a mechanism for removing documents from the search results without rebuilding the indexes. | NumberOfItems32
Heartbeat code | The heartbeat periodically checks all other servers, answering a broadcast for missing client IDs and missing packets. The heartbeat may suspend while synchronizing the fault tolerance storage with another indexer. | NumberOfItems32
LastSequence | Indicates the last processed sequence id in the indexer's internal sequence processor. | NumberOfItems64
NumberOfElements | The current number of unprocessed elements on the indexer queue. | NumberOfItems64
Row | Indicates the row number. | NumberOfItems32
Status code | Indicates the status code. | NumberOfItems32
TCP port | Indicates the TCP port. | NumberOfItems32
Time since last index | Indicates how many seconds have passed since the last index was built. | NumberOfItems64

Performance Counter Category - "OSS Search FAST Content Plugin"

The performance counters in this category apply to items that are crawled with the FAST Search Connector Content Search Service Application only. The counters do not persist between sessions. A service restart or other administrative operations will reset the values.

Name | Description
Items Total | The total number of documents submitted for processing.
Items Success | The number of items that have been successfully indexed.
Items Failed Individually | The number of items that failed prior to indexing.
Items Open | The number of items that are currently being processed, and that were neither successful nor have they failed.
Items Failed Batch | The number of items that failed because the batch that contains them has failed.
Items Ignored | Items that failed in an MSS component external to the content plug-in.
Items No Index | The number of items with a "noindex" flag.
Items Warnings | The number of received FAST Search Server 2010 for SharePoint item warnings.
Items Retried | The number of items that have been resubmitted individually after failing as part of a batch.
Items In Robot Threads | The number of open FAST Search Content Search Service Application sessions that are running via the content plug-in.
Batches Total | The total number of batches that were submitted for processing.
Batches Success | The number of batches that have been successfully indexed.
Batches Failed | The number of batches that failed prior to indexing
Batches Open | The number of batches that are currently being processed and that were neither successful nor have they failed.
Batches Ready | The number of batches that have been queued up and are not yet submitted to FAST Search Server 2010 for SharePoint.
Batches Submitted | The number of batches that have been submitted to FAST Search Server 2010 for SharePoint but that have not yet completed processing.
Avg End-To-End Time | The average time from item initialization to completion of indexing.
Avg Init Time | The average time from item initialization until the item has been placed into a batch for submission.
Avg Wait Time | The average time from when an item is placed into a batch until the first attempt to submit the batch to FAST Search Server 2010 for SharePoint.
Avg Submit Time | The average time from the first attempt to submit an item's batch until FAST Search Server 2010 for SharePoint acknowledges the batch has successfully been received for processing.
Avg Callback Time | The average time between acknowledgement of a batch submission and the receipt of a final success or failure callback
Transactions Add Started | The number of initialized ADD operations.
Transactions Add Completed | The number of completed ADD operations.
Transactions Update Started | The number of initialized UPDATE operations.
Transactions Update Completed | The number of completed UPDATE operations.
Transactions Delete Started | The number of initialized DELETE operations.
Transactions Delete Completed | The number of completed DELETE operations.
Submitter Threads | The number of threads that are submitting batches to FAST Search Server 2010 for SharePoint.
Submitter Thread Idle Time | The percentage of time an item has spent in submitter threads not doing anything.
Submitter Thread Submit Time | The percentage of time spent (compared to total time) registering batches with FAST Search Server 2010 for SharePoint.
Feed Rate Item | The number of items that were processed per second.
Feed Rate Data | The item feed rate in megabytes per second.
Items Failed Timeout | The number of items that timed out while waiting for a success or failure callback from FAST Search Server 2010 for SharePoint.
Items Failed Total | The total number of items that have failed for any reason.
Submission Timeouts | The total number of times a batch timed out while submitting to FAST Search Server 2010 for SharePoint (this number includes multiple retries for same batch).
Batches Failed Timeout | The number of batches that timed out while waiting for a success or failure callback from FAST Search Server 2010 for SharePoint.


About linguistic features (FAST Search Server 2010 for SharePoint)
See Also

Published: June 24, 2010

Microsoft FAST Search Server 2010 for SharePoint has many linguistic features that help improve search relevancy. Some features can be tuned, but other features have a default behavior that cannot be changed.

The following linguistic features are described in this article:

  • Tokenization

  • Automatic language and encoding detection

  • Stemming

  • Spell checking and spell tuning

  • Anti-phrasing

  • Property extraction

  • Offensive content filtering

For an overview of the supported languages for these linguistic features, see Linguistic features per language (FAST Search Server 2010 for SharePoint).

Tokenization

Tokenization is segmentation of text into individual words (tokens) that can be indexed. Spaces, tabs, periods, commas, dashes, question marks and quotation marks are considered delimiting characters. For East Asian languages (Chinese, Japanese), which do not have these delimiting characters between words, more sophisticated methods must be employed to produce the indexable tokens.

Tokenization is performed on text content both during item processing and query processing. The tokenization process in FAST Search Server 2010 for SharePoint consists of three stages:

  1. Language-independent input normalization, where input text is transformed into a unified format. This includes replacing complex characters such as ligatures with their canonical forms, and replacing less-used Unicode characters by compatible characters or sequences of characters (Trademark sign with tm or full width Latin characters with half-width equivalents).

  2. A language-specific tokenization engine based on the document language splits the text into individual words/tokens based on word breaker tokenizers.

  3. Indexed tokens are normalized according to language-independent rules to ensure cross-language retrieval. The normalization reduces the complexity of a character by changing it or removing parts of it. In FAST Search Server 2010 for SharePoint, all characters are lowercased and accented characters are reduced to their unaccented base characters.

Tokenization is supported for all languages.

Automatic language and encoding detection

During item processing, FAST Search Server 2010 for SharePoint automatically recognizes more than 80 different languages in all common encodings. The text language and encoding can be defined in the metadata of a document, or it can be determined by an automatic process during item processing.

The information is used to select the appropriate language-specific dictionaries and algorithms during item processing.

Stemming

Stemming merges multiple forms of the same word, for example the singular and plural forms of a noun. Stemming increases recall, and for languages that have many forms of the same word, stemming is very important to achieve sufficient recall. You cannot tune the stemming dictionaries.

Spell checking and spell tuning

Spell checking improves the quality of queries by comparing the query terms against language specific dictionaries and identifying misspelled terms.

Spell tuning fine-tunes the spell checking dictionaries to make sure that they are aligned with the frequency of words in the processed documents. Users will only get spell checking suggestions that are relevant within the processed content. Without this alignment, spell checking suggestions could lead to zero hit result sets.

You can define words to exclude from spell checking, for example a specific product or company name. The exclusion list is used for all languages.

  • For more information about spell checking and exclusion lists, see Add, remove and display spell checking exceptions by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

  • For more information about how to manage spell tuning, see Spell tuning cmdlets (FAST Search Server 2010 for SharePoint).

Anti-phrasing

Anti-phrasing is closely related to the concept of stop words, which are words that the search system ignores in end-user queries. The anti-phrasing feature does not remove single words, but complete phrases. Removing single words implies the risk of removing important words that are identical to stop words. Phrases are less ambiguous and can be removed from the query more safely. The anti-phrasing dictionaries that are delivered with FAST Search Server 2010 for SharePoint therefore do not contain single words. You cannot tune the anti-phrasing dictionaries.

Property extraction

FAST Search Server 2010 for SharePoint provides advanced, language-specific property extractors for person names, company names and geographic names/locations.

For more information, see Manage property extraction (FAST Search Server 2010 for SharePoint).

Offensive content filtering

FAST Search Server 2010 for SharePoint can provide filtering against offensive content for many languages.

Offensive content filtering is not provided out of the box, but can be configured.

See Also

Concepts

Linguistic features per language (FAST Search Server 2010 for SharePoint)
Spell tuning cmdlets (FAST Search Server 2010 for SharePoint)


Linguistic features per language (FAST Search Server 2010 for SharePoint)
See Also

Published: June 24, 2010

The following table shows linguistic features per language. For information about the different features, see About linguistic features (FAST Search Server 2010 for SharePoint).

Language | Code | Automatic language detection | Stemming | Spell checking: Pre-configured dictionary | Spellchecking: Spell tuning | Anti-phrasing | Property extraction | Offensive content filtering1
Afrikaans | af | yes |  |  |  |  |  |
Albanian | sq | yes |  |  |  |  |  |
Arabic | ar | yes | yes | yes | yes | yes | yes | yes
Armenian | hy | yes |  |  |  |  |  |
Azeri | az | yes |  |  |  |  |  |
Bangla | bn | yes | yes |  |  |  |  |
Basque | eu | yes |  |  |  |  |  |
Bosnian | bs | yes | yes |  |  |  |  |
Breton | br | yes |  |  |  |  |  |
Bulgarian | bg | yes | yes |  | yes |  |  |
Byelorussian | be | yes |  |  |  |  |  |
Catalan | ca | yes | yes |  | yes |  |  |
Chinese Simplified | zh-cn | yes |  |  |  | yes |  | yes
Chinese Traditional | zh-tw | yes |  |  |  | yes |  | yes
Croatian | hr | yes | yes |  | yes |  |  |
Czech | cs | yes | yes | yes | yes | yes |  | yes
Danish | da | yes | yes | yes | yes | yes |  |
Dutch | nl | yes | yes | yes | yes | yes | yes |
English | en | yes | yes | yes | yes | yes | yes | yes
Esperanto | eo | yes |  |  |  |  |  |
Estonian | et | yes | yes | yes | yes | yes |  |
Faeroese | fo | yes |  |  |  |  |  |
Farsi | fa | yes |  |  |  |  |  |
Filipino | tl | yes |  |  |  |  |  |
Finnish | fi | yes | yes | yes | yes | yes |  | yes
French | fr | yes | yes | yes | yes | yes | yes | yes
Frisian | fy | yes |  |  |  |  |  |
Galician | gl | yes |  |  |  |  |  |
Georgian | ka | yes |  |  |  |  |  |
German | de | yes | yes | yes | yes | yes | yes | yes
Greek | el | yes | yes | yes | yes | yes |  |
Greenlandic | kl | yes |  |  |  |  |  |
Gujarati | gu | yes | yes |  |  |  |  |
Hausa | ha | yes |  |  |  |  |  |
Hebrew | he | yes | yes | yes | yes | yes |  |
Hindi | hi | yes | yes | yes | yes | yes |  | yes
Hungarian | hu | yes | yes | yes | yes | yes |  |
Icelandic | is | yes | yes |  | yes | yes |  |
Indonesian | id | yes | yes |  | yes |  |  |
Irish Gaelic | ga | yes |  |  |  |  |  |
Italian | it | yes | yes | yes | yes | yes | yes | yes
Japanese | ja | yes |  |  |  | yes | yes | yes
Kannada | kn | yes | yes |  |  |  |  |
Kazakh | kk | yes |  |  |  |  |  |
Kirghiz | ky | yes |  |  |  |  |  |
Korean | ko | yes |  |  |  |  |  |
Kurdish | ku | yes |  |  |  |  |  |
Latin | la | yes |  |  |  |  |  |
Latvian | lv | yes | yes | yes | yes | yes |  |
Letzeburgesch | lb | yes |  |  |  |  |  |
Lithuanian | lt | yes | yes | yes | yes | yes |  | yes
Macedonian | mk | yes |  |  |  |  |  |
Malay | ms | yes | yes |  | yes |  |  |
Malayalam | ml | yes | yes |  |  |  |  |
Maltese | mt | yes |  |  |  |  |  |
Maori | mi | yes |  |  |  |  |  |
Marathi | mr | yes | yes |  |  |  |  |
Mongolian | mn | yes |  |  |  |  |  |
Norwegian |  | yes | yes | yes | yes | yes | yes | yes
Pashto | ps | yes |  |  |  |  |  |
Polish | pl | yes | yes | yes | yes | yes |  |
Portuguese | pt | yes | yes | yes | yes | yes | yes | yes
Punjabi | pa | yes | yes |  |  |  |  |
Rhaeto-Romance | rm | yes |  |  |  |  |  |
Romanian | ro | yes | yes | yes | yes | yes |  |
Russian | ru | yes | yes | yes | yes | yes | yes | yes
Sami Northern | se | yes |  |  |  |  |  |
Serbian | sr | yes |  |  | yes |  |  |
Slovak | sk | yes | yes | yes | yes | yes |  |
Slovenian | sl | yes | yes |  | yes |  |  |
Sorbian | wen | yes |  |  |  |  |  |
Spanish | es | yes | yes | yes | yes | yes | yes | yes
Swahili | sw | yes |  |  |  |  |  |
Swedish | sv | yes | yes | yes | yes | yes |  | yes
Tamil | ta | yes | yes |  |  |  |  |
Telugu | te | yes | yes |  |  |  |  |
Thai | th | yes |  |  |  |  |  |
Turkish | tr | yes | yes | yes | yes | yes |  | yes
Ukrainian | uk | yes | yes | yes | yes | yes |  |
Urdu | ur | yes | yes |  | yes |  |  |
Uzbek | uz | yes |  |  |  |  |  |
Vietnamese | vi | yes |  |  |  |  |  |
Welsh | cy | yes |  |  |  |  |  |
Yiddish | yi | yes |  |  |  |  |  |
Zulu | zu | yes |  |  |  |  |  |

1 Offensive content filtering is not provided out of the box

See Also

Concepts

About linguistic features (FAST Search Server 2010 for SharePoint)


About IFilters, files type and indexing connector protocols (FAST Search Server 2010 for SharePoint)

Published: December 2, 2010

Microsoft FAST Search Server 2010 for SharePoint can crawl content either by using the indexing connectors integrated in Microsoft SharePoint Server 2010 (through the FAST Search Content Search Service Application), or by using the FAST Search Server 2010 for SharePoint specific indexing connectors (the FAST Search Lotus Notes connector, the FAST Search database connector and the FAST Search Web crawler).

A number of indexing connector protocols are installed by default in SharePoint Server 2010, whereas other content sources require a different indexing connector and protocol which needs to be installed and configured separately.

All file types are crawled unless explicitly excluded. A number of file types with little relevance to search are excluded by default. You can find the file types that are excluded by default in the article IFilter and file type reference (FAST Search Server 2010 for SharePoint).

While most file types are crawled by default, FAST Search Server 2010 for SharePoint only provides filters for content and metadata extraction of a subset of them. The supported file types are listed in the article IFilter and file type reference (FAST Search Server 2010 for SharePoint). To extract content and metadata from additional crawled file types requires that you enable the Advanced Filter Pack or register a third-party IFilter.

In this section:

  • IFilter and file type reference (FAST Search Server 2010 for SharePoint)

  • Default indexing connector protocols (FAST Search Server 2010 for SharePoint)


IFilter and file type reference (FAST Search Server 2010 for SharePoint)
See Also

Published: December 2, 2010

After content is crawled (either through the FAST Search Content SSA or by using a FAST Search specific connector), the items are submitted to Microsoft FAST Search Server 2010 for SharePoint for item processing. IFilters are used to extract content and metadata from the crawled content.

If you want to index a file type that is not supported by an IFilter that is provided with FAST Search Server 2010 for SharePoint, you should inspect the Crawl Log to determine the necessary steps. For more information and procedures, see Include a file type in the content index (FAST Search Server 2010 for SharePoint).

If you crawl a file type without a registered filter, only the file properties are included in the index and not the actual content of the item. In this case, the index will only contain the metadata properties of the file in the content repository, for example the date the file was created or the title of the file.

File types and IFilters included by default

The following table lists the file types that are supported out-of-the-box by FAST Search Server 2010 for SharePoint.

File name extension:

.doc

.docm

.docx

.dot

.dotx

.eml

.html (and all other HTML files, regardless of file name extension)

.mht

.mhtml

.msg

.nws

.odp

.ods

.odt

.one

.pdf

.pot

.pps

.ppsx (support for this format requires that you have installed Microsoft Office 2010 Filter Pack Service Pack 1)

.ppt

.pptm

.pptx

.pub

.rtf

.txt (and all other plain text files, regardless of file name extension)

.vdw

.vsd

.vss

.vst

.vsx

.vtx

.xlb

.xlc

.xls

.xlsb

.xlsm

.xlsx

.xlt

.xlm

.xps

.zip

File types and IFilters excluded by default

The following table lists the file types that are not crawled by default. You can view the list of excluded file types by looking at the Manage File Types page in the Microsoft SharePoint Server 2010 Central Administration.

File name extension:

.aac

.asf

.asx

.avi

.bmp

.cab

.com

.css

.db

.dll

.dvi

.exe

.gif

.hgx

.ico

.img

.iso

.jar

.java

.jpeg

.jpg

.m4a

.midr

.mp3

.mpeg

.mpg

.msi

.mso

.ogg

.pdb

.png

.prz

.ra

.ram

.rpm

.swf

.sys

.ttf

.vmark

.wav

.wma

.wmf

.wmv

.wrl

See Also

Concepts

Include a file type in the content index (FAST Search Server 2010 for SharePoint)
Delete a file type from the content index (FAST Search Server 2010 for SharePoint)
Enable Advanced Filter Pack (FAST Search Server 2010 for SharePoint)
user_converter_rules.xml reference


Default indexing connector protocols (FAST Search Server 2010 for SharePoint)

Published: December 2, 2010

This article lists the indexing connector protocols that are installed by default in Microsoft SharePoint Server 2010 and are used when you crawl content with the FAST Search Content Search Service Application or people profiles with the FAST Search Query Search Service Application. SharePoint Server 2010 uses indexing connector protocols to connect to different content repositories so that the content in that repository can be crawled. You can install additional indexing connector protocols to crawl content created by other Microsoft products or third-party software. For more information, see Content sources that require additional configuration (FAST Search Server 2010 for SharePoint). You can also develop additional connectors by using the SharePoint Server search connector framework. For more information, see SharePoint Server Search Connector Framework (MSDN).

The following table shows the indexing connector protocols that are installed by default in SharePoint Server 2010.

This indexing connector protocol | Is used to connect to this type of content repository
Bcs | Business Connectivity Services (internal protocol)
Bcs2 | Business Connectivity Services URLs (internal protocol)
File | File shares
http | Web sites
https | Web sites over Secure Sockets Layer (SSL)
Notes | Lotus Notes databases
Rb | Exchange public folders for versions prior to Microsoft Exchange Server 2003
Rbs | Exchange public folders over SSL for versions prior to Exchange Server 2003
Sps | People profiles in Microsoft Office SharePoint Portal Server 2003
Sps3 | People profiles in Microsoft Office SharePoint Server 2007
Sps3s | People profiles in Office SharePoint Server 2007 over SSL
Spss | People profiles in SharePoint Portal Server 2003 over SSL
Sts | SharePoint Team Services 1.0 sites (internal protocol)
Sts2 | Windows SharePoint Services 2.0 and SharePoint Portal Server 2003 sites
Sts2s | Windows SharePoint Services 2.0 and SharePoint Portal Server 2003 sites over SSL
Sts3 | Windows SharePoint Services 3.0 and Office SharePoint Server 2007 sites
Sts3s | Windows SharePoint Services 3.0 and Office SharePoint Server 2007 sites over SSL
Sts4 | Microsoft SharePoint Foundation 2010 and SharePoint Server 2010 sites
Sts4s | SharePoint Foundation 2010 and SharePoint Server 2010 sites over SSL


About backup and restore scripts (FAST Search Server 2010 for SharePoint)
See Also

Published: August 23, 2011

Use the Microsoft FAST Search Server 2010 for SharePoint backup and restore scripts to back up and restore a FAST Search Server 2010 for SharePoint farm. See Backup and restore (FAST Search Server 2010 for SharePoint) for more information.

The backup and restore scripts are stored in the <FASTSearchFolder>\bin folder.

Where <FASTSearchFolder> is the path of the folder where you have installed FAST Search Server 2010 for SharePoint, for example C:\FASTSearch.

Use this script | To do this
backup.ps1 | Run a configuration backup or a full backup.
restore.ps1 | Restore a configuration backup or a full backup.
Tip:

To view a list of options that are available for the backup.ps1 and restore.ps1 scripts, use these commands: get-help backup.ps1 -detailed or get-help restore.ps1 -detailed.

In this article:

  • Backup script options (backup.ps1)

  • Restore script options (restore.ps1)

Backup script options (backup.ps1)

Option | Values | Description
-action | config|full | Specifies the backup type: config: configuration backup full: full backup
-backuppath | <UNC path of backup store> | Specifies the path of the backup store. Always use UNC encoded paths. You can specify multiple backup stores by using a comma-separated list of UNC paths.
-excludeindex |  | Excludes the searchable index from the backup. This option cannot be used with the -sqluser and -sqlpasswordfile options.
-force |  | Overwrites the existing backup directory.
-multithreaded |  | Enables Robocopy to run with multiple threads. This is recommended for full backup of large FAST Search Server 2010 for SharePoint farms. Only supported on Windows Server 2008 R2.
-passwordfile | <name and path of the password file> | Specifies a file that contains a saved password. This allows you to run the script without having to manually enter the password at runtime. See Run scripts with a saved password.
-sqluser | <SQL user account> | Specifies the user name that is used for SQL authentication. This option cannot be used with the -excludeindex option.
-sqlpasswordfile | <name and path of the sql password file> | Specifies the file that contains the password of the user that is used for SQL authentication. The file should contain the password as a secure string. This option cannot be used with the -excludeindex option.

Restore script options (restore.ps1)

Option | Values | Description
-action | config|full|configmigrate | Specifies the restore type: config: configuration restore full: full restore configmigrate: migration of the configuration to a different deployment
-backuppath | <UNC path of backup store> | Specifies the path of the backup store. Always use UNC encoded paths. You can specify multiple backup stores by using a comma-separated list of UNC paths.
-multithreaded |  | Enables Robocopy to run with multiple threads. This is recommended for full restore of large FAST Search Server 2010 for SharePoint farms. Only supported on Windows Server 2008 R2.
-passwordfile | <name and path of the password file> | Specifies a file that contains a saved password. This allows you to run the script without having to manually enter the password at runtime. See Run scripts with a saved password.
-sqluser | <SQL user account> | Specifies the user name that is used for SQL authentication.
-sqlpasswordfile | <name and path of the sql password file> | Specifies the file that contains the password of the user that is used for SQL authentication. The files should contain the password as a secure string.

See Also

Concepts

Backup and restore (FAST Search Server 2010 for SharePoint)


User roles required for deployment tasks
See Also

Published: March 27, 2012

The tasks involved in deploying FAST Search Server 2010 for SharePoint require different user roles.

Deployment

This table lists deployment related tasks and the required user roles for each task.

Task | No specific role required | Member of FASTSearchAdministrators group | FAST Search user | Local Administrator | SQL DBCreator | Domain user | Search Service Account | Query Admin app pool Account | Member of Farm Administrators SharePoint Group
Run the installer |  | x |  | x |  |  |  |  |
Run FAST Search Server 2010 for SharePoint |  |  | x |  |  | x |  |  |
Configure proxy settings |  |  | x |  |  |  | x | x |
Enable Windows PowerShell scripts |  |  |  | x |  |  |  |  |
Set up permissions for MS SQL Server |  |  |  |  | x |  |  |  |
Configure a stand-alone deployment |  |  |  | x |  |  |  |  |
Verify that all modules are running in a stand-alone deployment |  | x |  |  |  |  |  |  |
Prepare the deployment file | x |  |  |  |  |  |  |  |
Configure the administration server in a multiple server deployment |  | x |  | x |  | x |  |  |
Verify that all modules are running |  | x |  |  |  |  |  |  |
Configure the non-administration server(s) in a multiple server deployment |  |  |  | x |  |  |  |  |
Verify that all the modules are running on the non-administration server |  | x |  |  |  |  |  |  |
Create and setup the Content SSA |  |  |  |  |  |  | x |  |
Configure SSL enabled communication |  |  |  | x |  |  |  |  |
Navigate to the Manage Content Sources page |  |  |  |  |  |  |  |  | x
Create and setup the Query SSA |  |  |  |  |  |  | x |  | x
Verify user access to FAST Search Server 2010 for SharePoint administrative services |  | x |  |  |  |  |  |  |
Configure claims authentication |  |  |  | x |  |  |  |  |
Enable or disable Advanced Filter Pack |  |  |  | x |  |  |  |  |
Run scripted installation |  |  |  | x |  |  |  |  |
Change user password |  | x |  |  |  |  |  |  |
Change FAST Search user |  | x |  | x |  |  |  |  |
Change database |  |  |  | x |  |  |  |  |
Enable click-through relevancy after initial configuration |  |  |  | x |  |  |  |  |
Install a software update in a single server or multiple server deployment |  |  |  | x |  |  |  |  |
Uninstall FAST Search Server | x |  |  |  |  |  |  |  |

See Also

Concepts

Deployment for FAST Search Server 2010 for SharePoint


Conformance statement A-level (FAST Search Server 2010)

Published: July 22, 2010

Conformance statement for Microsoft FAST Search Server 2010 for SharePoint in regards to Web Content Accessibility Guidelines 2.0 (http://www.w3.org/TR/WCAG20/).

Conformance statement for configuration and administration experience at A-level

Claim | Details
Date of the claim | May 12th, 2010
Guidelines title, version, and URI | Web Content Accessibility Guidelines 2.0 (http://www.w3.org/TR/2008/REC-WCAG20-20081211/)
Conformance level satisfied: (Level A, AA or AAA) | A
A concise description of the Web pages, such as a list of URIs for which the claim is made, including whether subdomains are included in the claim. | Default content included with FAST Search Server 2010 for SharePoint and Administration and Configuration features of FAST Search Server 2010 for SharePoint. For other product features, see Conformance statement AA-level (FAST Search Server 2010).
A list of the Web content technologies relied upon. | HTML JavaScript CSS
Additional Web content technologies used, but not relied upon. | ARIA Silverlight RSS PNG GIF JPEG
A list of success criteria beyond the level of conformance claimed that have been met. This information should be provided in a form that users can use, preferably machine-readable metadata. | Conforms to 1.4.9 Images of Text (No Exception) - AAA. Conforms to 2.1.3 Keyboard (No Exception) - AAA. Conforms to 2.2.3 No Timing (No Exception) - AAA. Conforms to 2.3.2 Three Flashes - AAA. Conforms to 2.4.8 Location - AAA. Conforms to 3.3.6 Error Prevention (All) - AAA.

This conformance statement pertains to FAST Search Server 2010 for SharePoint.

Customization of the product voids this conformance statement from Microsoft. Customers may make independent conformance statements if they have conducted due diligence to meet all relevant requirements for their customization.

Please consult with Assistive Technology (AT) vendors for compatibility specifications of specific AT products. AT products that are compatible with all the rely-upon-technologies are expected to be functional with the product.

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

Revised 07/22/2010.

Microsoft regularly updates its websites and provides new information about the accessibility of products as that information becomes available.


Conformance statement AA-level (FAST Search Server 2010)

Published: July 22, 2010

Conformance statement for Microsoft FAST Search Server 2010 for SharePoint in regards to Web Content Accessibility Guidelines 2.0 (http://www.w3.org/TR/WCAG20/).

Conformance statement for core reader, author, and contributor experience at AA-level

Claim | Details
Date of the claim | May 12th, 2010
Guidelines title, version, and URI | Web Content Accessibility Guidelines 2.0 (http://www.w3.org/TR/2008/REC-WCAG20-20081211/)
Conformance level satisfied: (Level A, AA or AAA) | AA
A concise description of the Web pages, such as a list of URIs for which the claim is made, including whether subdomains are included in the claim. | All default content included with FAST Search Server 2010 for SharePoint and Web application features of FAST Search Server 2010 for SharePoint except where limited by Conformance statement A-level (FAST Search Server 2010).
A list of the Web content technologies relied upon. | HTML JavaScript CSS
Additional Web content technologies used, but not relied upon. | ARIA Silverlight RSS PNG GIF JPEG
A list of success criteria beyond the level of conformance claimed that have been met. This information should be provided in a form that users can use, preferably machine-readable metadata. | Conforms to 1.4.9 Images of Text (No Exception) - AAA. Conforms to 2.1.3 Keyboard (No Exception) - AAA. Conforms to 2.2.3 No Timing (No Exception) - AAA. Conforms to 2.3.2 Three Flashes - AAA. Conforms to 2.4.8 Location - AAA. Conforms to 3.3.6 Error Prevention (All) - AAA.

This conformance statement pertains to FAST Search Server 2010 for SharePoint.

Customization of the product voids this conformance statement from Microsoft. Customers may make independent conformance statements if they have conducted due diligence to meet all relevant requirements for their customization.

Please consult with Assistive Technology (AT) vendors for compatibility specifications of specific AT products. AT products that are compatible with all the rely-upon-technologies are expected to be functional with the product.

This document is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS OR IMPLIED, IN THIS DOCUMENT.

Revised 07/22/2010.

Microsoft regularly updates its websites and provides new information about the accessibility of products as that information becomes available.


Manage search scopes (FAST Search Server 2010 for SharePoint) (Obsolete)

This page has been moved to Manage search scopes (FAST Search Server 2010 for SharePoint).


Synonyms and query recall (FAST Search Server 2010 for SharePoint) (Obsolete)

This page has been moved to Synonyms and query recall (FAST Search Server 2010 for SharePoint) (Obsolete).


Add or remove user contexts for Best Bets and Visual Best Bets by using Windows PowerShell (FAST Search Server 2010 for SharePoint) (Obsolete)

This topic has been moved to Add, remove and display user contexts by using Windows PowerShell (FAST Search Server 2010 for SharePoint).


Add or remove user contexts for synonyms by using Windows PowerShell (FAST Search Server 2010 for SharePoint) (Obsolete)

This topic has been moved to Add, remove and display user contexts by using Windows PowerShell (FAST Search Server 2010 for SharePoint).


Tune search clickthrough (FAST Search Server 2010 for SharePoint) (Obsolete)

Published: May 29, 2010

This page has been moved to Change the query authority (click-through) weight and resend backlog clicks by using Windows PowerShell (FAST Search Server 2010 for SharePoint)

  • Tune the weight of the search clickthrough boost

  • Resend backlog clicks

Tune the weight of the search clickthrough boost

You can tune the weight of a search clickthrough boost by updating a rank profile. See Manage rank profiles (FAST Search Server 2010 for SharePoint) and Alter boost points for hits in anchor text or associated queries for more information. The rank factor queryauthority governs the weight of the search clickthrough boost.

Resend backlog clicks

The first time that you match a SharePoint Server front-end with a FAST Search Server 2010 for SharePoint back-end, 30 days of clicks are sent to the FAST Search Server 2010 for SharePoint back-end (even if no clicks have been done). This starts the search clickthrough functionality and keeps the system from running 30 days before clickthrough boosting becomes effective. However, if you re-install the FAST Search Server 2010 for SharePoint back-end, make sure that the 30 days of clicks are resent to the back-end by following these steps:

  1. Verify that you have Farm Administrator permissions.

  2. On Microsoft SharePoint Server, open a SharePoint 2010 Management Shell:

    1. On the Start menu, click All Programs.

    2. Click Microsoft SharePoint 2010 Products.

    3. Click SharePoint 2010 Management Shell.

  3. At the command prompt, type the following command(s):

    Copy Code
    $ctejd = Get-SPEnterpriseSearchExtendedClickThroughExtractorJobDefinition -SearchApplication "<SearchApplication>"
$ctejd.ResetLastExtractionDates()
$ctejd.Update()
$ctejd.RunNow()

    Where:

    • <SearchApplication> is the name of your FAST Search Query Search Service Application (SSA).


Configure content collection (FAST Search Server 2010 for SharePoint) (Obsolete)

This page has been moved to Manage crawling (FAST Search Server 2010 for SharePoint).


Synonyms and query recall (FAST Search Server 2010 for SharePoint) (Obsolete)

Published: July 8, 2010

This page has been moved toAdd, remove and display synonyms for keywords by using Windows PowerShell (FAST Search Server 2010 for SharePoint).


Display, create or remove user contexts by using Windows PowerShell (FAST Search Server 2010 for SharePoint) (Obsolete)

This topic has been moved to Add, remove and display user contexts by using Windows PowerShell (FAST Search Server 2010 for SharePoint).

Display all user contexts by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Contexts

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

Create a user context by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$context = $searchSettingGroup.Contexts.AddContext("<ContextName>")
$andExpression = $context.AddAndExpression()
$orExpression = $andExpression.AddOrExpression()
$orExpression.AddMatchExpression("<PropertyName0>","<PropertyValue0>")
$orExpression.AddMatchExpression("<PropertyName1>","<PropertyValue1>")
$notExpression = $andExpression.AddNotExpression()
$notExpression.AddMatchExpression("<PropertyName2>","<PropertyValue2>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <ContextName> is the name of the context you want to create.

    • <PropertyName0> is the name of a user context property to be added to the context.

    • <PropertyValue0> is the value of the corresponding user context property (<PropertyName0>) to be used in the context expression.

    • <PropertyName1> is the name of a user context property to be added to context.

    • <PropertyValue1> is the value of the corresponding user context property (<PropertyName1>) to be used in the context expression.

    • <PropertyName2> is the name of a user context property to be added to context.

    • <PropertyValue2> is the value of the corresponding user context property (<PropertyName2>) to be used in the context expression.

Note:

A user context forms a logical match expression which is evaluated when executing a search. You can make the expression arbitrarily complex by adding AND, OR, and NOT expressions. The procedure above creates an expression with the form:

(<PropertyName0>:<PropertyValue0>OR<PropertyName1>:<PropertyValue1>)AND(NOT(<PropertyName2>:<PropertyName2>))

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

  • Get-FASTSearchSearchSettingGroup

Remove a user context by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Contexts.RemoveContext("<ContextName>")

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

    • <ContextName> is the name of the context you want to delete.

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

  • Get-FASTSearchSearchSettingGroup

Remove all user contexts by using Windows PowerShell

  1. Verify that you meet the following minimum requirements: You are a member of the FASTSearchAdministrators local group on the computer where FAST Search Server 2010 for SharePoint is installed.

  2. On the Start menu, click All Programs.

  3. Click Microsoft FAST Search Server 2010 for SharePoint.

  4. Click Microsoft FAST Search Server 2010 for SharePoint shell.

  5. At the Windows PowerShell command prompt, type the following command(s):

    Copy Code
    $searchSettingGroup = Get-FASTSearchSearchSettingGroup -Name "<SearchSettingGroupName>"
$searchSettingGroup.Contexts.Clear()

    Where:

    • <SearchSettingGroupName> is the name of an existing search setting group.

See also

  • Microsoft.SharePoint.Search.Extended.Administration.Keywords Namespace

  • Get-FASTSearchSearchSettingGroup


Manage keywords (FAST Search Server 2010 for SharePoint) (Obsolete)

This page has been moved to Tune keyword rank (FAST Search Server 2010 for SharePoint).


Add or remove user contexts for document and site promotions by using Windows PowerShell (FAST Search Server 2010 for SharePoint) (Obsolete)

This topic has been moved to Add, remove and display user contexts by using Windows PowerShell (FAST Search Server 2010 for SharePoint).


Configure crawling with the FAST Search Content Search Service Application (Obsolete)

This page has been moved to Manage crawling with the FAST Search Content SSA (FAST Search Server 2010 for SharePoint).


Plan to deploy System Center Operations Manager (SCOM) 2007 (Obsolete)

This topic has been moved to Plan to monitor the health of the search system (FAST Search Server 2010 for SharePoint).