February 2009
Microsoft Office: Business Intelligence with SharePoint and Excel
A good business intelligence strategy can both streamline business processes and boost productivity and profit. And rolling out a solution with tools that you already have may be easier than you think. Find out how you can integrate Microsoft Office Excel and Excel Services with SharePoint to create a business intelligence solution.A good business intelligence strategy can both streamline business processes and boost productivity and profit. And rolling out a solution with tools that you already have may be easier than you think. Find out how you can integrate Microsoft Office Excel and Excel Services with SharePoint to create business intelligence solution. Ben Curry and Bill English with the Microsoft SharePoint teams
SQL Server: Understanding Logging and Recovery in SQL Server
Logging and recovery are critical parts of SQL Server, but they are often poorly understood. Paul Randal gives an overview of how the logging and recovery features work in SQL Server and explains the transaction log and recovery models to help you maintain a better database. Paul S. Randal
Virtualization: Automating Virtual Machine Host Deployment
Hyper-V-based environments are relatively easy to automate, making it a trivial task to deploy and redeploy entire test environments with just a few clicks. Here’s what you need to know to start deploying Hyper-V servers, virtual machines, guest operating systems, and server applications with no need for administrator interaction. Fergus Strachan
Microsoft Dynamics: Deploying Microsoft Dynamics CRM 4.0
CRM is not just a sales and marketing management tool. Microsoft Dynamics CRM is a scalable, enterprise-class system that when appropriately configured and deployed can handle small teams, enterprise-wide solutions and everything in between. Here is a guide to planning the right solution for your environment. Aaron Elder
Columns
From the Editor: Transform Information into Intelligence: | Letters: Readers Speak Out: | Toolbox: New Products for IT Pros: Greg Steen discusses using HTTP Debugger Pro to view and analyze HTTP data, using switcher to switch between applications, launching your applications with RocketDock, backing up your databases with SQLsafe, and performing fast recoveries with ReplayRA.Greg Steen
SQL Q&A: Backup Compression, Client Redirection with Mirroring, and More: Should you enable Backup Compression? How can you avoid dropped application connections during a failover? Paul Randal answers these questions and more in the latest installment of SQL Q&A.Paul S. Randal | Utility Spotlight: Web Page Error Toolkit: Discover how you can use the free Web Page Error Toolkit from Microsoft to replace the generic 404 page with a custom error pages, providing useful details for site visitors who bump into trouble.Lance Whitney | Utility Spotlight: Visual Round Trip Analyzer: The Visual Round-trip Analyzer utility can help you to discover the reasons why your Web pages may be slow to load, and what you can do to make the pages load faster.Jim Pierson
Inside SharePoint: Maintaining Security Account Credentials: In the first part of a two-part series on the standard procedures and tools for maintaining SharePoint security accounts, Pav Cherny explores the architectural details and the complicated process of accomplishing password changes.Pav Cherny | Windows PowerShell: Best Inventory Tool Ever!: In the past few installments of the Windows PowerShell column, Don Jones has been demonstrating how to develop a Windows PowerShell-based inventory tool. In this installment, Don brings it all together into a very powerful, full-featured tool, and offers some guidance on customizing the tool for your own needs.Don Jones | Geek of All Trades: Making Sense of Server Core: In the world of IT publications, the needs of the jack-of-all-trades IT professionals tend to be neglected. Do you work in a small environment where you are responsible for managing everything from network routing to changing toner cartridges? If so, this new column is dedicated to you. In this first installment, Greg Shields discusses using Windows Server 2008 Server Core in a small environment.Greg Shields
Hey, Scripting Guy!: Browsing Active Directory: This installment clears up some confusion over Active Directory Browser, ADSI Scriptomatic, and browsing Active Directory. And in doing so, the Scripting Guy creates a Windows PowerShell script that will allow you to browse the Active Directory schema.The Microsoft Scripting Guys | Security Watch: Malware Inspection at the Perimeter: Learn how to use the Malware Inspection feature of the Microsoft Forefront Threat Management Gateway to mitigate threats at your perimeter.Yuri Diogenes, Mohit Saxena, and Jim Harrison | Field Notes: IT’s Punctuated Equilibrium: What does change really mean in the world of IT? Romi Mahajan discusses three theories of change, offering a historical perspective on how IT has evolved and what we can expect to experience in the near future.Romi Mahajan
Field Notes: Media and the Future of Content: While new technologies and media can facilitiate the speed and breadth of content, it does not impact impact the meaning or the message.Romi Mahajan | Windows Confidential: The Case of the Disappearing Cursor: Have you ever noticed that the cursor blinks when you take a screen capture? Raymond Chen explains why.Raymond Chen


Microsoft Office
Business Intelligence with SharePoint and Excel
Ben Curry and Bill English with the Microsoft SharePoint teams
At a Glance:
  • The Microsoft BI architecture
  • Using Excel for analyzing data
  • Publishing and updating SharePoint lists
  • Working with Excel Services

Contents
The Microsoft BI Big Picture
BI Integration with SharePoint Server 2007
Using Excel for Analyzing Data
Excel Add-Ins
Office BI Integration with SharePoint Lists
Excel Services
Authentication between Client and Server
Publishing Excel Workbooks
Configuring Excel Web Access Web Parts
Unsupported Features in Excel Services
Options and Costs
Today's business decision-makers have a distinct advantage over their predecessors when it comes to business intelligence (BI) tools that they can use for analysis, trending, and presentation of data. A good BI strategy, supported by a solid base of BI technologies, can streamline business processes while also boosting productivity and profit.
The overall BI strategy of Microsoft comprises a suite of server and client-side data integration tools. Powerful analytical and reporting tools in SQL Server 2005 provide the backbone data management infrastructure, while Microsoft Office applications, specifically Microsoft Office Excel, provide the flexibility for information workers to remotely interact with centralized and secure data sources.
This article, which is adapted from the Microsoft Press book Microsoft Office SharePoint Server 2007 Best Practices, focuses on how you can use Excel and Excel Services to analyze business intelligence data within SharePoint sites. Note that some features discussed here, such as data connection libraries, are offered only with SharePoint Server 2007 Enterprise licensing.

The Microsoft BI Big Picture
There are many different ways to analyze external data and publish it to SharePoint sites, including using Excel Services. Depending on your particular business requirements, you may choose a data integration method that will allow an administrator to more selectively create and deploy data reports and minimize end user control, or you may choose to empower end users by enabling them to access data sources and build custom reports directly inside SharePoint. The Microsoft BI suite includes the flexibility to securely and selectively deploy data based on user credentials.
The Microsoft BI strategy includes three main components:
  • The BI platform
  • End-user tools and performance management applications
  • The delivery mechanisms
The BI platform contains the various database and data warehousing components for data storage and schema. SQL Server 2005 Analysis Services (SSAS) provides a semantic model, referred to as a unified dimensional model (UDM). The UDM defines business entities, business logic, calculations, and metrics and provides a bridge between end users and data sources. End users can run queries directly against the UDM using BI tools, such as Excel and Office PerformancePoint Server 2007. Figure 1 illustrates how a user can query SQL Server data sources via the UDM.
Figure 1 Interaction between end user tools and the BI platform
Figure 2 illustrates the overall integrated Microsoft BI solution. SharePoint Server 2007 is at the top of the stack and acts as the hub through which information workers can collaboratively access and analyze data.
Figure 2 Architecture of the integrated MicrosoftBI solution

BI Integration with SharePoint Server 2007
Excel, Excel Services, SharePoint Report Center, and the Business Data Catalog (BDC) are the default, or core, BI features within the Office suite of products. SQL Server 2005 Reporting Services, Analysis Services, and PerformancePoint Server 2007 extend the core BI functionality by offering enhanced reporting capabilities, data warehousing, in-depth data analysis, and real-time monitoring. Figure 3 shows the overall relationship between the various BI applications. Note how pivotal SharePoint Server 2007 is as the negotiation point for data between server and client applications.
Figure 3 SharePoint Server 2007 at the heart of the Microsoft BI solution

Using Excel for Analyzing Data
Excel integrates well with both SQL Server 2005 and SharePoint Server 2007 and plays a pivotal role in providing end users with the tools for data analysis, data-mining construction, and the creation of PivotTables. Excel 2007 enhances the user's ability to access and analyze data from SSAS cubes using PivotTables and PivotCharts. Analysis Services data can also be accessed with Excel 2003 via PivotTables, but this involves some additional configuration.
Excel 2007 provides new features, such as the ability to view and work with Analysis Services KPIs and flexibility for building KPI scorecards. (Note that we refer to reports, dashboards, scorecards, and key performance indicators—or KPIs—as the means for analyzing, preparing, and presenting data within SharePoint Server 2007.) The screen in Figure 4 shows an Excel PivotTable. The workbook is directly linked to an SSAS cube. Fields from the PivotTable Field List, shown to the right of the figure, can be dragged and dropped onto the main workspace to create scorecards, including KPIs.
Figure 4 Excel workbook connected directly to SSAS showing PivotTable functionality
Excel workbooks can be stored within SharePoint document libraries, where they can be shared and versioned. Historical details can be dynamically captured and stored for each workbook. More importantly, Excel workbooks containing active connections to Analysis Services can be published to Excel Services and made available for viewing through a Web browser without the need for an Excel client.

Excel Add-Ins
Two Excel BI add-ins assist with remote data analysis and end-to-end data submission. The SQL Server 2005 Data Mining add-ins for Office 2007 include custom data mining and forecasting tools for Excel, such as the ability to estimate and predict based on certain patterns within selected data fields. Figure 5 shows the Data Mining menu on the Excel 2007 Ribbon.
Figure 5 Excel 2007 Ribbon showing the Data Mining menu options
A data mining Visio template is also included as part of the Data Mining add-in, and you can easily generate dependency network, cluster, and decision trees based on mining models from SSAS. The data mining add-in can be used to create data mining models on existing analysis server data or to perform additional queries in Excel workbooks. Figure 6 shows an example of a predictive model based on existing analysis server data.
Figure 6 Data-mining model decision tree based on an existing Excel 2007 workbook analysis server connection
As a best practice, use the PerformancePoint add-in for Excel to enable users to access assigned tasks remotely using Excel and submit updated worksheets to PerformancePoint Server for approval. (Figure 7 shows the PerformancePoint add-in for Excel actively selected in the Excel 2007 Ribbon.) The add-in provides equivalent functionality in both Excel 2007 and Excel 2003, with the exception that some formatting features specifically related to pivot styles will not be available in Excel 2003. Note that the PerformancePoint add-in for Excel requires an existing installation of either Excel 2003 (SP2) or Excel 2007. Further details about prerequisites can be found in the article "Install the PerformancePoint Add-in for Excel."
Figure 7 Excel 2007 Ribbon showing the PerformancePoint menu options

Office BI Integration with SharePoint Lists
Both Excel and Access continue to play a key role in the negotiation between client and server BI functionality within SharePoint Server 2007, and they provide the ability to expose data within SharePoint sites. For example, Excel worksheets can be published as a SharePoint list in which the data can be accessed and updated by multiple users.
The following are a few scenarios you should consider when using Access databases and assessing your data analysis needs:
  • You can publish an Access database as a SharePoint list and perform additional data analysis in SharePoint. Other key benefits include filtering and sorting.
  • Access 2007 databases can be upgraded to SQL Server 2005 databases using the Access 2007 Upsizing Wizard. For instance, you could upsize an existing Access 2007 database to SQL Server 2005 for a scalable database solution and then use Reporting Services to create custom reports, which you could then publish directly to SharePoint Server 2007.
  • You could use Access 2007 as the front-end reporting mechanism for SharePoint lists. Access 2007 includes the ability to import a SharePoint list, or multiple SharePoint lists, into an Access 2007 database to perform additional reporting while maintaining an active link back to the list on the SharePoint server.
There are some key differences between working with data between Excel 2003 and Excel 2007, which are covered in the following sections.
Excel-to-SharePoint and Data Synchronization New lists within SharePoint sites can be created directly from existing Excel spreadsheets (for example, by selecting Create  | Custom Lists | Import Spreadsheet). SharePoint will import all columns and any existing data, or you can specify a range of cells to import, such as a range of cells, a table range, or a named range. Existing formulas within the source Excel worksheet will be converted to calculated columns.
If you are using Excel 2003, you can create a SharePoint list directly from within Excel by clicking a single cell in the Excel spreadsheet and selecting List | Publish List. This creates a new SharePoint list in Datasheet view. Note that when you are publishing from Excel 2003 to a SharePoint list, only a single worksheet will be published—in other words, if you have an Excel workbook with multiple worksheets, not all worksheets will be published.
Linkage can be maintained between the original Excel 2003 file and the newly created SharePoint list and synchronization can be performed both ways. That is, you can add new rows to the source Excel 2003 worksheet and have the linked SharePoint list updated. Likewise, you can update the SharePoint list and have the source Excel worksheet updated.
Using Excel 2007, you have two options for publishing data to SharePoint:
  • You can publish to Excel Services. If Excel Services is installed as part of your SharePoint Server 2007 deployment, a one-way sync will exist between the source Excel 2007 file and the published server version. In other words, updates to the source file can be pushed to an existing published version and changes reflected within a Web page.
  • You can export a table within Excel 2007 to a SharePoint list. Once a table has been exported, no linkage remains between the source and destination data. That is, updating the original Excel file will not then update data in the SharePoint list, and data updated in the SharePoint list will not update the original Excel 2007 file.
The main difference between publishing from Excel 2003 and publishing from Excel 2007 to a SharePoint site is that you can publish not only a single Excel worksheet but an entire Excel workbook if you are using Excel 2007 with Excel Services installed. In Excel 2007, the publishing option has been built into the main File menu and will publish an entire Excel workbook to Excel Services.
SharePoint-to-Excel and Data Synchronization Lists within SharePoint sites can be exported to Excel, and you can choose to maintain a link between the exported data and the original SharePoint list. Updates to the SharePoint list will be synced to the external Excel spreadsheet. This is a one-way sync—updates to the exported Excel spreadsheet will not replicate to the original SharePoint list. Note that using the Export To Excel option from the Actions menu within a SharePoint Server 2007 document library or list, either in Standard or Datasheet view, requires the Excel 2007 client to be present.
Using the Datasheet view, you can directly interact with and manipulate data (such as selecting and deleting multiple rows of data) within the Datasheet view in the browser. Datasheet views also enable you to perform additional data analysis, such as exporting and reporting on list data within Access and using Excel to query, print, chart, and create PivotTables, as shown in the right task pane in Figure 8. Both Excel 2003 and Excel 2007 are supported when you select the Query List with Excel task pane option within Datasheet view.
Figure 8 The Datasheet view
Keep in mind that the Datasheet view requires the presence of a client-side Windows SharePoint Services-compatible list datasheet control (such as Excel 2003, Excel 2007, Access 2003, or Access 2007) and ActiveX control support. This differs from the Excel Web Access Web Part, which is used in conjunction with Excel Services, does not require ActiveX control and does not have client-side dependencies.
Consequences of Excel Add-Ins
Be aware that certain Excel add-ins may cause an issue when you attempt to integrate Excel with SharePoint lists. Specifically, if you're attempting to create a new SharePoint (SharePoint Server 2007 or Windows SharePoint Services 3.0) list from either an Excel 2007 or Excel 2003 spreadsheet and you receive the error "Method 'post' of object 'IOW-SPostData' failed," the problem could be due to one or more Excel add-ins.
I've experienced a problem specifically with the Analysis ToolPack add-in, which is included as part of Excel 2007 out-of-the-box add-in options. There is a client-side solution to this problem that involves editing the EXPTOOWS.XLA file on the client and editing the file in Excel using the Visual Basic code editor. For details on how to accomplish this, see my blog post at "Method 'post' of object 'IOWSPostData' failed."
—Kathy Hughes, Microsoft MVP

Excel Services
Excel Services is part of SharePoint Server 2007 Enterprise Edition. A benefit of this edition is that you can publish an Excel workbook to a Web page and end users can view workbook contents without the presence of the Excel client. However, for users to interact with Excel Services, clients require the Office SharePoint 2007 Enterprise Client Access License (CAL).
The published workbook is effectively a read-only version of the source Excel workbook. This provides a good way to share information while protecting the integrity of the data—a published Excel workbook, such as an annual or fiscal budget, is a centralized, single version accessed by all. Here we will cover the key features of Excel Services.
Unlike creating a new SharePoint list by importing an Excel spreadsheet or publishing an Excel 2003 worksheet, you can use Excel 2007 to publish an entire Excel workbook to Excel Services. Within each workbook, you can limit the number of worksheets to be published or elect to publish all.
The three main Excel Services components are detailed here:
  • The Excel Calculation Services (ECS) component loads the Excel workbook into a Web page, performs server-side calculations, and refreshes external data.
  • The Excel Web Access (EWA) Web Part displays a workbook or interconnecting parts of a workbook on a Web page or dashboard. It uses DHTML and JavaScript for Web page interaction, thus avoiding the need to download ActiveX controls to the client. EWA Web Parts can be configured to filter on specific values. EWA is unlike the Datasheet view available in SharePoint lists, which is dependent upon ActiveX controls along with the Excel or Access client. And EWA offers additional features, such as the ability to create a snapshot of an Excel workbook.
  • Excel Web services (EWS) is a Web service API that developers can use to develop custom or add-on applications for Excel Services. The typical address to access these Web services is server_site_name/_vti_bin/excelservice.asmx.
Numerous configuration options are available when you deploy Excel Services. Typically, Excel Services installs EWA and EWS on the front-end Web server and one ECS on the application server. In a single-server deployment, EWA, EWS, and ECS are installed on the same server. In multi-server deployments, the components can be installed on separate servers. A major consideration in deploying Excel Services, and one you will want to plan carefully, is how the authentication between servers and clients is negotiated.

Authentication between Client and Server
In a distributed server environment where SQL Server 2005 and Analysis Services are on a different machine than SharePoint Server 2007 and Excel Services, and where you are leveraging external data connections—that is, an Office Data Connection (ODC) file—to your SSAS as the basis for your published Excel workbook, you need to configure Kerberos to impersonate the user accessing the database server and trust/delegation between servers. The same also applies when you are implementing Reporting Services integration where SQL Server 2005 and Reporting Services are on a different machine than SharePoint Server 2007.

Publishing Excel Workbooks
As a best practice, be sure to establish trusted file locations and trusted data connection libraries, as defined in the Excel Services Settings in Shared Services, before publishing your Excel workbooks. They form the basis for successfully publishing Excel workbooks, such as publishing an online analysis processing (OLAP) PivotTable, which uses an ODC file located within a Trusted Data Connections Library. Excel 2007 includes an option within the main File menu to publish to Excel Services. (Note that the Excel workbook must first be saved to a trusted file location.)
ECS performs calculations included within the source Excel workbook on the server, which means that actual functions and formulas are not directly accessible for end users.
The main inference in utilizing Excel Services is that an Excel workbook can be published to a Web page, but the published version is not editable. Users cannot make edits or enter new data and then save the changes to the source Excel file/workbook. Instead, users can perform run-time calculations via the browser based on parameters created in the source Excel workbook. Once a Web page containing EWA is refreshed or a user ends her browser session, any calculated changes are lost.
Snapshots of Excel workbooks can be captured and saved as separate Excel files to the client machine. A snapshot includes the data values within the published Excel workbook and excludes any formulas or other business logic from the source workbook. End users with viewer permissions to the document library containing the source Excel file and location of the published Excel file are able to copy the published Excel workbook to an Excel file (including data values and formulas) and open a snapshot of an Excel workbook. Figure 9 shows both the Open In Excel and Open Snapshot In Excel options available from the EWA Web Part toolbar.
Figure 9 Excel export options available within the EWA Web part toolbar
Excel 2007 is required for Excel snapshots and Excel copies from an EWA Web Part. If Excel 2007 is not installed on the client machine where a snapshot or copy is being attempted, the following message is displayed: "The workbook cannot be opened. Excel may not be installed properly (or was set to install on first use), the path to the document may be incorrect, or your session may have timed out. Try opening the workbook again, or clicking Reload on the Excel Web Access toolbar."
Excel 2007 workbooks that contain multiple worksheets can be published to Excel Services. The person publishing the workbook can choose which worksheets within a workbook to display at the time of saving and publishing to Excel Services. Portions of an Excel worksheet can also be published based on named cells, or parameters, defined within the source Excel workbook.
As a best practice, you should enable auditing, versioning, and content approval on the Reports document library (or the nominated document library location) for published Excel workbooks to monitor edits and updates to the source Excel workbooks.

Configuring Excel Web Access Web Parts
Once an Excel 2007 workbook is published to Excel Services, the EWA Web Part provides the mechanism for viewing and manipulating the published workbook within the browser. (The EWA does not have client-side dependencies, nor does it require an ActiveX control to be downloaded to the client.)
An entire Excel workbook can be displayed in a single EWA. End users can navigate worksheets within a published workbook just as they would when using the Excel client. Sections of Excel workbooks can also be divided into individual EWA Web Parts on a Web page (also referred to as a dashboard) based on named parameters within the source Excel 2007 workbook. Figure 10 shows several EWA Web Parts on a single Web page, which includes filtering and Web Part connections between each EWA Web Part to dynamically refresh and display data based on the name of the account representative selected.
Figure 10 Dashboard showing multiple instances of EWA Web parts
Using filtering, Excel parameters, and Web Part connections between EWA Web Parts, you can generate powerful and intuitive dashboards within SharePoint Server 2007.

Unsupported Features in Excel Services
Not all Excel client-application features are supported in Excel Services. For example, attempting to publish Excel files that contain code—such as Visual Basic for Applications macros, embedded pictures, or clip art—will result in failure to publish. Other non-supported client-side functionality includes, but is not limited to, data validation and external references to linked workbooks.
It is not always apparent whether an Excel workbook is compatible with Excel Services until you try to publish it. However, there is an open source Excel Services Compatibility Checker (see "Excel Services Compatibility Checker Addin - Beta" for details and get the download at Excel Services Compatibility Checker Download Page).

Options and Costs
If you currently have the standard version of SharePoint Server 2007 and are considering integrating Excel workbooks with SharePoint sites as part of your overall BI solution, consider the following options:
  1. Upgrade to SharePoint Server 2007 Enterprise Edition and use Excel Services.
  2. Keep SharePoint Server 2007 Standard Edition and use the built-in document management features within SharePoint document libraries to store, version, and share Excel workbooks in their native format.
  3. Keep SharePoint Server 2007 Standard Edition and create a new SharePoint list from an existing Excel spreadsheet.
Option 1 incurs additional costs and deployment considerations, such as changing to Kerberos (if you're not already running in Kerberos mode) and upgrading to Excel 2007 (if you're currently using Excel 2003). However, it provides a more scalable Excel solution without the need for the Excel client or ActiveX control.
Option 2 means that the Excel workbook will be retained in its original native format and can be checked out and edited by authorized users. However, the Excel client must be present on those clients responsible for updating Excel workbooks. Excel workbooks uploaded to a SharePoint document library can be secured using the security features in Excel, but there is a risk that end users will still be able to republish the workbook elsewhere, resulting in duplicate copies of the same workbooks.
Option 3 means that, although a new list will be created from an Excel worksheet, there are implications in continuing to update the list from an Excel client or the portability of data within the list. SharePoint lists have limitations for data types and Excel functions, and it is difficult to append updates to an existing list from an Excel workbook. Also, if your organization is currently running Excel 2003, users will experience compatibility issues when you upgrade to Excel 2007. Unlike Excel 2003, Excel 2007 does not provide two-way synchronization with SharePoint.
If you plan to stay with the Standard Edition and you are using multiple Excel workbooks throughout your organization (and custom formulas), then we recommend that you maintain the existing Excel workbooks in their native Excel format and utilize the document management features within SharePoint document libraries, such as versioning and approval, so you can monitor who has updated what and when. In addition, you should lock down access to Excel workbooks using the built-in SharePoint security model and give edit rights only to those users who will actually need to update existing workbooks and upload new workbooks.
If you are contemplating upgrading at a later stage and the basis of that upgrade is BI—specifically, Excel features—carefully consider the business requirements. For instance, are you planning to deploy a read-only version of Excel workbooks, or are you planning to have a distributed Excel authoring environment? If it's the latter, consider purchasing PerformancePoint Server 2007.
Remember, while Excel Services enables you to publish an entire Excel 2007 workbook as a Web page in which end users can access Excel data and perform run-time calculations (without being able to save data back to the source Excel workbook), the onus of updating and republishing an Excel workbook falls upon one or several people. PerformancePoint Server 2007, on the other hand, offers the ability to distribute Excel tasks to end users, providing centralized and secure control over user access and Excel templates.
Using Analysis Services Data in Excel Services
I started working with Excel Services, in combination with Analysis Services data, in my virtual environment. This basically meant that everything including client, SQL Server 2005, SharePoint Server 2007, and Excel Services were all configured on the one virtual machine. The Web application was configured to use NTLM and everything worked seamlessly. However, enter a real-world scenario where the customer's deployment consists of a multiple-server environment—for instance, where SharePoint Server 2007 and Excel Services are hosted on one server and SQL Server 2005 and Analysis Services are hosted on another—and you suddenly realize that NTLM just won't cut it!
In order for Excel Services to successfully access a separate SQL server, you need to impersonate and delegate the user account. You can do this by implementing Kerberos authentication, or single sign-on (SSO). Simply using NTLM is not sufficient because the Web server cannot delegate the current user to the SQL server, also referred to as a double hop.
A best practice when testing an Excel Services and Analysis Services scenario is to test within a multi-server environment so you can accurately test authentication and ensure that clients and servers can successfully access back-end Analysis Services cubes and data. This will better prepare you for real-world scenarios and deployments.
—Kathy Hughes, Microsoft MVP
This article is adapted from the book Microsoft Office SharePoint Server 2007 Best Practices (Microsoft Press, 2008).

Ben Curry and Bill English are the authors of Microsoft Office SharePoint Server 2007 Best Practices (Microsoft Press, 2008).


SQL Server
Understanding Logging and Recovery in SQL Server
Paul S. Randal
At a Glance:
  • How logging and recovery function in SQL Server
  • How the transaction log works and what you need to know about managing it
  • Recovery models and their effect on logging

Contents
What Is Logging?
What Is Recovery?
The Transaction Log
Recovery Models
Wrapping Up
Some of the most misunderstood parts of SQL Server are its logging and recovery mechanisms. The fact that the transaction log exists and can cause problems if not managed correctly seems to confound many "involuntary DBAs." Why is it possible for the transaction log to grow unbounded? Why does it sometimes take so long for the database to come online after a system crash? Why can't logging be turned off completely? Why can't I recover my database properly? Just what is the transaction log and why is it there?
These are all questions I see repeatedly on SQL Server forums and newsgroups, so in this article I want to provide an overview of the logging and recovery system and explain why it is such an integral part of the SQL Server Storage Engine. I will explain the architecture of the transaction log and how the three recovery models available for a database can change the behavior of the transaction log and the logging process itself. Along the way, I'll also provide some links to resources covering transaction log management best practices.

What Is Logging?
Logging and recovery are not concepts that are unique to SQL Server—all commercial relational database management systems (RDBMSs) must have them to support the various ACID properties of transactions. ACID stands for Atomicity, Consistency, Isolation, and Durability, which are the fundamental properties of a transaction-processing system (such as an RDBMS). You can read more about this in the ACID Properties section of the MSDN Library.
Video
Paul Randal demonstrates the importance of managing your transaction log properly in the full recovery model, and he shows you techniques for doing so in SQL Server.

Operations in an RDBMS are logged (or recorded) at the physical and logical level in terms of what happens in the storage structures of the database. Each change to the storage structures has its own log record, which describes the structure being changed and what the change was. This is done in such a way that the change can be replayed or reversed, if necessary. The log records are stored in a special file called the transaction log—I will describe this in more detail later, but for now you can think of it as a sequential-access file.
A set of one or more such changes can be (and in fact, always are) grouped together in a transaction—which provides the basic unit of making changes (atomicity) to a database as far as users, application developers, and DBAs are concerned. A transaction either succeeds (committing) or fails/is canceled (rolling back). In the first case, the operations that form the transaction are guaranteed to be reflected in the database. In the second case, the operations are guaranteed not to be reflected in the database.
Transactions in SQL Server are either explicit or implicit. An explicit transaction is one where the user or application issues a BEGIN TRANSACTION T-SQL statement, signaling the start of a group of related changes by that session. An explicit transaction succeeds when a COMMIT TRANSACTION statement is issued, signaling the successful completion of the group of changes. If a ROLLBACK TRANSACTION statement is issued instead, all changes made by that session since the BEGIN TRANSACTION statement was issued are reverted (rolled back) and the transaction is aborted. A transaction rollback could also be forced by an external event, such as the database running out of disk space or a server crash, as I will explain later.
An implicit transaction is one where the user or application does not explicitly issue a BEGIN TRANSACTION statement before issuing a T-SQL statement. However, given that all changes to the database must be transactional, the Storage Engine will automatically start a transaction under the covers. When the T-SQL statement completes, the Storage Engine automatically commits the transaction that it started to wrap around the user's statement.
You might think that this is not necessary because a single T-SQL statement won't generate a large number of changes to the storage structures of the database, but consider something like an ALTER INDEX REBUILD statement. Although this statement can't be contained within an explicit transaction, it could generate an enormous number of changes to the database. So there must be some mechanism for ensuring that if something does go wrong (the statement is canceled, for instance), all of the changes are properly reversed.
As an example, consider what happens when a single table row is updated in an implicit transaction. Imagine a simple heap table with an integer column c1 and a char column c2. The table has 10,000 rows, and a user submits an update query as follows:
UPDATE SimpleTable SET c1 = 10 WHERE c2 LIKE '%Paul%';
The following operations take place:
  • The data pages from SimpleTable are read from disk into memory (the buffer pool) so they can be searched for matching rows. It turns out that three data pages hold five rows that match the WHERE clause predicate.
  • The Storage Engine automatically starts an implicit transaction.
  • The three data pages and five data rows are locked to allow the updates to occur.
  • The changes are made to the five data records on the three data pages in memory.
  • The changes are also recorded in log records in the transaction log on disk.
  • The Storage Engine automatically commits the implicit transaction.
Note that I didn't list a step where the three updated data pages are written back out to disk. This is because they don't yet need to be; as long as the log records describe the changes are on disk in the transaction log, then the changes are protected. If the pages need to be subsequently read or changed again, then the most up-to-date copy of the page is already in memory, just not on disk (yet). The data pages will be written out to disk when the next checkpoint operation occurs or if the memory that they use in the buffer pool is required for another page image.
Checkpoints exist for two reasons—to batch up write I/Os to improve performance and to reduce the time required for crash recovery. In terms of performance, if a data page were forced out to disk each time it was updated, the number of write I/Os occurring on a busy system could easily overwhelm the I/O subsystem. It's better to periodically write out dirty pages (pages that have been changed since being read from disk) than to write out pages immediately as they are changed. I'll discuss the recovery aspect of checkpoints in a moment.
One common misconception about checkpoints is that they only write out pages with changes from committed transactions. This is not true—a checkpoint always writes out all dirty pages, regardless of whether the transaction that changed a page has committed or not.
Write-ahead logging is the mechanism wherein the log records describing a change are written to disk before the changes themselves are written. It provides the durability part of the ACID properties. As long as the log records describing changes are on disk, in the event of a crash, the log records (and hence the changes themselves) can be recovered and the effects of the transaction are not lost.

What Is Recovery?
Looking for SQL Server Tips?

For tips on SQL Server, visit the TechNet Magazine SQL Server Tips page.
For more Tips on other products, visit the TechNet Magazine Tips index.
Logging exists to support a variety of operations in SQL Server. It ensures that if a crash occurs, a committed transaction will be correctly reflected in the database after the crash. It ensures that an uncommitted transaction will be correctly rolled back and not reflected in the database after a crash. It ensures that it is possible to cancel an in-flight transaction and have all its operations rolled back. It allows a backup copy of the transaction log to be taken so that a database can be restored and the transaction log backups replayed to bring the database to a specific point in time with transactional consistency. And it supports features that rely on reading the transaction log, such as replication, database mirroring, and change data capture.
The majority of these uses of logging involve a mechanism called recovery. Recovery is the process of having the changes described in log records replayed or reverted in the database. Replaying log records is called the REDO (or roll forward) phase of recovery. Reverting log records is called the UNDO (or roll back) phase of recovery. In other words, recovery will make sure that a transaction and all its constituent log records are either redone or undone.
The simple form of recovery is when a single transaction is canceled, in which case it is undone and there is no net effect on the database. The most complex form is crash recovery—when SQL Server crashes (for whatever reason) and the transaction log must be recovered to bring the database to a transactionally consistent point. This means that all transactions that were committed at the time of the crash must be rolled forward to ensure their effects are persisted in the database. And all in-flight transactions that had not committed at the time of the crash must be rolled back to ensure their effects are not persisted in the database.
This is because there is no facility for a transaction in SQL Server to continue after a crash. Thus, if the effects of a partially complete transaction were not rolled back, the database would be left in an inconsistent state (possibly even structurally corrupt, depending on what the transaction was in the middle of doing).
So how does recovery know what to do? All recovery processes depend on the fact that each log record is stamped with a log sequence number (LSN). A log sequence number is an ever-increasing, three-part number that uniquely defines the position of a log record within the transaction log. Each log record in a transaction is stored in sequential order within the transaction log and contains the transaction ID and the LSN of the previous log record for the transaction. In other words, each operation that is recorded as part of the transaction has a "link" back to the operation that immediately preceded it.
For the simple case of a single transaction being rolled back, the recovery mechanism can easily and quickly follow the chain of logged operations from the most recent operation back to the first operation and undo the effects of the operations in the opposite order from which they occurred. The database pages that were affected by the transaction are either still in the buffer pool or on disk. In either case, the image of the page that is available is guaranteed to be one where the effect of the transaction is reflected on the page and must be undone.
During crash recovery, the mechanism is more complicated. The fact that database pages are not written to disk when a transaction commits means that there is no guarantee that the set of database pages on disk accurately reflects the set of changes described in the transaction log—either for committed or uncommitted transactions. However, there is one final piece of the puzzle that I haven't mentioned yet—all database pages have a field in their page header (a 96-byte portion of the 8192-byte page that contains metadata about the page) that contains the LSN of the last log record that affected the page. This allows the recovery system to decide what to do about a particular log record that it must recover:
  • For a log record from a committed transaction where the database page has an LSN equal to or greater than the LSN of the log record, nothing needs to be done. The effect of the log record has already been persisted on the page on disk.
  • For a log record from a committed transaction where the database page has an LSN less than the LSN of the log record, the log record must be redone to ensure the transaction effects are persisted.
  • For a log record from an uncommitted transaction where the database page has an LSN equal to or greater than the LSN of the log record, the log record must be undone to ensure the transaction effects are not persisted.
  • For a log record from an uncommitted transaction where the database page has an LSN less than the LSN of the log record, nothing needs to be done. The effect of the log record was not persisted on the page on disk and as such does not need to be undone.
Crash recovery reads through the transaction log and ensures that all effects of all committed transactions are persisted in the database, and all effects of all uncommitted transactions are not persisted in the database—the REDO and UNDO phases, respectively. Once crash recovery completes, the database is transactionally consistent and available for use.
I mentioned earlier that one of the uses of a checkpoint operation is to reduce the amount of time that crash recovery takes. By periodically flushing out all dirty pages to disk, the number of pages that have changed because of committed transactions but whose images are not on disk is reduced. This, in turn, reduces the number of pages that need to have REDO recovery applied during crash recovery.

The Transaction Log
Crash recovery is only possible if the transaction log is intact. In fact, the transaction log is the most important part of the database—it's the only place where all changes to the database are guaranteed to be described in the event of a crash.
If the transaction log is missing or damaged after a crash, then crash recovery cannot complete, leading to a suspect database. In that case, the database must be restored from backups or recovered using less desirable options, such as emergency mode repair. (These procedures are beyond the scope of this article but will be covered in depth in articles later in the year.)
The transaction log is a special file that each database must have to function correctly. It holds the log records that are produced from logging and is used to read them again during recovery (or any of the other uses of logging I've already mentioned). As well as the space occupied by the log records themselves, a transaction will also reserve space in the transaction log for any potential log records required if the transaction were to be canceled and required to roll back. This accounts for the behavior you may observe where, say, a transaction that updates 50MB of data in the database may actually require 100MB of transaction log space.
When a new database is created, the transaction log is essentially empty. As transactions occur, log records are written sequentially to the transaction log, which means there is no performance gain from creating multiple transaction log files—a very common misconception. The transaction log will use each log file in turn.
Log records for concurrent transactions can be interspersed in the transaction log. Remember that log records for a single transaction are linked by their LSNs, so there is no need for all log records for a transaction to be grouped together in the log. LSNs can almost be thought of as a timestamp.
The physical architecture of the transaction log is shown in Figure 1. It is split internally into smaller chunks called virtual log files (or VLFs). These are simply an aid to easier internal management of the transaction log. When a VLF becomes full, logging automatically proceeds to use the next VLF in the transaction log. You might think that eventually the transaction log will run out of space, but this is where the transaction log is so different from data files.
Figure 1 Physical architecture of the transaction log
The transaction log is really a circular file—as long as the log records at the start of the transaction log have been truncated (or cleared). Then when logging reaches the end of the transaction log, it wraps around to the start again and begins overwriting what was there before.
So how do log records get truncated so the space they occupied can be reused? A log record is no longer needed in the transaction log if all of the following are true:
  • The transaction of which it is part has committed.
  • The database pages it changed have all been written to disk by a checkpoint.
  • The log record is not needed for a backup (full, differential, or log).
  • The log record is not needed for any feature that reads the log (such as database mirroring or replication).
A log record that is still needed is called active, and a VLF that has at least one active log record is also called active. Every so often, the transaction log is checked to see whether all the log records in a full VLF are active or not; if they are all inactive, the VLF is marked as truncated (meaning the VLF can be overwritten once the transaction log wraps). When a VLF is truncated, it is not overwritten or zeroed in any way—it is just marked as truncated and can then be reused.
This process is called log truncation—not to be confused with actually shrinking the size of the transaction log. Log truncation never changes the physical size of the transaction log—only which portions of the transaction log are active or not. Figure 2 shows the transaction log from Figure 1 after truncation has occurred.
Figure 2 The transaction log after log truncation
Active VLFs make up the logical log—the portion of the transaction log that contains all the active log records. The database itself knows where crash recovery should start reading log records within the active portion of the log—the start of the oldest active transaction in the log, the MinLSN (this is stored in the database boot page).
Crash recovery does not know where to stop reading log records, so it continues until it reaches a zeroed section of the transaction log (if the transaction log has not yet wrapped) or a log record whose parity bits do not fit the sequence from the previous log record.
As VLFs become truncated and new ones become active, the logical log moves within the physical transaction log file and eventually should wrap around to the start again, as shown in Figure 3.
Figure 3 The circular nature of the transaction log
The check whether log truncation can take place under either of the following circumstances:
  • When a checkpoint occurs in the SIMPLE recovery model or in other recovery models when a full backup has never been taken. (This implies that a database will remain in a pseudo-SIMPLE recovery model after being switched out of SIMPLE until a full database backup occurs.)
  • When a log backup completes.
Remember that log truncation may not be possible because of the many reasons a log record must remain active. When log truncation cannot occur, the VLFs cannot be truncated and eventually the transaction log has to grow (or another transaction log file be added). Excessive transaction log growth can cause performance problems through a phenomenon known as VLF fragmentation. Removing VLF fragmentation can sometimes lead to a dramatic improvement in the performance of log-related activities.
For more information on this, see Kimberly Tripp's blog post "8 Steps to Better Transaction Log Throughput." Kimberly discusses best practices pertaining to transaction log capacity planning, management, and performance improvements—it's well worth reading!
There are two common issues that can prevent log truncation:
  • A long-running active transaction. The entire transaction log since the first log record from the oldest active transaction can never be truncated until that transaction commits or aborts.
  • Switching to the FULL recovery model, taking a full backup, and then never taking any log backups. The entire transaction log will remain active, waiting to be backed up by a log backup.
For a complete list of factors and instructions on how to determine what is preventing log truncation, see the SQL Server Books Online topic "Factors that Can Delay Log Truncation." I've also created a video demonstration that shows the effect of uncontrolled transaction log growth and how to remove VLF fragmentation—check out this video screencast (and previous screencasts on SQL topics) at technetmagazine.com/videos.
If the transaction log does grow to capacity and cannot grow any further, then error 9002 will be reported and you will need to take steps to provide more space, such as growing the log file, adding another log file, or removing any impediment to the log being truncated.
Under no circumstances should you delete the transaction log, try to rebuild it using undocumented commands, or simply truncate it using the NO_LOG or TRUNCATE_ONLY options of BACKUP LOG (which have been removed in SQL Server 2008). These options will either cause transactional inconsistency (and more than likely corruption) or remove the possibility of being able to properly recover the database.
For more information on how to troubleshoot a full transaction log, check out the Books Online topic "Troubleshooting a Full Transaction Log (Error 9002)."

Recovery Models
As you can see, the behavior of the transaction log depends in part on the recovery model the database is using. There are three recovery models available, and they all have an effect on transaction log behavior or how operations are logged or both.
The FULL recovery model means that every part of every operation is logged, which is called being fully logged. Once a full database backup has been taken in the FULL recovery model, the transaction log will not automatically truncate until a log backup is taken. If you do not want to make use of log backups and the ability to recover a database to a specific point in time, do not use the FULL recovery model. However, if you wish to use database mirroring, then you have no choice, as it only supports the FULL recovery model.
The BULK_LOGGED recovery model has the same transaction log truncation semantics as the FULL recovery model but allows some operations to be partially logged, which is called being minimally logged. Examples are an index rebuild and some bulk-load operations—in the FULL recovery model the entire operation is logged.
But in the BULK_LOGGED recovery model only the allocation changes are logged, which drastically reduces the number of log records produced and, in turn, reduces the potential for transaction log growth. For more information on minimally logged operations, see the Books Online section "Operations that Can Be Minimally Logged."
Finally, the SIMPLE recovery model actually has the same logging behavior as the BULK_LOGGED recovery, but the transaction log truncation semantics are quite different. Log backups are not possible in the SIMPLE recovery model, which means the log can be truncated (as long as nothing else is holding log records active) when a checkpoint occurs. There are pros and cons to each of these recovery models in terms of which backups are possible (or needed) and the ability to recover to various points in time (I will cover this in another article later this year).

Wrapping Up
This article has really been a more academic explanation of how a critical part of SQL Server works. I hope I've managed to clear up any misconceptions you might have had. If this is your first introduction to logging and recovery, these are the key points I'd like you to take away from this article:
  • Do not create multiple log files, as it will not lead to a performance gain.
  • Be aware of the recovery model your database is using and the effect it has on the transaction log—especially around whether it can automatically truncate or not when a checkpoint occurs.
  • Be aware of the potential for transaction log growth, the factors that can lead to it, and how to get it back under control.
  • Know where to look for help when troubleshooting a full transaction log.
On my blog, I have a lot more information about the transaction log and factors that affect it—see my blog post category"Transaction Log" for more details. The various Books Online topics concerning the transaction log are also very good—starting with Transaction Log Management.
As always, if you have any feedback or questions, please drop me a line at Paul@SQLskills.com.

Thanks to Kimberly L. Tripp for providing a technical review of this article.

Paul S. Randal is the Managing Director of SQLskills.com and a SQL Server MVP. He worked on the SQL Server Storage Engine team at Microsoft from 1999 to 2007. Paul wrote DBCC CHECKDB/repair for SQL Server 2005 and was responsible for the Core Storage Engine during SQL Server 2008 development. Paul is an expert on disaster recovery, high-availability, and database maintenance and is a regular presenter at conferences around the world. He blogs at SQLskills.com/blogs/paul.


Virtualization
Automating Virtual Machine Host Deployment
Fergus Strachan
At a Glance:
  • Hyper-V deployment and automation
  • Resolve Sysprep generalization issues
  • Provision virtual resources
  • Configure unattended deployment of guest OS and server technologies
Code download available at: StrachanHyperV2009_02.exe (1,026 KB)

Contents
Deployment Architecture
WDS-Based Hyper-V Deployment
WMI-Based Hyper-V Configuration
Virtual Lab Deployment
Server Application Deployment
Conclusion
You are probably familiar with many of the good reasons to set up a Hyper-V environment, but one that may be of particular interest is how Hyper-V can facilitate product evaluations and training in test labs and learning environments without sacrificing 64-bit compatibility. Hyper-V runs even on entry-level 64-bit hardware provided you have a capable CPU and a recent BIOS update to support hardware virtualization. This makes it a blast to deploy full-scale test environments based on fully supported software versions, such as the 64-bit version of Microsoft Exchange Server 2007. And once you have set up the deployment, it is easy to redeploy anytime you need to start from scratch, whether to look at a new product or start a new class session.
Even if you're deploying a test environment for a customer with two domain controllers (DCs), a computer running SQL Server, two SharePoint front-end servers, an Exchange 2007 Mailbox server, a Hub Transport server, and a Client Access server, it's going to take a lot of effort. Suppose you have a much larger environment, with perhaps 600 Virtual Machines (VMs). Can you imagine reinstalling these VMs every week or whenever a new lab environment is needed? It is imperative to automate such deployments, and this is where Hyper-V can make a difference.
Hyper-V is Windows technology, and you can combine it with Windows Management Instrumentation (WMI), Windows PowerShell, Windows Deployment Services (WDS), Windows Automated Installation Kit (AIK), and Windows Preinstallation Environment (Windows PE) 2.0 to enable full-scale deployments at the blink of an eye—or at least without requiring much attention. You might find it interesting to look at the installation screens and progress bars while the systems deploy and configure themselves, but you don't have to do this if you have more important things on your list.
In this article, I show you how to deploy Hyper-V servers, VMs, guest operating systems, and server applications without any administrator interaction by using WDS, custom install images, unattend.xml files, and WMI scripts. The idea is to preconfigure a WDS environment once and then install the test systems whenever the need arises, such as when reinstalling a teaching environment, troubleshooting complicated problems in different configurations, and developing and testing custom solutions.
The only required interaction during the deployment is to press the F12 key to launch the Preboot eXecution Environment (PXE), and you can eliminate even this step if you use the Startrom.n12 instead of the default Startrom.com boot file in your WDS configuration, as explained in the TechNet article "Fully Automated Installation Design Background."
The remaining tasks are then already in the hands of WDS, AIK, and WMI, once you have auto-started Hyper-V VMs. You will be able to find the configuration files and scripts in the companion material, available from the February 2009 Code Downloads section of the TechNet Magazine Web site at technet.microsoft.com/magazine/cc135868. The actual install images are not included, as they are too large, but you should be able to adapt the companion files to your own lab environment.

Deployment Architecture
At the core of my lab deployment infrastructure is a WDS server running Active Directory Domain Services (AD DS), Domain Name System (DNS), Dynamic Host Configuration Protocol (DHCP), and, of course, WDS. For administrative convenience, I also installed AIK and Hyper-V tools for remote administration on this server. That's all it takes to get ready for Hyper-V deployment efficiency. You don't need more, though you can introduce additional WDS servers if high availability through redundancy is important to you. The remaining physical computers are Hyper-V servers, deployed through WDS and hosting the VMs that form the actual test environment, as illustrated in Figure 1.
Figure 1 A full-scale lab environment based on Hyper-V and virtual machines
For instructions on deploying the WDS server, check out the companion worksheet "Deploying Windows Deployment Services" at the same download site mentioned earlier. As you'll see, the installation is straightforward. It's the deployment and configuration of the Hyper-V hosts that's challenging, but more about this in a moment.

WDS-Based Hyper-V Deployment
One of the advantages of deployments of WDS for Hyper-V is that WDS simplifies updating the Windows Server 2008 installation media, which is necessary because the original media only includes a pre-release version of Hyper-V. The actual release version is available as a separate update on the Microsoft Download Center.
In a nutshell, here are the steps you take: Deploy Windows Server 2008 on a reference computer, update the installation with the latest Hyper-V files, install Hyper-V, use Sysprep.exe to generalize the installation, capture and upload the generalized install image to the WDS server, and then automate the default Hyper-V deployment for all hosts in the physical infrastructure. I prefer to use Windows Server 2008 Server Core for Hyper-V because my Hyper-V servers are dedicated to hosting VMs and Server Core offers a small operating system footprint as well as security, reliability, and manageability advantages. And for capturing install images, I use WDS, of course. Check out the companion worksheet "Deploying Default Hyper-V Hosts by Using Windows Deployment Services" to see how easy it is to create, upload, and use an updated Hyper-V install image. It couldn't be simpler.
So far, so good; the default Hyper-V deployment based on WDS is easy, but the unattended configuration is not without challenges. The issue is that you need to run Sysprep.exe to generalize the reference installation prior to capturing and uploading the install image, yet Sysprep.exe removes vital configuration information from the generalized Hyper-V image.
Among other things, Sysprep.exe generalizes the Boot Configuration Data (BCD) and removes the hypervisor launch directive from the BCD store. BCD is supposed to be firmware independent, but that's not the case with Hyper-V. The hypervisor depends on the virtualization capabilities of the underlying hardware and BIOS, so the hypervisor launch directive must go in order to generalize the install image. It is possible to modify the BCD store offline after the Sysprep generalization, but this isn't a solution.
If you mount the install image by using ImageX.exe (a tool included in the AIK), you can re-enter the launch directive by using BCDEdit.exe; however, Windows Setup removes this directive again during the generalization phase of the actual installation routine. You are back at the beginning.
It's a bit of a tricky situation because you need the launch directive to start the hypervisor. Without a running hypervisor, the Hyper-V server is not functional. Figure 2 shows the error message displayed when trying to start a VM on a Hyper-V server, deployed by using a custom install image but without an adjustment of the boot configuration.
Figure 2 Unable to start virtual machines because the hypervisor is not running
One way to re-enter the hypervisor launch directive is to add it manually after the server installation by running the command
bcdedit /set hypervisorlaunchtype auto
and restarting the Hyper-V server, but this manual step would be a serious roadblock to a fully unattended test lab deployment. Fortunately, the AIK includes Windows System Image Manager, which you can use to create an unattend.xml file for the install image that WDS applies during the installation in addition to its own WDSClientUnattend.xml file. In this unattend.xml file, you can specify that Setup should automatically log on to Windows with administrative credentials supplied by the WDS client and then run a script that inserts the hypervisor launch directive back into the BCD store and then restarts the server.
Figure 3 illustrates the general approach, and the companion material includes the full version of the unattend.xml file as well as a complete hypervconfig.vbs script. You can include the hypervconfig.vbs script directly in the install image so that it is available during the installation. You simply need to mount the image with ImageX.exe, as outlined in the companion worksheet "Customizing Hyper-V Deployments."
Figure 3 Reconfiguring and starting the hypervisor

WMI-Based Hyper-V Configuration
Re-enabling the hypervisor isn't too difficult, but if you analyze my hypervconfig.vbs script, you will find that it contains somewhat more than these five simple lines of code. In addition to the launch directive, you must configure the entire Hyper-V environment, and this is the challenging part of the deployment story.
You cannot simply create VMs on the reference system before image capture, include them in the install image, and then expect everything to work after fixing the hypervisor launch directive. The server will have the VMs, of course, but the hardware dependencies are missing.
The image generalization disconnects the Ethernet ports of your VMs from the physical network interface cards (NICs) and pass-thru drives from the underlying hard disks and CD/DVD devices. You could skip the generalization, but it's not a good idea to include preinstalled VMs in an install image. Preinstalled VMs inflate the image tremendously, evaluation licenses of deployed test servers eventually expire, and Active Directory domains don't like to be taken offline for extended periods of time either. If you restore a lab environment by using backups of VMs installed several months ago, you have a fair chance of experiencing Active Directory authentication and replication issues. It's better to start from scratch each time.
So let's provision VMs and the associated resources, such as NICs, hard disks, and DVD drives, in the Hyper-V environment before tackling the actual deployment of the test lab. As you might have guessed, provisioning these virtual resources is the main task of the hypervconfig.vbs script.
The approach is fairly straightforward. The script determines the name of the local Hyper-V server and then configures a host-specific set of VMs. Each VM receives two virtual DVD drives mapped to a server-specific .iso file and a general installation .iso file. The server-specific .iso file corresponds to the boot DVD. It includes all the necessary scripts and configuration files for an unattended installation of a particular lab server.
The general installation file provides the actual installation media. Sharing the general .iso file among all VMs on a server helps to keep the size of the Hyper-V install image moderately under control. You could place the .iso files on a network server, but one way or another you must copy the files to the Hyper-V server for installation, so I decided to include them directly in the install image. In this way, you have the .iso files available locally whenever the need arises. This can be useful, for example, when installing additional components or reinstalling a particular VM without tearing down the entire lab environment.
I'll discuss the server-specific installation DVDs in a moment. First, I'll focus on the configuration of the Hyper-V infrastructure by means of a WMI-based script. As illustrated in Figure 4, there are a variety of virtual resources that you must provision, including a virtual switch with internal and external switch ports, the VMs themselves with their virtual Ethernet cards, virtual IDE drives connected to virtual hard disk (VHD) files, and the virtual DVD drives connected to the .iso files for the installation of guest operating systems and server applications.
Figure 4 Provisioning virtual resources for the lab environment
You also need to adjust the auto-start configuration of the VMs and change the boot order to use the virtual disk drive first, followed by the virtual DVD drive that is connected to the server-specific .iso file. In this configuration, the VMs boot from the installation DVD until the OS is installed on the virtual hard disk. I'm sure you are familiar with this sequence because it is standard for any modern personal computer.
The hypervconfig.vbs script configures the VMs to start automatically when the physical computer starts, so the VMs come online after the HypervisorLaunchType reboot, and that's how lab installation commences. The VMs eventually boot into the Setup routines of their guest operating systems. This is the key to a fully unattended lab deployment.
For the most part, the VM configuration follows the same principles you take into consideration when configuring physical computers with multiple drives connected to multiple IDE controllers. The virtual switch, however, requires further explanation because it is the key to enabling communication between VMs on the same Hyper-V server and between VMs on separate servers over the computer network. Essentially, you can compare a virtual switch to its physical counterpart. You create it by calling the CreatedVirtualSwitch method, but a switch without ports is not very useful.
In order to connect the switch to a physical network, you must create a switch port by calling the CreateSwitchPort method and associate this port with an available Ethernet network card on the server. You can connect a physical network card to only one virtual switch, but you can have multiple switches connected to each other directly or through VMs running router software. For the purposes of this article, however, a basic LAN environment without network routers is sufficient, so I configured a single virtual switch on each Hyper-V server connected to the first available physical Ethernet card.
You also must connect the VMs to the virtual switch. Again, you must create a separate switch port for each VM by calling CreateSwitchPort. You can then associate each switch port with a VM's virtual network adapter. Don't forget to connect the parent partition to the virtual switch as well if you want to provide external network connectivity. You can accomplish this task conveniently by calling the SetupSwitch method, which expects an external and internal switch port, a reference to an available physical Ethernet card, and a unique device name and a display name as parameters.
By calling the SetupSwitch method, you convert the virtual switch from a private switch to an external switch, as demonstrated in the hypervconfig.vbs script in the companion material. The script contains all the details to set up external network connectivity for VMs. For more information, read the Virtualization WMI Provider documentation on MSDN. Substantial parts of my hypervconfig.vbs script are based on the samples available under "Using the Virtualization WMI Provider."

Virtual Lab Deployment
Now that the Hyper-V deployment is accomplished and the VMs boot automatically after each system restart, I can turn my attention to the actual deployment of the lab environment. For training centers, it probably suffices to deploy the virtual network infrastructure and guest operating systems and let the trainees deploy the remaining server applications later on. But for development, test, and evaluation purposes it is better to automate the entire deployment of the lab environment.
The overall approach is similar to the Hyper-V method. Following the unattended installation of the OS, let the Administrator account log on automatically and run any additional setup commands. However, you must orchestrate the deployment.
All VMs boot into their setup routines practically at the same time, yet some servers depend on other servers so it isn't possible to perform all installations concurrently. For example, you need to install AD DS before you can add any other servers to the domain, Exchange Server 2007 also requires AD DS, SharePoint server farms require SQL Server, and so forth, so the only VM in your scenario that can run Windows Setup immediately is DC01.Litware.com. All other VMs must wait until the DC is up and running.
There are several ways to implement an installation sequence. You can configure a boot delay for VMs, but this technique is notoriously unreliable. Are you willing to bet that the Active Directory installation always completes within 15 minutes? And how long does it take to install the first Exchange Server after that?
Another possibility is a WMI-based script to switch on VMs when the installation prerequisites allow it. This is a better alternative, yet it requires you to coordinate centralized script execution with distributed VM deployment. It's less complicated to customize each individual setup routine and let the VMs check any given installation prerequisites themselves before initiating their Windows Setup routines, as shown in Figure 5.
Figure 5 Implementing a deployment sequence based on installation prerequisites
Windows PE enables us to implement these custom setup routines. It is a minimal Win32 operating system with limited services, but with support for Windows Script Host (WScript), WMI, and Microsoft Data Access Component (MDAC). You need only to create a customized Windows PE image, add the required Windows feature packages, include a custom script, and then edit the Startnet.cmd file located at %SYSTEMROOT%\System32 of the Windows PE image to run the custom script.
The companion worksheet "Creating Custom Boot Images for Server Deployments" outlines how to create a customized Windows PE image for each server in the test lab environment. Figure 6 shows you how to use this technique to orchestrate the deployment of a second DC.
Figure 6 Orchestrated deployment of a second domain controller in a test lab
The Startnet.cmd file includes a netsh command to assign a static IP address to the network interface of the VM and then calls the StartSetup script. The netsh command is not strictly necessary in a DHCP-enabled environment, yet it helps to highlight network-related errors. For example, if you provision a standard network card (Microsoft Synthetic Ethernet Port) for your VM in your Hyper-V configuration script instead of a legacy network card (Microsoft Emulated Ethernet Port), the netsh command will inform you that Windows PE can't recognize the NIC.
The StartSetup script does not inform you about this issue when attempting to access network resources because the On Error Resume Next statement enables the script to sustain runtime errors. If DC01 is unavailable for any reason, the connection attempts fail and the script continues to loop indefinitely. The loop ends only if a connection attempt succeeds and if DC01 is a global catalog server, which implies that AD DS has been installed.
When the loop ends, the script calls the actual Setup command, specifying an unattend.xml file with server-specific configuration settings. The diagram in Figure 6 illustrates how to wait for a global catalog server to come online, but the same principle is also applicable in other scenarios, such as when checking the availability of file shares or SQL Server databases. Simply attempt to access the resource and exit the loop if the attempt succeeds.
Hyper-V Resources

Windows Server 2008 Hyper-V Web site

Windows Virtualization Team Blog

Windows Deployment Services Guide

Windows Automated Installation Kit

Unattended Windows Setup Reference

Server Application Deployment
The only remaining task is to configure the unattend.xml file to add the server to the domain, configure TCP/IP settings, enable the Remote Desktop Protocol (RDP), and configure <FirstLogonCommands> for the installation of any desired server applications. Most Microsoft server applications support unattended deployments.
For AD DS, you must provide an answer file, as explained in the Microsoft Knowledge Base article "How to Use Unattended Mode to Install and Remove Active Directory Domain Services on Windows Server 2008-Based Domain Controllers." For Exchange Server 2007, you should use command-line parameters instead (see "How to Install Exchange 2007 in Unattended Mode" in the online help). For SQL Server 2008, you should follow the online help instructions outlined in the article "How to: Install SQL Server 2008 from Command Prompt." And for Windows SharePoint Services 3.0, pay attention to "Config.xml reference for Windows SharePoint Services."
The requirements vary in complexity, but you can deploy these systems without any administrator interaction. The final task is to press the F12 key to engage the WDS-based deployment system.

Conclusion
Hyper-V is exciting technology. It is fully 64-bit compatible, so you don't need to deploy 32-bit software versions for evaluation or training purposes anymore if a 64-bit version is available. It is Windows technology, so you can take full advantage of WDS, AIK, and Windows PE for deployment. It supports WMI and Windows PowerShell through a Virtualization WMI Provider, which you can use to manage all aspects of the virtualized environment including the provisioning of resources and VMs during the deployment process. It uses a hypervisor instead of a Virtual Machine Monitor (VMM) to provide high performance and increase scalability, and it is included in Windows Server 2008 at no extra cost.
Hyper-V-based environments are relatively uncomplicated to deploy. It only takes a few mouse clicks to get started with the first VMs, and in conjunction with Windows deployment technologies it is fun to automate even the most complex scenarios.
The only downside I can see is the online documentation of the Virtualization WMI Provider, which is still at an infant stage and, thus, the sample code does not cover all relevant tasks. The results, however, are well worth the effort. It is fun to see an IT environment deploy itself—even if it includes far fewer than 600 VMs.

Fergus Strachan is a London-based independent consultant specializing in Microsoft server infrastructure design and implementation for corporate customers in the UK. Fergus has written technical articles about Microsoft server technology and authored Integrating ISA Server 2006 with Microsoft Exchange 2007. He is also co-author of the Microsoft Exchange Server 2003 Resource Kit.


Deploying Microsoft Dynamics CRM 4.0
Aaron Elder
At a Glance:
  • Software components of a CRM system
  • The development lifecycle
  • Elements of a CRM solution
  • A look at multi-tenancy

Contents
The Solution Development Lifecycle
Elements of a CRM Solution
What about Multi-Tenancy?
Design Considerations
Key Takeaways
If you're used to thinking of CRM as just a sales and marketing management tool, think again. Microsoft Dynamics Customer Relationship Management is a platform for developing applications that manage and track information and processes related to real-world objects. Those objects might be customers, but they can also be just about any entity (and related activities) that you need to manage.
As with any large-scale custom solution, there are some basics related to deployment that need to be understood. In this article, I am going to cover a few fundamental concepts related to Microsoft Dynamics CRM deployment, including how these concepts can be used to support a full product lifecycle deployment. I will also discuss managing multiple deployments as part of a single-solution release, and how multi-tenancy should and should not be used as part of the entire solution lifecycle.
I want to make clear at the outset of this article that when I refer to a Microsoft Dynamics CRM "solution," I mean the totality of all customizations, extensions, custom coding, schema changes, and so forth. A solution is not just one thing; it is all of your changes together.
Behind the scenes, a Microsoft Dynamics CRM solution is a standard ASP.NET 2.0 and Microsoft .NET Framework 3.5 data-driven Web application. The three-tier system includes the following major components:
Data Tier SQL Server 2005 or SQL Server 2008 database. Using SQL Server 2008 requires a hot fix as described in the Knowledge Base article "Support for Running Microsoft Dynamics CRM 4.0 together with Microsoft SQL Server 2008."
Middle Tier Microsoft Internet Information Services (IIS) 6.0 or later Web front end; SQL Server Reporting Services (SRS) 2005 or SRS 2008; ASP.NET 3.5; custom Windows Services.
Client Tier Microsoft Internet Explorer 6.0 or later client; ASP.NET 2.0 or later; Microsoft Office Outlook 2003 or Office 2007 client (with optional offline access); others such as SDK consumers and third-party mobile clients.
Microsoft Dynamics CRM also relies on a variety of external systems including Microsoft Exchange Server 2003 or later and Active Directory.

The Solution Development Lifecycle
A Microsoft Dynamics CRM solution goes through the same lifecycle that a custom application development project would, which might look something like the process depicted in Figure 1.
Figure 1 The application development cycle
This entire process would be supported by several environments that together comprise the development, test, and production systems. In the world of any multifaceted enterprise application, this, of course, can turn out to be surprisingly complex. If, for example, you were to mirror your environments, you might end up with something that looks like Figure 2.
Figure 2 Mirroring your dev, test, and production environments
That's three domains, three domain controllers, three mail servers, three Web servers and three database servers—and this model assumes that SRS and CRM are on the same box, and it does not take into account things such as load balancing. Now let's imagine you add redundancy and a few external services such as Microsoft Office SharePoint Services (MOSS), and you could end up with a scheme like the one in Figure 3.
Figure 3 Increasing complexity
For cost and complexity reasons, trade-offs might be considered to balance the need for environment isolation against the need to keep costs down and manageability up. Organizations have thus looked to a variety of techniques, such as virtualization and the Microsoft Dynamics CRM built-in multi-tenancy features, to address these challenges.
When designing a set of environments to support your CRM project's lifecycle, there are several schools of thought and, depending on what principles are important to you, you might choose to go one way or the other. At one end of the spectrum, designers promote total isolation using exact replication. These folks believe that the only way to validate that something will work outside of production is to have a test environment that is 100% identical to the production environment. Every server, every bit, and every setting must be identical and completely isolated from development and production for testers and IT to accept and believe something will work in production.
In contrast, others think that sort of isolation doesn't really matter at all. If they could, they would develop and test directly in the production environment. They tend to see the redundancy as a waste of time and money, and they are certain delivery would be easier if they could just get in there and make things work.
Hopefully, you fall somewhere in between these extremes and will be open to the idea that when it comes to a Microsoft Dynamics CRM-based solution, it is possible to develop a hybrid that balances complexity, cost, isolation, and manageability.

Elements of a CRM Solution
Microsoft Dynamics CRM solutions components can be divided into four major buckets, and your solution may include one, two, three, or all four.
Customizations They include form, grid, schema, and metadata changes; security roles; workflows; system settings; and templates. Microsoft Dynamics CRM customizations are provided as one or more (typically one or two) zipped XML files. They are imported into a CRM deployment via the Outlook or Web client "Settings | Customization" area and then are "published" to make them active. All of this can be automated using code that is written against the Microsoft Dynamics CRM SDK.
Extensions These include reports and custom code such as plug-ins that must be deployed separately from the customizations. Plug-in registration information is stored as an XML file and can be deployed via either a command-line or Windows Form application provided by Microsoft. This also can be automated via code written against the Microsoft Dynamics CRM SDK.
Custom Code Anything developed as part of your solution, and it might consist of external Web services, custom Web application components, and so forth. The rules and practices for deploying the custom code should be no different than for any other custom Web application.
Data Any information that needs to be imported into an environment for that environment to function. This might include domain data (such as a list of product codes) or users. The data that your solution needs can be deployed into your Microsoft Dynamics CRM instance using scripted code or CRM's Bulk Import feature, or with some form of external process using BizTalk or other ETL (extract, transform, load) tool. Some data, such as Users, needs to be created manually or through Microsoft Dynamics CRM SDK calls.
I like to think of CRM solution deployments just as though they were custom application development deployments. This means that during development and test, each new build of the solution is installed from a clean base system and the process is as repeatable and scripted as possible.

What about Multi-Tenancy?
Now let's discuss what the environment you are going to deploy them into should look like. You may have read about the Enterprise Edition of Microsoft Dynamics CRM 4.0 support for a feature called multi-tenancy, which lets you partition multiple instances of Microsoft Dynamics CRM within a single deployment. This means that several completely distinct organizations with their own reports, workflow, customizations, and schemas can be run on the same set of hardware using the same physical servers and the same database instances and IIS Web sites.
At first glance, this might appear to be the panacea that solves all of our manageability, isolation, and cost conundrums. Such a solution might be visualized as in Figure 4.
Figure 4 A multi-tenancy-only solution
This seems logical because each organization gets its own physical database on the shared SQL Server or instance (which includes customizations, workflows, users, roles, and settings) and its own SQL Reporting Services folder.
This model works perfectly well if those distinct organizations are part of different team or departmental solutions. This, after all, is what multi-tenancy was designed for. While it is true that each organization (or tenant) gets its own database, they all share the same organizational unit (OU) and Active Directory groups, and they will all share the same platform services and front-end application as well. This means that the same asynchronous service and IIS Web site will be shared among organizations. The front-end servers are able to "host" these different organizations through a URL provider that determines, based on the URL, which organization to host.
Take these URLs as an example: crmserver/ContosoDevOrg/loader.aspx and crmserver/ContosoTestOrg/loader.aspx. The CRM server looks at the root directory to determine the name of the organization to serve up. If no root organization name is found, as in the case of crmserver/loader.aspx, the server defaults to the first organization created in the deployment or the one where the calling user has access.
Because the same Web site is used for both organizations, if you have custom code as part of your solution, it too will be shared by both organizations; for example, crmserver/ContosoDevOrg/ISV/mycustomdialog.aspx and crmserver/ContosoTestOrg/ISV/mycustomdialog.aspx.
Both point to the same physical file on disk, such as C:\inetpub\wwwroot\isv\mycustomdialog.aspx. Since it is likely that the version of a custom extension would be different between Dev, Test, and Production, this can pose a serious problem. Let's assume, for example, that Build 11 of your application is currently being developed, while Build 9 is in UAT (user acceptance testing) for test. If you attempt to use multi-tenancy to solve your environment problem, you will have a hard time isolating these two builds. In such situations, some of you might be tempted to try the solution shown in Figure 5.
Figure 5 Attempting to use different IIS servers to segregate your custom solutions code
In that model (if you are no longer using the Network Load Balancing address), users might hit a URL that looks like this:
Development 192.168.1.100/ContosoDevOrg/loader.aspx
Test 192.168.1.105/ContosoTestOrg/loader.aspx
Production 192.168.1.110/Contoso/loader.aspx
This model lets you have three separate front-end servers, hosting three different organizations, with three different code bases on disk. As long as a user doesn't inadvertently hit the wrong organization on the wrong server, everything should work out perfectly.
Unfortunately, since all front-end servers are considered a part of the same deployment, difficulties start to arise a little further downstream than you might realize at first glance. This really becomes a challenge if your solution uses asynchronous plug-ins or workflows, because while you can control which servers your users hit, you cannot control which asynchronous service will process events and requests for which organizations.
This is because all asynchronous services in a deployment work in a round-robin manner, and, as such, your development server's asynchronous service might process a workflow, system job, or asynchronous plug-in response to a request from your test server, thus blowing the isolation requirement right out of the water. In addition, if your custom code that is run by this asynchronous process relies on files that must be deployed on disk to the server (such as a configuration file or a file in the Global Assembly Cache, or GAC), you will get version conflicts.
It is important to note that, for the most part, these challenges arise only when you are writing custom code that needs to be deployed on disk or if your custom code relies on resources that would be available only on or from a particular server. If your solution is simple and only uses customizations (schemas, forms, views, and so forth), workflows, and reports, you won't have any problems using the approach in Figure 4.
So what is multi-tenancy for and when is it a good solution for product lifecycle environments? Multi-tenancy was originally designed to solve a hardware problem related to the hosting of multiple distinct tenants in a production environment, and it does this very well. Previously, in Microsoft Dynamics CRM 3.0, each deployment, or tenant, had to have its own dedicated SQL Server or SQL Server instance, as well as a front-end server.
This was true for many reasons, including the fact that deployment-specific settings used to be stored in the registry and on disk. All of these configurations have now moved to the database, so a single application server is able to handle multiple organizations. Multi-tenancy comes in handy for hosted versions of CRM including Microsoft Dynamics CRM Online.

Design Considerations
Now that you're aware of some potential issues, let's discuss some points to keep in mind as you design your deployment. The answer, of course, is that it depends. It is certainly possible to run a full CRM environment (including the domain controller, SQL server, and Web server) on a single box, as you can see on the Microsoft Dynamics CRM 4.0 Virtual Machine demonstration (see the "CRM Resources" sidebar for the URL). It is very common to use a single machine virtual image for a development environment. For test, however, it is important to validate key production environment challenges, and, for this reason, I recommend having your test environment mirror your production environment in terms of structure but not capacity. Your environment might look like the one in Figure 6.
Figure 6 The test environment structure should mirror the production structure
In this approach, you try to minimize physical hardware of the infrastructure using virtualization and further attempt to minimize virtualization resources by virtualizing only key scenarios that need to be tested. You will be able to allow your developers to develop on a single server image (or images, if they have their own virtual machine on their personal desktops) if you ensure that they will pay attention and be aware of the environment to which their solution will be deployed. The issues that developers should be paying attention to are the same issues you should be building your test environment to validate, including:
Make the Settings Configurable Don't, for example, assume the server will respond to localhost or a particular port.
Be Aware of Multiple Servers Don't assume things will work without setting up proxy users or trust for delegation.
Keep Load Balancing in Mind Be very careful with session state and caching. Note that Microsoft Dynamics CRM is designed to be completely stateless and work well with a round-robin load balancer.
Think about Multi-Tenancy When multiple tenants are hosted on a single machine, they share the same process space. This means that elements such as caches need to be keyed by the organization name so that users from one organization will not inadvertently utilize data from another organization. In addition, when you have client-side code that has links or calls back to the server, you need to be sure that the calls preserve the organization name in the URL; otherwise, you might hit the default organization or an organization you do not expect.
CRM Resources

Microsoft Dynamics CRM 4.0 Implementation Guide

Microsoft Dynamics CRM 4.0 as a Business Application Development Platform

Microsoft Dynamics CRM Team Blog

Microsoft Dynamics CRM 4.0 Virtual Machine

Optimizing and Maintaining Microsoft Dynamics CRM 4.0

Key Takeaways
Isolation Is Important When designing your solution, keep in mind which approach (as illustrated in Figures 4, 5, or 6) will work best for you and be aware of when and where your custom code might run. It's also worth noting when you do not need to worry about such issues because of the type of extensions your solution uses.
Virtualization Virtualization helps reduce complexity in building an environment that mirrors key test scenarios of production. Here is some guidance about the setup. Put CRM and SQL Server on separate servers. This helps verify trust for delegation and other related issues. CRM servers should be load-balanced, which will help identify session, caching, and cross-server issues. Finally, put the domain controller and e-mail on separate servers; this helps in identifying connectivity issues.
Refresh Environment for Each Build As a general rule, it's a good idea to create backups either of the virtual environments or simply of the Microsoft Dynamics CRM databases (Data and Config) that can then be restored to get the server back to a "vanilla" state. You then do a full clean deployment into a fresh environment each time, including custom code, customizations, plug-ins, and domain data.
Redundancy/Performance Testing Can Be Done Separately Except for very large organizations, you can usually tackle failover and performance testing through isolated simulations and not through "real-world" build-outs. This means that it is not necessary for you to attempt to build out a test environment that enables testing of these scenarios. As an alternative, you can rely on testing them either in production or in separate one-off environments.
In closing, Microsoft Dynamics CRM is a scalable, enterprise-class system that, when appropriately configured and deployed, can handle small teams, enterprise-wide solutions, and every option in between. Trying to determine which product lifecycle environment is right for you will depend on a variety of factors.
Generally speaking, multi-tenancy is not an ideal way to address the product lifecycle development challenges of complex solutions and is best used when it is fully understood. Simple solutions that require only basic customization or that make use of properly coded and isolated custom extensions that don't rely on disk resources or server-specific access should do just fine following the model depicted in Figure 4.
If your solution demands more isolation, server-specific resources, or access (perhaps an external service is only allowed through your VLAN from one specific server to another), and so forth, I recommend going with the model shown in Figure 6. And I'd recommend avoiding the approach that Figure 5 illustrates, as it is a hybrid hack at best.
Ultimately, Microsoft Dynamics CRM can be deployed in thousands of configurations, and exactly what is right for you will depend on what your solution requires. With a better understanding of multi-tenancy, single-server development environments, virtual test environments, and what testable scenarios are important to you, you should be able to design a product lifecycle deployment that is both functional and cost-effective.

Aaron Elder (Microsoft Dynamics CRM MVP) works for Ascentium, a technology consulting and interactive marketing agency. Visit the Ascentium blog atascentium.com/blog/crm.


From the Editor Transform Information into Intelligence
Joshua Hoffman

There's an old public speaking adage that goes something like, "It's ten percent about what you say and ninety percent about how you say it." Certainly it's an exaggeration to make a point, but I think it's a point well made. It doesn't matter how right you are or how important your information is if nobody understands or nobody cares.
Nowhere is this more clear than in the presentation of data. You could have the best idea in the world. Maybe you figured out how to turn water into fuel. Or, more realistically, you came up with a way to save your company a bunch of money. Good luck trying to get anyone to pay attention by inundating them with a bunch of raw data. (Though truth be told, if you really figure out how to turn water into fuel, you probably won't have any trouble getting people's attention.) This brings us to the important distinction between information and intelligence, and it also brings us to the lead topic this month.
Information is a powerful tool, but it's transformed into intelligence when it's presented in a way that helps people make decisions. This month we've partnered with our good friends at Microsoft Press to bring you some great information on business intelligence from the recent book Microsoft Office SharePoint Server 2007 Best Practices. This adapted article discusses some powerful ways to leverage SharePoint and Excel to build business intelligence solutions for your organization. Just remember—it's ten percent about what you say and ninety percent about how you say it.
In this issue we also cover a topic that hasn't received nearly enough attention in these pages to date: Microsoft Dynamics. Microsoft Dynamics is a suite of software for small and medium businesses, which includes many of you. We're fortunate to be joined by Aaron Elder, a Microsoft Dynamics CRM MVP, as he discusses deployment planning and considerations for Dynamics CRM 4.0.
Also in this issue, as promised, we give you the premiere of our new monthly column, Geek of All Trades with Greg Shields. We try to provide useful and practical information in every article for all of our readers, but we certainly recognize that not everyone is a SharePoint expert, or an Exchange expert, and so on. Greg's goal is to provide information focused on the IT guy or gal who has to do it all. He begins this month with a discussion of Server Core. Future topics include a roundup of crucial IT pro utilities, simple network access protection, and new backup and restore techniques. We look forward to your feedback on this new addition. Drop us a line at tnmag@microsoft.com.
—Joshua Hoffman
Thanks to the following Microsoft technical experts: Yury Berezansky, Jamie Fiorda, Andrew Mason, Michael Murgolo, Sanjeev Nair, Jez Sadler, Jim Truher, Brad Wilson, and Edwin Yuen.


Letters Readers Speak Out

Moving Mailboxes
Our group currently has Exchange Server 2003 running, but we are planning to install Exchange Server 2007 on another server. My question is—can I move the mailboxes from Exchange Server 2003 to Exchange Server 2007 without integration?
—Jon T.

When you say you want to move mailboxes to the Exchange 2007 server without any form of integration, I assume you mean moving the mailboxes to an Exchange 2007 server in another Active Directory forest? If that is the case, the answer is yes. You can move your mailboxes directly from an Exchange 2003 server in one Active Directory forest to an Exchange 2007 server in another.
In order to do this, you have to use the Move-Mailbox cmdlet (this is because the Exchange Management Console does not allow you to perform cross-forest mailbox moves). If you would like to find further details on exactly how this is accomplished, I recommend that you take a look at the "How to Move a Mailbox Across Forests" topic in the Exchange Server 2007 documentation on TechNet.
—Henrik Walther, TechNet Magazine Columnist, Exchange Q & A

Interop
Regarding the December 2008 issue of TechNet Magazine, I am writing to say thank you to Joshua Hoffman for going out on a limb and allowing this to be published.
I would also really like to thank Wes Miller for writing "Interacting with Windows from a Mac Environment", Don Jones for "Managing Macs in Your Windows Environment", and Gil Kirkpatrick for "Authenticate Linux Clients with Active Directory." And to the rest of the TechNet Magazine team in general, thank you for that issue.
I work in an environment where Mac, Linux/Unix, and Windows machines all need to interoperate happily. It's been a tragedy trying to get the various flavors of Linux, Mac, and Windows to all properly authenticate via Active Directory. Countless late-night expeditions to the Internet reading thousands of forums, .config files, help-me-out Web sites, and braving the chatter on IRC to arm myself with the technical kung-fu required to be the guy who "makes it work" in a cutthroat IT market.
Speaking from this experience, Gil has done a tremendous job in this step-by-step guideline for IT pros and hobbyists alike to integrate non-Windows systems into the Windows environment. And he made sure to include all of the "gotchas" that come along (many of which have consequently caused me to stumble when working on integration projects).
—KC M.

Thank you for your very kind note, KC. It's certainly gratifying to hear that you're finding interop articles from the December 2008 issue useful. While this the topic of interop is certainly a challenge for us to cover in a balanced way, our goal is to always try to provide the most practical and relevant content to help IT pros be successful in their jobs.
If we were able to help you and other readers in that regard, we couldn't be more pleased. Of course, we can't let a compliment go unpunished! Please make sure to keep coming back to us with constructive criticism, feedback, and suggestions for new topics you'd like to see covered in these pages. Keep pushing us, and I promise we'll deliver. Thanks again!
—Joshua Hoffman, Editor-in-Chief, TechNet Magazine

Macs and Windows
Wes Miller's December 2008 article "Interacting with Windows from a Mac Environment" was great. We have seen a big influx of Apple Macs in our Active Directory Domain, and yes, it can be done quite easily.
—Dan A., Posted on the TechNet Blog

Thanks, Dan. As we pointed out in the December 2008 issue, our job here is not to take sides on which platform is best (I think you probably know our opinion there!) but rather to make sure that you, as IT professionals, are equipped to address the needs of your business. Hopefully that's what we have achieved.
—Joshua Hoffman, Editor-in-Chief, TechNet Magazine

Got something to say? E-mail us at tnmag@microsoft.com, or visit our blog forum at blogs.technet.com/tnmag.


Toolbox New Products for IT Pros
Greg Steen

The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft. All prices were confirmed December 3, 2008, and are subject to change.

View and Analyze HTTP Data
HTTP Debugger Pro
httpdebugger.com
There's no shortage of tools available to provide insight into HTTP requests and responses. In fact, I have already covered some in this column. I've been using another HTTP debugging tool recently, HTTP Debugger Pro, that's worth covering here.
Like most HTTP debugging tools, HTTP Debugger Pro places itself between your applications (such as Internet Explorer) and the network to record the request and response traffic for analysis. This gives you an at-a-glance view of what exactly is transpiring when you request a site's URL.
Since HTTP Debugger Pro isn't tied directly into the browser as a plug-in or extension, you can also use it to see how applications such as MSN Messenger, AIM, and Microsoft Office Outlook use HTTP requests on your system. And if there are any rogue processes on your system using HTTP to communicate, you'll even be able to identify them quickly and see what kind of information is being transmitted.
The default layout of the application's UI is pretty well thought out, but you can tailor it to your own liking. The default view provides five views into the request and response streams as they are collected. The main request view gives a data grid of the request sequence, showing such details as the domain, URL, content type, response status and times, and sizes of each component of the request. It also indicates the program that initiated the request and the user account under which the application process is running—this is great information for analysis, especially on multi-user systems and when you're trying to track down a rogue process.
On that same component of the UI, you can switch to see the raw request and response streams or switch to a chart view that displays a number of statistics for the current request logs in either a bar or pie-type chart. For example, you can see how many images or JavaScript files were requested, average speeds, and domain breakdowns. The default layout of HTTP Debugger Pro offers a summary view for a quick overview of a particular request and a cookie/URL parameter view that shows you any cookies or query string parameters that were issued with the request.
The other two default UI components are dedicated to the request and response components, both of which can be switched among a view of the header, the raw content, and a preview of what the request or response looked like, letting you isolate individual components of each. You can set highlighting rules based on status, size, content, end user, or text. There is also search functionality that lets you search components of all the requests and responses in the log for a string of text.
If you are troubleshooting and need to recreate a scenario, you can create custom requests with any custom header, user agent, cookie, or query string values to get the data you want for GET and POST methods. The built-in URL/Text converter tool can encode or decode URLs, Base64, and HEX data. For the latter two, you can also save the results directly to a file. HTTP Debugger Pro also lets you export the request log to Microsoft Office Excel, allowing for more filtering and analysis.

Price: $99.95 (direct) for purchase of a single user license.
Monitor HTTP traffic with HTTP Debugger Pro

Launch Your Applications
RocketDock
rocketdock.com
Not having to navigate the start menu to get to your favorite and most-used applications can be a big time-saver. The enhanced start menu in Windows Vista helps, as do the tried-and-true, quick-launch buttons on the taskbar. But I like a little extra boost, and if you agree, check out the free RocketDock utility.
RocketDock works on Windows 2000 up through Windows Vista (though not on x64). It's a fast, eye-pleasing application launch pad for the apps you use most often. If you are familiar with Macs, you might notice that it's quite similar to the OS X Aqua GUI's application dock.
After installing RocketDock, add your favorite (or should I say most commonly used) applications to the app. The tool makes this easy, allowing you to drag and drop shortcuts onto the dock, placing and ordering the icons as you want them to appear. Right-click on the dock to separate your applications into groups.
The right-click context menu also lets you delete items, lock the dock, set specific icon settings, and set the dock's anchor location on your desktop. And if you don't like the default auto-hide setting of the dock, you can choose to make it always visible.
Other customizations are available—you can choose to have application windows minimize to the dock instead of your taskbar, nearly eliminating the need for the traditional Taskbar/Start menu view. And you can choose to have the dock activate a running application instance when you click the launch icon instead of launching a new application instance as a default.
There are also customizations to adjust how icons and animations look and act—tone them down for a slower system or go all out on a very powerful machine. You can assign RocketDock to dock to the monitor and edge of your choosing, and set specific centering and edge offsets to maximize usability. And to match your current desktop theme, you can select the font, color, outline, and shadow of the application text that pops when you scroll over each icon on the dock—especially useful if you've chosen to minimize your applications to the dock.

Price: Free.
Launch applications with RocketDock

Book Review
Windows Group Policy Resource Kit
microsoft.com/mspress/books/9556.aspx
Group Policy is an incredibly powerful management tool for maintaining centralized control over your Windows domains. But many IT professionals don't fully understand the vast capabilities of Group Policy, which now has more than 5,000 individual settings and almost 40 Client-Side Extensions (CSEs).
One valuable reference to help you get up to speed on Group Policy and learn about the new functionality available with the latest editions of Windows is Windows Group Policy Resource Kit: Windows Server 2008 and Windows Vista (Microsoft Press, 2008) by Derek Melber and the Windows Group Policy Team at Microsoft.
The book defines Group Policy and demonstrates how it can help you as a system administrator. It then covers the new features within Windows Server 2008 and Windows Vista and delves into administering Group Policy with the Group Policy Management Console (GPMC), explaining how to perform the most common administrative tasks.
It also discusses how you can automate Group Policy with both Windows PowerShell and VBScript, as well as how to work with the new ADMX files and the ADMX central store. There is also an excellent section on troubleshooting Group Policy Objects (GPOs) with such tools as GPLogView, GPMonitor, and GPOTool.
Whether you are just getting started with Group Policy or have been using it for years and want to learn about the new features, this is a good reference worth considering for your bookshelf.

Price: $49.99.

Back Up Your Databases
SQLsafe
idera.com/Products/SQLsafe
The more complex your application infrastructure becomes, the harder it is to find available backup windows for those applications and the data they store. This is especially true if you are dealing with large SQL Server databases. One tool out there that aims to help by speeding up the backup process for those large data stores is SQLsafe from Idera.
SQLsafe gives you faster, smaller backups and restores with settings for more precise control. Two of the features that help achieve this are the IntelliCompress and IntelliRestore. IntelliCompress will pick a compression algorithm suited to achieve the maximum compression based on your data, while IntelliRestore provides graphic point-in-time recovery.
SQLsafe lets you restore items at the object level, allowing you to restore individual tables if need be. And you can encrypt your backups with SQLsafe, choosing from DES (Data Encryption Standard), TripleDES, RC2, and Rijndael algorithms to balance speed with security.
The centralized management console is handy if you have numerous SQL Server instances in your environment. You can manage backup policies across your environment from either the management application's UI or a Web interface. It also allows you to set up e-mail notifications for backup and restore events as needed for your environment.
As with any backup solution, you need to test your backup policies and compression settings as well as validate the restore capabilities of those backups before letting this tool loose in your production environment. This is necessary to ensure that everything will work properly in your environment so you aren't caught off guard when you first encounter a real disaster recovery situation.

Price: Starts at $1,195 (direct) per SQL Server instance. A stripped-down version is available for free.
Back up large data stores with SQLsafe

Recover Quickly from System Crashes
Replay RA—Recovery Accelerator Standard Edition
appassure.com/Products/RA/Replay-RA.aspx
Let's face it—the world isn't perfect. Systems go down unexpectedly. Nightly backups and intraday differentials of file systems and databases are helpful for minimizing data loss. But when a catastrophic failure occurs, those measures won't necessarily protect the state of the server applications that are running on the affected system.
Replay RA from AppAssure Software (the "RA" stands for Recovery Accelerator) aims to speed up the recovery from outages affecting Microsoft Exchange, BlackBerry Enterprise Server, SQL Server, SharePoint, domain controllers, and more. To do this, the application takes an image of the system and then captures block-level changes to the system's disks, replicating those changes to its disk images. Block-level data is also "de-duplicated" on the fly, allowing the application to transfer the data to the image location quickly and reduce the amount of space needed for that image.
In the event of a system failure, you can use that image to restore your system to the point in time just prior to the failure or to the point in time you determine to be when the system was in a stable state.
If you want the basic application, pricing for Replay RA starts at about $500 per server. If you decide you want to use the application in connection with Exchange, you'll also pay a per-user charge of about $12 as a one-time fee with volume discounts available. If you're running Exchange clusters, you'll want to take a look at the Enterprise Edition.

Price: Starts at about $500.

Switch between Applications
Switcher
insentient.net
If you're anything like me, you have several applications running simultaneously on your desktop so you can multitask while your machine multitasks in the background. The Alt-Tab and the new Windows Key-Tab (in Windows Vista) hotkeys let you shift between applications quicker than using a mouse. But sometimes you need a little bit more. One tool that gives you just that is a side project by Bao Nguyen called Switcher.
Switcher is a free utility written in C# (though not open source) for Windows Vista with Aero enabled. As the name suggests, the application gives you a great twist on the standard Window activation and visualization otherwise known as window switching. Where Alt-Tab and Windows Key-Tab just give you basic cyclic rotation of the available Windows open on your desktop, Switcher gives you a number of different styles, flows, and capabilities to let you switch in more ways.
First, you can choose between three different window views when you press the Switcher shortcut key combination: Tile, Dock, and Grid. Tile places your windows across all your monitors as big as a non-overlapping view will allow, offering easier recognition. Dock brings each window option to the forefront as you cycle through the options. While it's much like an Alt-Tab with a "preview" of the current option, dock also lets you choose which edge of the screen to dock to as well as behavior options for the preview, both of which are very useful in multi-monitor desktops. Grid is similar to Tile, but makes each window the same size on the preview.
In terms of the animation and appearance, Switcher lets you choose whether to show the desktop as a window option, change the brightness of the Switcher view, change the number of windows, set a preview zoom percentage, and set the durations of the animations as you cycle through each option. You can also add custom window headers with specific fonts, font styles, locations, and gradient color schemes. You can also set switcher to activate on a custom keyboard or mouse shortcut to your preference if you don't like the defaults.
One of the coolest features offered by Switcher is its filtering capability. Once you press the mouse or keyboard shortcut to activate the switch, you can also type on the keyboard to filter which windows should be options. And if your system is anything like mine, with one or two keystrokes you'll have the window you want up front in much less time than by pressing Alt-Tab or using a mouse. So if Alt-Tab just doesn't cut it for you or you just want to gain a few more seconds in your day, you should consider trying the free Switcher utility as an addition to your Windows Vista desktop.

Price: Free.
Quickly change applications with Switcher

Greg Steen is a technology professional, entrepreneur, and enthusiast. He is always on the hunt for new tools to help make operations, QA, and development easier for the IT professional.


SQL Q&A Backup Compression, Client Redirection with Mirroring, and More
Paul S. Randal

Q We're going to be upgrading most of our servers to SQL Server 2008, and one of the features that I'm looking forward to putting into production is backup compression. I know that I can turn it on by default for all databases on each server, but I've also heard that I might not want to do that. I'm not sure why I wouldn't want to have the feature enabled by default, as it seems like I've got nothing to lose. Can you help explain the reasoning behind what I've heard?
A The answer is my perennial favorite: it depends! Let me give some background to explain.
The key point to consider is the compression ratio that each database backup will have when backup compression is enabled. The compression ratio of anything being compressed by any algorithm is determined by the actual data being compressed.
Looking for SQL Server Tips?

For tips on SQL Server, visit the TechNet Magazine SQL Server Tips page.
For more Tips on other products, visit the TechNet Magazine Tips index.
Random data (small integer values, for instance) will not compress very well, so the contents of the tables and indexes in the database will determine, for the most part, the compression ratio that can be achieved.
Here are some examples of when backup compression may not produce a high compression ratio:
  • If the database has transparent data encryption enabled, then the compression ratio will be very low because the data being compressed is random small values.
  • If most of the data in the database is encrypted at the column level, then the compression ratio will be low, again because the column encryption essentially randomizes the data.
  • If most tables in the database have data compression enabled, then the compression ratio will be low; compressing data that is already mostly compressed usually has little effect.
In the case when the compression ratio is low, the problem is not the low ratio but the fact that CPU resources are used to run the compression algorithm for no gain. No matter how well a chunk of data can be compressed, CPU resources are always used to run the compression and decompression algorithms.
This means that you need to check how well each database compresses in a backup before deciding to use backup compression for that database all the time. Otherwise you may possibly be wasting CPU resources. This is the basis for what you've heard.
To summarize, if the majority of the databases will benefit from backup compression, it makes sense to enable backup compression at the server level and manually change a few backup jobs to specifically use the WITH NO_COMPRESSION option. Alternatively, if the majority of the databases will not benefit from backup compression, it makes sense to leave backup compression turned off at the server level and manually change a few backup jobs to specifically use the WITH COMPRESSION option.
Q Last year we upgraded our databases to have database mirroring so that if a failure occurs, we can failover to the mirror and the application continues. While we were designing the system, we practiced doing failovers of the database and everything worked fine. Last week we had a real failure and the database failover occurred, but all of the application transactions stopped and the application didn't connect to the failover server. In the future, how can I set up SQL Server so that it doesn't drop the application connections during the failover so the transactions can continue?
A Let me break this down into two parts—how applications can cope with failovers and how to manage client redirection with database mirroring.
When a failover occurs using any of the high-availability technologies available with SQL Server, the client connection to the failed server is dropped and any in-flight transactions are lost. It is not possible to migrate an in-flight transaction between servers (in a failover situation or otherwise). Depending on the high availability technology, the in-flight transaction either will not exist at all on the failover server or it will exist as an in-flight transaction but will be rolled back as part of the process to bring the database online on the failover server.
With regard to database mirroring, which continually ships transaction log records from the principal server to the mirror server, it is usually the latter case—any in-flight transactions are rolled back as part of bringing the mirror database online as the new principal.
Therefore, there are two things that an application must be able to do gracefully when running on a server with the possibility of having to failover to another server:
  1. It must be able to gracefully handle the server connection being dropped and then try reconnecting after a small time interval.
  2. It must be able to gracefully handle a transaction being aborted and then retrying the transaction after a connection is established with the failover server (possibly using a mid-tier transaction manager).
The only high-availability technology here that does not require client changes specifically to allow redirection of the client connection after a failover is failover clustering. Clients connect to a virtual server name and are transparently redirected to whichever physical cluster node is active.
With high-availability technologies, such as log shipping and replication, the server name of the failover server is different, which means that manual redirection of client connections is required after a failover. This manual redirection can be done in several ways:
  • You can hardcode the failover server name into the client so the reconnection attempts are made to the failover server.
  • You can use Network Load Balancing with a 100/0—0/100 configuration, which will then allow the connection to be switched to the failover server.
  • You can use something like a server name alias or switching entries in a DNS table.
With database mirroring, any of these options will work. But database mirroring also has built-in client direction capabilities. The client connection string is able to explicitly specify the name of the mirror server, and if the principal server cannot be contacted, the mirror will then automatically be tried. This process is known as explicit redirection.
If the client connection string cannot be changed, then implicit redirection may be possible if the failed server is now running as the mirror server. Any connections to it will be automatically redirected to the new principal—but this only works if the mirror server is running.
The SQL Server 2005 white paper "Implementing Application Failover with Database Mirroring" explains these options in more detail.
Q When we upgraded to SQL Server 2005, we redesigned our large tables to be partitioned so that we could take advantage of partitioned maintenance and the sliding-window mechanism. You described this in the August 2008 installment ("Partitioning, Consistency Checks, and More"). But we've encountered a problem. Occasionally, concurrent application queries are experiencing blocking across the whole table when the queries aren't even accessing the same partitions. I've heard that SQL Server 2008 fixes this problem—can you please explain how I can stop this blocking?
Figure 1 Examining locks on a partitioned table
A The problem you're seeing is caused by a mechanism called lock escalation. SQL Server acquires locks on data to protect them while a query is reading or writing the data. It can acquire locks on entire tables, data file pages, or individual table/index rows, and every lock takes up a little memory.
If a query causes too many locks to be acquired, SQL Server can decide to replace all the locks on rows or pages in a table with a single lock on the entire table (the threshold, when this takes place, is approximately 5000 locks, but the exact algorithm is complicated and configurable). This process is called lock escalation.
In SQL Server 2005, if query A is operating on a single partition of a table and causes enough locks to be taken to trigger lock escalation, then the entire table becomes locked. This can prevent query B from being able to operate on a different partition of the same table. Hence, query B is blocked until query A completes and its locks are dropped.
In SQL Server 2008, the lock escalation mechanism has been improved to allow a table to have partition-level lock escalation. Using the example above, this means that the lock escalation caused by query A would only lock the single partition query A is using, rather than the entire table.
Query B will then be able to operate on another partition without being blocked. Query B could even trigger lock escalation itself, which would then lock just the partition that query B is operating on, rather than the entire table.
This model of lock escalation can be set using the following syntax:
ALTER TABLE MyTable SET (LOCK_ESCALATION = AUTO);
GO
This syntax instructs the SQL Server Lock Manager to use partition-level lock escalation if the table is partitioned and regular table-level lock escalation if the table is not partitioned. The default behavior is to use table-level lock escalation. Care should be taken when setting this option, as it could lead to deadlocks depending on the behavior of your queries.
For example, if queries A and B both cause lock escalation on different partitions of a table, but then they each try to access the partition the other query has locked, one of the queries will be aborted by the Deadlock Monitor process.
The figure shows an example of querying the sys.partitions system catalog view (the first set of results) and the sys.dm_os_locks DMV (the second set of results) to examine the locks being held for queries against a partitioned table where partition-level lock escalation has taken place. In this case, there are two partition-level exclusive locks (the HOBT locks in the output), but the table locks (the OBJECT locks in the output) are not exclusive, so multiple queries can access partitions even though lock escalation has taken place. Notice that the resource IDs for these two partition locks match the partition IDs for the first two partitions of the table in the output from sys.partitions.
Earlier this year I blogged about an example script showing how partition-level lock escalation works and the potential for deadlocks. The SQL Server 2008 Books Online topic called "Locking in the Database Engine" has an in-depth explanation of all aspects of locking in SQL Server 2008.
Q One of our servers had some issues with the disk holding the transaction log for a database, and the database became suspect. The most recent full backup was from five weeks ago, and it was going to take too long to restore all the log backups, too. It was out of hours when the problem occurred, so we rebuilt the broken transaction log to avoid the downtime. Under some circumstances, this could cause problems. But if nothing was accessing the data, then I think we're safe. Did we do the right thing?
A The simple answer is that the only time I would consider rebuilding a transaction log is when it's not possible to recover from backups. Although you are aware of the dangers of rebuilding a transaction log (for those readers who are not, see my blog post, "Last resorts that people try first...," the fact that the database went suspect means that it failed during recovery—either when running crash recovery or while rolling back a transaction. This means that there is the real possibility of data corruption in the database.
Although the problem occurred during your quiet period, did you consider scheduled jobs and background tasks? A maintenance job could have been running that was rebuilding or reorganizing a clustered index when the log became corrupted. A background task might have been running ghost cleanup on pages in a heap or clustered index. Either of these, for example, could have been making changes to clustered index structures that, if not properly rolled back, would result in corruption in the database and possible data loss.
The bottom line is that rebuilding a transaction log should always be the absolute last resort in any disaster recovery scenario due to the enormous potential for causing more corruption and data loss. At the very least, you should run a full DBCC CHECKDB on that database to check whether any corruption exists.
Moving forward, you should change your backup strategy so that you are able to perform timely restores and don't have to resort to drastic measures, such as transaction log rebuilding. The steps to design a backup strategy extend beyond the scope of this column, but I plan on covering this topic in a full-length feature article at some point in the coming year. So please stay tuned!

Paul S. Randal is the Managing Director of SQLskills.com and a SQL Server MVP. HeHe worked on the SQL Server Storage Engine team at Microsoft from 1999 to 2007. Paul wrote DBCC CHECKDB/repair for SQL Server 2005 and was responsible for the Core Storage Engine during SQL Server 2008 development. Paul is an expert on disaster recovery, high availability, and database maintenance and is a regular presenter at conferences around the world. He blogs at SQLskills.com/blogs/paul.


Utility Spotlight Web Page Error Toolkit
Lance Whitney

Contents
Requirements
Download and Install
Setting Up Your Web Server
Setting Up the Web.config File
Customizing the ErrorPage File
Working in IIS
Most of us who work on the Web, whether as webmaster, IT administrator, or ordinary surfer, are probably familiar with the dreaded 404 error page. This error pops up when the Web page you want can’t be found, and it does so for a variety of reasons, including broken links, a missing page, and a misspelled URL. But 404 errorpages aren't very helpful—they tell you that the page can't be found and list a few possible causes, but they don't point you to the right information (see Figure 1).
Figure 1 A “not found” error
You undoubtedly get frustrated when you run into one of these errors, and so, of course, do the visitors to your organization's own Web site. As an IT admin, is there anything you can do to make your site more user-friendly when a page cannot be found?
In fact, there is. You can set up a custom error page that replaces the generic 404 page and instead displays useful details for site visitors who bump into trouble. And there is a handy tool that can help you design and implement custom error pages—the free Web Page Error Toolkit from Microsoft.
The Web Page Error Toolkit is an ASP.NET application that taps into IIS. You modify an ASPX Web page and other files included with the Toolkit and then set IIS to deploy your custom page. The error page will display search results so your visitors can more easily find the right information in the event of an error.
By default, the page uses the Microsoft Live Search engine to generate search results (though you can use another search engine if you prefer). The page can also offer alternate spelling suggestions if the URL or search string was mistyped.
Microsoft uses the Web Page Toolkit at its own site to help visitors. Let's say you search for "Microsoft Office" at microsoft.com and mistype it as "Microsoft Ofice". The custom error page displays a message asking if you meant to type "Microsoft Office" and provides a list of search results for the correct term, as shown in Figure 2.
Figure 2 An error page displaying search results

Requirements
Requirements for the Web Page Error Toolkit include:
  • Windows 2000, Windows Vista, Windows XP, Windows Server 2003, or Windows Server 2008
  • IIS 5.0 or higher
  • ASP.NET 2.0 or higher
  • Visual Studio 2005 or higher

Download and Install
The Web Page Error Toolkit can be downloaded from Live Services. The documentation suggests that you install the downloaded file, WebPageErrorToolkitSetup.msi, to the root of your Web site. However, I recommend you install it to the default directory—c:\program files\Microsoft\Web Page Error Toolkit—and then copy the necessary files and directories to your server's root. This gives you control over which files are stored in the root. After installation, a new folder will appear in the Programs menu called Web Page Error Toolkit with two shortcuts: User Guide and Web Page Error Toolkit Files.
The Toolkit files include:
Web.config.merge This is the file that contains the necessary settings for your web.config file. On an ASP.NET Web site, the web.config file contains certain core settings organized into different sections.
Errorpage.aspx This is the custom error page that will replace the generic 404 error page.
Errorpage.aspx.cs The codebehind page for errorpage.aspx written in C#.
Errorstyle.css The stylesheet for your error page.
UserGuide.docx The documentation in Microsoft Office Word 2007 format.
Readme.rtf This is a brief document that explains what you must do with the web.config.merge file.
MSPL license.rtf A document with the software's licensing information.
You'll also find three subdirectories in the installation directory: App_Code, bin, and Web References.

Setting Up Your Web Server
First, you must copy the necessary files to the root of your Web server if you chose not to install them there. Errorpage.aspx, errorpage.aspx.cs, errorstyle.css, and web.config.merge should be copied to the root. Also copy the three subdirectories: App_Code, bin, and Web References. Be aware when you copy these three subdirectories that they may already exist in your Web server's root. However, you should be safe, as the files in these directories are specific to the Toolkit and unlikely to conflict with any existing files.

Setting Up the Web.config File
Next, you must set up the web.config file. If no web.config file exists in the root of your web server, rename web.config.merge to web.config and use that as your default. But if your Web site runs on ASP.NET, a web.config file should already reside in the root. In that case, you'll want to merge the contents of the web.config.merge file to your existing web.config file.
Before you do this, however, you'll need to change a few of its settings. Open the web.config.merge file in a text editor or in Visual Studio. Look for and modify the parameters for the following sections:
ErrorPage.AppID If you would like to use Live Search as your search engine, you first have to get an ID at the search.live.com Web site. You then enter that ID in this section.
ErrorPage.CultureInfo Here you can enter the culture/locale where you want the search performed. The default value is en-US for US English.
ErrorPage.Domain In this section you enter the name of your domain to tell the Toolkit where to retrieve the search results to display in the error page.
ErrorPage.WebSearchResultsNumber The value you enter here lets you indicate the maximum number of search results to list on the error page. The default is 10; the maximum is 50.
ErrorPage.PerformSpellCheck Here you enter a value of True if you want the error page to display spelling suggestions in the case of mistyped URLs or search strings.
ErrorPage.SpellingMessage In this section you type the syntax of the question you want displayed in the event of a spelling error. The default text string is: Were you looking for {suggestion} instead?
ErrorPage.SearchTitle This is the title of the page that is displayed if subsequent search queries don't deliver the correct content. Otherwise, your site visitors will keep receiving this phrase: "We are sorry, the page you requested cannot be found."
ErrorPage.PrivacyURL, ErrorPage.Sitemap.URL, and ErrorPage.CopyrightURL You can add links to these other pages, which are displayed in the footer of your error page.
After you've modified the web.config.merge file, you then must merge its settings to your existing web.config file. The web.config.merge file contains two sections, which are appSettings and system.web. The settings in the appSettings section are unique to the Toolkit, so you can copy and paste those strings directly into the same section in your web.config file. The system.web section contains a single string for customErrors. If the customErrors string already exists in your web.config file, comment it out and replace it with the string from the web.config.merge file. Remember to back up your existing web.config file before you make any changes.

Customizing the ErrorPage File
As your next step, you may want to modify the ErrorPage.aspx file. The file is already fully configured, so there's little you need to do here. You may want to change some of the error responses in the page, such as "We are sorry; the page you requested cannot be found" and "See below for results close to your request," to text strings of your own choice. Finally, you can modify the errorstyle.css stylesheet by tweaking the color, font, point size, and similar attributes for your errorpage.aspx file.

Working in IIS
Your final step is to modify the custom error settings in IIS to point to your new custom error page. This process varies depending on which version of IIS you are using.
For IIS 5.0 or IIS 6.0:
  1. Open IIS.
  2. Select the Web site to customize.
  3. Right-click on the site and select Properties from the pop-up menu.
  4. From the Web Site Properties window, click the Custom Errors tab.
  5. Scroll down the list of HTTP errors and select the one you wish to customize—in this case, 404.
  6. Click on the Edit button.
  7. Select URL from the Message type dropdown box.
  8. Type /ErrorPage.aspx in the URL field. Click OK.
For IIS 7:
  1. Open IIS.
  2. Select the Web site you wish to customize.
  3. Double-click on the icon for Custom Error Pages.
  4. Scroll down the list of status codes and double-click on the 404 code.
  5. Select Execute URL from the Path Type dropdown box.
  6. Type /ErrorPage.aspx in the Path Field. Click OK.
You can now test your error page. Open the home page of your own site. In the address field, add a subpage of your site to the URL, but misspell it. Your custom error page should appear with a list of alternate suggestions and potential search results.
You may not be able to completely ensure that visitors will always find what they're looking for on your site, but presenting a custom error page should ease their frustration.

Lance Whitney is an IT consultant, trainer, and technical writer. He has spent countless hours tweaking Windows workstations and servers. Originally a journalist, he took a blind leap into the IT world 15 years ago.


Utility Spotlight Visual Round Trip Analyzer
Jim Pierson

Web engineers! Are you looking at performance problems in your application caused by the network? Most performance engineering efforts focus on measuring server response time as the primary cause of slow Web pages. Server response issues certainly need to be tracked and resolved, but many Web sites are still perceived as slow even when the servers are fast. Focusing on the network download pattern between client and server(s) can produce huge performance wins, and that's something the free Web performance analysis tool, Visual Round-trip Analyzer (VRTA), can help you with.
Recently, we announced the public release of VRTA in the November issue of MSDN Magazine. In the column, "12 Steps to Faster Web Pages," I discussed many of the visual aspects of the tool, such as how the tool shows files downloading in series or parallel. That column really focused on the deep-dive troubleshooting that VRTA brings to the table, but suppose you're more interested in generating a simple business summary of a Web site's performance issues. In this column, I will look at the Expert Analysis report that is built into VRTA.
This new expert makes it easy to quickly identify download performance problems. VRTA evaluates a Web site against a list of known best practices and grades the site for each of these, A through F. Let me show you how it works.
The TechNet Magazine Web site is a good site to analyze. First, let's make sure we are all starting with the same download. I want to look at the site as a first time user would see it, with no cached files already downloaded. We call this Page-Load-Time 1 (PLT1), a first time user. A user coming back the next day would have cached files. We call that PLT2.
If you are testing the same site over and over as PLT1, you will need to delete the cache each time and close all TCP connections. To do this, go to an About:blank, delete the cached files, then close and reopen the browser to the test site, in this case technet.microsoft.com/magazine/default.aspx. Before I navigate to this site, I will start the VRTA tool. See the VRTA download site for the tool. When the page has finished loading, I will stop VRTA and open the Analysis report. Figure 1 shows a summary of the results from the Expert.
This high level summary of the best practices indicates the Web page has several problems with performance. The overall score is a 'C'. The site is said to "perform moderately." It appears there are problems with the efficiency of the download, expiration dates, too many small images, and too much white space.
Let's have a look at the exceptions reports under the Analysis tab of Visual Round-trip Analyzer. This is a list of known Best Practices, as in Figure 1. For each, we give you an idea of how the Best Practice works and what parameters we are looking for.
Bandwidth Efficiency (Score = F) Bandwidth Efficiency Rate (BER) is a primary indication of how well a Web page is able to utilize the available bandwidth. Applications that do not follow such best practices as downloading on parallel TCP ports and using Keep-Alives and larger files will not sustain high bit rates on longer latency wide area networks (WANs). A score of A is given to 80% BER and above. BERs of 40% and below are given a score of F. The math for this rule is BER = AvgBitRate / MaxBitRate.
For our test of the TechNet Magazine site the analysis indicates that though the Web site is downloading at a maximum bit rate of 3Mbps (3022Kbps), it is averaging only 523Kbps. That's an average of only 17%of the max—not very efficient. The ideal Web page ramps up to max bit rate and then stays at max bit rate until the page is completed. Instead, many sites download at a trickle or in short bursts. The score for this rule reflects of all the other best practices (known and unknown).
Expiration Date (Score = F) This rule checks for expiration dates. When a user returns to a site, many of the needed files are already in the cache and will be reused, but first the browser will check to see if a file is still good. If no expiration date is found, the browser will send a Get-If-Modified request to the server.
Most files will not have been updated and will get a 304-Not-Modified. This wastes time and server resources. A better idea is to set expiration dates to three years out, then change the filename, path, or argument string to force a new file version download. Max-Age is an alternative to Expiration dates and is also checked. A grade of A is given for files whose Max Age is greater than one year.
The files in Figure 2 were found to either have no expiration date or to have one that is less than one year away. Many of the files in this site do not have expiration dates set, or they have them set to really short periods. Max Age is specified in seconds and 7,200 seconds is only two hours, so a user coming back the next day would request these files again (Get if-modified), instead of using their cached files from today. This wastes server capacity and network resources as well as user time.
Moreover, as I click through several pages on the site, I notice the same files are used on many pages. Each time a user goes to the next page, the browser sends a "get-if-modified" request for each file, to which "304-Not modified" is a common response from the server, again wasting time and resources.
Time to First Byte(Score = D) This rule checks the server response time. TTFB is the time from the Get packet until the first response packet. Subtracting the TCP connect time from the TTFB gives the server response time. A response time of less than 30ms is scored as an A, while a time greater than 100ms is an F.
The files in Figure 3 have long server time (seconds). Here we see several files that take longer than 30ms for a response from the server. The VRTA tool has already subtracted the network delay time by looking at the TCP connect time. Most servers can respond in less than 10ms when only a disk seek is required, so 400ms for the server to respond is a very long time.
Image clustering (Score = F) This rule checks the number of small image files. The site we looked at has 28 files that are under 4KB. TCP does a poor job downloading smaller files. It's much like going grocery shopping and only getting one item at a time, then going home and coming back later for another one item, over and over again. It's far better to consolidate many smaller files into fewer large files so that TCP can send more packets per round-trip. With images, we call this image clustering, and it involves putting many small images side-by-side in one file and then cropping them out and placing them on the page on the client side.
These are but a very small set of examples of what the VRTA tool can show you about your Web page. The VRTA download is available online. Try it on your site and see how many problems can be solved. You'll find that many of them are "low hanging fruit" and can be fixed with just a little effort.

Jim Pierson has been with Microsoft for 12 years, 5 of which were focused on Web Performance Engineering. Jim is Principal Group Manager in the GlobalPerf team that provides tools for measurement, analysis, and training of best practices to the online services of MSN and Windows Live.


Inside SharePoint Maintaining Security Account Credentials
Pav Cherny

Code download available at: ChernyInsideSharePoint2009_02.exe (413 KB)

Contents
Password Changes in a SharePoint Environment
Troubleshooting Farm Credential Issues
Conclusion
Frequently changing farm credentials and security account passwords are important measures for keeping a Microsoft Office SharePoint Server (MOSS) farm secure. Yet for administrators, these tasks are tedious and repetitive. The tools are difficult to use, the underlying processes are complex, good documentation is hard to find, and there is a certain risk of corrupting the server farm even when following all of the proper procedures.
Adding to the mix is problematic technical advice by Microsoft Product Support Services (PSS). On the one hand, there are useful Knowledge Base (KB) articles, such as "How to Change Service Accounts and Service Account Passwords in SharePoint Server 2007 and in Windows SharePoint Services 3.0." And on the other hand, there are disasters, such as "Error Message When You Try to Use the SharePoint Products and Technologies Wizard: 'Exception: System.ArgumentException: Error during Encryption or Decryption.'"
The above article states that you must create a new configuration database if the account credentials of a Web app can't be decrypted anymore. This can happen if you use a critical command switch at the wrong time or if the process of updating farm credentials completes unsuccessfully for any reason. This KB article did a fine job of instilling fear in admins that changing farm credentials can corrupt a server farm irreparably. Who can expect customers to maintain security in their SharePoint environments by frequently changing farm and security account passwords if they face that kind of risk?
Customers don't wipe out Active Directory to reset user accounts, and they should not be forced to wipe out their SharePoint configuration databases either just because a Web application password was lost. It isn't necessary to create a new configuration database. In most cases, it is sufficient to use the standard Stsadm.exe tool to overwrite corrupted credentials. And in all other cases, you can reset the passwords and retain your configuration database by using the tools included in the companion material, available in the February 2009 Code Download. Resetting passwords in Windows SharePoint Services (WSS) 3.0 and MOSS 2007 environments doesn't require access to old credential keys nor a new configuration database. But it does require better support tools from Microsoft and better documentation of the underlying processes that facilitate password changes.
This is the first column of a two-part series in which I review the standard procedures and tools for maintaining SharePoint security accounts, discuss their limitations, highlight the risks, and suggest an alternative approach to achieve better security, reduce administrative overhead, and therefore lower total cost of ownership (TCO) in SharePoint environments. This first part deals with the architectural details and the complicated process of accomplishing password changes.
It is vital to understand how SharePoint deals with security accounts and passwords whether you maintain security manually using scripts or by using a fully automated solution, such as the one I will introduce in next month's column. To keep my explanations realistic and practice oriented, I once again rely on a test environment. In this first part, a straightforward single-server installation is sufficient, as outlined in the companion worksheets. As always, my worksheets and companion tools are for test labs only and are not to be used in production environments. Use my tools at your own risk.

Password Changes in a SharePoint Environment
Changing the password of a SharePoint security account or the farm's configuration account is a remarkably complex undertaking. Among other things, you must apply changes in Active Directory and in the SharePoint server's local Security Account Manager (SAM) database, in the Service Control Manager (SCM) database, in the IIS metabase, in SQL Server, and, of course, in SharePoint content and configuration databases.
You also must replicate the changes to all other servers in the farm to apply the changes consistently. You might have to re-encrypt all the passwords of all SharePoint services and application pools for all servers in the farm if your changes affect the farm's credential key, which is the encryption key that SharePoint uses to protect the passwords of security accounts in the configuration database. When you change the farm's config account password, you implicitly change the farm's credential key. Figure 1 shows the process in a server farm with two front-end servers. Frankly, it is impressive that this process works as well as it does.
Figure 1 Changing the password of a SharePoint security account
All security account password changes begin in Active Directory, and from that moment forward, until you update the affected password in SharePoint, the farm is in an inconsistent state. The password is now outdated in IIS and elsewhere, but SharePoint is still operational.
This is good news for organizations with high-availability requirements. Password changes do not require system restarts. Windows services and IIS can continue to use the security token they obtained by logging on the affected security account with the old password when you last started the server. But don't restart IIS or the entire server during the transitional phase of password inconsistency.
IIS and other services would not be able to log on with the old password during the restart, and the affected application pool or service would not be able to come online anymore. In this situation, IIS writes a warning into the server's event log (see Figure 2). So don't wait too long with the password update in SharePoint following the password change in Active Directory.
Figure 2 IIS warns about outdated application pool credentials
In order to update the password of an application pool account, you should use SharePoint 3.0 Central Administration (_admin/FarmCredentialManagement.aspx) or the following Stsadm.exe command:
stsadm -o updateaccountpassword -userlogin <DOMAIN\USER>
  -password <PASSWORD> -noadmin
In response, SharePoint encrypts the new password by using the farm's credential key and overwrites the old encrypted password in the configuration database. Next, SharePoint updates the account information in the IIS metabase and in all other necessary places. Once this is accomplished, SharePoint generates a timer job of type SPContentAppPoolCredentialDeploymentJobDefinition to deploy the new credentials on the remaining servers in the farm, which it places into the configuration database.
As seen in Figure 1, SharePoint relies on timer jobs to apply administrative settings globally on all servers in the farm. The SharePoint Timer services on the remaining servers in the farm pick up the job and accordingly update the local security settings on their servers with the help of the WSS Administration (SPAdmin) service to bring the farm back into a consistent state.
This is the process for application pool accounts, but there are many other types of SharePoint services that use security accounts, such as the SharePoint Timer service itself, the WSS Help Search service, and possibly Shared Services Providers (SSPs), Office SharePoint Server Search service, and single sign-on (SSO) service. There may be even more services, depending upon the solutions installed in your farm, and each service type has different password update requirements. The article at support.microsoft.com/kb/934838 contains a list of commands that you must use for the service accounts in WSS 3.0 and MOSS 2007. Check the product documentation of any additional solutions that you use for further tools, commands, and update procedures.
This highly diversified and uncoordinated landscape of tools and commands is one of the downsides of the current SharePoint security architecture. It doesn't scale well and can drive up TCO depending upon the number of custom solutions in your server farm that use security credentials. In the second part of this series, I will show you how to bring this under control so you can use a single solution to accomplish all the necessary updates regardless of the underlying service types.
The most important update scenario concerns the farm credentials. The password of the farm account is special because it affects the farm's credential key, which is used to encrypt all passwords in the farm, as mentioned earlier. So, following a change of the farm account's password in Active Directory, you must update SharePoint using the command:
stsadm -o updatefarmcredentials -userlogin <DOMAIN\USER>
  -password <PASSWORD>
SharePoint then must re-encrypt all existing (encrypted) passwords in the configuration database, it must update the SharePoint Timer service account (which uses the farm account as its identity), and it must propagate these changes again to all servers in the farm by means of a timer job of type SPAdminAppPoolCredentialDeploymentJobDefinition.
Many things can go wrong during this phase. The timer job might get stuck in the queue, as illustrated in Figure 3, or the update process might fail unexpectedly, due perhaps to a sudden power outage, leaving behind old encrypted passwords that SharePoint now cannot decrypt anymore because the credential key has changed.
Figure 3 A credential deployment job stuck in the queue because the SharePoint Timer service is not running on all servers in the farm
In another scenario, you might update the passwords of application pool accounts before the new farm credentials have reached all servers in the farm, causing servers with the outdated credential key to fail because they cannot decrypt the updated password in the configuration database yet, as illustrated in Figure 4. This is an interesting scenario and the reason for the -noadmin switch in the updateaccountpassword command. If the farm account is also used as an application pool account (not recommended), then you should update the farm credentials first, wait until all servers in the farm have processed the timer job, and then update the application pools.
Figure 4 Hold application pool updates until the Administration Application Pool Credential Deployment job has been processed
Correspondingly, the updateaccountpassword command checks whether the specified security account is the farm account and informs you about the dependencies if this is the case without performing the update. By using the -noadmin switch, you disable this check and apply the changed password to the account in the application pool configuration, but it is difficult to automate these procedures in a script with appropriate lag time.

Troubleshooting Farm Credential Issues
Now let's take a closer look at the updatefarmcredentials command. It comes with a dangerous switch that can cause great pain, especially if it's used as described in the article "Error Message When You Try to Use the SharePoint Products and Technologies Wizard: 'Exception: System.ArgumentException: Error during Encryption or Decryption”. I'm referring to the switch called -local. The SharePoint developers introduced this switch for performing local farm credential updates manually. The idea is that you can delete a timer job from the queue in Central Administration (_admin/ServiceJobDefinitions.aspx) if the job is corrupted or not processed for any other reason and then perform the necessary update step directly by using the command:
stsadm -o updatefarmcredentials -userlogin <DOMAIN\USER>
  -password <PASSWORD> -local
The -local switch instructs the updatefarmcredentials command to apply the password change only on the local computer. It is important to recognize, however, that this update affects the credential key and the SharePoint Timer service, but not the application pools, Search service, SSPs, and so forth. It is assumed that you already ran the updatefarmcredentials command without the -local switch on another server in the farm and thus re-encrypted all the passwords in the configuration database. It isn't necessary to perform this re-encryption step again. But what if you used the -local switch without taking this step?
Using the -local switch without first running the updatefarmcredentials command without this switch causes trouble because the -local switch changes the credential key. The application pool passwords are encrypted with the old key in the config database, but this key is now overwritten.
Take a look at Figure 5. You cannot run the updatefarmcredentials command without the -local switch anymore because this requires re-encryption of passwords that can no longer be decrypted. When the command fails, you can find entries in the Application event log stating: "Error re-encrypting credential Id 022e607e-b49e-40e4-bd3f-f56a3c69f94d with owner Id 431b6897-16eb-4b9a-be65-60f1f603008d during deploying of administration application pool credentials, please recreate credential manually. Operation is not valid due to the current state of the object."
Figure 5 Unable to re-encrypt application pool passwords because the passwords can no longer be decrypted
If you just changed the farm account in a single-server deployment from the Network Service account to a domain account, you are in real trouble because you can't change back to the old credential key in this case. The Network Service doesn't use a password, and the credential key is therefore random.
In a search for help, you might encounter that "Error Message When You Try to Use the SharePoint Products and Technologies Wizard: 'Exception: System.ArgumentException: Error during Encryption or Decryption” article I mentioned earlier, and without further knowledge, your trouble worsens because you now learn that you must create a new configuration database to get rid of these passwords that can no longer be decrypted. It is an unbelievable constellation of unfortunate circumstances. You shouldn't be able to run the updatefarmcredentials command with the -local switch in the first place, or the command should create a backup copy of the old credential key so you can re-encrypt the passwords later on. Or it should simply detect that the passwords are not yet re-encrypted and re-encrypt them at this point.
Instead, the -local switch goes ahead and happily corrupts your SharePoint configuration database without any warnings, as shown in Figure 5. A support tool from Microsoft to reset those passwords that can no longer be decrypted would come in handy. And, of course, proper product documentation that warns you about the critical nature of certain command-line operations and that guides you out of this trouble would also be appreciated.
The good news is that the updateaccountpassword command can encrypt new passwords of application pool accounts without having to decrypt the old passwords. So, use this command to update all application pools that use domain accounts. This should take care of most, if not all, corrupted passwords. Unfortunately, you can't use this command to update application pools that use the Network Service account. This account doesn't require a password, so the updateaccountpassword command doesn't apply.
Interestingly, the Network Service account might be associated with password data in the configuration database. New application pools that use the Network Service have no passwords, yet if you ever change the application pool to use a domain account and then revert back, the Network Service inherits the domain account's password reference. SharePoint doesn't set the password to null (bad coding practice) and this rubbish data now causes the trouble.
It's an ironic twist that customers are supposed to throw away their configuration databases because rubbish and completely useless data can no longer be decrypted! If you are lucky, you can change the application pool configuration in Central Administration (_admin/FarmCredentialManagement.aspx) and specify a domain account. If you are unlucky, you encounter the encryption/decryption error displayed in Figure 6. You can't change the account in Central Administration, you can't use the updateaccountpassword command, you can't run the SharePoint Products and Technologies Configuration Wizard, and you can't update the farm credentials by using the updatefarmcredentials command. Now what do you do?
Figure 6 Trouble caused by Network Service password junk left behind in the configuration database
To solve this problem, you need a tool that goes straight into the configuration database and removes the junk, such as the Reset AppPool Password tool, illustrated in Figure 7 and included with source code in the companion material. This tool is very straightforward. It pulls the data of application pools that use accounts with associated encrypted passwords directly out of the configuration database and then uses the SharePoint object model to determine if the application pool password can be decrypted.
Figure 7 Resetting corrupted passwords to null
If accessing the password through the object model fails with an argument exception, the password is corrupted. Here, the tool gives you an opportunity to replace the password's array of encrypted byte values with a null reference and saves the changes back into the configuration database. Empty strings don't need to be decrypted and therefore don't cause an argument exception. Problem solved.
In order to complete the repair process, I recommend that you update the farm credentials and subsequently all application pool accounts by using Stsadm.exe and Central Administration. Your farm is back in a consistent state, and you didn't have to throw away your configuration database.

Conclusion
Despite the fact that changing farm credentials and security account passwords is a tedious and error-prone process, you need not fear that changing farm credentials will corrupt your server farm irreparably. The configuration database can be repaired even if you happen to lose your current credential key. You simply need to reset the affected passwords, and this is possible through the standard Stsadm.exe tool or a low-level database tool, such as Reset AppPool Password. So keep changing farm credentials and security accounts frequently, use strong passwords, and don't use the Network Service as a security account because it complicates the farm configuration and troubleshooting. Use dedicated domain accounts.
Now that you've addressed the risks of database corruption through password changes, you can focus your attention on the real issue in the SharePoint security architecture: the current architecture does not accommodate password changes very well. You need to apply too many commands in a more or less specific order depending on the service types you must update in your farm. Some commands have dangerous switches; some don't. Some commands can corrupt your configuration database; others are harmless. Some services need to be updated globally across the entire farm and others are local to a specific server.
In any case, administrative overhead is high due to the complexities involved, and security is generally low due to infrequent change schedules, weak passwords, and scripts that deal with passwords in clear text. In my next column, I'll show you how to tackle these issues and eliminate them—and for all service types, including those not yet developed, whatever their password update requirements. No more (manual) password changes!

Pav Cherny is an IT expert and author specializing in Microsoft technologies for collaboration and unified communication. His publications include white papers, product manuals, and books with a focus on IT operations and system administration. Pav is President of Biblioso Corporation, a company that specializes in managed documentation and localization services.


Windows PowerShell Best Inventory Tool Ever!
Don Jones

Contents
Exploring the Tool
Test Drive
Computers En Masse
What Do You Do with It?
Going Further
Powerful Do-It-Yourself Tools
In the past few installments of this column, I've demonstrated various techniques for using Windows PowerShell to collect computer inventory information. In this installment, I will assemble everything into a ready-to-run tool that can gather any kind of information from Windows Management Instrumentation (WMI).
Video
Don Jones wraps up his series on building a very powerful, full-featured inventory tool. In this streaming video, he demonstrates the final touches you'll need to add.

And the tool will be designed to work with either a text list of computer names or computer names queried from Active Directory.

Exploring the Tool
I will begin with the script itself (see Figure 1), stepping through the elements of its structure and looking at some of the tricks it uses. Some of these techniques may seem a bit odd or out of place, but I'm lumping them in here so I can demonstrate a variety of cool tricks being used.
Figure 1 The script
1. function Get-WmiInventory {
2.  param (
3.  $wmiclass = "Win32_OperatingSystem"
4.  )
5.  PROCESS {
6.   $ErrorActionPreference = "SilentlyContinue"
7.   $computer = $_
8.   trap {
9.    $computer | out-file c:\errors.txt -append
10.    set-variable skip ($true) -scope 1
11.    continue
12.   }
13.   $skip = $false
14.   $wmi = Get-WmiObject -class $wmiclass -computer $computer -ea stop
15.   if (-not $skip) {
16.    foreach ($obj in $wmi) {
17.     $obj | Add-Member NoteProperty ComputerName $computer
18.     write $obj
19.    }
20.   }
21.  }
22. }
On line 1, you can see that the function is named Get-WmiInventory. Right below that, I define an input parameter named $wmiclass and give it a default value of "Win32_OperatingSystem". The PROCESS scriptblock indicates that this is a filtering function, one designed to accept a collection of computer names from the pipeline—you'll see in a moment how I pass that information into the function.
Line 6 turns off the shell's normal error-reporting behavior since I want to provide my own error logging. And line 7 simply grabs the current computer name (as piped into the function) into the variable $computer.
Now go to line 13, where I create a variable named $skip and set it to the Boolean value False (more on that in a moment). On line 14, I then attempt to grab the desired WMI information from the current computer by using the Get-WmiObject cmdlet. Note that I've specified the –ErrorAction (or –EA) parameter, which tells the shell to generate an exception if the WMI retrieval fails for any reason.
Line 15 checks to see if the $skip variable still contains False. If it does, then line 16 enumerates through whatever came back from WMI and adds (on line 17) a ComputerName property to each WMI object. That way, when the script is returning WMI objects from multiple computers, each one is labeled with its parent computer name in a convenient property. Line 18 outputs each object to the pipeline so another cmdlet can deal with it or so the shell's formatting subsystem can take over to display some of the object's properties.
But what if something goes wrong? Since I specified –EA Stop, the shell will execute the trap on line 8. First, it writes the computer name to a text file so that I have a log of computer names that failed the inventory. Then, on line 10, I set the $skip variable to the Boolean value True. This prevents the script from attempting to output anything at line 15—if I hadn't done this, then each computer that fails would result in the previous computer's inventory information being output again.
Windows PowerShell Q&A
Q Can I use Windows PowerShell to manage Windows Server 2008 Server Core?
A Absolutely. Oh, you've probably heard that Windows PowerShell can't be installed on Server Core because Server Core doesn't currently support the .NET Framework, which the shell requires. But this isn't a problem. A lot of wise admins don't install anything on their servers if they can help it, and Windows PowerShell is no exception. Instead, you can install Windows PowerShell on your client computer and use it to remotely manage Server Core by using such technologies as Windows Management Instrumentation (WMI), Active Directory, and much more. You can, for instance, easily manage most aspects of Active Directory on a Server Core-based domain controller, all without leaving the comfort of your desk.
Line 10 uses a different technique to set the variable: The trap itself is a private variable scope and it doesn't contain a $skip variable. The $skip variable I want is from the trap's parent scope, one level up. The Set-Variable cmdlet allows me to modify the variable by using the -scope parameter, indicating that the $skip I'm interested in is one level up. Also note that when you use Set-Variable, the variable name doesn't include the $ symbol in front of it.

Test Drive
A simple way to test this is to just pipe in a single computer name, like this:
"localhost" | Get-WmiInventory
Because I have not specified a WMI class name, the default of Win32_OperatingSystem is used. To specify a different class, do this:
"localhost" | Get-WmiInventory "Win32_LogicalDisk"
Figure 2 shows the results. And if you want to include more computer names, you simply make a comma-separated list of them like so:
"localhost","server2","client17" |
Get-WmiInventory "Win32_LogicalDisk"
Figure 2 Get-WmiInventory results

Computers En Masse
Specifying individual computer names is pretty tedious if you have a long list. If you already have a list of computer names in a text file—one computer name per line—then you can pipe those names right to the inventory function by doing this:
Get-Content c:\names.txt | Get-WmiInventory "Win32_Service"
Another technique involves the use of the Get-QADComputer cmdlet, which is provided as part of the free Active Directory cmdlets available from quest.com/powershell. You need to install the Active Directory cmdlets, open Windows PowerShell, and run:
Add-PSSnapin Quest.ActiveRoles.ADManagement
Run Get-QADComputer to retrieve all of the computers from Active Directory. But be aware that in a large domain, this will take a while! You can run Help Get-QADComputer to see some of the options available for filtering the list of computers, such as only grabbing those computers that reside in a specific organizational unit (OU). You also need to make a minor change to the script, changing line 7 from
$computer = $_

$computer = $_.Name
This is necessary because the objects produced by Get-QADComputer store the computer name in a Name property rather than as a simple string object.

What Do You Do with It?
This script's default output will take whatever form the Windows PowerShell formatting subsystem will create by default. For example, the Win32_OperatingSystem class is a small subset (just six properties) of the information actually available from that class. You can pipe this script's output to any standard shell cmdlet to customize that output. So, to display build number and service pack information, along with the computer names, do this:
Get-Content c:\names.txt | Get-WmiInventory |
Format-List BuildNumber,ServicePackMajorVersion,ComputerName
If you want to get an inventory of installed services and produce an HTML table, do this:
Get-Content c:\names.txt | Get-WmiInventory "Win32_Service" |
ConvertTo-HTML | Out-File c:\services_inventory.html
And if you want to get a complete inventory of logical disk information and write it to a comma separated values (CSV) file, do this:
Get-Content c:\names.txt |
Get-WmiInventory "Win32_LogicalDisk" | Export-CSV c:\disks.csv
Note that because the Get-WmiInventory function produces objects, not simple text, the shell can do quite a bit to help you get the data into the form you need, whether it's for statistical trending, management reports, or just a quick look at critical information.

Going Further
One weakness in the current script is that it retrieves all of the WMI objects from the class you specify. With some classes, that may be too much data. I typically prefer to restrict Win32_LogicalDisk so that I only retrieve local disks, which have a DriveType property of 3. I can easily modify the script to include filter criteria like that—the revised script is shown in Figure 3. To call this new script, I use this:
Get-Content c:\names.txt |
Get-WmiInventory "Win32_LogicalDisk" "DriveType='3'"
Figure 3 The revised script to support filtering
1. function Get-WmiInventory {
2.  param (
3.   $wmiclass = "Win32_OperatingSystem"
4.   $filter = ""
5.  )
6.  PROCESS {
7.   $ErrorActionPreference = "SilentlyContinue"
8.   $computer = $_
9.   trap {
10.    $computer | out-file c:\errors.txt -append
11.    set-variable skip ($true) -scope 1
12.    continue
13.  }
14.  $skip = $false
15.  $wmi = Get-WmiObject -class $wmiclass -computer $computer -ea stop –filter $filter
16.  if (-not $skip) {
17.    foreach ($obj in $wmi) {
18.     $obj | Add-Member NoteProperty ComputerName $computer
19.     write $obj
20.    }
21.   }
22.  }
23. }
The changes, at lines 4 and 15, simply collect a new parameter named $filter and pass it to the –filter parameter of Get-WmiObject. When calling the function, the $filter parameter is second, so it's passed as a second value after the WMI class name. There's plenty more that you can do to customize this function for your specific environment, but there are a few things you should avoid doing:
  • Don't have the function send its output to anything but the pipeline, which is done on line 19 of the revised script. In other words, don't have the function output information to a file. Instead, take the pipeline output of the function and send it to a cmdlet, such as Out-File, that moves the output to a file or other medium. This will ensure that the function retains its flexibility.
  • Don't filter out information within the function—such as removing WMI object properties or only outputting a subset of properties. This too would reduce the function's flexibility for future scenarios. Instead, if you don't need all of the information being output, pipe that output to a Format cmdlet or to Select-Object so that you can get just the information you need for the task at hand.
  • Don't add additional tasks to the function. Keep each function single-tasked, rather than building super-functions that do a lot of things. If you have a complex process, accomplish it by building each discrete task as a standalone function and then piping information from one to another. This keeps each function more flexible, easier to maintain, and much easier to debug.

Powerful Do-It-Yourself Tools
While a script like this is no replacement for a powerful configuration management system such as System Center Configuration Manager, it does provide an easy way to inventory select information from a set of remote computers. This script is a good example of one of the things Windows PowerShell is best at: providing a quick, ad hoc solution to problems that Microsoft didn't foresee when creating Windows or whatever other product you're using.
Because this particular script uses WMI, it works with computers all the way back to Windows NT 4.0—even though Windows PowerShell itself can't be installed on a computer that old! And, because many Microsoft products expose management information through WMI, you can collect information from various products. I like to use a WMI explorer tool (a free one is at scriptinganswers.com in the "Tools Zone") to browse the WMI information available on a computer and then use a script like the one I've shown here to collect that information from multiple computers.

Don Jones a partner at Concentrated Technology and co-author of Windows PowerShell: TFM (SAPIEN Press). Get weekly Windows PowerShell tips, and contact Don, at ConcentratedTech.com.


Geek of All Trades Making Sense of Server Core
Greg Shields

Contents
Windows Server 2008, MS-DOS Edition
The Cheat Sheet
Core Management
In the world of IT publications, the needs of the jack-of-all-trades IT professional are perhaps the least represented. With all the books and articles that explain today's technology in deep and gory detail, too often the needs of the smaller-environment IT pro don't get met. Does this describe you? Are you ultimately responsible for every facet of your computing environment? Is it your job to manage everything from network routing to server administration to changing toner cartridges in the printers?
If so, this column is dedicated to you. My name is Greg Shields and I've spent the last 15 years working in IT environments both large and small, and have devoted the last few to writing about what I've seen and learned. I'm excited about this monthly opportunity to share the best real-world tips and tricks that you—the master of none—can implement today to improve your small IT environment.
You'll note that I'm specifically not saying "your small business" here. While those of us in small businesses today typify the jack-of-all-trades IT professional, there are others whose job it is to manage small parts of a much bigger organization. No matter, if you're in charge of it all, you've likely got a "just-get-it-done" outlook on work. In this column, I'll attempt to tell you exactly how to do that, and hopefully get you home on time every night as an added bonus.
To start you out right, this first column will talk about a new form of Windows Server you might not have paid attention to yet. Server Core in Windows Server 2008 is a compelling new addition that makes sense for the small computing environment. In this column, I'll tell you why that might be the case and where it might fit best. I'll also take you through the steps you'll need to set up your first instance of Server Core on Windows Server 2008.

Windows Server 2008, MS-DOS Edition
The MSDN Web site refers to Server Core as "a minimal server installation option for computers running on the Windows Server 2008 operating system. Server Core provides a low-maintenance server environment with limited functionality." Lots of words, but what they really say is that Server Core arrives as another installation option for Windows Server 2008, one that strips away vast amounts of functionality from the operating system. There is no Explorer shell. There is no Internet Explorer. There is no support for managed code.
Server Core is one result of some componentization activities that were completed by Microsoft in the development of Windows Server 2008. From a high-level perspective, this componentization looked at the codebase that was Windows Server and broke it down into logical groups that connect to each other. The detailed knowledge thus gained of the components that make up Windows Server, along with the interfaces between those components, better enabled Microsoft to make decisions about which components could be safely set side.
By stripping Windows down to its bare essentials, even eliminating nearly all its graphical user interfaces, the end result is a smaller OS with a reduced attack surface and fewer requirements for hardware resources. Those reduced hardware requirements mean Server Core can be installed on less-powerful servers. With Server Core, you can continue using older hardware that you might otherwise dispose of.
As noted earlier, this slimmer and trimmer OS eliminates virtually every graphical tool used to administer a server at its console. For all practical purposes, Server Core can be (jokingly) considered Windows Server 2008, MS-DOS Edition. So if you're to use it in your environment, you'll need to dust off your old command-prompt skills and relearn the text commands you may not have used in years.
For some, that's the bad news. But once you've gotten through that initial re-learning process, you'll find you've developed some command-line techniques you can immediately use in managing the other "real" Windows servers in your environment. This is the case because, with few exceptions, every command you use to manage your Server Core instance can be used on the full Windows Server to accomplish the same task. More command-line management results in more scripts and batch files, which lets you solve problems and administer your environment both faster and with more reliability.
If Server Core's command-line requirements still scare you, you will be happy to know that most of the command-line configuration happens when you first build your server. Once the server is built, networked, and connected to your domain, you'll use your regular administrative consoles like those you use for your other servers.

The Cheat Sheet
Toward that end, let me help you through your first Server Core installation. This process includes all the commands that you'll need to run from the command prompt after the initial installation in order to get your server ready for operation.
Server Core arrives not as a separate edition of Windows Server 2008, but instead as an installation option. This means that no matter what edition of Windows Server you purchase—Standard, Enterprise, or Datacenter—you have the option of installing that instance as either a full version or a Server Core version. You'll see these options when you drop the installation DVD into a candidate server's drive and boot the machine.
After answering some initial questions about the installation, you'll see a screen similar to Figure 1, where you can select the installation option for the edition you've chosen. In this case, we'll install the Standard Edition of Windows Server 2008 with the Server Core Installation option. Finish answering WinPE's preinstallation questions to kick off the installation.
Figure 1 Server Core is an installation option
Once the installation routine completes, log in as administrator with a blank password. You'll immediately be asked to change that password. When that's done, you'll be logged in and presented with Server Core's undeniably minimalist user interface. Figure 2 shows the sparse style of the console of your Server Core instance.
Figure 2 Server Core’s interface shows little more than a command prompt
A freshly installed Server Core instance arrives with none of the needed settings preconfigured. To get that instance on the network, in your domain, and ready for work involves a set of configurations that you'll accomplish through the command prompt. Though this may sound daunting, feel free to use what follows as a cheat sheet to get you through the initial configuration:
Change the Computer's Name Your computer may have some random name. Change that name with the following command:
netdom renamecomputer %computername%
  /newname:{newComputerName}
After that, you need to reboot the computer, which you can do with the following command:
shutdown /r /f /t 0
Set Static IP Address, Subnet Mask, and Default Gateway If you use DHCP for your servers, ignore this step. To set a static IP address and configure the default gateway for your server, use this command:
netsh interface ipv4 set address {interfaceName} static
  {ipAddress} {subnetMask} {defaultGateway}
For most servers the value for {interfaceName} will be set to "Local Area Connection." You can verify that name first with this command:
netsh interface show interface
Set a DNS Server Finish up your network configuration by setting your server's DNS server address. Do it with this command:
netsh interface ipv4 add dnsserver name={interfaceName}
  address={dnsServerIpAddress} index=1
Join the Domain Once the computer's name and networking are set, join it to the domain with this command:
netdom join {computerName} /domain:{domainName}
  /userD:{domainUsername} /passwordD:{domainPassword}
Note that you'll need to reboot the computer again after you've completed this process.
Configure a Proxy Server If your organization makes use of a proxy server to connect to the Internet, set its configuration with this command:
netsh winhttp set proxy {proxyServerName}:{proxyPortNumber}
Although Server Core doesn't have the ability to run a Web browser, environments that use proxies will need to configure this setting if they will be using Automatic Updates to download updates from the Internet.
Add and Activate a License Key Server Core doesn't include Server Manager, so the command line is the mechanism for licensing your server. Enter its license key with this command:
slmgr –ipk {licenseKey}
You can separate the character strings in that key with dashes. Activate that license with this command:
slmgr –ato
Be patient during the long pause you are likely to experience after entering these commands, before you see a pop-up window with their result.
Configure Automatic Updates Automatic Updates can be configured via the command line or through Group Policy. Note that it is usually easier to use Group Policy to configure your Server Core instance along with the rest of your environment. However, if your Group Policy configuration for Automatic Updates is set to notify before downloading or installing, it will not work with Server Core.
With no explorer shell, the balloon notifications used by Automatic Updates do not function with Server Core. However, you can use the scregedit.wsf script found in C:\Windows\System32 to set Automatic Updates to download updates and install them at 3:00 A.M. every day, like so:
Cscript c:\windows\system32\scregedit.wsf /au 4
Manage the Windows Firewall with Advanced Security As with Automatic Updates, the Windows Firewall with Advanced Security is best managed with Group Policy. However, if you wish to turn the firewall on or off on a standalone instance, you can do so with this command:
netsh advfirewall set {profileName} state {on|off}
Use the command
netsh advfirewall set /?
to determine the values for {profileName}.
Enable Remote Desktop Finally, if you ever want to manage your Server Core instance's console via Terminal Services, you would use this command:
cscript c:\windows\system32\scregedit.wsf /ar 0
That's it. Now you're ready to install whatever components you need onto your Server Core instance. Remember that Server Core is limited to only a subset of the possible roles that can be installed to a full Windows Server 2008 instance:
  • Active Directory Domain Services
  • Active Directory Lightweight Domain Services
  • DHCP Server
  • DNS Server
  • File Services
  • Media Services
  • Print Services
  • Hyper-V
  • Web Services (IIS) (Limited)
The limited roles available mean that your Server Core instance will likely fulfill only a few needs in your environment. Server Core instances make great lightweight infrastructure servers like domain controllers, DNS servers, or file servers, for example. Because you need only limited direct, on-console interaction with these types of roles, you can generally get by with lesser and/or older hardware.
Server Core's reduced hardware also needs help when hosting Hyper-V. With Hyper-V, your goal is to squeeze as much performance out of your hardware as possible for the running of virtual machines. Since a primary partition operating system instance is required for the processing of Hyper-V's virtualization layer, Server Core's more limited hardware needs means more hardware resources will be available for the operation of your virtual machines.
To take advantage of this, there are two commands you'll need to know to install components into Server Core. The first, oclist, lists the available and installed components on the server. The second, ocsetup {componentName} installs a desired component. For example, to install the Hyper-V role, use this command:
ocsetup Microsoft-Hyper-V
Be aware that unlike virtually every Windows command, ocsetup is case-sensitive. Thus, to install a component, you'll need to use oclist to get the exact name with proper case to later install it with ocsetup.
Another command, Wusa, is handy for the installation of patches. To manually install MSU or MSP patches, use this command:
wusa {patchFileName}

Core Management
Once Server Core is installed, the remainder of your administration is likely to be done through the management consoles already installed to your desktop. For example, once the Hyper-V role is installed, all other work with Hyper-V is then done through the Hyper-V Manager. This makes working with Server Core instances easy once they're built.
On occasion, you'll still need to interact with the server itself to work with files or install a new component or patch. The most obvious and easiest way to accomplish this is by connecting to the server using Terminal Services. But there is another set of command-line tools, the Microsoft PsTools, that can help.
The PsTools have long been used by smart administrators to accomplish command line-based actions remotely on systems all around the network. The tools are of particular use with Server Core since its installation is all about the command line.
If you haven't yet downloaded these tools, you can get them from the PsTools site at technet.microsoft.com/sysinternals/bb896649.aspx. The download includes an HTML help file with usage information for all of the tools.
Then copy them after unpacking to a location in the path of your management desktop. Helpful hint: I copy them to c:\windows where they'll always be available when I open a command prompt.
There are many PsTools, but two in particular are useful for working with Server Core. The first, PsExec, provides a way for you to run a command on another computer remotely. This is useful for initiating individual commands on remote Server Core instances, but even more compelling when you use it to launch a remote command prompt. As shown in Figure 3, you can do this from your management desktop with this command:
psexec \\{computerName} cmd
Figure 3 PsExec brings Server Core’s command prompt to your local desktop
If you run this command against your Server Core instance, you'll see that your command prompt happens to be the very same command prompt on the remote server.
Sometimes you merely want to discover how processes are behaving on your server. Maybe a process is using more than its fair share of the processor or memory. PsList can help. To launch a regularly updating text-based view of processes and view their resource use (as shown in Figure 4), use this command:
pslist \\{computerName} /s
Figure 4 PsList lets you display a list of running processes
The resulting list of processes repeatedly updates and is ordered by processor use. If you happen to find some processes that are misbehaving, you can kill any offending processes with this command:
pskill \\{computerName} {processNameOrPid}
Finally, you will also find the two scripting languages VBScript and Windows PowerShell very useful for managing your Server Core instance. While Server Core comes equipped with the necessary VBScript runtime code to launch commands directly from its console, it does not presently include the ability to use Windows PowerShell. So, while you can run Windows PowerShell commands on your management desktop against a Server Core instance, you can't directly launch them within your Server Core instance.
So, there you have it. I've provided you with enough information to get started using Server Core in your small environment today.
Got a problem in your small environment or something you'd like a little help or insight into? Send your questions and suggestions to me at gshields@concentratedtech.com.

Greg Shields, MVP, is a co-founder and IT guru with Concentrated Technology. His latest book, Windows Server 2008: What's New/What's Changed, is available through SAPIEN Press. Get to know him at ConcentratedTech.com.


Hey, Scripting Guy! Browsing Active Directory
The Microsoft Scripting Guys

Code download available at: HeyScriptingGuy2009_02.exe (151 KB)

Contents
Giving Help
The Nitty-Gritty
Helper Functions
As you may know, the Scripting Editor and I have only recently become the new full-time Microsoft Scripting Guys, and we have spent much of our time lately sifting through the thousands of e-mails that come to the scripter@microsoft.com e-mail account. Without having actually commissioned a study, we'd venture to guess that the number one e-mail question we've seen is "Where can I get the Active Directory Browser." But guess what? We have never seen the thing. We looked and looked but could not find it.
Finally, after dropping a number of secret messages on an Internet message board, we got a coded response that, when deciphered, indicated to us that the ADSI Scriptomatic is in fact the Active Directory Browser. Well, it's not called the Active Directory Browser, and its purpose is not actually browsing Active Directory (it's helping you create ADSI scripts), so I decided to write a Windows PowerShell script that would allow you to browse the Active Directory schema. It is called, oddly enough, BrowseActiveDirectorySchema.ps1, and it is shown in Figure 1.
Figure 1 BrowseActiveDirectorySchema.ps1
Param($action,$class, [switch]$help)

Function GetHelp()
{
  $helpText= '
@"
 DESCRIPTION:
 NAME: BrowseActiveDirectorySchema.ps1
 Browse Active Directory Schema. Lists Classes, and properties.

 PARAMETERS:
 -Action <L(ist all classes), M(andatory), O(ptional), F(ind)>
 -Class class to search: user, computer, person, contact etc
 -Help displays this help topic

 SYNTAX:
 BrowseActiveDirectorySchema.ps1 -A L
 Lists the name of each class defined in the AD schema

 BrowseActiveDirectorySchema.ps1 -A M -c user
 Lists the mandatory properties of the user class

 BrowseActiveDirectorySchema.ps1 -A O -c computer
 Lists the optional properties of the computer class

 BrowseActiveDirectorySchema.ps1 -A F -c user
 Lists all Active Directory Classes that contain the word user
 in the actual class name

 BrowseActiveDirectorySchema.ps1 -Action Find -c user
 Lists all Active Directory Classes that contain the word user
 in the actual class name

 BrowseActiveDirectorySchema.ps1 -help
 Prints the help topic for the script
"@ #end helpText
  $helpText
} #end GetHelp
Function GetADSchema($Action, $class)
{
 $schema = [DirectoryServices.ActiveDirectory.ActiveDirectorySchema]::GetCurrentSchema()
 Switch ($Action)
  {
   "classes" {
              $schema.FindAllClasses() |
              Select-Object -Property Name
             }
   "Mandatory"
             {
              "Mandatory Properties of $class object"
              ($schema.FindClass("$class")).MandatoryProperties |
               Format-Table -Property Name, Syntax, IsSingleValued -AutoSize
             }
   "Optional"
             {
              "Optional Properties of $class object"

              "This might take a few seconds ..."
              ($schema.FindClass("$class")).OptionalProperties |
              Format-Table -Property Name, Syntax, IsSingleValued -AutoSize
             }
   "Find"
             {
              $schema.FindAllClasses() |
              Where-Object { $_.name -match "$class" } |
              Select-Object -property name
             }
   DEFAULT {"$action is not a valid action." ; GetHelp ; Exit}
  }
} #end GetADSchema

Function GetAllClasses()
{
 GetAdSchema("classes")
} #end GetAllClasses

Function GetMandatory($class)
{
 GetAdSchema -action "Mandatory"  -class $class
} #end GetMandatory

Function GetOptional($class)
{
 GetAdSchema -action "Optional"  -class $class
} #end GetOptional

Function FindClasses($class)
{
 GetAdSchema -action "Find" -class $class
} #end FindClasses
# *** Entry Point to Script ***
if($help) { GetHelp ; Exit }

Switch ($action)
{
 "L" {GetAllClasses ; Exit}
 "M" {GetMandatory($class) ; Exit}
 "O" {GetOptional($class) ; Exit}
 "F" {FindClasses($class) ; Exit}
 DEFAULT { "$action is not a valid action." ; GetHelp ; Exit}
}
As you can see, the script begins with the Param statement. This is used to create three command-line parameters that allow you to modify the script by running it, instead of having to edit the script to see different behavior, which is pretty helpful:
Param($action,$class, [switch]$help)
If, for example, you would like to see the mandatory properties of the user class in Active Directory Domain Services (AD DS), you specify the –action M and the –class user, as shown in Figure 2.
Figure 2 Viewing the mandatory properties of the user class
You could just as easily choose the optional properties of the group class. This is the advantage of using command-line parameters: you can modify the behavior of a script at run time instead of having to go back to the drawing board. This is a best practice when writing scripts you intend to use as utilities, instead of those you will use only once.

Giving Help
The next section of the script in Figure 1 shows the GetHelp function. When you write a script that exposes command-line parameters, it is a best practice to include a function that will display help for using the script. You don't want the users of your script to have to open it and read through all of the text to figure out what it does. True, a guiding principle of script design is that a script should be readable, but if you include a help function, you make life easier for users.
The GetHelp function displays information in much the same way as the Get-Help cmdlet. Three sections of text are displayed: description, parameters, and syntax. GetHelp does two things. It creates a here-string that contains the text to display, and then it displays the contents of the variable that holds the here-string (in this case, it's $helpText).
A here-string is a Windows PowerShell construct that allows you to type information and format your output without worrying about quoting rules. Everything that you type in the here-string is treated as text. GetHelp is called when the script is run with the –help switch or if someone types an incorrect parameter. Figure 3 shows the result of calling the script with the –help switch.
Figure 3 Calling the script with the–help switch

The Nitty-Gritty
Now we come to the GetADSchema function—the one that does most of the real work for the script in Figure 1. The here key is using the DirectoryServices.ActiveDirectory.ActiveDirectorySchema Microsoft .NET Framework class. By putting the class in square brackets and following that with double colons, you can access the static methods of the class (in this case, GetSchema and GetCurrentSchema).
The GetCurrentSchema static method returns an instance of an ActiveDirectorySchema class that represents the schema to which you are currently connected. Figure 4 shows the ActiveDirectorySchema class members.
After creating an instance of the DirectoryServices.ActiveDirectory.ActiveSchema .NET Framework class and store the resulting schema object in the $schema variable, you must decide which action to perform. To do this, a Switch statement evaluates the value that is passed to it from the $action variable.
Based upon the condition that is met, the function will find all of the classes in AD DS, display mandatory or optional properties of a specific class, or search for a class that meets a given criteria. When you are using Switch, it is a best practice to always include a default condition.

Helper Functions
To break up some of the code and make it easier to expand the script, I've included a number of helper functions that are called based on the parameters passed to the script when it is run. Each of the helper functions calls GetADSchema and passes a different set of parameters, depending upon the value supplied for the –action parameter from the command line.
The first helper function is GetAllClasses, which calls GetADSchema and passes the word "classes". When the Switch statement in the GetADSchema function matches the string "classes", it calls the FindAllClasses method from the ActiveDirectorySchema class. Here is the code called by the Switch statement:
"classes" {
              $schema.FindAllClasses() |
              Select-Object -Property Name'
And here's the GetAllClasses function:
Function GetAllClasses()
{
GetADSchema("classes")
} #end GetAllClasses
The next helper function is GetMandatory, whose purpose is to return mandatory properties of the object specified in the –class parameter when the script is run. The GetMandatory function receives the value for $class from the command line via the –class parameter. When GetMandatory calls the GetADSchema function, it passes two parameters. Because two parameters are being passed, I consider it a best practice to specify the full parameter name for both, making the code easier to read and understand. In contrast, in the GetAllClasses function, we did not use the –action parameter name when calling the GetADSchema function. In that case, the value "classes" was passed in a positional fashion.
When the GetADSchema function is called, it uses the FindClass method from the ActiveDirectorySchema class to retrieve the class that is specified in the $class variable. It returns an instance of an ActiveDirectorySchemaClass class. The members of this class are shown in Figure 5.
The GetADSchema function next queries the MandatoryProperties property of the object and pipelines the results to the Format-Table cmdlet, where it chooses the Name, Syntax, and IsSingleValued properties. The –Autosize switched parameter of the Format-Table cmdlet automatically sizes the columns to avoid cutting off property values, if possible. The code that runs when the string "Mandatory" is matched is shown here:
"Mandatory"
       {
       "Mandatory Properties of $class object"
       ($schema.FindClass("$class")).MandatoryProperties |
       Format-Table -Property Name, Syntax, IsSingleValued -AutoSize
       }
And here's the GetMandatory function:
Function GetMandatory($class)
{
 GetAdSchema -action "Mandatory" -class $class
} #end GetMandatory
Now let's look at how the GetOptional function can display the optional properties of an AD DS class. GetOptional accepts the $class value that was received from the command line via the –class parameter, then it passes the $class value to the GetADSchema function along with the action named "Optional".
When the string "Optional" is matched in the Switch statement, a message is printed to the screen in order to explain that displaying optional properties might take a few seconds. Then, the FindClass method from the ActiveDirectorySchema class is called. The FindClass method returns an instance of an ActiveDirectorySchemaClass class. If you were to query the OptionalProperties property of the ActiveDirectorySchemaClass class, it'd return the optional properties defined for the chosen class in AD DS.
The results are pipelined to the Format-Table cmdlet, which displays the information in the same fashion as it did for the mandatory properties. Here's that section of the code:
"Optional"
      {
      "Optional Properties of $class object"
      "This might take a few seconds ..."
      ($schema.FindClass("$class")).OptionalProperties |
      Format-Table -Property Name, Syntax, IsSingleValued -AutoSize
      }
The following code is the complete GetOptional function:
Function GetOptional($class)
{
 GetAdSchema -action "Optional"  -class $class
} #end GetOptional
The final helper function is FindClasses, which calls the GetADSchema function and passes two values. The first is the action Find, and the second is the class to locate. The purpose of this function is to help the user identify classes in the AD DS schema that might merit further exploration.
Suppose you were interested in working with e-mail. In that case, you would want to see which classes relate to e-mail, then which properties exist for an applicable class. As Figure 6 shows, first you run the script with the –action f and –class mail parameters. This returns all classes that have the string "mail" contained somewhere in the name. After you locate a class you are interested in, you explore the properties of the class by choosing the action m (for mandatory) and specifying the –c (for class) parameter, followed by the exact name of the class.
Figure 6 Finding a class in the AD DS schema
When the FindClasses function calls the GetADSchema, the FindAllClasses method is called and the resulting collection of ActiveDirectorySchemaClass classes is sent over the pipeline. The Where-Object uses a regular expression pattern match to look for classes that match the value stored in the $class variable. Here's the code:
"Find"
    {
    $schema.FindAllClasses() |
    Where-Object { $_.name -match "$class" } |
    Select-Object -property name
And this is the FindClasses function:
Function FindClasses($class)
{
 GetAdSchema -action "Find" -class $class
} #end FindClasses
After all the helper functions are defined, you arrive at the entry point to the script. The entry point does only one thing—it examines the command line and determines which function to call. The first thing that is checked is the presence of the –help parameter. If the script is run with –h or –help, the script will call the GetHelp function to display help and then exit, as follows:
if($help) { GetHelp ; Exit }
Because the –help parameter is looked for first, its presence on the command line trumps everything else. If the –help parameter was not used, the script must evaluate the value supplied for the –action parameter—which is the default parameter. Anything typed on the command line will be interpreted as a value for –action if nothing else is used. This gives the added advantage of allowing the capture of bogus input to the script.
The Switch statement is a natural tool for evaluating the values supplied for –action. Four actions are defined, each of which call the appropriate function. The fifth condition that is defined is the default action, and it displays a message that the action is not permitted, whereupon it calls the GetHelp function.
One thing to keep in mind about the Switch statement is that it could find multiple matches. For this reason, the Exit statement is used after each call to the functions. The Switch statement code is shown here:
Switch ($action)
{
 "L" {GetAllClasses ; Exit}
 "M" {GetMandatory($class) ; Exit}
 "O" {GetOptional($class) ; Exit}
 "F" {FindClasses($class) ; Exit}
 DEFAULT { "$action is not a valid action." ; GetHelp ; Exit}
}
Well, that's about it folks. There is no Active Directory Browser, other than the ADSI Scriptomatic. But the BrowseActiveDirectorySchema.ps1 script should serve you well for the time being. For more Active Directory scripting fun, visit the Active Directory scripting hub. And you know we're open every moment of the year on the Script Center.

Ed Wilson is a Senior Consultant at Microsoft and a well-known scripting expert. He is a Microsoft Certified Trainer who delivers a popular Windows PowerShell workshop to Microsoft Premier customers worldwide. He has written eight books including several on Windows scripting, and he has contributed to almost a dozen other books. Ed holds more than 20 industry certifications.

Craig Liebendorfer is a wordsmith and longtime Microsoft Web editor. Craig still can’t believe there’s a job that pays him to work with words every day. One of his favorite things is irreverent humor, so he should fit right in here. He considers his greatest accomplishment in life to be his magnificent daughter.


Security Watch Malware Inspection at the Perimeter
Yuri Diogenes, Mohit Saxena, and Jim Harrison

Contents
How Malware Inspection Works
Configuring TMG Malware Inspection Feature
Setting Policy for Web Access
Update Center
Testing and Monitoring
Conclusion
The new Microsoft Forefront Threat Management Gateway Medium Business Edition (TMG MBE), available as part of Essential Business Server and as a standalone product, provides significant improvements for the Microsoft Firewall service operation. One of the most important features of this new firewall is the capability to inspect HTTP traffic that crosses it for malware. By using this new feature, you will be able to:
  • Improve your ability to protect your internal network against malware coming from the Internet.
  • Keep the perimeter updated with the latest malware signatures by using TMG Update Center.
  • Keep an eye on suspicious traffic via real-time monitoring of log entries and get post-mortem malware statistics using the new set of reports.
This approach lets you mitigate potential threats in HTTP traffic crossing the perimeter, adding a new layer of security to your anti-malware solution. The need for client and server antivirus is not eliminated, but because this inspection is performed before the client workstation receives the data, the malware threat is greatly minimized.
This is especially useful if you have unmanaged computers on your network, such as guest computers. With Forefront TMG, you ensure that if those computers try to download a suspicious file, the file will be blocked even if the unmanaged computer is not running antivirus software.

How Malware Inspection Works
When a user accesses a Web site and tries to download a file, TMG will intercept that traffic and check whether or not the rule that allows that user to access the target site has the malware inspection feature enabled on it. If it does, TMG will start the inspection. (Obviously, if this feature is not enabled, TMG will not scan the traffic.) Figure 1 summarizes the basic flow of the malware inspection as the client is downloading a file:
  1. Client sends an HTTP request to the target Web site to download a file.
  2. Forefront TMG receives the request, determines whether or not any rule matches and, if this rule has malware inspection enabled, scans the request for malware.
  3. If the request is valid and clean, Forefront TMG then sends the request to the destination server.
  4. The destination server receives the request and responds accordingly.
  5. Forefront TMG receives the response from the destination server and processes it first through the proxy engine.
  6. If the rule specifies malware inspection, the proxy engine sends the body of the HTTP request to the Malware Inspection Filter. Responses smaller than 64KB are accumulated in memory. (Based on the Internet statistics, approximately 98% of downloads are smaller than 64KB and can be scanned without disk I/O.) The Malware Inspection Filter accumulates the content, timing the download and inspection, then returns the control to the proxy engine.
  7. If the content is allowed, Forefront TMG sends the original file to the user. If the file is infected and TMG cannot clean the file, TMG sends an HTML page to the user saying that the content was blocked.
Figure 1 Malware inspection flow
During the accumulation (Step 6), TMG improves the user experience using one of the following content delivery methods:
  • HTML progress page, which shows dynamic progress indication and allows the user to download the content from the TMG machine when scanning completes.
  • Standard trickling, in which Forefront TMG initially sends the content at a very slow rate to the client and then, when the scanning is complete, sends the data at the highest possible rate.
  • Fast trickling, in which the parameter you define is a number indicating the trade-off between the user experience (less buffering on Forefront TMG and more scans) and performance (more buffering on Forefront TMG and less scans). This is usually used for media files played by online players (not for media streaming).
To maintain the standards found in other Microsoft security solutions, the malware protection feature in TMG takes advantage of the same Malware Protection Engine (MPE) that is used in Forefront Client Security, Windows Defender, and One Care. Later in this column, we will demonstrate how to keep the definitions up-to-date using Update Center.

Configuring TMG Malware Inspection Feature
To configure malware inspection, you need to enable it first at a global level and then also at the rule level. The first step is to go to the Web Access Policy node and click on Configure Malware Inspection on the Tasks pane, as shown in Figure 2.
Figure 2 Web access settings
When you do, a dialog box appears that lets you enable malware inspection globally, as shown in Figure 3. This dialog box also contains other settings for malware inspection that are pre-populated with default settings. Some of these settings can be controlled at the access rule level while some can only be set globally.
Figure 3 Configuring malware inspection at the global level
The Exceptions tab in Figure 4 allows you to control which sites will be exempted from malware inspection. You can also do this through policy settings, but configuration using this dialog overrides any access rule: if the site is listed here, it will not be inspected for malware even if defined for inspection at the rule level. Such global settings are useful for Web sites hosted by organizations in their DMZ for their internal users or for any other trusted, frequently used site.
Figure 4 Site exceptions
The Inspection Settings tab shown in Figure 5 lets you specify what kind of content will be blocked. You can define the default action for this type of content, such as whether Forefront TMG should attempt to clean infected content and then send it to the end user or whether the content will be completely blocked with no attempt to clean it.
Figure 5 Setting inspection parameters
You can also choose to block suspicious, corrupted, or encrypted files or files that cannot be scanned. And you can set file size limits to preserve bandwidth and prevent users from downloading large files or files that take too long for inspection. Note that these are global settings and are not exposed in the rule.
The Content Delivery tab, shown in Figure 6, allows you to configure the user experience during file download, including specifying whether an end user will receive a trickled response or a progress notification page for files that take longer than 10 seconds (configurable in COM) to download and scan. (No notification occurs if the process takes 10 seconds or less.)
Figure 6 Specifying how content is delivered
You can also select what type of content will receive a progress notification as opposed to a trickled response by clicking on Select Content Types and adding or removing content types from that dialog box. This is a global setting and is not exposed in the rule.
The Storage tab defines the folder where the files will be temporarily accumulated while being scanned and served to the end user. The default folder is the %SystemRoot%\Temp, but this can be changed. Again, this is a global setting and is not exposed in the rule.
Forefront TMG malware inspection performance for content that must be accumulated to disk will improve when this folder is located on a different spindle than the drives used for OS paging or Forefront TMG logging. It is important to exclude this folder from being scanned if you have antivirus software on the TMG Server so that the files are not locked for scanning by the antivirus software while in use by Forefront TMG Server. For best practices on what folders to exclude, please refer to "Considerations When Using Antivirus Software on ISA Server."

Setting Policy for Web Access
The Forefront TMG administrator can configure rules controlling user access to the Internet via either the Web Access Policy or the Firewall Policy. When using the Web Access Policy, the rules explicitly allow only the HTTP and HTTPS protocols, and they let administrators allow or deny user access to Web sites. This can also be achieved through Access Rule under the Firewall Policy, where you can manually allow HTTP and HTTPS access based on source, destination, and user. Note that in an Access Rule, the Malware Inspection option is visible only if the selected protocols include the Web protocols.
To enable malware inspection at the rule level, you can check the box "Inspect content downloaded from Web servers to clients," which you will find on the Web Access Default Rule Properties dialog. Be aware that this inspection applies only to HTTP content that is downloaded by the rule. Malware inspection needs to be enabled globally first before it can be applied at the rule level.

Update Center
Forefront TMG maintains the definitions of known viruses, worms, and other malware. To keep these important definitions up-to-date, Forefront TMG has built in a centralized mechanism called the Update Center that allows the administrator to configure the update frequency as well as the automatic update action. The Update Center can be accessed from the Forefront TMG console.
The Definition Updates panel shows the status of the last update and the time when the last check for new updates was performed. The Tasks pane on the right side is where you can configure update parameters. Figure 7 shows the various options for definition updates that can be accessed by clicking on Configure Update Settings in the Tasks pane.
Figure 7 Definitions update window
By default, Forefront TMG uses the Automatic Updates agent to pull updates from the Microsoft Update service in order to refresh the antimalware definitions. The update agent uses the machine's default update server selection; therefore, if the machine uses updates from Windows Server Update Services (WSUS), the agent will also get updates from WSUS, otherwise it will get them directly from Microsoft Update. These transactions are recorded in the %systemroot%\windowsupdate.log file (as are the regular Windows Updates).
The frequency settings in the Forefront TMG Update Center do not override the Windows Update settings. These settings are completely separate; therefore, Windows downloads software updates while Forefront TMG only downloads signatures.
You can force Forefront TMG to look for updates by clicking on Check for Updates in the Task pane. If new updates are detected and installed, an informational alert will appear in the Alerts tab, as shown in Figure 8. As you can see, the bottom part of the window shows details about the update as well as the versions of the files that were updated.
Figure 8 Malware Inspection Filter alert

Testing and Monitoring
After configuring the inspection and Update Center settings, the next step is to test the functionality. Let's assume that you have a client workstation that accesses the Internet through Forefront TMG and wants to download a file from a Web site. The first step, before the file download starts, is to configure the monitoring in Forefront TMG by filtering the IP address of the client workstation that is trying to access the external resource, as show in Figure 9.
Figure 9 Monitoring the client workstation that is trying to download a file
For this example, suppose the client is sending a HTTP GET to files.fabrikam.com/suspicious.exe. Then Forefront TMG evaluates the request, and after detecting that the file in the request is suspicious, it writes a Failed Connection Attempt event to the log (see Figure 10).
Figure 10 Suspicious file detected
Notice that in the Malware Inspection Result column, this file is categorized as suspicious; the Threat Name column indicates the malware name, and the Threat Level is severe. The details pane for the error shows more information about why the connection attempt failed.
The user who was trying to download this file also experiences this error but receives a more descriptive and friendly message explaining that "access to suspicious files is blocked due to security policy settings" (as shown in Figure 11). This approach allows the user to understand what is happening and why the file he is trying to access was not available.
Figure 11 A more user-friendly error message

Conclusion
Our main goal in this column was to explain how using the malware inspection feature of the Microsoft Forefront TMG can improve edge security. This feature gives you a centralized view of any suspicious traffic that potentially crosses your firewall and allows you to take actions based on the results of the inspection. While this is an important approach toward a more secure environment, there are always concerns about how this may impact the user experience while browsing the Web.
The Microsoft Forefront TMG also addresses these concerns in such a way that the scan process can be more transparent to the end user. For more information, review the Forefront TMG Documentation at the Microsoft Forefront Edge Security TechCenter.

Yuri Diogenes (MCSE+S, MCTS, MCITP, Security+, Network+, CCNP) works for Microsoft as Security Support Engineer on the ISA Server/IAG Team. He also writes articles for ISA Server Team Blog and TechNet Magazine. Yuri is a co-author of the Forefront Community Page called "Tales from the Edge."

Mohit Saxena is the Technical Lead for Microsoft ISA Server Support Team. He leads a team of Support Engineers and Escalation Engineers to provide support for customers on break fix issues, bugs, and design change requests.

Jim Harrison joined the ISA Server Sustained Engineering team as a QFE tester in January 2003. He is now an avid ISA server supporter and systems implementer and the co-author of the Forefront Community Page called "Tales from the Edge."

Yuri, Mohit, and Jim are writing the next Microsoft Press book about Microsoft Forefront Threat Management Gateway (TMG); the book will be available in 2009.


Field Notes IT’s Punctuated Equilibrium
Romi Mahajan

Debates about rates of evolutionary change have raged now for more than a century, and therein lie bountiful lessons for the examination of IT and its future. There are three theories of evolutionary change worth exploring here: catastrophism, uniformitarianism, and punctuated equilibrium.
Catastrophism holds that the earth has been rocked by large-scale, violent events that are very wide in geographical scope and, therefore, affect life on the planet. In contrast, uniformitarianism says that change occurs in a slow, continuous process. Punctuated equilibrium contends that sexually reproducing populations are largely static for most of their evolutionary cycle, but that when change does occur, it occurs in rare and short-lived bursts. Each of the three theories has important lessons for IT organizations.
According to the catastrophism theory, IT will be forced to change fundamentally, though rarely. When change occurs, however, the new paradigm that emerges will fully supplant the modes of thinking and operating that existed before the major event.
For example, in the early 1980s, the advent of the personal computer did away with the old paradigm of mainframes and microcomputers, much like the hypothesized meteor collision did away with dinosaurs 65 million years ago. When Y2K was upon us, computing as we knew it made major changes—people invested massively in mitigation efforts and in disaster recovery plans. Recently, many big thinkers have said that Web 2.0 technologies will change IT forever. A large and vocal camp has argued that outsourcing is creating tectonic changes in IT and that IT pros must learn to change with it. While this theory appears to have validity, it seems to me to be hyperbolic and rather useless with regard to planning and preparation.
Then there are those who think that uniformitarianism is the logic that best describes the evolution of IT. In this scenario, IT changes slowly in a continuous and predictable fashion, and keeping up with change is the best way to be competitive. Change occurs, but in routinized ways. While there is some rhetorical value in this scenario, it ultimately is not very realistic given what we've learned from history.
Personally, I view punctuated equilibrium to be the best analogy for the evolution of IT. What this theory holds for IT is that over a significant period of time, change might appear to be continuous, but in reality there have been periods of intense and frenetic innovation. When this happens, practitioners of IT must rapidly adapt or perish. To see if this theory holds, let's look at the historical record.
After the incredibly fecund late 1960s (the Internet was "invented" in 1969), no real massive changes occurred in the 1970s. The years from 1982 to 1984 saw massive changes in the computer industry, but the 1985-to-1993 era witnessed little that was truly innovative. However, from 1994 to 1995 there were massive disruptions as the client/server computing model entered a phase of maturity, and the emergence of Windows 95 created a fundamental rupture in IT.
We have been in a massive innovation spurt since 2005, and the world of computing is being challenged once again to reinvent itself. Whether it's "computing in the cloud" or the massive changes IT will very soon need to make to reduce our enormous tax on the environment, rapid change is upon us. IT is being forced to support social networks and mobile phones. The need for incredible flexibility is now the order of the day for our industry. Some have called for the demise of packaged software and the growth of software delivered as a service. And if this is true, what indeed does it mean for those folks who support applications? It means change. Big change.
The theory of punctuated equilibrium holds, however, that this spurt of innovation will soon give way to stasis. If this is true, we'll risk becoming less relevant at the exact juncture when the relevance of IT is considered by many to be fundamental. It is up to us, therefore, to control our fate.

Romi Mahajan is Chief Marketing Officer of Ascentium Corporation. Before joining Ascentium, he spent 7+ years at Microsoft where his last role was as Director of Technical Audience and Platform Marketing. Romi is widely published in the areas of technology, politics, economics, and sociology.


Field Notes Media and the Future of Content
Romi Mahajan

The famous communications theorist and professor of English literature Marshall McLuhan wrote that "the medium is the message," meaning that the form in which communications are relayed is more important than the content of the message itself. McLuhan also invented the term "global village," and with these two intellectual strokes, established his importance in the field of intellectual thought. I wonder what McLuhan would think of Unified Communications or books written for Amazon Kindle. I wonder, too, what he would view as the relationship between new technologies and the cultural change that accompanies them.
I bring up the concept of Unified Communications with some humor. Indeed, if the same message is to be transferred over different media, or in other words, if the cost of transferring from medium to medium (for the same message) is trivial, then what becomes of McLuhan's homily? If we are to believe McLuhan, do we then believe that the promise of technology to radically simplify meaning simply isn't true? Or have the proponents of said technologies only really promised the speedy and plentiful transmission of messages with nary a view towards creating meaning?
I believe that technologies do both, but that on the whole they are intellectually neutral. They both facilitate communications of some varieties and open the possibilities of new meanings being created at the juxtaposition of form, content, and context. (They actually reduce communications of some varieties as well—witness the demise of the handwritten letter.)
Consider for a moment e-book readers and their users. Although I don't use e-book readers, I know many people who swear by them—and not just for the convenience. Some people feel that they can absorb content in-depth if they can surround their reading experience with context (such as being able to find online dictionaries or Wikipedia references just a click away). Some authors write books specifically for distribution to the Web and will soon do so for e-books.
In the very near future, books will become rich Internet experiences through which one will be able to both see and hear what is happening in the text. Imagine a story about the havoc created by an explosion if one could both see and hear the explosion on the page. Some readers might be enthralled, while others will bemoan the atrophying of the imagination when literature must appeal to the senses of both sight and sound.
Many people have called for the demise of printed books. There are still others who continue to stock their library shelves and for whom the smell and feel of newly printed books evokes strong emotions. The market for first editions and autographed books remains quite strong, even in our challenging economy. Neither point of view is right or wrong—the specific technology in this case is neutral.
Similarly, look at the current debate around the future of print magazines and newspapers. Whereas there are numerous valid arguments about the deleterious environmental impact of these mediums, most of the debates consist of platitudes about the world moving to digital. That said, I (an IT pro) love printed magazines—they serve a specific need for me. For others, the burden of authorship inherent in print confers a sense of legitimacy to content rendered in that form, whereas the lack thereof (or perceived lack) on the Web creates the opposite effect. On some subjects, people enjoy short, clipped, unedited content (such as blogs), while for other subjects, they prefer longer, edited communication.
Both types have good reason to exist. Technology allows for the distribution of this content, but does not dictate expression; it can magnify it, offer a forum for it, but not enable it.
Ultimately, the future of written content is complex—it can take many forms and involve many types of media. Meaning and form do matter. Technology in this case plays a significant though not essential role in the making of meaning.

Romi Mahajan is Chief Marketing Officer of Ascentium Corporation. Before joining Ascentium, he spent more than seven years at Microsoft, where his last role was as Director of Technical Audience and Platform Marketing. Romi is widely published in the areas of technology, politics, economics, and sociology.


Windows Confidential The Case of the Disappearing Cursor
Raymond Chen

You may have noticed that when you press the PrtSc key to take a screenshot, the cursor blinks briefly. Believe it or not, it actually does this for the same reason that the Windows XP fade-out shutdown screen does not include translucent windows.
These translucent windows, known as layered windows, are not normally included by the BitBlt function when reading pixels from the screen. In order to get them, you have to pass the CAPTUREBLT flag. When you do, you may also notice that the mouse cursor blinks. So why is that?
Once upon a time, the only graphical object that floated "above" the rest of the objects on your screen was the mouse cursor. Originally, the system supported only monochrome mouse cursors. These cursors were supported either in software or, if you happened to have an awesome video card, in hardware.
When you use a video card-supported cursor, the Graphics Device Interface (GDI) gives the video card a bitmap and a mask and says, "OK, this is the mouse cursor. Overlay this on the screen at the coordinates I specify." When the user moves the mouse, the GDI sends the video card updated coordinates, and the video card does the hard work of moving the pixels around the screen.
On the other hand, when you use a software-supported cursor, then the GDI is responsible for saving the pixels under the cursor before drawing it into the frame buffer. The procedure works in the following way: when the user moves the mouse, the GDI manually restores the original pixels, saves all of the pixels under the cursor's new position, and then it draws the cursor at that new position.
When a hardware cursor is employed, the pixels of the mouse cursor do not actually exist in the frame buffer because the hardware is responsible for performing the overlay. Consequently, the BitBlt function is able to just copy pixels from the frame buffer without fear of picking up pixels from the mouse cursor by mistake since those pixels don't exist in the frame buffer to begin with.
Conversely, if a software cursor is being used, then the GDI must remove the mouse cursor from the screen before performing a BitBlt from the screen if the region being copied overlaps the mouse cursor.
When animated cursors are employed, hardware cursors just don't quite cut the mustard, since hardware cursors don't animate. Therefore, animated cursors are implemented in software.
OK, great. But what does this have to do with CAPTUREBLT? Hang on, I'm getting there.
In Windows 2000, the composition mechanism that was used for software cursors was generalized so applications could also take advantage of it. These pseudo-cursors ultimately became known as layered windows. Like software cursors, layered windows don't show up when you do a BitBlt from the screen since they do not really exist in the usual sense. The pixels from the cursor and layered windows are composited onto the screen at the last moment.
But what if you want to capture the pixels of a layered window via BitBlt? That's where the new CAPTUREBLT flag comes into play. This captures the pixels after the composition engine has had its say. Since the code that generates the Windows XP dimmed shutdown screen doesn't pass the CAPTUREBLT, layered windows don't appear.
That still doesn't seem to explain why the cursor flickers, though. Actually, it does. The mouse cursor is just another composition object and therefore would be captured by the CAPTUREBLT flag. To prevent this from happening during a screen capture, the composition engine has to hide the cursor, do the CAPTUREBLT, and then re-show the cursor.
By exposing the Windows 2000 composition engine to applications, cursors have lost their special status. We've come full circle. It's now just like the good old days, back when all you had was a frame buffer and a software cursor.

Raymond Chen's Web site, The Old New Thing, and identically titled book (Addison-Wesley, 2007) deal with Windows history, Win32 programming, and misreading newspaper headlines.