June 2009
Application Compatibility: Inside the Application Compatibility Toolkit 5.5
The Application Compatibility Toolkit (ACT) lets users know whether applications are compatible with a new version of Windows before deployment and can also help them determine how an operating system update will impact these applications. By taking a look under the hood of the Application Compatibility Toolkit, readers are better able to evaluate the potential of each tool and invest in them appropriately. Chris Corio and Chris Jackson
Application Compatibility: Planning Your Application Compatibility Project
One of the biggest obstacles that hinder operating system upgrades is application compatibility, so it makes sense to design a project that will reveal the applications you have and figure out what will work or how to fix things ahead of time. This article describes what you have to consider to implement a successful project. Chris Jackson
Group Policy: Automating Group Policy Management with Windows PowerShell
Group Policy is used in almost every environment and relied on by many to secure and their Windows environment. Surprisingly few have introduced automation into the process, however. This article explains how to take advantage of the GPMC APIs and Windows PowerShell to automate management of Group Policy in your organization. Darren Mar-Elia
Active Directory: Using Catch-All Subnets in Active Directory
In an ideal world, users are directed to the appropriate domain controller for Active Directory authentication, but this is not necessarily what happens in most organizations due to IP subnet information not being properly defined in Active Directory. This article presents a solution to ensure users locate the appropriate DC for authentication—a catch-all subnet to catch the authentication from clients on subnets are not defined in Active Directory. John Policelli
Web Infrastructure: Providing Scalability for ASP.NET Applications
Although Web technology provides an elegant architecture of Web farms and load balancers for ASP.NET applications, data storage technologies have not kept pace, so the applications are unable to scale effectively. The solution is an in-memory storage mechanism that can grow linearly—a distributed cache, as this article discusses. Iqbal Khan
Columns
From the Editor: It’s All Gotta Work: | Toolbox: New Products for IT Pros: Greg Steen highlights new tools and utilities for the IT Professional.Greg Steen | SQL Q & A: Removing Index Fragmentation, Synchronizing vs. Synchronized, and More: Paul Randal answers your SQL questions: How does removing index fragmentation affect statistics? Does data compression interfere with backups? Can you explain database mirroring states? And more!Paul S. Randal
Utility Spotlight: PendMoves and MoveFile: You can’t move, rename or delete a file that’s in use—but sometimes you have to. Luckily, two free utilities called PendMoves and MoveFile let you handle these tasks with ease.Lance Whitney | Inside SharePoint: Creating an External Storage Solution for SharePoint: Storing unstructured data in SharePoint content databases is not always ideal. This article discusses the advantages and disadvantages of using an external storage solution; extending the SharePoint storage architecture using the ISPExternalBinaryProvider API in managed and unmanaged code; and how to avoid data loss during lazy garbage collection.Pav Cherny | Windows PowerShell: Automating User Provisioning, Part 4: Don Jones concludes his four-part series on automating user provision with Windows PowerShell. In this final edition, Don discusses adding users to domain groups and modifying directory attributes.Don Jones
Geek of all Trades: The Benefits of Big, Cheap Disks: Windows Server Backup is a quick, easy, and inexpensive way to back up computers running the Windows Server 2008. Greg Shields explains some of the design decisions that went into this tool, an why it may be useful in small environments.Greg Shields | Hey, Scripting Guy!: Creating A Self-Documenting Script: In this article, the Microsoft Scripting Guys develop a script that will gather comments from other scripts (providing those comments follow a certain pattern) and then save the comments in a text file.The Microsoft Scripting Guys | Security Watch: Thoughts on Identity, Part 1: In this first part of a two part series, security expert Jesper Johansson delves into the concept of identity - What defines an identity? Who gets to control the information, and how do we ensure it is properly secured?Jesper M. Johansson
Business of IT: Successful Project Management: IT organizations today face challenges whose complexities rival those of advanced software development projects and, to boot, require doing a lot more with substantially less. Certain practices, however, can help to ensure the success of a project.Alessandro Muti | Blog Tales: Monitor the Default Management Pack: It seems almost inevitable that the Default Management Pack will become cluttered over time, and that can build a dependency for this MP on any MP it references. In this column, Jonathan Almquist gives you a sample script and Management Pack that will alert you whenever someone saves an override to the default MP.Jonathan Almquist | The Cable Guy: NAP on the Internet: Network Access Protection (NAP) on the Internet is the extension of Internet Protocol security (IPsec) enforcement to the Internet. It allows roaming computers to validate and correct their health state, remain healthy, and mitigate security risks. NAP on the Internet helps fill the security gap that exists for mobile computers that are only evaluated for system health when connected to the intranet.Joseph Davies
Field Notes: Building Strategic Vision into IT: Rather than fighting against the introduction of consumer technologies into IT organizations, IT pros should consider the ways they themselves use these technologies, and how this knowledge can be translated into a business resource.Romi Mahajan | Windows Confidential: When Windows Audio Goes Haywire: Raymond Chen discusses how to handle unexpected trouble with the Windows Audio Engine (AudioDG).Raymond Chen


Application Compatibility
Inside the Application Compatibility Toolkit 5.5
Chris Corio and Chris Jackson
At a Glance:
  • Application Compatibility Manager and the Agent Framework
  • Compatibility Evaluators
  • Storing and analyzing the data
  • The Standard User Analyzer
  • Internet Explorer Compatibility Test Tool

Contents
Application Compatibility Manager
Compatibility Evaluators
ACT Database
Analysis
Online Sync
Standard User Analyzer
Internet Explorer Compatibility Test Tool
Conclusion
The Application Compatibility Toolkit (ACT) helps you manage the process to determine whether your applications are compatible with a new version of Windows before you deploy them. It can also help you determine how an operating system update will impact these applications. There have been a number of articles and white papers written on how to use the ACT 5.5. What we want to do in this article is go under the hood and explore how each of the tools in this toolkit works. We'll start with the tool most people associate with the toolkit: the Application Compatibility Manager.

Application Compatibility Manager
The Application Compatibility Manager provides a framework for discovering information about your production environment, organizing your analysis of this data, and helping to drive the testing process. Let's dig into this tool to understand how it works.
Evaluator deployment packages (MSIs) You deploy ACT 5.5 data collection packages as MSIs, but if you dissect the MSI, you'll find that it doesn't actually do much work. Instead, it extracts an executable file that performs the setup. Figure 1 shows the process tree for a typical run of the setup process.
Figure 1 Agent Framework Installation Process Tree
The third instance of msiexec.exe, shown in red, is an interesting one: it's a call to uninstall the original MSI. Essentially, the MSI extracts an .exe (shown as a .tmp file), and then uninstalls the outer MSI. Later, a second MSI (called by msiexec) is installed. In one test, the deployment MSI finished installing at 1:19:46, and it finished uninstalling at 1:19:50. The MSI appears installed for only 4 seconds! Consequently, when deploying a data collection package, don't look for the installation of the MSI to determine if you ought to install it to a target computer, because you're not likely to find it. You need to look for other evidence that you have installed agents to each workstation.
For example, while the MSI you deploy will have a new generated product code each time, the inner MSI will always have the same product code (DC93B45B-D4F5-4FFE-9B47-042BD6FA8CC5), and this can be used as evidence that an ACT data collection package is installed (although you won't be certain which one, as all ACT data collection package have this footprint). Note that we do not recommend uninstalling a data collection package using this MSI entry; we discuss the recommended way to uninstall an ACT data collection package next.
Afsetup.exe performs the bulk of the work, and you'll want to be aware of this binary in the event you have any issues with your agent deployment. You can uninstall the agent framework using the following command:
%program files%\Microsoft Agent Framework\Agent Framework\afsetup.exe /uninstall
Agent Framework The ACT 5.5 Agent Framework provides a structure for gathering data from a large collection of computers in an enterprise. A design goal of agents is to be unnoticed and suitable for deployment into a production environment. Consequently, the work of the individual agents is heavily performance-optimized (which you'll notice in some of the design decisions made in the agents).
The framework itself is straightforward. When the Agent Framework is installed, you can find its files in %program files%\Microsoft Agent Framework. It runs as a service (actdcsvc.exe) that allows the scheduling and launching of agents. The XML found in the Agent Framework\Data subdirectory configures the agents to launch according to a specified schedule. You can find the agent executable files under Agent Framework\Agents.
Agents are nothing more than executable files—the Agent Framework service simply manages the schedule and launches the appropriate executable files as specified in the installation and configuration XML. Figure 2 shows an example of the process tree for the first 5 minutes of a typical Agent Framework install.
Figure 2 Agent Framework Process Tree
What we have here is a straightforward service that simply schedules and executes agents. The agents do the work, so let's explore them.
The ACT 5.5 Inventory Agent, collect.exe, is one of the most significant. Essentially, this agent searches client systems and compiles an inventory of applications. In terms of the data required for planning an application compatibility project (an application name aligning to a vender, version, and eventually a support statement), Inventory Agent consistently performs extremely well compared to other software inventory tools.
What's particularly helpful is not just how well the agent scours the system looking for evidence (detailed below), but how well it then cleans up that data before sending it along. The tool creates a series of "buckets" and then consolidates all of the evidence found in the searches detailed below, flattening, removing duplicates, and categorizing it. The agent looks for evidence in a number of locations to find applications.
The MSI Database search uses the MsiEnumComponents API to enumerate all applications installed using Windows Installer.
The add/remove programs search opens both HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\Current Version\Uninstall and HKEY_CURRENT_USER\Software\Microsoft\Windows\Current Version\Uninstall and collects data from each subkey found here.
The Windows Shell search works by enumerating the contents of HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders, as well as HKEY_USERS\<…>\Software\Microsoft\Windows\CurrentVersion\Explorer\Shell Folders for each user with a profile on the computer. The agent then searches for all of the files and links in these directories or their subdirectories (specifically excluding desktops, which tend to be "parking lots" for executables downloaded from the Web, to reduce noise). This search captures software that, for example, does not register itself in Add/Remove Programs but does add a Start menu shortcut.
The app paths search enumerates all of the subkeys in the HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\App Paths registry key, picking up any applications that have registered themselves to receive a program-specific application path.
The path environment variable search looks for executable files in each directory (but not subdirectories) pointed to in any user's path environment variable. It first looks for executable files in paths returned from a call to ExpandEnvironmentStrings (which returns both system environment variables and the current user's environment variables) plus the contents of HKEY_USERS\<…>\Environment for all other users. This enables the inventory collector to find applications (typically command-line applications) installed by simply placing the file in the file system and then appending to the path environment variable.
The file-extension handlers search looks at each registered file extension and captures programs registered to handle those file extensions. It first opens HKEY_LOCAL_MACHINE\Software\Classes and enumerates each entry that begins with a period (.) as these are file extensions. For each entry that's found, the agent enumerates the program IDs and locates the shell\open\command subkeys of each program ID to find executable files. It then does the same for each user, enumerating the contents of HKEY_USERS\<…>\Software\Microsoft\Windows\CurrentVersion\Explorer\FileExts. This allows the agent to find software where the installation simply dropped binaries and then registered itself as a handler for a file extension.
The registry run/runonce search looks at the following registry keys to locate executable files that may represent applications:
  • HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\Run
  • HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVersion\RunOnce
  • HKEY_LOCAL_MACHINE\Software\Microsoft\Windows\CurrentVerison\RunOnceEx
Examining these registry keys allows the agent to find software that runs with each startup but that may not leave other evidence for the collector to find.
The Service Control Manager search queries the Service Control Manager (using the EnumServiceStatus API) and then queries each service for details (using the QueryServiceConfig API). This allows the agent to find software whose installer manually places executable files on the file system and sets the registry keys to register the service.
The Windows Components search queries the optional components installed for Windows, to capture these applications as well. It does this by querying sysocmgr.inf (in %windir%\inf) and enumerating the optional components selected. This allows the agent to find software that is likely not to exist in a standard installation of Windows.

Compatibility Evaluators
While knowing what software you have installed on computers across the enterprise is, typically, the most critical data for running an application compatibility project, the Agent Framework also lets you gather additional data from the computers with agents installed. The compatibility evaluator agents provided with ACT are performance-optimized for production use, and consequently collect a limited amount of data. Review the data collected and run a proof-of-concept to measure the value of the data before investing heavily in a massive compatibility evaluator deployment.
UAC Compatibility Evaluator The UAC Compatibility Evaluator (uacce.exe) installs a custom shim database (uacce.sdb) and custom shims (uacdetct.dll) in order to determine whether an application can run as a standard user (or as a protected administrator) on Windows Vista or Windows 7. (Shims are small pieces of application code that are inserted between the application and Windows, which are typically used in Windows to resolve application compatibility issues but here are used to detect them.) It applies these shims to processes running on the system by setting a compatibility layer for explorer.exe; it sets the __COMPAT_LAYER environment variable to include UACCEDetection, a layer defined in the custom SDB. Because child processes inherit layers, any process created using Explorer will have these shims applied.
What do the shims do? The FileOperations shim intercepts calls to the _lcreat, CopyFile, CopyFileEx, CreateFile, DeleteFile, MoveFile, MoveFileEx, MoveFileWithProgress, ReplaceFile, LZOpenFile, EncryptFile, DecryptFile, and DuplicateEncryptionInfoFile APIs. For each call, it checks to see if the ACL restricts access or if the file lives in a special path (in case you have loosened the ACLs).
The ACL check is determined by taking the existing ACL, removing "administrator" ACEs and privileges, and then calling AccessCheck on the resource to determine if it succeeds for a non-administrator user. The special path check looks to see if the path is in the root directory, %program files% directory, %system% directory, or %windows% directory. Any failures are recorded.
The RegistryOperations shim intercepts calls to the RegCreateKey, RegCreateKeyEx, RegDeleteKey, RegDeleteKeyEx, RegDeleteValue, RegOpenCurrentUser, RegOpenKey, RegOpenKeyEx, RegCloseKey, RegOpenUserClassesRoot, RegReplaceKey, RegRestoreKey, RegSetValue, RegSetValueEx, and RegUnloadKey APIs. For each call to an intercepted API, the shim checks to see if the ACL would deny access to a standard user. This ACL check is the same one used for file operations—stripping down the token and calling AccessCheck. Failures are recorded.
The ProfileOperations shim intercepts calls to the WritePrivateProfileSection, WritePrivateProfileString, WritePrivateProfileStruct, WriteProfileSection, and WriteProfileString APIs. For each call to these APIs, the shim checks to see if the ini file is mapped using an ini file mapping. If not, the shim checks the permissions on the file, as indicated in the FileOperations shim (since, if not mapped, an ini file is just another file), and failures are recorded.
The RestrictedNamespace shim intercepts calls to the CreateFileMapping API. If the name of the object includes either the Global\ or Session\ namespace, then a failure is recorded.
The ElevatedRunLevel shim intercepts calls to the CreateProcess, CreateProcessAsUser, CreateProcessWithLogon, and CreateProcessWithToken APIs. All intercepted APIs check the target executable to determine if the installer detection feature of Windows will trigger an elevation event (which returns ERROR_ELEVATION_REQUIRED when using the CreateProcess APIs). Furthermore, the shim checks the target identity for APIs that change the user credentials to determine if they are administrator accounts.
If you look closely at how detection works, you'll see that it is assumed that you are running elevated while the agents are working. Since the agents were designed for use on production systems, the assumption is that the application is already working. And if it's working and would have a problem with UAC, then you're probably running as an administrator!
Windows Vista/Windows 7 Compatibility Evaluator The Windows Vista/Windows 7 Compatibility Evaluator is actually split into two separate agent executables, since one unit of work can be detected statically, while the other is monitored and analyzed at run time.
The GINA Session 0 agent (ginasession0.exe) performs a straightforward registry check. It looks at HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Winlogon\GinaDLL to determine if you have installed a custom GINA. It also enumerates services, looking at the type value in the registry to see if the 0x100 bit is set (the SERVICE_INTERACTIVE_PROCESS flag). Any failures are recorded.
Much like the UAC Compatibility Evaluator, the Deprecation Agent (dep.exe) installs a custom shim database (dep.sdb, or win2kagents.sdb on Windows 2000) and custom shims (depdetct.dll), setting the __COMPAT_LAYER environment variable for Explorer.exe to include DeprecationAgentLayer, which ensures that any processes launched through Explorer are shimmed. (Note that the shim database also excludes specific binaries from the shim that are known to fail if they detect that they have been shimmed.)
What are the deprecation agents looking for? Let's examine the shims it uses.
The DllLoadOperations shim intercepts LoadLibrary and LoadLibraryEx and compares the library name passed to the list of deprecated DLLs. This list is stored in DepManifest.csv, which you can find in %program files%\Microsoft Agent Framework\Agent Framework\Agents\DEP. The entries where the first column is DllType are used.
The ExeLoadOperations shim intercepts the CreateProcess, CreateProcessAsUser, CreateProcessWithLogon, CreateProcessWithToken, ShellExecute, ShellExecuteEx, and WinExec APIs, comparing the executable name passed to the list of deprecated EXEs, also stored in DepManifest.csv where the first column is ExeType.
The RegistryLoadOperations shim intercepts the RegCreateKey, RegCreateKeyEx, RegCloseKey, RegDeleteKey, RegDeleteKeyEx, RegDeleteValue, RegOpenCurrentUser, RegOpenKey, RegOpenKeyEx, RegOpenUserClassesRoot, RegReplaceKey, RegRestoreKey, RegSetValue, RegSetValueEx, and RegUnloadKey APIs, comparing the registry key passed to the list stored (again) in DepManifest.csv—this time with a first column of RegType.
The ApiLoadOperations shim directly shims specific APIs that have been deprecated, noting that an application called it. For Windows Vista, there is only one: StiCreateInstanceA in sti.dll. (And there's even a story behind this; see http://blogs.msdn.com/tomarcher/archive/2006/03/22/windows-vista-sti-and-a-story-about-customer-service.aspx.)
One thing you'll notice about this agent is that it depends on running down level: the fact that you called something is recorded only if you succeed in calling it.
Update Compatibility Evaluator The Update Compatibility Evaluator is useful in determining which Windows binaries a particular application is using, which enables you to target your testing more precisely when Windows Updates are released. The Update Compatibility Evaluator works by installing a kernel-mode driver (fdrtrace.sys) that implements both a File System Filter Driver and registry hooks to record the file system and registry activity generated by applications. It leverages a service (uiaservice.exe) and an executable file (uiaconvert.exe) to interpret the data and align it to applications and access to Windows binaries.
Bucketizer Agent The Bucketizer agent maps the data collected from compatibility evaluator agents against the computer's inventory in order to categorize any issues detected into buckets for each application.
Compressor Agent Just like the name sounds, the purpose of the Compressor agent (compressor.exe) is to compress the files to be uploaded to the specified file share into a CAB file to save network bandwidth.
Uploader Agent The Uploader agent is responsible for uploading the data collected from other agents to the specified file share. The work of this agent is relatively straightforward—copying data collected locally to the indicated location. If the location is inaccessible, the agent waits 5 seconds before retrying. The agent will retry three times before giving up.
An important new feature in ACT 5.5 is that Data Collection Packages can be "tagged"—configured so that all logs uploaded by a particular data collection package will have a particular tag applied to them. This allows you to more effectively address some scenarios, such as thoroughly understanding which software is being used for each group inventoried (previously this was hard to measure for overlapping applications) or for consolidating data from several groups or organizations.
Log Processing Service The Log Processing Service runs actdcsvc.exe as service. If you think that sounds familiar, you're exactly right—this is the same agent framework service discussed earlier! In this case, it runs the following three agents:
  • The Decompressor agent (decompressor.exe) is the companion to the Compressor agent used on the workstations you collect from; it extracts the XML files generated on the client from the CAB files used to upload them. Once extracted, the Listener agent detects the change and can proceed.
  • The Listener agent (listener.exe) monitors the specified directory for changes. It waits for changes to take place in the monitored directory, then processes them by sending an RPC call to the Queuing agent.
  • The Queuer agent (queuer.exe) sets up ports to listen for RPC calls from the Listener agent. When it receives these calls, it processes the file and uploads them to the database. The Queuer agent moves completed XML files from the root or uncompressed folder to the Processed folder when it succeeds. If it fails, it moves the files to the Failed folder. Quite often, if a log fails to process, you can simply move it back to the root directory of the monitored share to try again, and often this will succeed.
A question that comes up quite frequently is whether you need to save these processed files. Since the data is already in the database, you technically do not need to save them, and most people can safely delete them. However, these files can come in handy to rebuild the database or to consolidate two databases, so it is worth considering your future need for regenerating the database before deleting this data.

ACT Database
The Application Compatibility Manager may be where most of the magic happens, but the ACT database is where all the data lives. There are a number of tables in the ACT database. While the schema is undocumented, it is very straightforward ( but, alas, subject to changes with almost every new release of the ACT ). Important tables include Applications, Machines, Devices, and Issues, and they are used to organize all of the information returned by the various agents and processed by the Log Processing service.
Applications are tied to data with a unique Application ID, generated using the name, version, vendor, and language (NVVL) of the application.
You can host the ACT database on Microsoft SQL Server 2005 or later, including the Express versions. (SQL Server 2000 is no longer supported.) The Application Compatibility Manager creates the database during the initial configuration phase of ACT's Enterprise Configuration. The script used to create the database resides in %allusersprofile%\Microsoft\Application Compatibility Toolkit 5\CreateDB.sql.

A key aspect of the ACM is the ability to create reports. You create reports by filtering the ACT database using a query builder control in ACM, which is shown when you click Toggle Filter. You can filter based on criteria related to entities in the ACT database and link these clauses to create AND and OR relationships. You can save your report as an .adq file—an .xml file that defines the filters used. The .adq file is a portable format that neither contains data nor depends on a particular set of data.

Online Sync
When performing an online sync, ACT uses a public Web service. In previous versions of ACT, unique Application IDs for all of your software was sent, but you could hide your compatibility assessment from the community if you wanted. With ACT 5.5, when you decide not to share information about an application, ACT never sends the Application ID (which also means you don't receive data in return).
The community data comes from the Microsoft Compatibility Exchange database and includes logo certification data and community data (other users' votes). A separate system today powers windows.com/compatibility, and this data adds the outcome of manual research into the compatibility state of commercially available software; ACT 5.5 will be the first release of ACT to incorporate this data , which provides far more information about the known state of compatibility than was previously available.

Standard User Analyzer
The Standard User Analyzer (SUA) consists of several binaries to monitor and analyze application execution and to communicate concerns that arise during application testing. The main SUA UI allows you to target an application to analyze and select the launch conditions for this application. SUA will then employ Application Verifier as its underlying platform to monitor applications. After monitoring the application's execution, SUA presents the acquired data in the SUA UI where the user can work to understand the data and even create mitigation packages that plug into the Windows application compatibility infrastructure.
Application Verifier Application Verifier is a tool that offers developers a way to monitor application execution. SUA uses Application Verifier to track application behavior that would require administrator privileges by hooking specific APIs and watching their usage.
Application Verifier implements a plug-in model (which, as of this writing, is not yet documented for public use) where specific dlls can be registered and employed to monitor targeted applications. (Plug-ins are configured using the registry key HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\{ApplicationVerifierGlobalSettings}.) The dll that SUA uses to monitor applications is vfLuaPriv2.dll, which is installed into the %windir%\System32 directory during installation. The vfLuaPriv2.dll binary supersedes vfLuaPriv.dll installed by Application Verifier.
When SUA launches an application for testing, it initially checks to ensure Application Verifier is installed on the system and prompts the user to install it if necessary. Next, SUA uses the Application Verifier command line interface to register an application; for example, if notepad.exe is targeted for monitoring, the command line SUA calls would be the following:
appverif.exe -enable luapriv -for "notepad.exe"
Application Verifier matches the luapriv string with the name specified in vfLuaPriv2.dll.
When SUA configures an application, it also tells Application Verifier to monitor any child processes created by that application. Application Verifier creates a separate log file for each process, but SUA combines these log files to create a single, consolidated report. SUA can also save the collection of logs in a .cab file for viewing on another computer.
Standard User Analyzer UI SUA can test an application in one of three different contexts: launching the application with the current user's token, with the current user's token disabling file and registry virtualization, or with elevated permissions. In each case, SUA will launch an elevated process, SUAnalyzerSrv.exe, since Application Verifier requires elevated permissions to configure. Figure 3 shows a typical process tree when testing notepad.exe.
Figure 3 Testing Notepad.exe
SUAnalyzerSrv.exe and AppVerif.exe will always run with administrator privileges in order to prepare the environment for the target application to be tested. SUA will then launch notepad.exe under the specified security context and wait for the application and all child processes to finish executing. SUA then aggregates and processes the Application Verifier logs and displays the results in the UI.
The SUA results are broken down into categories of issues including File, Registry, INI, Token, Privilege, Name Space, Other Objects, and Process. These categories correspond to a tab at the top of the SUA UI and allow you to focus on different issues as you test the application.
The File and Registry tabs show you file system or registry operations that would require Administrator privileges. The luapriv Application Verifier plug-in monitors the behavior of relevant APIs such as CreateFile or CreateRegistryKey for cases where their use would require Administrator privileges. In some cases, the UAC File and Registry Redirection feature would allow these operations to succeed on Windows Vista.
Each tab shown in the UI generally results from luapriv monitoring the behavior of a set of APIs. For example, the INI tab focuses on finding issues related to the use of the WriteProfile APIs, originally used in 16-bit applications. A discussion of each tab is available in the SUAnalyzer.rtf documentation included as part of the SUA installation. You can expand the events under each tab to view detailed information about the event, including a stack trace for where in the application the event occurred.
After reviewing an application's test results, SUA allows you to test and create mitigation packages for an application. A mitigation package is an MSI that registers an application compatibility database on a system and can use loosen.exe, a utility that ships with SUA, to adjust file or registry ACLs. The mitigation packages can be deployed throughout your environment.

Internet Explorer Compatibility Test Tool
The Internet Explorer Compatibility Test Tool works by enabling the Internet Explorer Compatibility Evaluator (iece.exe). In ACT 5.0, you could deploy the evaluator as part of a data collection package, but since the collection required that you already have the target browser installed, this was removed for ACT 5.5. (After all, discovering that something doesn't work after it is installed is suboptimal; most people would prefer to discover any issues before they deploy the new browser!)
The Internet Explorer Compatibility Evaluator is fairly simple—the only work it has to do is to modify the registry so that Internet Explorer enables logging. Because the browser itself provides the detection, the fidelity of the data is extremely high. For details on all of the events that Internet Explorer can detect (along with remediation recommendations for each), see Internet Explorer Application Compatibility.
Internet Explorer logs information to the Event Log, and the test tool collects information from the registry and consolidates it. You can analyze the information directly in the tool, or you can upload events to a consolidated ACT database. If you choose to upload them to an ACT database, the Internet Explorer Compatibility Evaluator runs agents just like the agents run in a data collection package, first running collect.exe, then bucketizer.exe, and finally compressor.exe, after which it prompts you for a location to save the CAB file.

Conclusion
Well, that concludes our under-the-covers tour of the Application Compatibility Toolkit 5.5. The product provides a number of useful tools to support your migration to Windows Vista or Windows 7, and by understanding more about what each one does, you'll be better able to evaluate the potential of each tool and invest in them appropriately.

Chris Jackson is the technical lead for the Windows Application Experience SWAT team at Microsoft. He has worked with enterprise customers around the world to help them investigate and mitigate application compatibility issues, as well as providing instructional training about Windows application compatibility for numerous industry events. Chris can be reached at http://blogs.msdn.com/cjacks.

Chris Corio was a member of the Windows Security team at Microsoft for more than five years. His primary focus at Microsoft was application security technologies and management technologies for securing Windows. You can reach Chris at winsecurity@chriscorio.com.


Application Compatibility
Planning Your Application Compatibility Project
Chris Jackson
At a Glance:
  • Defining vision and scope
  • Building a team
  • A project’s three key phases
  • Remediation approaches

Contents
Defining a Vision and Scope: Windows
Defining a Vision and Scope: Application Management
Building a Team
Collect
Analyze
Test
Remediate
Conclusion
My job takes me around the world to work with customers who are in the process of migrating to the latest version of Windows. Often, the biggest obstacle they encounter is application compatibility, which raises the following questions: How much is it going to cost to fix things? How long is it going to take? What do I need to know? How do I accelerate the process if it's taking too long?
An application compatibility project is similar to a software development project: cost and time estimates refine over time as you discover what you are facing. I have seen people propose estimates of time and cost based on the number of applications, but the best this system can offer is an average. While the average is often close, almost nobody actually hits it; most organizations are notably better than average, or disappointingly worse.
So, let's talk about how to go about the planning phases of an application compatibility project—understanding cost estimates early, refining them over time, and minimizing them throughout. I will share with you the tips and tricks our most successful customers have used, hopefully saving you rework in the process. I especially want to maximize the productivity of those participants who aren't directly involved in the process, such as end users testing and business owners prioritizing applications, since you'll need their cooperation to be successful.

Defining a Vision and Scope: Windows
The going can get tough when you're dealing with application compatibility. When you find the pocket of Visual Basic 3.0 applications that are mission critical, broken, and without source code, it's easy to get discouraged. You'll want a motivating rallying cry. You'll want overarching goals to guide you in making hard, strategic decisions.
For example, if you are fixing an application that requires admin rights, and a major goal of upgrading Windows is to enable running as a standard user, than applying the RunAsAdmin shim is not a good mitigation. If your primary goal is to make finding, organizing, and using information easier, and you need to fix a broken application using the "buy a new one" fix, you'll take into consideration information visualization and shell integration features of the products you consider.
After you write down your vision, define the scope. If your vision includes moving a percentage of desktops to standard users, define that percentage. This may sound obvious, but it is frequently overlooked, and strongly correlated with success.

Defining a Vision and Scope: Application Management
An operating system migration is the perfect time to set goals for application management. You're touching every application in your enterprise—what a great time to think about how you manage your application portfolio.
Review your application management goals from two perspectives:
  • Agility How rapidly can you respond to technology change as a competitive advantage? How good are you at testing software to minimize both risk and time to production? How well do you manage the lifecycle of applications? Do you understand what business problem your software is solving, and is it the optimal software solution? You want to use software deliberately and strategically.
  • Productivity Are you maximizing user productivity? Do you provide a familiar, modern, and consistently high-quality software experience? Do you have quality standards to manage productivity and helpdesk costs? Can your users collaborate using the same software platforms? You want to leverage software to enable business productivity.
The single biggest application management challenge almost everywhere I go is application proliferation. Some organizations have close to 100,000 pieces of software. That's completely unmanageable! Think about it—you could hire a 50-person team, spend only one hour testing each application, and it would still take you an entire calendar year to finish your testing. An application compatibility project is a scale project, and one of the biggest wins is to minimize the scale.
Whatever your greatest issue is, write down how you will improve. You probably won't go from 100,000 applications to 500 this time. You probably won't get rid of all the old VB6 applications with buttons so big they take two hands to push. Nevertheless, you should set objectives, incorporate them into your vision, and quantify them in your scope.

Building a Team
With your vision and scope in place, it's time to assemble the team to do the work. Crucial roles (a person can fill more than one role) include:
  • Project Manager coordinates a team spanning many disciplines and organizations.
  • Business Coordination Lead works with business application owners to obtain data on application priority, locate and coordinate user acceptance testing, and coordinate with pilot users (this role is frequently executed by the project manager).
  • Technical Lead works with developers to identify training gaps, and work with debuggers to resolve complex compatibility issues.
  • Lab Manager ensures that the team and the user base have a comfortable chair and a current OS image on which to test software.
  • Application Research Team determines current support status for third-party software (frequently outsourced).
  • Testing Team runs install and smoke tests to ensure basic compatibility before user testing (frequently outsourced).
  • Mitigation Team resolves compatibility issues that surface during testing.
  • Packaging Team creates installation packages after testing is completed (frequently outsourced).
The entire team will need to work closely with each other and with anyone else involved in the operating system migration. For example, you'll want current images and up-to-date group policies and configurations.
With a vision and scope defined and a team identified, it's time to start planning the work you intend to do. The process can be divided into three phases:
  • Collect What do I have? (Current State)
  • Analyze What do I want? (Desired State)
  • Test and Mitigate What works?

Collect
Before you can plan your work, you must understand what software you have, the current state. The more sophisticated your software management tools, the more visibility you have into your current state. If you already have an application inventory from your software management tool, use it. If not, the free Application Compatibility Toolkit (ACT) from Microsoft can provide an excellent inventory. The tool you use doesn't matter; what does matter is that your tool gives you all the data you need to the answer questions that will come up later in the process. To help you structure your inventory, here's what you need to know:
  • Division or organization Where does the user work? This information helps identify which software solves which business problem, to either drive prioritization or identify redundancy. I've seen this collected in a number of ways, using machine name, IP subnet, and so forth.
  • Role What does the user do? This information helps identify what the software is used for, again to inform prioritization or identify redundancy. This data is typically harder to locate unless it is already encoded someplace, such as in Active Directory.
  • Usage How many users have the software installed? Even better, how much are they using it? It's easier to retire an application if you know nobody has it installed, or that the people who have it installed never use it.
Later, you'll want to help the people testing the application with the following data:
  • OS/Patch Level You need details of the user configuration where the software is (presumably) working so the tester can compare the working case to the broken case when things go wrong.
  • Subject Matter Experts You want your tests to represent real user scenarios, for which you'll need real users. If you haven't identified SMEs, perhaps the usage data can help identify candidates.
  • Other Applications Installed Sometimes, application issues are caused by conflicts between applications, which you'll want to be able to identify.
Finally, you'd like to have data in your software inventory to support the actual deployment. If you are planning to deploy by role (see the sidebar "Deploying by Role" to learn why this is a good idea), then tagging the applications by role helps guide you to test those applications first. Similarly, if you are planning to deploy by division or geography, you'll want to start testing the applications used by the people you'll be deploying to first.
Do you completely understand your current state? Great—skip to the next section. If not, let's choose a tool to get you there.
You may need raw numbers early to build the business case for the application compatibility effort. Microsoft offers an agentless tool to help you get some rough numbers, the Microsoft Assessment and Planning Toolkit. While typically not sufficient to drive the entire project, this tool does provide some indication of the amount of effort you'll need to invest.
Deploying by Role
Our most successful customers deploy Windows Vista by role. This works particularly well when you have structured task worker roles. Roles have the greatest commonality in software usage than any other easily distinguishable division. You can start with a structured task worker role, such as the call center. This group may have a relatively small number of applications, perhaps a dozen. You can test all of these applications, finish, and be ready to deploy this role. You can brag to management, and move on to the next role. Team morale is high because of the success, management is happy because progress is visible, and the team gets practice doing the work on manageable numbers before they tackle roles with lots of software, such as information worker roles.
To obtain detailed data, you'll need an agent. Collecting an inventory is non-trivial as there are many ways to install applications (MSI, xcopy, setup.exe, and so forth). You want to find all of the applications that matter to you. Most tools surface rich client applications, but what about the Web applications, ActiveX controls, and Microsoft Office applications? I haven't found a tool that surfaces everything.
If you don't already have an application inventory, the Application Compatibility Toolkit (ACT) is best of breed for surfacing desktop applications in a way that matters to you: giving you an application and a version that you can map up to a support statement.
By the way, with regard to ACT and its inventory, I want to clear up one misconception: the idea that the Compatibility Evaluators are a substitute for testing. Weigh the cost of this data against its value when deciding how much to collect. You will not eliminate the need to test by using compatibility evaluators. Evaluators are tuned for performance (so they can be run in a production environment), so they're explicitly not designed to catch every possible problem (this would slow down users too much, were it even possible).
Compatibility evaluators will effectively report deprecations (features that have been removed from the operating system, such as a GINA). The Internet Explorer compatibility evaluator is fantastic, except it only runs on Internet Explorer 7 and above (and, since you're not likely to deploy before you test, that pretty much limits its use to lab or pilot machines). Because the UAC evaluator doesn't catch much that file and registry virtualization doesn't automatically fix, it's a rather coarse predictor of additional bugs running as a standard user. In summary, the things it catches are all absolutely compatibility bugs that you should address, but because of the limited scope, it's a poor predictor of if and how badly an application as a whole is broken.
We weigh this against the costs. With sophisticated software deployment systems in place, it is easier and cheaper to deploy agents. I also have to price out a server to support the data collection, as well as the time to collect the data. With an average of 17 seconds to process each log, I could do a collection of 1,000 machines for 3 days, uploading every 8 hours, and process this data in under 2 days. But, if I wanted to collect from every machine in a 200,000 seat enterprise for 30 days, uploading every 8 hours, I'd have to wait close to 10 years (!!!) for that data to process.
If you're collecting inventory anyway, it makes sense to collect compatibility data from a subset of computers, but not to invest relentlessly. I've seen too many massive cost estimates relating to collecting this data.

Analyze
Once you know what you have, you need to determine what you want—your desired state inventory. This requires collaboration between business and IT, and a number of tough choices. As a result, I've seen the process proceed along the lines of what's shown in Figure 1 far too many times, where you only figure out what you want after you've spent a lot of money on things you don't want.
Figure 1 Unwise application analysis—what works is not necessarily what you want.
The problem with deciding what you want based on what works is that you end up keeping (and supporting) redundant applications just because they happen to work, or deciding to eliminate an application after you have already spent money researching it.
It makes more sense to figure out what you want first, and then invest your money researching and testing only those applications that you have determined add value to your business.
To make the application analysis process more productive, we recommend that you set some explicit goals and measure your progress against them. Recommended goals include:
  • Maximum number of applications Set an explicit goal for the number of applications you'd like to support.
  • Managed application tolerance Set a tolerance level for when an application becomes a "managed" application, based on both business priority and number of users.
  • Management level In a decentralized organization, set organization-wide goals for application management, affording business units the autonomy to implement this guidance as appropriate for their business.
  • Commercial software versioning standards It can be prohibitively expensive to always buy the latest version of all of your software, but you incur risk having very old software. Consider setting standards of n (current version) or n-1 (previous version) for your business critical applications
  • Supported platforms Limiting supported platforms helps you manage complexity. While you don't want to be running in place (creating new versions whose only feature is compatibility with the latest platform), upgrading everything becomes prohibitively large and expensive.
  • Application prioritization goals People have very different perspectives on what "business critical" means. You'll want to set a percentage goal or some objective criteria to make this clear
With these goals in mind, it's time to involve folks on the business side of things—the people who know how and why the software is used. For smaller organizations, this can be one-on-one. For larger organizations, you can use SharePoint portals to collect data to inform the analysis process. While you want to simplify, you also want to make sure you can capture data like, "we need to keep the previous seven versions of this tax software working somewhere, by federal law."
One important practice: optimize for the time of people who aren't officially on your team. Business owners typically are helping you with little to no immediate reward.
How do you collect the information that is already known about your applications? Information about commercial software is shared on the Windows Compatibility Center, but you need to match up your list with the data on this site. The Application Compatibility Toolkit 5.5 will automate this matching.
This principle of cutting early and often applies throughout. A high-value resource eliminating an application in 30 seconds is less expensive than a low-value resource spending an hour researching it. Remove obvious noise before you collect data from business owners, and only research applications that business owners helped determine you want to keep around. The preferred approach to researching applications is shown in Figure 2.
Figure 2 Filter down the list of applications early, when it’s inexpensive
How well does this work? With one customer, we took an inventory of around 1,200 applications from 54 different computers. Over a lunch hour, we removed obvious noise according to their business rules. We narrowed down the list to about 450 applications, and could probably narrow it down even more with additional time. That's over 700 unimportant applications removed in an hour—a significant cost savings.
You can now refine your cost estimate based on your desired state. You can further inform that estimate with known compatibility state for commercial software, and perhaps use a static analysis tool to help understand what you expect to work or to have issues.

Test
Next, you need to determine who is going to be involved in the testing process. Considerations for the team include:
  • Internal team makeup Have a strong project manager and a technical expert leading the team internally, ensuring you can coordinate several roles (testers, debuggers, development teams, users, business owners, and so on).
  • Partners involved Many organizations involve partners to assist in the process. Think about where they fit (targeted skill augmentation, staff augmentation, factory approach, and others) and how they'll integrate with your users for business functionality testing
You'll also want to plan the technology you intend to use. Consider the following technologies:
  • Virtual machines Undo disk and snapshot capabilities will save you a lot of time (from, for example, "first run" bugs, bugs that permanently destroy the state of the machine)
  • Terminal Services/Remote Assistance These are very useful for user testing, providing an easy way to give people access to a Windows Vista computer quickly. And Remote Assistance helps with bug reproduction and investigation.
  • Pilot machines Giving users first access to your hotrod new laptops in exchange for testing apps can be very motivating.
Next, map out the testing process. Figure 3 shows a skeleton workflow:
Figure 3 The application testing process
Do everything you can to ensure that nothing obvious breaks before you involve the user. There is nothing more frustrating than finally convincing a reluctant user to come to your lab, only to have the installer blow up in their face.
Likewise, make sure testers don't end up testing something you're unwilling to fix. If support is required, only test supported versions.

Remediate
To have efficient testing, you'll want to be testing with a fix in mind. Debug a failing application until you determine which remediation bucket it fits into; once you have a bucket, stop.
Of course, to do that, testers must know which buckets you're considering, and when. Crisply define your strategy for remediation. Remediation options most organizations consider include:
  • Get a new one. This is extremely likely to work, and offers you vendor support (which probably matters for some of your applications). This tends to be the most expensive approach, with either development or acquisition costs. Typically, this approach is used any time you can afford it!
  • Shim it. This is the cost-saving route—help the application by modifying calls to the operating system before they get there. You can fix applications without access to the source code, or without changing them at all. You incur a minimal amount of additional management overhead (for the shim database), and you can fix a reasonable number of applications this way. The downside is support as most vendors don't support shimmed applications. You can't fix every application using shims. Most people typically consider shims for applications where the vendor is out of business, the software isn't strategic enough to necessitate support, or they just want to buy some time.
  • Change policy. When a particular feature breaks a number of applications, you may want to disable that feature. The advantage is similar to using shims—you don't have to change or even have access to the source code. And the disadvantages are similar as well—lack of support and inability to fix everything. Some people consider this approach for Web applications, where shims aren't an option. Some of the security features can be controlled individually and disabled as a stopgap solution. A common choice is to disable protected mode for the Local Intranet zone (which Internet Explorer 8 does by default). Note, however, that any time you modify the default security of the system, you want to take that decision very seriously. For example, disabling UAC can decimate the business value of the OS migration.
  • Application virtualization. There is a lot of confusion around application virtualization as an application compatibility solution. I have heard it described as a complete separation of the application from the underlying OS, and therefore a complete and foolproof solution. This is emphatically untrue today. With the exception of the file and registry calls, the application still calls the underlying OS, and any compatibility issues outside of the file system or registry remain unfixed. It is great for application-to-application conflicts, but not a generic solution for application to OS conflicts. Support status is unknown but likely not in your favor, as not every company supports software within application virtualization even if it is supported natively on the OS. The typical scenarios where customers use this solution are: when the issues are with the file system and registry, when the issue is caused by a conflict with another application in the core load, or because they like the deployment story behind virtualized applications and it's just good fortune that it also fixes a compatibility issue.
  • Machine virtualization and terminal services. Machine virtualization is your brute force method. You know it's going to work, because you are actually running it on a previous version of the OS, whether on your local machine or on a server somewhere. It almost always puts you in a supported scenario, since you're actually running it on a supported operating system. But, while some say "virtualize it all, migrate today, and fix things later," I tend to be more cautious. There is management overhead, since you're managing potentially double the number of operating systems per user. If you're using local virtualization, then you need machines with the resources (particularly memory) to support two simultaneous operating systems. The user experience today isn't always that great, as most users are perplexed when they see two start buttons (although there are solutions from both Microsoft and partners to improve this). Most of my customers tend to use this as their last resort for application issues. (In fact, many customers set testing thresholds; if the remediation team can't fix a problem within the amount of time it estimated for each application, instead of carrying on potentially forever, it just stops and puts the application into a previous OS environment.)
  • Get rid of it. Don't forget this option! Sometimes it's not worth it to remediate a low-business-value application or a redundant piece of software. Retire it instead.

Conclusion
We have walked through some of the most important considerations for planning an application compatibility project. I have done this planning either in one solid chunk (generating a complete project plan before the project actually begins), and I have worked with customers who plan for each stage after the preceding one is completed. The critical point is to understand what you can be doing at each step to save you time and money later.
Though there is an aspect of engineering craftsmanship in the process, the big challenge with an application compatibility project is managing bulk and motivating people who aren't incented to help you. These tips and guidelines should help.

Chris Jackson is the technical lead for the Windows Application Experience SWAT team at Microsoft. He has worked with enterprise customers around the world to help them investigate and mitigate application compatibility issues, as well as providing instructional training about Windows application compatibility for numerous industry events. Chris can be reached at blogs.msdn.com/cjacks.


Group Policy
Automating Group Policy Management with Windows PowerShell
Darren Mar-Elia

Parts of this article are based on pre-release software and is subject to change.
At a Glance:
  • Using GPMC APIs
  • Creating Group Policy Reports with Windows PowerShell
  • Cmdlets in Windows 7 and Windows Server 2008 R2

Contents
What Does GPMC Offer?
Automating the GPO Lifecycle
Automating Group Policy Reporting
Generating an HTML-Based Report
Generating an XML-Based Report
Reporting on RSoP logging
An Easier Way
Group Policy is a powerful yet complex technology. It is used, to some extent, in almost every environment. And for many who rely on it heavily to secure and lock down their Windows environment, Group Policy is a key part of their infrastructure.
That said, I am always surprised to see how little automation is used for Group Policy management in many IT organizations. When the Group Policy Management Console (GPMC) shipped, Microsoft made a set of APIs and sample scripts available for automating tasks that were performed with that console. There is much that can be done using these APIs, as well as opportunities to automate other aspects of Group Policy management, such as troubleshooting and diagnostic tasks. And, with the advent of Windows PowerShell, some of these tasks have become easier.
Thorbjörn Sjövold discussed using Windows PowerShell for getting at some of the GPMC APIs in his article "Simplify Group Policy Administration with Windows PowerShell." In this article, I want to build on that foundation with some additional automation techniques you can use to further automate management of your Group Policy environment.

What Does GPMC Offer?
The GPMC is focused on operating against the whole Group Policy Object (GPO), and its associated permissions, links, and so on. It does not provide automation or management of the actual settings within the GPO. However, it can be useful to perform automation against the whole GPO as a way of managing the change process within your Group Policy environment. For example, you can use the GPMC APIs to modify links to GPOs. If you have a new GPO that you want to deploy, you can script the creation of the GPO and then you can script the linking process after its settings have been populated. You can also script the changing of a GPO's permissions, in case you want to modify which security groups are targeted by a GPO or who can edit that GPO.
And of course, you can always use the APIs to query information about GPOs, as opposed to just making changes. This includes generating HTML- and XML-based reports for GPO settings as well as Resultant Set of Policy (RSoP) reports against remote workstations and servers to determine if Group Policy was successfully applied.
It's also worth mentioning that when Microsoft released the updated GPMC that shipped with Windows Vista SP1 and Windows Server 2008, there were some updates to the APIs in order to support some of the new features that GPMC and Group Policy in general support. These include the ability to create new GPOs from "Starter GPOs" and to add comments to a GPO. Starter GPOs are like templates—they let you create a set of Administrative Template policy settings that you can then apply to a new GPO, pre-populating some of its settings. I want to start by looking at how you can automate the process of creating, permissioning, and linking a GPO and then show how you can leverage some of these new GPMC features in this automation.

Automating the GPO Lifecycle
To demonstrate how you can automate creating and managing GPOs, I am going to use Windows PowerShell and the GPMC APIs. In my example, I am creating a GPO called "TechNet Marketing Policy." When I create the GPO, I'm going to use a Starter GPO called "User Lockdown Template" as the starting point and add a comment indicating that I created the GPO. I could create the Starter GPO using GPMC APIs, but in this example, I'm going to assume it already exists.
The next step that I want to automate is the permissioning of the GPO. I'm going to permission the GPO so that only users within the "Marketing Users" group will process the policy, and I will add a group called the "GPO Admins" with the permissions to edit the GPO. Finally, I am going to link the GPO to the Marketing OU in my Active Directory domain.
The entire Windows PowerShell script, which I've called gpoCreate.ps1, is shown in Figure 1. I've added the line numbers simply for reference.
Figure 1 The Windows PowerShell gpoCreate.ps1 scriptt
1. $gpmc = New-Object -ComObject GPMgmt.GPM
2. $constants = $gpmc.GetConstants()
3. $domain = $gpmc.GetDomain("cpandl.com",$null,$null)
4. $starter = $domain.GetStarterGPO("{CDFD6B94-BF4E-4D07-8D99-3D416EC7C9A0}")
5. $gpo = $domain.CreateGPOFromStarterGPO($starter)
6. $gpo.DisplayName = "Technet Marketing Policy"
7. $gpo.Description = "Created by Darren for Technet Demo"
8. $permissions = $gpo.GetSecurityInfo()
9. $permissions.RemoveTrustee("Authenticated Users")
10. $applyPermission = $gpmc.CreatePermission("Marketing Users",$constants.permGPOApply,$false)
11. $editPermission = $gpmc.CreatePermission("GPO Admins",$constants.permGPOEdit,$false)
12. $permissions.Add($applyPermission)
13. $permissions.Add($editPermission)
14. $gpo.SetSecurityInfo($permissions)
15. $som = $domain.GetSOM("OU=Marketing,DC=cpandl,DC=com")
16. $som.CreateGPOLink(1,$gpo)
Line 1 is required to get started using the GPMC APIs. You will use this in any GPMC script you write. This line creates an instance of the base GPMC object and assigns it to the $gpmc variable. Line 2 is another commonly used command. The GPMC provides a set of handy constants that are used across the spectrum of tasks to indicate a particular state. You'll see how I use it later in the script, but for now, I'll just assign the constants to the $constants variable.
In Line 3, I need to get a reference to the Active Directory domain that I'll be operating on. In my example, that is a domain called cpandl.com. To do that, I call the GetDomain method on the $gpmc variable. The two $null parameters are optional and allow you to specify a particular domain controller to connect to when you connect to the domain. By leaving these null, I'm essentially choosing the default, which is the PDC emulator DC.
In Line 4, I need to get a reference to my Starter GPO (User Lockdown Template). The GetStarterGPO method only supports calling the Starter GPO by its GUID, so I have to go into the GPMC console to look for that. (I could have scripted it also.) That's the GUID I'm passing to the GetStarterGPO method.
In Line 5, once I have my Starter GPO reference, I use it to create the new GPO, using the CreateGPOfromStarterGPO method that is available on the $domain variable. I assign the newly created GPO to $gpo so I can continue to use it. Note that at this point, the GPO has no name (well, it has the default name of "New Group Policy Object"). So in Line 6, I modify the displayName property on $gpo to give it a new name. Line 7 is where I add a comment to the GPO, by setting the description property on $gpo.
Now that I've got my GPO created, the next set of tasks is to modify the permissions on the GPO. In Line 8, I start off by getting the current list of permissions on my newly created GPO, using the GetSecurityInfo method on the GPO. The approach for modifying permissions on a GPO is to get the list of current permissions on the GPO, add and delete entries from that list as needed, and then re-apply the list to the GPO. To that end, in Line 9, I remove the Authenticated Users default permission from the newly created GPO.
In Lines 10 and 11, I create the two new permissions that I want to add to the GPO. I create those using the $gpmc CreatePermission method, supplying the Trustee (user group) name and the permission that I want the group to have. Note that I'm using the $constants variable to define the permission. The $constants.permGPOApply property grants the "Read and Apply Group Policy" permissions that allow members of a group to process a GPO, while the permGPOEdit property grants the ability for that group to Edit a GPO. The $false parameter at the end of the CreatePermission method call simply says that the permission should not be inherited, which is the default for GPO permissions.
Once the two permissions are created, Lines 12 and 13 add them back to the $permissions list and Line 14 calls the SetSecurityInfo method on the GPO to apply the new list back to the GPO.
The final two lines link the GPO to the Marketing OU. In Line 15, I call the GetSOM method (SOM stands for "scope of management") on the $domain variable to "connect" to the OU. In Line 16, I call CreateGPOLink on the $som object I just created and pass it two parameters. The first parameter indicates the order on the OU that I want the GPO to be linked (an OU can have multiple GPOs linked to it). The "1" stated for the first parameter indicates that I want the GPO to be linked first in the list. The second parameter (in this case, the $gpo variable) is the reference to the GPO I want to link. And now I've successfully created, permissioned, and linked a GPO using automation. The result is shown in Figure 2.
Figure 2 Viewing the Newly Created GPO

Automating Group Policy Reporting
Another aspect of Group Policy management you can automate is the reporting. In this respect, there are at least two types of reports that GPMC delivers. The first is the ability to report on the settings within a GPO. This allows you to generate either an HTML- or XML-based report of the settings that are currently enabled in the GPO, as shown in Figure 3.
Figure 3 Reporting on GPO Settings
The second reporting capability lets you generate Resultant Set of Policy (RSoP) or Group Policy Results reports. There are two types of RSoP reports—logging and planning. A logging report is run against a remote desktop or server, indicating what policies were delivered to that remote system and whether they were successful. The RSoP planning report lets you perform what-if analysis against a particular OU, computer, or user to determine what policies will apply. In both cases—logging and planning—you can generate HTML or XML output using the GPMC APIs and Windows PowerShell.

Generating an HTML-Based Report
To generate a GPO settings report in Windows PowerShell, I start off with two familiar initialization commands, which I used in the previous script:
1. $gpmc = New-Object -ComObject GPMgmt.GPM
2. $constants = $gpmc.GetConstants()
Next, I need to get a reference to the GPO that I want to report on, like so:
3. $domain = $gpmc.GetDomain("cpandl.com",$null,$null)
4. $gpo = domain.GetGPO("{31B2F340-016D-11D2-945F-00C04FB984F9}")
In Line 3 here, I am again connecting to the domain and then in Line 4, I use the GetGPO method on the domain to get a reference to the GPO I want to report on. In this case, I have to pass the GUID of the GPO, which happens to be the "Default Domain Policy."
Next, I need to generate the settings report:
5. $gpo.GenerateReportToFile($constants.ReportHTML,"c:\GPReports\DDPSettings.html")
Here, I am calling the GenerateReportToFile method on the GPO to create the settings report. The first parameter uses the $constants variable to specify an HTML report type. The second parameter points to the path where I want to save the report.

Generating an XML-Based Report
Another way to access the settings data using Windows PowerShell is to take advantage of the built-in XML parsing capabilities in Windows PowerShell and its ability to generate a settings report in XML. So, instead of Line 5 above, I change that to the following:
[xml]$report = ($gpo.GenerateReport($constants.ReportXML)).Result
In this example, I'm using a different method on the GPO, called GenerateReport. This method takes a single parameter, which is the report type. But in this case, I am assigning the output of the method call to a variable called $report and I am preceding that variable name with the Windows PowerShell type accelerator [xml], which tells PowerShell to take the output of the command that I am storing in $report and convert it to an XML document instead of just a bunch of text. However, in order to get the actual XML from GenerateReport, I have to use the Result property on that output, which is what you see at the end of that statement. So the Result property holds the actual XML that I am using to generate my XML document.
Once I have the XML in the $report variable, I can do lots of interesting things with it. For example, consider the following command:
$report.GPO
This returns the "GPO" element within the document that provides overview information about the GPO I am reporting on. Or this command:
$report.GPO.LinksTo
This returns a list of all of the places that this GPO has been linked.
But more interestingly, I can use the structured nature of XML to investigate the actual settings within the GPO from the command-line. For example, since this is the Default Domain Policy GPO, I know it probably contains some security settings related to password policy. By navigating the GPO namespace as represented in the XML file, I can quickly get to those settings, as follows:
$report.GPO.Computer
This returns information about the Computer side of the GPO. If I view the properties on this Computer property, I notice a collection of policy areas under the ExtensionData property, as shown in Figure 4.
Figure 4 Viewing GPO Settings Through XML
In this example, there are two extension areas within the computer side of this GPO: registry and security. So I issue the following command:
$report.GPO.Computer.ExtensionData[0].Extension
This gives me a list of policy areas implemented under the Security policy section. I see two properties called Account and Security Options. Now I enter the following command:
$report.GPO.Computer.ExtensionData[0].Extension.Account
This gives me a list of each of the Account Policy settings for this GPO, as shown in Figure 5.
Figure 5 Viewing Account Policy Settings Within the GPO
Notice that some of the settings do not actually show the value in Figure 5. Each setting is returned as a member of a collection under the Account property. So in order to see, for example, the Minimum Password Length setting, I need to index into the collection, like so:
$report.GPO.Computer.ExtensionData[0].Extension.Account[4]
Once you get the hang of navigating the XML namespace for GPO settings, you can easily get to particular settings and use this capability in conjunction with Windows PowerShell to locate the values of settings within a GPO quite quickly.

Reporting on RSoP logging
Just as with GPO settings, you can generate an XML- or HTML-based report in Windows PowerShell that shows the Group Policy settings that have been applied to a given machine. The approach is a little different, though. Starting again with the two initialization commands, I take the script in a slightly different direction:
1. $gpmc = New-Object -ComObject GPMgmt.GPM
2. $constants = $gpmc.GetConstants()
3. $rsop = $gpmc.GetRSOP($constants.RSOPModeLogging,$null,0)
4. $rsop.LoggingComputer = "xp2"
5. $rsop.LoggingUser = "cpandl\dpmtest"
6. $rsop.CreateQueryResults()
7. $rsop.GenerateReportToFile($constants.ReportHTML,"c:\gpreports\XP2Rsop.html")
After Lines 1 and 2 set up the process, Line 3 creates the $rsop variable by calling the GetRSOP method on $gpmc. In that method call, I indicate that I want to create an RSOP logging and not a planning report. Lines 4 and 5 set properties on that $rsop object to tell it what computer and user I want to collect RSOP data against. Then, the sole purpose of Line 6 is to connect to the computer specified in line 5 and generate the RSoP namespace for this query. Finally, in Line 7, I output the results of the query to an HTML file.
Note that the RSoP object also has a GenerateReport method similar to the GPO settings example. And you can output the RSOP report to an XML document and navigate through it in Windows PowerShell to find out what's happening from a Group Policy perspective on that remote client. For example, say I want to quickly find out if Group Policy processing succeeded for the Computer side of GPO processing. I can use the previous script, but replace Line 7 with this:
[xml]$rsopReport =
  ($rsop.GenerateReport($constants.ReportXML)).Result
This puts my RSoP report into an XML document.
Next, I can navigate into the XML name-space to find out the Client-Side Extension (CSE) status for the computer as follows:
$rsopReport.Rsop.ComputerResults.ExtensionStatus
When I do that, I get a listing that shows me the status on each CSE that was processed by the computer!

An Easier Way
As I've shown so far, there is a lot of powerin leveraging the GPMC APIs to automate Group Policy lifecycle, reporting, and diagnostic tasks. But it does take quite a few steps to get to the desired result. However, there is somerelief and even more coming in the future.
I've created a set of 25 free GPMC Windows PowerShell cmdlets that wrap most of the common GPMC functions into easy-to-use cmdlets. You can download these GPMC cmdlets at www.sdmsoftware.com/freeware. To give you an example of the simplified process, from our first example of creating and then permissioning and linking a GPO, the following script using my GPMC cmdlets would accomplish the same tasks as the script I presented earlier, but with fewer commands:
$gpo = New-SDMgpo "Technet Marketing Policy"
  -FromStarterGPO "User Lockdown Template" –native
$gpo.Description =
  "Darren's Technet Demo GPO" Remove-SDMgpoSecurity
$gpo.DisplayName -Trustee "Authenticated Users"  –PermApply
Add-SDMgpoSecurity $gpo.DisplayName
  -Trustee "Marketing Users" –PermApply
Add-SDMgpoSecurity $gpo.DisplayName
  -Trustee "GPO Admins" –PermEdit
Add-SDMgplink "Technet Marketing Policy"
  -Scope "OU=Marketing,DC=cpandl,DC=com" -Location 1
Even better, Windows 7 and Windows Server 2008 R2 will provide built-in Windows PowerShell cmdlets for GPMC. For example, to create my example GPO from a Starter GPO with a comment, I can issue one Windows PowerShell command, as follows:
new-gpo "Darren's Technet Policy" -starterGPOName
 "User Lockdown Template" -Comment "Darren's Demo"
Microsoft is also adding support for reading and writing a subset of Group Policy settings. Specifically, there will be support for reading and writing registry settings into either native Administrative Template policy or the newer Group Policy Preferences registry extension. For example, to write a new registry value into Group Policy Preferences, I would issue the following command :
Set-GPPrefRegistryValue "Darren's Technet Policy"
  -key 'HKEY_LOCAL_MACHINE\Software\SDM Software'
  -ValueName "Path" -Value "2" -Type String
  -Context Computer -Action Update
Here, the set-GPPrefRegistryValue cmdlet takes a number of parameters to create a registry policy setting within my "Darren's Technet Policy" GPO for the registry value HKEY_LOCAL_MACHINE\Software\SDM Software\Path= REG_SZ 2. The Context parameter tells the policy whether it should be put under the Computer or User side of the GPO, and the Action parameter specifies how to apply the registry value and corresponds to the options within the GPP UI (other options include Replace, Create and Delete).
As of the time of writing this article, Windows 7 and Windows Server 2008 R2 are planned to ship with 25 cmdlets for managing Group Policy. Once these are available, managing Group Policy through Windows PowerShell will be much easier.

Darren Mar-Elia is a Microsoft Group Policy MVP, creator of the popular Group Policy site www.gpoguy.com and coauthor of Microsoft Windows Group Policy Guide (Microsoft Press, 2005). He is also CTO and founder of SDM Software, Inc. Reach him at Darren@gpoguy.com.


Active Directory
Using Catch-All Subnets in Active Directory
John Policelli
At a Glance:
  • The domain controller locator
  • The hub and spoke topology
  • Implementing a catch-all subnet

Contents
How Clients Locate Domain Controllers
The Problem
The Catch-All Subnet
Wrapping Up
In an ideal world, users are directed to the appropriate domain controller (DC) for Active Directory authentication. However, this is not necessarily the case in most organizations due to IP subnet information not being properly defined in Active Directory. As a result, a number of users authenticate against an arbitrary domain controller, which is not optimal for Active Directory authentication.
In this article, I provide a potential solution to ensure users locate the appropriate DC for authentication when IP subnet information is not fully defined in Active Directory.

How Clients Locate Domain Controllers
When a computer attempts to locate a domain controller, a process called the domain controller locator (Locator) is initiated so the appropriate Active Directory domain controller can be located. Locator uses information that is stored in Active Directory and DNS to attempt to find a domain controller with the desired roles and that is located in a site closest to the client.
The Locator uses information that is defined in the Configuration container in the forest root domain, which is replicated to every domain controller in the forest. Site objects, subnet objects, and domain controller server objects are all imperative for the Locator to find the closest domain controller for a client computer. Site objects are used to represent Active Directory sites. Subnet objects are used to represent IP address segments and are associated with the appropriate site object. Domain controller server objects are used to represent domain controllers and are associated with a site object.
Active Directory domain controllers register DNS records that specify the site in which the DC resides. The number of DNS records that each domain controller registers depends on the roles the DC has. For example, a DC that is a Global Catalog server will register an additional DNS record that advertises itself as such. Similarly, a domain controller that holds an Operations Master role will register a DNS record that advertises itself as this.
The process for a client computer to locate a domain controller is as follows:
  1. The Locator is initiated on the client computer as a remote procedure call (RPC) to the local Net Logon service.
  2. The client collects the information that is needed to select a domain controller and passes the information to the Net Logon service.
  3. The Net Logon service on the client computer uses the collected information to build a query to send to DNS to identify the appropriate domain controller.
  4. The Net Logon service on the client computer sends a datagram to the discovered domain controllers.
  5. The directory service intercepts the query and passes it to the Net Logon service on the domain controller.
  6. The Net Logon service on the domain controller looks up the client IP address in its subnet-to-site mapping table by finding the subnet object that most closely matches the client IP address and then returns the following information to the client: the name of the site in which the client is located, or the site that most closely matches the client IP address; the name of the site in which the current domain controller is located; and a bit that indicates whether the found DC is located in the site closest to the client.
  7. The client inspects the information to determine whether it should try to find a better domain controller. The decision is made as follows: If the returned domain controller is in the closest site, the client uses this domain controller; If the client has found a DC in the site in which the DC claims the client is located, the client uses this domain controller; If the DC is not in the closest site, the client updates its site information and sends a new DNS query to find a new DC in the site. If the second query is successful, the new DC is used. If the second query fails, the original DC is used.
  8. If the domain that is being queried by the client is the same as the domain to which the computer is joined, the site in which the computer resides is stored in the registry on the client computer.
  9. After the client locates a DC, the domain controller entry is cached. If the domain controller is not in the optimal site, the client flushes the cache after fifteen minutes and discards the cache entry. It then attempts to find an optimal domain controller in the same site as the client.
In the case where a client computer uses an IP address that is not represented in the subnet-to-site mapping table, the DC returns a NULL site name and the client uses the returned domain controller, which may reside in any Active Directory site.

The Problem
If the domain controller cannot determine the closest site, due to the IP subnet information not being defined in Active Directory, authentication then uses a DC that is not optimal.
Figure 1 illustrates a sample Active Directory site design that uses a hub and spoke topology. Users who log on from a computer that is on one of the subnets that are defined in Active Directory are directed to a domain controller in their closest Active Directory site for authentication. However, users who log on from a computer that is on any other subnet, for example 10.1.4.0/24, are directed to an arbitrary DC.
Figure 1 Sample hub and spoke topology site design
The appropriate fix to address this issue, of course, is to define the additional subnets in Active Directory and associate them with the appropriate site. However, organizations (especially medium and large organizations) often face issues obtaining the information required to add these additional subnets to Active Directory. More specifically, Active Directory administrators usually become aware of the existence of the additional subnets through errors and log files, but do not know which physical office these subnets belong to, preventing them from identifying which site to associate the subnet with.

The Catch-All Subnet
A more practical solution to the issue is to use one or more catch-all subnets in Active Directory. A catch-all subnet is an Active Directory subnet that is used to catch the authentication from clients that are on subnets that are not defined in Active Directory. A catch-all subnet is essentially a super-subnet.
As shown in Figure 1, the 10.1.1.0/24 subnet is defined in Active Directory and associated with the Toronto Active Directory site. Say you notice that a number of clients on the 10.1.4.0/24 and 10.1.5.0/24 subnets are authenticating against Active Directory. You believe these subnets are in the Toronto office based on the prefix (10.1.x.x). You want to ensure that computers on these subnets, as well as any other subnets that use the 10.1.x.x prefix, authenticate against a DC in the Toronto site. So you create a catch-all subnet, which uses the 10.1.0.0/16 prefix, to direct authentication from all subnets you believe to be located in the Toronto office, and you ensure the authentication uses a DC in the Toronto site. The catch-all subnet mapping for this example is shown in red text in Figure 2.
Figure 2 Using a catch-all subnet for hub sites
In most organizations, IP subnet information for remote offices is communicated to Active Directory administrators. The gap usually exists for IP subnets for central offices, hub locations, and data centers. The catch-all subnet can be used to improve authentication in these cases as well. Since the IP subnets for remote offices are fairly static, and you usually have a good grasp on these subnets, you can create a catch-all subnet for all other subnets.
In this example, you can create a catch-all subnet, which uses the 10.0.0.0/8 prefix, to catch authentication from all subnets you believe do not belong to a remote office and ensure the authentication uses a DC in the central Toronto site. The catch-all subnet mapping for this example is shown in red text in Figure 3. And you can even use multiple catch-all subnets, as shown in Figure 4, if you desire.
Figure 3 A catch-all subnet for all locations
Figure 4 Using multiple catch-all subnets
Notice in Figure 3 and 4 that the catch-all subnets use a prefix that overlaps the prefixes used in the remote sites (10.0.0.0/8 covers 10.1.1.0/24 for example). You may be wondering whether authentication from computers in the remote offices will still be directed to a domain controller in the appropriate remote office as a result. The answer is yes. When overlapping IP subnets exist in Active Directory, the IP subnet with the smallest matching subnet mask is used. For example, 10.1.1.0/24 will be used instead of 10.0.0.0/8 if the computer has an IP address of 10.1.1.5/24 subnet. However, 10.1.1.1./32 will be used instead of 10.1.1.0/24 for a computer that has an IP address of 10.1.1.5.

Wrapping Up
Before you go ahead and implement catch-all subnets, you need to be aware that the concept of a catch-all subnet is not suited for all environments. The feasibility of such a solution depends greatly on your IP addressing scheme.
In the examples I've provided here, the IP addressing scheme is fairly straightforward. However, as the IP addressing scheme becomes more complex, the feasibility of using catch-all subnets becomes less likely. For example, if your environment consists of multiple network segments and your IP addressing scheme is not sequential across each location, it will be virtually impossible to create catch-all subnets that will provide any value. As with all solutions, you need to consider the technical, business, and environmental factors specific to your environment when determining the viability of implementing catch-all subnets.
And note that the use of catch-all subnets does not resolve the issue of missing subnets entirely. In fact, if you introduce catch-all subnets in your environment, it is even more important that you explicitly define all known subnets in Active Directory.
The use of catch-all subnets will increase the number of users that locate the closest domain controller. However, your goal should still be to address the source of the issue and ensure you receive the appropriate information for all additions, modifications, and removals of network subnets so you can maintain an up-to-date Active Directory site design that provides efficient authentication for all users.

John Policelli (Microsoft MVP for Directory Services, MCTS, MCSA, ITSM, iNet+, Network+, and A+) is a solutions-focused IT consultant with over a decade of combined success in architecture, security, strategic planning, and disaster recovery planning. John has spent the past nine years focused on Identity and Access Management and providing thought leadership for some of the largest installations of Active Directory in Canada. John maintains a blog at http://policelli.com/blog.


Web Infrastructure
Providing Scalability for ASP.NET Applications
Iqbal Khan
At a Glance:
  • Scalability bottlenecks in ASP.NET applications
  • Session-state storage options
  • Available caching topologies
  • Necessary distributed cache features

Contents
Where the Problems Are
Why These Problems Exist
What's the Answer?
Caching Topologies
Different Choices
The Real World
The popularity of ASP.NET, the Web application framework from Microsoft, continues to grow by leaps and bounds within the developer, enterprise, and IT ranks. There is one area of difficulty, however: scaling ASP.NET applications out of the box is simply not possible.
Scalability has two meanings in this context. First, you need to be able to effectively handle peak user loads because every application goes through peaks and valleys in terms of the number of users logged in at any point in time. When you design the infrastructure, you want to design the application so it can handle the peak loads as efficiently and as fast as nonpeak loads.
Second, you have to be able to increase the total capacity of your system. Today you may have only 5,000 users. Six months, a year down the road, you may have 10,000 or 15,000 or 20,000, and in a few years you could end up with 100,000 users. Being able to grow with the number of users without grinding the application to a halt is what scalability is all about. It means you are able to add more users without negatively impacting the performance in any noticeable way or, if there is any degradation, it should be within an acceptable range.
A typical ASP.NET application is deployed on one or more Web servers that are linked together in a Web farm, with a load balancer that distributes traffic to all the Web servers. In theory, the more Web servers you add, the more requests you should be able to process per second. The architecture of a Web farm is meant to give scalability to ASP.NET. That's the theory; the reality is a bit different.
The problem for ASP.NET applications is that while Web technology provides an elegant architecture of Web farms and load balancers, data storage technologies have not kept up. Certainly you can scale out in a Web application by adding more servers or by increasing the strength of individual servers with more memory and CPU power.
But as you do that, data storage is not able to scale in the same proportions. It does scale, but not as much as the Web application tier. Consequently, anything in your ASP.NET application associated with data storage or data access is a potential scalability bottleneck. More to the point, a database server does not scale for either sessions or applications data.

Where the Problems Are
Let's take a look at the different access or storage actions that take place within an ASP.NET application, starting with session storage. For every user request, a session is read from the storage at the beginning and written back to the storage at the end of the response. At the beginning of the user request, the page has to execute and, for that, it needs the session data. The entire session data, called a "session object," is loaded so that the page, while it executes, can reference that data. The page will read some of that data from the session object. It will put some more data into the session object. This all happens within the ASP.NET process and no session storage trips are being made.
When the page is done executing, some result needs to be sent back to the user. The session is probably updated during this time so that session now has to be saved back to the storage. It will be kept stored until the next user request for the same session and the same process repeats itself.
From a user's perspective, you click on a link; by the time the user sees the page that results from that link, a session has been read and a session has been written back to the storage once. So there are two trips to a session storage that the ASP.NET application has made.
Now you do the math. If you've got 10,000 users all accessing pages at the same time, you will have maybe 1,000 requests per second. Every user will be clicking something every few seconds, so every second you'll have at least 1,000 and probably more requests going to the Web farm.
Let's say 1,000 or more requests are going to the Web farm and every request going to the Web server results in two trips to the session storage. From a Web perspective, that means 2,000 trips to the session storage. You can see how quickly the load can increase. This is one place a scalability bottleneck can happen.
Scalability bottlenecks can also happen while a page is executing and needs to read or write some application data. Let's use airline flight availability as an example. The user clicks on a page to search for flights from one location to another, which can result in multiple read trips to the application database. Then, of course, the user wants to make a flight reservation, which requires some data be put into the database. This data is called "application data," and it is stored in the database; this save operation might make multiple database trips to store multiple data elements.
Thus, you could, in the end, wind up with the number of database trips being 5, 10, 15, or 20 times more than the actual number of user requests. So the stress on the database is that much more, and this can be a major bottleneck.
A third scalability bottleneck comes about if you are using an SOA (service-oriented architecture) environment and your application is making calls to another service layer, which could be within your data center or in a different data center.
The server layer architecture typically involves a server farm and is therefore scalable in the same way that Web application architecture is. But the service layer has the same scalability bottlenecks for the application because it relies on its own database.
So your application has dependencies on other services, which have dependencies on their databases, which in turn have scalability bottlenecks, and the chain is only as strong as its weakest link. If a service does not scale because of its database, your application cannot scale (see Figure 1).
Figure 1 The database becomes a bottleneck as the Web farm grows.
It doesn't really matter whether the database is a mainframe or a relational database. The data access or data storage simply is not able to scale, and it can't keep pace with the scalability of the Web technology. And these bottlenecks in the data storage prevent ASP.NET applications from scaling.

Why These Problems Exist
Why can't data storage scale? Let's first tackle the three session-state storage options Microsoft provides: InProc, StateServer, and SqlServer. InProc has limitations. It was designed to be used in a single-server, single-process environment, and it does not work in a multi-server or multi-process ASP.NET environment. The session is not retained.
Here's what happens. The user starts on one server, and the session is created there. If the load balancer then sends the user to a different server, the application won't find that session; it will think the user is starting fresh and ask her to log in again. Every time a user clicks on anything, she'll have to log in so she will not be able to proceed. The application will not work.
One way you can solve this is by using the "sticky sessions" feature, which lets you always route the user request back to the same server so the application will find that session on the server.
You can also handle InProc limitations by not creating a Web garden on the server. A Web garden is when your application has multiple ASP.NET worker processes running on the same server. If you avoid using a Web garden, there's only one process and that at least allows InProc to be used in a Web farm.
These two workarounds are far from ideal, however. Sticky sessions can cause a major scalability bottleneck because the load on some servers increases more than others because the length of a user session is not uniform. Some users log in for only one minute, others for 20 minutes. Some servers will get a lot of sessions, but those will be virtually empty or free or idle. Even if you add more boxes, you are not necessarily improving the throughput.
Moreover, InProc has memory limitations. Every session in your ASP.NET process requires memory. As you increase the number of sessions, the memory requirements of your worker process grow significantly. In a 32-bit platform, however, there is a 1GB memory limit for a worker process, and that's a problem. You cannot grow session data beyond what fits in a one gigabyte worker process memory, along with other data and application code. So InProc causes bottlenecks. And the more users you have, the more you will encounter these problems.
StateServer stores session state in a process that is separate from the ASP.NET worker process, but it too has limitations. You can configure it so that every Web server contains its own StateServer or you can dedicate a separate box and maintain the state completely in that box.
With the first option, the problem is you still have to use sticky sessions. Wherever the session was created, that's where you always have to go back to it. This first option only mitigates the InProc Web garden limitation. It doesn't resolve the sticky sessions problem that can leave you with extra boxes that don't get used even while others are clogged. The net effect to the user is that his session and response time become extremely slow.
The other drawback to this configuration option is that if any Web server goes down, the StateServer on that Web server box also goes down, so you lose all those sessions. True, you don't lose all the sessions in a Web site, but you do lose any sessions stored on that box, and that's not acceptable. Ideally, you never want to lose any sessions.
If you choose the other configuration StateServer offers—a dedicated StateServer box—you no longer have to use a sticky session because every Web server goes to the same dedicated box. But now you have a bigger problem: if that box ever goes down, the entire Web farm is down because every Web server is trying to get its session from that box.
That's not all. This dedicated StateServer box gets overwhelmed as more Web servers are added and transactions per second escalate. Consequently, it quickly becomes a scalability bottleneck. So the scalability issue is not solved with StateServer, not with either configuration.
Now we come to SqlServer, which stores session state in a SQL Server database and can be thought of as a dedicated StateServer. This is Microsoft's premier database server designed for high transaction environments. It is more scalable than a StateServer because you can create a cluster of database servers.
In SqlServer configuration, all the Web servers actually connect to a dedicated SqlServer box where all the sessions are stored. It's as if each of the Web servers were connected to a dedicated StateServer box. The concept behind this is that SqlServer will be more scalable than a StateServer. But SqlServer is not as fast as a State Server because a StateServer is an in-memory data store and accordingly, has acceptable performance. On the other hand, SqlServer is not an in-memory data store. It is a disk-based data store. All databases are kept on disk because they grow so large that memory is not sufficient to hold the entire database. Thus, a database stores its data on persistent storage, which is a disk. Due to disk storage, SqlServer performance is not as fast, resulting in a performance drop.
SqlServer can come in multiple configurations. In a standalone configuration, which is the most common, there is only one database server that all the Web servers talk to, and as you grow the size of the Web farm and add more Web servers, you put more and more load on the database (see Figure 2).
Figure 2 ASP.NET sessions still a bottleneck and database also not fully scalable
Plus, you have a performance issue because SqlServer isn't based on memory, and you have a scalability issue because it's not able to scale out as much. You could scale up by making the hardware more powerful by adding CPUs to that box, but you cannot keep adding more database server boxes as you grow the Web farm. You can maybe go from one to two or two to three servers so that SqlServer provides a database clustering capability, which does scale to be more than a StateServer, but it also has its limitations.
The other problem SqlServer storage has is that all of the sessions are kept in a single table. The locking contention for concurrent access and concurrent updates of the session data becomes obvious as soon as you scale up. As you have more and more transactions per second, you have more and more lock delays because everything is kept in one table.
So, while SqlServer scales more than State-Server, it hands you a performance problem and doesn't scale sufficiently. Moreover, it does not scale linearly. You are supposed to be able to grow a Web farm from a 5- to 50- to 100-server farm, and the Web farm itself is supposed to grow quite smoothly; however, the data access does not grow correspondingly. As I noted earlier, a database is one of those data storage accesses that does not grow, so storing sessions in a database doesn't make any major improvement. It's only an incremental improvement over a State-Server environment. Moreover, a SqlServer becomes a bottleneck for application data as well as for sessions data. Hence, a database server does not scale for either sessions or applications data.

What's the Answer?
The solution is an in-memory storage mechanism, so it can be extremely fast, as fast as a StateServer. However, it should be virtually linearly scalable. Linear scalability means that as you add more servers, you are almost multiplying the capacity. For example, if you could handle 10,000 transactions per second with one box, adding a second box should give you close to 20,000 transactions per second total. Note that "virtually linear" doesn't mean exactly 20,000—it could be 19,000; it won't, however, be 12,000 or 15,000. And this is what we need: storage that can grow almost linearly, and it should also be in memory.
Because of these two needs, we are not talking about persistent storage, which has other requirements and is intended for the long term. A database is intended for long-term storage, while in-memory storage is always transient and temporary. But our needs are temporary. We only need to store data in this temporary storage during a user session or perhaps for the duration of an application, a few hours to a few days, or maybe a few weeks at most. Then that data can go away because there is always permanent master storage, which is the database where we can load the data from again.
So, with all of this in mind, we can think of a storage mechanism called a "distributed cache," a concept that has become popular because it provides the benefits cited above, as Figure 3 shows.
Figure 3 Distributed cache relieving pressure on the database server
A distributed cache is in-memory, so it is fast, and it is designed to distribute growth fairly linearly, especially if you have the right distribution mechanism (also called caching topology).
A distributed cache must provide high performance and linear scalability, and since it exists in-memory, it must provide replication so that if any machine goes down (the memory in that machine becomes available), another machine will have the data and you won't lose any. Replication provides more than one copy of the same data in different locations on different boxes and by doing that, you achieve 100 percent up time for the duration of your data storage.
A distributed cache stores a .NET object or a Java object or, for that matter, any other data like an XML document. It stores data in a prepared format. It does not have the concept of tables and rows and primary keys and foreign keys that a database has. For programmers, a distributed cache is essentially a HASH table, where there is a key and each key has a value and that value is an object. You need to know the key, and based on the key, you can fetch the object you want. This is one logical cache that can span multiple servers. You can add servers at the same time to grow cache cluster size, and you can remove boxes at the same time to shrink the cache cluster without stopping anything.

Caching Topologies
The various topologies that an effective cache should provide are replicated, partitioned, a hybrid of replicated and partitioned, and client or local cache. The idea is to have different caching topologies for different types of usage, making the cache extremely flexible. A replicated topology replicates the cache many times, depending on how many times you need (see Figure 4). It is intended for situations where you have a read-intensive cache usage, but not a lot of updates.
Figure 4 The replicated cache is ideal for read-intensive usage
A partitioned cache is the highly scalable topology for update-intensive or transactional data that needs to be cached. This could be ASP.NET session data, which is very transactional. As mentioned earlier, for every Web request, the session is read once and updated once, so it has an equal number of reads and writes.
A partitioned topology (see Figure 5) is excellent for environments where updates must be done at least as many times as you are doing reads, or fairly close to that. In this topology, the cache is divided. As you add more and more cache servers, the cache is further partitioned in such a way that almost one Nth (N means the number of nodes) of the cache is stored on each cache server.
Figure 5 A partitioned cache is ideal for write-intensive usage.
A third topology is a hybrid of the partitioned and replicated versions. You can partition the cache and, at the same time, every partition can be replicated. Hence, you achieve the best of both worlds. You'll be able to partition and grow, plus you can replicate for availability to make sure that no data is lost (see Figure 6).
Figure 6 Partition-replica caches are ideal for write-intensive usage with reliability.
With the help of partitioned and partitioned-replication hybrid topologies, you can grow your cache linearly in terms of scalability.
A client or local cache is the fourth highly useful topology that sits on the application server. This type of cache is very close to the application and can even be InProc. It's usually a small subset of the actual large distributed cache and is based on whatever the application at that moment in time has been requesting. Whatever the application requests, a copy is kept in the client cache. The next time that application wants the same data, it will automatically find it in the client cache. It won't have to go to distributed cache, to save even that as a trip, because the distributed cache is often across the network on a separate caching server or cluster of cache servers. A client cache gives you an additional performance and scalability boost.
The data in a client cache must be kept synchronized with the distributed cache. If that same data is changed in the distributed cache, the distributed cache has to synchronize that change with the client cache. This is an important aspect—you don't want to have just a local cache that is totally disconnected. That's just the equivalent of an InProc cache, which is unacceptable because you have data integrity problems. You have multiple copies of the same data that gets out of sync.

Different Choices
There are multiple distributed caching features available... And, as in most situations, free solutions provide a more limited feature set while commercial ones offer a lot more options and features.
Aside from high performance, scalability, and high availability, an efficient distributed cache must include several key features to help you to keep the cache fresh and synchronized with the master data source, whether database or mainframe. The cache should have an expiration option so you can tell it to perform automatic expiration, which may be either an absolute time or what's called "sliding time." Basically, that's idle time; if no one uses the data, it is automatically expired.
A cache should also be able to manage relationships between different types of data. Most data is relational. For example, if you've got a customer, you have orders for that customer so there is a relationship between customer data and order data. If you cache both customer and order data and you inadvertently delete the customer data from the cache, it makes sense that the order would be automatically deleted. In this instance, you don't know if you removed the customer data from the cache or you permanently deleted it. In case you permanently deleted it, the order is also invalid now because the order has to be from a valid customer.
There are other similar types of relationships that must be managed in the cache. If the cache doesn't do it, then the application has to keep track and that is very cumbersome. A Microsoft cache object in ASP.NET that is very useful is called a "cache dependency concept." One cached item depends on another one. If that other cached item is ever removed from the cache or even updated, the first cache item is removed as well. This is a powerful cache dependency concept that should be available in all caches that cache relational data.
Synchronizing with the database is another important ability for the cache. A database is usually shared by multiple applications. If an application using the cache is the only one updating the database, you probably don't need the database synchronization feature. But quite often, other applications, sometimes third-party applications, are updating data in the database because the database is a shared, common store and those applications are not using your cache. They may not even be .NET applications. They may be third-party applications you don't control, but they are updated in the database. So you have to allow for situations where the database might be updated outside your application, but some of that data that has been updated in the database is also cached. Therefore, the cache has to be able to synchronize. It has to be able to know whenever the data it has is no longer the same in the database. It must remove that data from the cache and maybe even reload the latest copy from the database. Database synchronization can be done either through events fired by the database server or by the cache polling the database. Events are of course more real-time, and polling has a slight delay. But polling can be more efficient if a lot of data is changing.
Event notification is among the most important features an effective distributed cache should have. A cache is often shared among multiple applications and even within an application among multiple users. Therefore, the cache should have an event notification mechanism in case, for example, a cached object is updated or removed. If your application is using that same data, you may want to be notified so you can reload either from the database or a new copy from the cache itself. A notification mechanism improves collaboration among multiple users or multiple applications through the cache.

The Real World
IT management faces all the performance issues associated with databases, and in the case of bottlenecks, if you're lucky enough to be able to report them to developers, they may attempt to resolve them. Unfortunately, development isn't always in-house. Often you are living with and managing a third-party application.
In any case, the best place to start implementing distributed caching to open bottlenecks and turbo-charge your applications is with ASP.NET Session storage, because you don't need to depend on developers. There's no programming involved. It's a simple matter of replacing existing Session storage with an in-memory distributed cache. Furthermore, implementing a distributed cache for ASP.NET Session storage gives you the opportunity to see the benefits that accrue for performance and scalability, and then you can decide whether to do the same for your application data.
To experience the scalability improvement, you either have to run distributed cache storage in production or you have to simulate that load in your test environment. You may have access to QA, which can help perform a stress test in a test environment to simulate a large load before putting a distributed cache in production. Most IT managers would probably not be comfortable putting a distributed cache in production unless they had tested it first in their QA environment, even if they couldn't simulate the same amount of load. So that's a good place to start.
Once you're up and running with a distributed cache and reaping its benefits, you can share your new ASP.NET Session performance and scalability findings with your in-house or third-party-vendor development team. With the hard evidence in hand, you can ask the development team to analyze areas where they can cache application data in this distributed cache as well.
Caching application data provides a further boost and in many cases a lot more boost than simply using a distributed cache for ASP.NET Session storage. The developers would be able to identify all the data elements that are read more frequently than they are updated. Even transactional data (such as customers, orders, and the like) is a good candidate for caching, even if it remains in the cache for a few minutes before expiring. This is because within this brief period of time, the data may be reread many times, and if this rereading is from the cache and not from the database, it relieves your database of a lot of read load.
However, for developers to cache application data, they'll have to do a bit of programming by making API calls to the distributed cache. The idea is very simple. Whenever their application tries to fetch data from the database, it should first check the cache. If the cache has that data, the data is taken from the cache. Otherwise, the application fetches the data from the database, caches it, and then gives it to the user. This way, the data will be found in the cache the next time it is read. Similarly, whenever data is modified in the database, it should also be updated in the cache. And, if the cache lives on multiple servers, it must therefore be synchronized automatically to ensure that if your application runs in a Web farm, the same cache data is accessible from all servers in the farm. For more information on the topic of how to develop an application that uses a distributed cache for better scalability, look for my upcoming article in the July issue of MSDN Magazine.

Iqbal Khan is the President and Technology Evangelist of Alachisoft, the company that provides NCache—the industry's leading .NET distributed cache for boosting performance and scalability in enterprise applications. Iqbal received an MS in Computer Science from Indiana University, Bloomington, in 1990. You can reach him at iqbal@alachisoft.com.


From the Editor It’s All Gotta Work
Joshua Hoffman

What's the most unusual excuse you've ever heard for not upgrading the operating system in your organization? Sure, we've all heard comments about expense, complexity, and end-user training, all of which are completely valid. Certainly an enterprise-wide desktop infrastructure upgrade can be time-consuming and challenging. But who forestalled upgrading to Windows XP because the CEO liked the way his taskbar looked in Windows 2000? Who held off because of that one business-critical application that couldn't be upgraded—not because the application was too sensitive, but because the developer who wrote it (and never documented the code) quit? Or was fired? Or died?
I spent a number of years in Microsoft Consulting Services focusing on desktop engineering and deployment, and trust me—I've heard every reason in the book. The truth is, no matter how ridiculous the excuse, it doesn't matter any less. A business-critical application that can't be upgraded is still business-critical. No matter what advantages a new operating system may offer, if that's the one application needed to bill your customers or manufacture your widgets, it has to work. It's all got to work.
The good news is, there's help to be had. This month we're joined by two experts on application compatibility—Chris Corio and Chris Jackson. Chris & Chris take us through two equally important topics. The first is how to plan an application compatibility project. When you're dealing with one or two applications, planning may not be such a big deal. But, as we all know, that's rarely the case. When you're evaluating tens or hundreds or even thousands of applications, real planning is required, and Chris's article provides an essential foundation. After the planning comes the doing, and that means digging into the Application Compatibility Toolkit, the powerful framework that Microsoft supplies for solving the most common application-compatibility problems. Our experts take you through the ins and outs of the toolkit, helping you ensure that your business-critical applications are ready for the next upgrade.
Knowing your appetite for great technical content, though, there's no way we would stop there! Group policy expert Darren Mar-Elia joins us to discuss automating group policy with Windows PowerShell. John Policelli is back this month with a walk through of using a catch-all subnet in Active Directory. And Iqbal Khan discusses tips for improving the scalability of ASP.NET applications.
We'd love to hear your application compatibility stories, challenges, and questions. Drop us a line anytime at tnmag@microsoft.com, or visit our blog at blogs.technet.com/tnmag. Enjoy!
—Joshua Hoffman
Thanks to the following Microsoft technical experts: Ajith Alexander, Prabu Ambravaneswaran, Eric Charran, Joe DeFazio, Aaron Dunnington, Uday Hegde, Stephen Kreyenbuhl, Jingmei Li, Greg Lindsay, Scott McArthur, Mitch Prince, Brent Scallan, and Kimberly L. Tripp.


Toolbox New Products for IT Pros
Greg Steen

The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft. All prices were confirmed at the time of writing, and are subject to change.

Manage Hard Disk Partitions
EASEUS Partition Master Home Edition
partition-tool.com
While it's not likely an everyday activity, moving, changing, copying, and removing partitions on your users' desktops does happen, and you need a utility to help you do these tasks. Windows, of course, has its own built-in partition management tool (available as a command-line tool and via an MMC snap-in), but sometimes you need a tool that's a bit more powerful to handle a chain of configuration changes or to copy the exact contents of one partition to another partition. One free tool to help you manage disks on 32-bit Windows desktop systems is YIWO Tech's EASEUS Partition Master Home Edition.
Partition Master lets you chain operations together, creating a single execution plan for multiple disk events. The application has two very useful utility wizards for copying: the Partition Copy Wizard and the Disk Copy Wizard, which guide you through complete bit-by-bit copying of partitions as well as complete physical disks. Both of these abilities can be useful when upgrading disks and systems.
As you'd expect, Partition Master also lets you resize, move, delete, label, and create partitions. Each of the tools allows you to either enter exact numbers or drag and drop partition details to specify your sizes and changes. You can format partitions, specifying labels, desired cluster sizes, and file systems. You can even mark partitions active, hide partitions, and run an integrity check.
To get the details of existing partitions, you bring up the properties window with a right-click. You get basic usage information: the partition type, disk serial numbers, first and last physical sectors, the total number of physical sectors, and the physical geometry of the partition. For NTFS partitions, you can also view cluster sizes, MFT cluster start points, the NTFS version being used, and the number of bytes per sector.
Once you have your execution plan in place, you click Apply and the changes are made, locking drives and rebooting as necessary. All in all, the application is easy to use and functional—not bad for a free tool. If you're looking for a partition management solution for your 64-bit Windows systems or need to support Windows Server, check out the Professional and Server editions, which start at $32 and $149, respectively.

Price: Free
Manage Hard Disk Partitions with EASEUS Partition Master

Manage Your Hard Disks
HD Tune Pro
hdtune.com
No matter how fast your processor or I/O bus may be, if your hard disk is slow, you are going to be bound by its constraints. So how well is your hard disk performing under load on your system? One way to find out is to use a benchmarking tool like HD Tune Pro from EFD Software. This low-cost benchmarking utility gives you a number of different metrics to measure the performance of your drives.
To get up and going after installation, you simply pick a drive and select a test or action to run on that disk. Next to the drive selection drop-down, you will notice that HD Tune Pro also shows the current operating temperature of your disk—if your disk reports that data.
The UI splits 10 functions into tabs for easy access. The Benchmark tab measures Read and Write performance for your selected disk over time, graphing the results as well as displaying the minimum, maximum, and average transfer rates. You'll also see how much work your CPU had to do, the burst rate, and the average access time for the drive.
The Info tab gives detailed statistics on each of the hard disks in your system, showing capacity, usage, file system, serial numbers, and firmware version. It also indicates which of 14 specific capabilities your drive supports. These 14 capabilities include S.M.A.R.T. (Self-Monitoring and Reporting Technology), AAM (Automatic Acoustic Management), and firmware upgradeability. If your drive supports S.M.A.R.T., the Health tab will show you all the measured parameters with their current, threshold, status, and worst values as reported by the disk.
The Error Scan tab lets you perform a quick or in-depth surface scan on your hard disk and shows you how many damaged blocks (if any) are on the disk. Folder Usage gives you a very useful view of aggregate folder size, with a drill-down so you can see which are taking up the most room on your disk.
If you're concerned about wiping data on your hard disks, the Erase tab lets you erase a drive using a zero fill, a random fill, the Gutmann algorithm, or the Department of Defense 5220.22-M algorithm. For the latter two, you can specify how many passes over the disk you want to make to ensure the data is really gone. If your drive supports it, you can also enable or disable the AAM feature via the AAM tab. For greater control, you can also use the slider bar to determine a balance between performance and noise, testing it as you go from the tab.
The File Benchmark tab performs file read/write tests with different block sizes. A little more akin to the real-world, the Random Access tab tests a large number of random reads or writes to your disk showing the number of operations per second, the average access time, and the average speed for the test. Here you will be able to specify the transfer size for the test, choosing one or more of 512 bytes, 4KB, 64KB, 1MB, or random sizes, and the results are graphed to illustrate performance. Finally, the Disk Monitor tab actively monitors one or all of the drives in your system, showing a graph of the performance of your hard disks over time, as well as showing you the current and maximum read and write values.
HD Tune Pro also makes saving a screenshot of the graphed results a simple click, saving the result as a PNG so you can easily repurpose the performance visualization in a presentation or e-mail message. If you care more about the raw information than the graphed results, you can also copy the test results to your clipboard and paste them into e-mail or a document for reuse. And the application even has command-line functionality, letting you run tests and log results without using the UI.

Price: $34.95 (direct)
Manage Hard Disks with HD Tuner Pro

Manage Shell Extensions
ShellExView
nirsoft.net/utils/shexview.html
Is your system feeling a bit bogged down because your Explorer has too many hooks into it? Or perhaps you have a control panel or context menu addition that just won't go away? Shell extensions are COM objects that run in-process with your Windows system, extending the features and functionality of the operating system. If you want to see what shell extension hooks are installed and activated on your system, check out ShellExView from NirSoft (otherwise known as programmer, Nir Sofer).
This handy, simple, and free utility scans your registry at startup and fills a data table within the UI containing a great deal of useful information, such as the friendly name, description, version, product, company, file name and path, size, GUID or CLSID, file attributes, and timestamps. There are also a number of useful Yes/No columns for sorting, such as where the extension will appear to the end user, including My Computer, the Desktop, Control Panel, and My Network Places, as well as whether the extension is enabled. Your system will undoubtedly have hundreds of these extensions, and finding what you're looking for can be extremely difficult without features like this.
If you are overwhelmed by the number of columns in the default view, you can also filter the columns to your liking, paring it down to only those you really need. You can also use the basic Find function to search for strings within the data set.
Once you've selected the target extensions, you can easily enable or disable them, add or remove them from the control panel, or pop up the file properties window with a right-click and select. In addition, you can save their details to a text file for reuse or copy them to the clipboard to paste the info into a document or e-mail message. ShellExView also lets you export the results of a selection or the whole result set as an HTML report. Besides enabling and disabling extensions, you can have the tool bring you to the registry key for that extension so you can do your own tweaking.
ShellExView runs as a stand-alone application, requiring no installer and no writes to the registry. This makes it a handy tool for troubleshooting and virus hunting as you can store it on your USB key or CD of troubleshooting apps. (You do have a USB key or CD of troubleshooting tools, don't you? If not, consider it.) Nevertheless, if you're looking to keep an eye on or clean up all those shell extensions on your system, you might want to consider ShellExView.

Price: Free
Manage Shell Extensions with ShellExView

Manage Your Remote Connections
VisionApp Remote Desktop 2009
visionapp.com
As a systems engineer, you probably have a host of machines that you need to connect to in your day-to-day operations. Keeping those machines of different environments, operating systems, and authentication domains straight can be a chore in larger environments. One tool out there to help you manage all those connections is VisionApp's Remote Desktop 2009.
Those of you who support a heterogeneous server environment that contains multiple operating systems will appreciate that VisionApp's Remote Desktop 2009 supports a set of protocols that include VNC, SSH, telnet, Citrix's ICA, and HTTP/s, in addition to the standard Windows Remote Desktop Protocol (RDP). When creating a new connection, you can choose to enter the connection by name, import a CSV file of several computers, or import the list directly from Active Directory, choosing to use a computer name suffix to limit the import.
Once you pick a method of creation, you can then specify the computer name, MAC address, management board URL, and protocol that you wish to use to connect to the remote machine. Here too you can specify whether you want the display to be full screen, in a separate window at a specified resolution, or in the default tab view, which can either be set to a specific resolution or auto size to the application window.
Each of the protocols has a number of configuration items you can tweak to your specific needs. For example, for RDP, you can specify the port, whether you want to connect to the console or want to use Network Level Authentication to protect your connection. The application also supports Microsoft TS Gateway over SSL, so you can be sure your remote connections are secure. And for VNC, you can specify such things as the custom compression and JPEG qualities, the authentication mode, and the encoding method. You can then save your customizations of connections as defaults to speed entry for the future or when you use the Quick Connect feature, which lets you type in a computer name or IP address and protocol and then use those defaults to connect to the remote machine.
Many users will find the tab view appealing as it lets you quickly switch between multiple simultaneous open connections, speeding up your multitasking. In addition to saving connections, VisionApp's Remote Desktop 2009 also lets you specify and save a collection of connection credentials so you don't have to remember those obfuscated account and service names (saving passwords is optional).
Rather than storing your connections and credentials locally, you can have Remote Desktop 2009 store them in a central SQL Server database. By doing so, you can share the lists with others on your staff. You can also use database account permissions to protect the structure from being modified by certain users if you wish or need to do so. Another useful option if you want to keep discrete copies of the connection information for each of your users is to create the connection list and then use the Export tool to export the details to an XML file, which can then be imported by other users or backed up in case your own data is ever lost.
Once you have your connections set up in the application, you can connect to them by double-clicking the icon. Or, if you want, you can click on a folder of machines to simultaneously connect to or disconnect from all of them. There is also a search feature and a sorting feature, which let you find machines faster and organize numerous connections. Your connection history is also saved, which can be used for auditing or simply as a reference. The connection history report shows the connection name, the computer, the protocol, the credentials used, the operations taken, and the date connected.
There are also simple date and name filters to help you find operations faster. This feature is especially useful for the centralized database option to which many users connect to manage machines, helping you monitor your team's activities.
As you can see, if you manage a ton of servers, or if you have a number of system administrators or users that all need to connect to the same set of machines, VisionApp's Remote Desktop 2009 may be the tool to keep you organized and, well, sane.

Price: Starts at $88 (direct for a single user license)
Manage Remote Connections with VisionApp Remote Desktop 2009

If you have a favorite tool or utility you would like to see featured here, please write to the author at tntools@microsoft.com.

Greg Steen is a technology professional, entrepreneur, and enthusiast. He is always on the hunt for new tools to help make operations, QA, and development easier for the IT professional.


SQL Q & A Removing Index Fragmentation, Synchronizing vs. Synchronized, and More
Paul S. Randal

QI'm confused about how removing index fragmentation affects statistics. I've heard that sometimes I should rebuild my statistics after rebuilding an index and sometimes I shouldn't, and that rebuilding a clustered index will affect all the other indexes too. Can you shed some light on this please, as I want to make sure I'm not inadvertently making performance worse?
AThis certainly is an issue that causes a lot of confusion, but you are right that a comprehensive database maintenance strategy includes removing index fragmentation and updating statistics. I'm not going to go into the details of exactly when and why to remove index fragmentation—for that, see the August 2008 article "Top Tips for Effective Database Maintenance." Instead, I'll assume you know which indexes to work with and I'll just focus on how things work.
The first part of the confusion is around which fragmentation removal operations affect statistics. Rebuilding an index (using ALTER INDEX … REBUILD, DBCC DBREINDEX, or CREATE INDEX … WITH DROP_EXISTING) does update statistics with the equivalent of a full scan, but reorganizing an index (using ALTER INDEX … REORGANIZE or DBCC INDEXDEFRAG) does not update statistics at all, even though both remove index fragmentation.
The reason that rebuilding does update statistics and reorganizing does not has to do with the algorithms used for these operations. An index rebuild operation has a complete view of the index and therefore can update the statistics correctly. A reorganize operation, however, only operates on a few pages of the index at a time and cannot correctly update the statistics for the whole index.
The second part of the confusion is about which statistics are updated when an index is rebuilt. There are two general kinds of statistics for a table—those for the table's indexed columns and those for the non-indexed columns. An index rebuild operation only updates the statistics for the index being rebuilt. The statistics for non-indexed columns must be updated manually in the maintenance plan. Furthermore, as I mentioned in the "Top Tips for Effective Database Maintenance" article, you must be careful not to manually update index statistics after rebuilding the index, as the manual update may use a sample rate of less than 100%, whereas the index rebuild will use the equivalent of a full-scan (100%) sample. In other words, this could lead to overwriting a full-scan statistic with a sampled statistic.
The last part of the confusion is around what effect rebuilding an index has on other indexes. The answer is always that rebuilding an index only affects that particular index and its statistics. The exception is for a non-unique clustered index in SQL Server 2000, where rebuilding such an index will also cause all non-clustered indexes on the table to be rebuilt. This has been fixed from SQL Server 2005 onwards. You can find more information on this point in my blog post at "Indexes from Every Angle."
To summarize, your index and statistics maintenance should do the following:
  • Rebuild or reorganize your indexes to remove fragmentation
  • Update index statistics for those indexes that were not rebuilt
  • Update non-index statistics
QI've been experimenting with SQL Server 2008 and I've found some weird behavior. It seems that if I enable data compression in the production database and then backup the database and try to restore it on a Standard Edition instance, the restore fails! Is this expected behavior? If so, is this limited to data compression or are any other features affected? And why doesn't the restore fail immediately instead of seeming to do the entire restore before failing?
AThe behavior that you're seeing is by design. Most "Enterprise-only" features just limit the SQL Server Editions with which you can make use of the feature to Enterprise Edition, Enterprise Evaluation Edition, and Developer Edition. There are some features, however, that also limit which SQL Server Editions can restore a backup of a database containing the feature. You've found one of the SQL Server 2008 features that has this restriction—data compression.
The behavior is actually not new for SQL Server 2008. In SQL Server 2005, if a database contains any partitioned tables or indexes (explicitly using the partitioning feature), then a backup of that database can only be restored using an Edition in the list I just mentioned. There are two issues with this, though, in SQL Server 2005. Firstly, it can be hard to tell whether there is any partitioning in a database, and secondly, the restore doesn't fail until just before it would have completed.
This means you don't find out that your restore is going to fail until you've waited through the entire (potentially long-running) restore operation. This is because the database isn't transactionally consistent until the recovery part of restore has completed—and so an operation during recovery might add or remove partitioning. Encountering this situation can be really annoying in a disaster recovery situation where the only server available to quickly get the database restored happens to be Standard Edition.
In SQL Server 2008, the list of "Enterprise-only" features with this behavior has grown to four: data compression, change data capture, transparent data encryption, and partitioning. This means that more people are going to encounter this problem. For this reason, a new DMV was added, sys.dm_db_persisted_sku_features, which allows a DBA to easily check whether any of these features are enabled in a database.
For instance, in a database where a table has had data compression enabled, running the DMV will give the following results:
SELECT * FROM sys.dm_db_persisted_sku_features;
GO
feature_name    feature_id
--------------  -----------
Compression     100
However, SQL Server 2008 does not address the issue with the restore operation having to almost complete before the DBA is informed that the database cannot actually be restored. In fact, this issue is unlikely to be addressed given the nature of how restore works—see the SQL Q&A column from the October 2008 issue for more details.
If it's possible that a database needs to be restored on Standard Edition (or a lower edition), it's imperative that the DBA either disallows these features or uses the DMV regularly to avoid getting a nasty surprise when trying to restore in a critical situation.
QI've implemented synchronous database mirroring and was under the impression that this meant the mirror database is always synchronized with the principal database. However, I've occasionally seen the mirror state become SYNCHRONIZING instead of SYNCHRONIZED, as well as when we first set up a mirror. Why does this happen?
ABefore I answer the question, I'll define the SYNCHRONIZED and SYNCHRONIZING database mirroring states. In a nutshell, database mirroring works by continually sending physical transaction log records between the SQL Server instance hosting the principal database and the instance hosting the mirror database. If there are no transaction log records waiting on the principal to be sent to the mirror, the mirroring state is SYNCHRONIZED (in other words, the two databases are in sync). If there are transaction log records that have not yet been sent to the mirror (the mirror database is "behind" the principal), the mirroring state is SYNCHRONIZING.
The method for initializing a database mirror is to take a full database backup, plus at least one transaction log backup, and restore them (using WITH NO_RECOVERY) on the mirror. Then when mirroring is enabled, the mirroring state is initially SYNCHRONIZING. This is because there may have been some transaction activity on the principal since the last log backup that was restored on the mirror. This is very likely for a database with a constant workload.
The trick to minimizing the time in which the mirroring state stays SYNCHRONIZING is to try to make the mirror database as up-to-date as possible using log backups. However, for a database with a constant workload, this may be very difficult and so the mirroring state will initially be SYNCHRONIZING for some time. Once the mirror catches up, the mirroring state will change to SYNCHRONIZED.
After that point, it is possible for the mirror database to again fall behind the principal, in which case the mirroring state will drop back to SYNCHRONIZING. One cause of this could be if the mirror and principal are unable to communicate for a period of time. Or it could happen if the principal is generating the transaction log faster than the log can be sent to the mirror. In either case, and depending on how database mirroring is configured, the principal database may continue to process transactions and a queue of transaction log records will build up (called the SEND queue). This must be sent to the mirror in order for it to catch up. Until the mirror database has caught up, the mirroring state will remain SYNCHRONIZING.
If the network link is unreliable between the principal and mirror, you may see the mirroring state switching back-and-forth between SYNCHRONIZING and SYNCHRONIZED. You can find a full discussion of these states in the whitepaper Database Mirroring in SQL Server 2005.
QWe have synchronous database mirroring set up for our main application database and our monitoring shows that the redo queue on the mirror is usually near-zero. We'd like to use the mirror database for reporting and consistency checking so we can make better use of the redundant hardware. We know this is possible, but are there any drawbacks to doing this that we should know of?
AThis is a pretty common practice, but there are a few drawbacks that you may need to watch out for.
The first issue is licensing. The license for the SQL Server instance hosting the mirror database is free if the instance is only used for that purpose. As soon as you create a database snapshot on the mirror database or do anything else with that SQL Server instance, you must purchase a license.
As far as consistency checking is concerned, running them on the mirror database (through a database snapshot) doesn't give you any guarantees about the consistency of the principal database. Only transaction log records are mirrored between the databases, so if the I/O subsystem corrupts a page in the principal database, that corruption will not be mirrored on the mirror database. This means that a consistency check of the mirror database would not see that corrupt page on the principal database.
Reporting is a good use of a database mirror. The first issue that people run into is how to refresh the database snapshot that the reports are running against. A database snapshot cannot be refreshed so a new snapshot must be created and the reporting application must connect to the new snapshot. This requires extra logic to be built into the reporting application. The second issue is determining how the reporting application should behave in the event of a mirroring failover—should it remain connected to the new principal or should it move to the new mirror? Going into details on this is beyond the scope of this article.
With any extra use of the mirror database, the main concern is the potential for causing performance problems on the mirror. Creating a database snapshot adds extra I/O load in that the first time a page changes in the mirror database, it must be pushed into the database snapshot. Also, any workload you then place on the database snapshot increases the I/O load on the mirror database—as many of the pages read from the database snapshot will really be read from the mirror database, as they will not have changed since the snapshot was created.
This extra I/O load on the mirror database can slow down the replaying of transaction log records and in turn lead to a backlog. This backlog is called the REDO queue and is the amount of transaction log that must be replayed before the mirror database can come online after a failover. The larger the I/O load on the mirror database from the database snapshot, the larger the REDO queue will potentially grow and the longer the database will be unavailable after a failover.
You did mention that your REDO queue length is usually near-zero so this may not be an issue for you, but it is definitely something to watch out for so you don't end up compromising the database's availability in return for the increased ability to run reports.

Paul S. Randal is the Managing Director of SQLskills.com and a SQL Server MVP. He worked on the SQL Server Storage Engine team at Microsoft from 1999 to 2007. Paul wrote DBCC CHECKDB/repair for SQL Server 2005 and was responsible for the Core Storage Engine during SQL Server 2008 development. Paul is an expert on disaster recovery, high-availability, and database maintenance and is a regular presenter at conferences around the world. He blogs at SQLskills.com/blogs/paul, and you can find him on Twitter at Twitter.com/PaulRandal.


Utility Spotlight PendMoves and MoveFile
Lance Whitney

After you install or uninstall a Windows application, specific files associated with that app often have to be replaced or deleted. But sometimes those files are locked in memory and can't be removed until Windows shuts down. Whether you're troubleshooting a problem with a bad install or are just curious, you may at times want to see a list of those files. Microsoft's free PendMoves tool can help—by showing you the names of all scheduled file moves and deletions before they occur. Additionally, you might sometimes need to move or delete a locked file yourself, either to totally remove an application or to clean up a PC. The MoveFile tool fits that bill.
Written by Mark Russinovich, PendMoves and MoveFile are part of the Windows Sysinternals library. To download PendMoves and MoveFile, go to the Code Downloads section of our Web site, .com. Extract the downloaded PendMoves.zip, and inside you'll find three files: EULA.text, pendmoves.exe, and movefile.exe. Like most Windows Sysinternals tools, no installation is required. Just trigger the programs from their EXE files.
PendMoves displays a list of any fi les scheduled to be moved or deleted
Both PendMoves and MoveFile run from a command line, so you'll need to open a prompt. At the prompt, move to the directory in which you extracted PendMoves.zip. Type pendmoves, and a console window opens to display basic information about the tool, including its name, date, and author, followed by a list of any files scheduled to be moved or deleted. Each file will contain a Source line, which displays the path name of the file, and a Target line, which displays the action or location set for the file—DELETE for deletions or the directory path for moves or renames, as shown in the screenshot. If a program has been installed or uninstalled and requires a reboot, you'll typically see the names of files belonging to that app.
How does PendMoves work? Windows uses an API called MoveFileEx, which lets an application queue up the files that need to be removed or replaced as part of its installation. That information is stored in the Registry key: HKLM\System\CurrentControlSet\Control\Session Manager\Pending FileRenameOperations. PendMoves reads and displays that Registry value.
The second half of this team, MoveFile, comes into play if you need to delete, remove, or rename a file in use. To run MoveFile, first copy movefile.exe to the directory containing the file you wish to move or delete. Open a command prompt and move to that directory. Type movefile followed by the name of the file and the location to which you want to move it—for example, movefile config.log c:\backups. To rename the file, specify the new name—for example, movefile config.log config.bak. To delete the file, type two quotation marks as the location—for example, movefile config.log "".
MoveFile will not accept wildcards. Therefore, to remove several files in one shot, you can create a script with a separate command string for each file. You can also run MoveFile on an entire directory to delete or rename it.
If you plan to use PendMoves or MoveFile on a regular basis, copy them to the Windows/System32 directory or other location in your path so that you can easily run them from any directory. Both PendMoves and MoveFile are compatible with all NT-based versions of Windows, including Windows XP, Windows Vista, and Windows Server.

Lance Whitney is an IT consultant, trainer, and technical writer. He has spent countless hours tweaking Windows workstations and servers. Originally a journalist, he took a blind leap into the IT world 15 years ago.


Inside SharePoint Creating an External Storage Solution for SharePoint
Pav Cherny

Code download available at: ChernySharePoint2009_06.exe (2,006 KB)
Contents
Internal Binary Storage
External Binary Storage
Building an Unmanaged EBS Provider
Building a Managed EBS Provider
Registering an EBS Provider in SharePoint
Implementing Garbage Collection
Conclusion
Microsoft estimates that as much as 80 percent of the data stored in Microsoft Windows SharePoint Services ( WSS ) 3.0 and Microsoft Office SharePoint Server (MOSS) 2007 content databases is non-relational binary large object ( BLOB) data, such as Microsoft Office Word documents, Microsoft Office Excel spreadsheets, and Microsoft Office PowerPoint presentations. Only 20 percent is relational metadata, which implies a suboptimal use of Microsoft SQL Server resources at the database backend. SharePoint does not take advantage of recent SQL Server innovations for unstructured data introduced in SQL Server 2008, such as the FILESTREAM attribute or Remote BLOB Storage API, but provides its own options to increase the storage efficiency and manageability of massive data volumes.
Specifically, SharePoint includes an external binary storage provider API, ISPExternalBinaryProvider, which Microsoft first published as a hotfix in May 2007 and incorporated later into Service Pack 1. The ISPExternalBinaryProvider API is separate from the Remote BLOB Storage API. Third-party vendors can use this API to integrate SharePoint with advanced storage solutions, such as content-addressable storage (CAS) systems. You can also use this API to maintain SharePoint BLOB data on a central file server outside of content databases if you want to build a custom solution to increase storage efficiency and scalability in a SharePoint farm. Keep in mind, however, that this API is specific to WSS 3.0 and MOSS 2007. It will change in the next SharePoint release, which means that you will have to update your provider.
In this column, I discuss how to extend the SharePoint storage architecture using the ISPExternalBinaryProvider API, including advantages and disadvantages, implementation details, performance considerations, and garbage collection. I also discuss a 64-bit compatibility issue of Microsoft Visual Studio that can cause SharePoint to fail loading managed ISPExternalBinaryProvider components despite a correct interface implementation. Where appropriate, I refer to the ISPExternalBinaryProvider documentation in the WSS 3.0 SDK. Another reference worth mentioning is Kyle Tillman's blog.
Kyle does a great job explaining how he mastered the implementation hurdles in managed code, but neither the WSS 3.0 SDK nor Kyle's blog post includes a Visual Studio sample project, so I decided to provide ISPExternalBinaryProvider samples in both unmanaged and managed code in this column's companion material. The purpose of these samples is to help you get started if you are interested in integrating external storage solutions with SharePoint. Remember, though, that these samples are untested and not ready for production use.

Internal Binary Storage
By default, SharePoint stores BLOB data in the Content column of the AllDocStreams table in the content database. The obvious advantage of this approach is straightforward transactional consistency between relational data and the associated non-relational file contents. For example, it's not complicated to insert the metadata of a Word document along with the unstructured content into a content database, nor is it complicated to associate metadata with the corresponding unstructured content in select, update, or delete operations. However, the most obvious disadvantage of the default approach is an inefficient use of storage resources. Despite an I/O subsystem optimized for high performance, the SQL Server storage engine is not exactly a file-server replacement.
A SQL Server database consists of transaction log and data files, as illustrated in Figure 1. In order to ensure reliable transactional behavior, SQL Server first writes all transaction records to the log file before it flushes the corresponding data in 8KB pages to the data file on disk. Depending on the selected recovery model, this requires more than twice the BLOB size in storage capacity until you perform a backup and purge the transaction log. Moreover, SQL Server does not store unstructured SharePoint content directly in data pages. Instead, SQL Server uses a separate collection of text/image pages and only stores a 16-byte text pointer to the BLOB's root node in the data row. Text/image pages are organized in a balanced tree, yet there is only one collection of text/image pages for each table. For the AllDocStreams table, this means that the content of all files is spread across the same text/image page collection. A single text/image page can hold data fragments from multiple BLOBs, or it may hold intermediate nodes for BLOBs larger than 32KB in size.
Figure 1 Default SharePoint BLOB storage in SQL Server
Let's not dive too deeply into SQL Server internals, though. The point is that when reading unstructured content, SQL Server must go through the data row to get the text pointer and then through the BLOB's root node and possibly additional intermediate nodes to locate all data fragments spread across any number of text/image pages that SQL Server must load into memory in full to get all data blocks. This is because SQL Server performs I/O operations at the page level. These complexities impair file-streaming performance in comparison to direct access through the file system. SQL Server also imposes a hard size limit of 2GB on SharePoint because this is the maximum capacity of the image data type. The Content column of the AllDocStreams table is an image column, so you cannot store files larger than 2GB in a SharePoint content database.

External Binary Storage
The ISPExternalBinaryProvider API offers a clever alternative to internal BLOB storage in SharePoint content databases. It is a straightforward COM interface with only two methods (StoreBinary and RetrieveBinary), which you can use to implement an External Binary Storage (EBS) provider. For architecture details, see the topic "Architecture of External BLOB Storage" in the WSS 3.0 SDK.
SharePoint loads your EBS provider when you set the ExternalBinaryStoreClassId property of the local SPFarm object (SPFarm.Local.ExternalBinaryStoreClassId) to the provider's COM class identifier (CLSID). SharePoint then calls the provider's StoreBinary method whenever you submit BLOB data, such as when you're uploading a file to a document library. The EBS provider can decide to store the BLOB in its associated external storage system and return a corresponding BLOB identifier ( BLOB ID) to SharePoint, or it can set the pfAccepted parameter in the StoreBinary method to false to indicate that it did not handle the BLOB. In the latter case, SharePoint stores the BLOB in the content database as usual. On the other hand, if the EBS provider accepted the BLOB, SharePoint only inserts the BLOB ID into the Content column of the AllDocStreams table, as indicated in Figure 2. The BLOB ID can be any value that enables the EBS provider to locate the content in the external storage system, such as a filename, a file path, a globally unique identifier (GUID), or a content digest. The sample providers included in the companion material, for instance, use GUIDs as filenames for reliable identification of BLOBs on a file server.
Figure 2 Storing a SharePoint BLOB in an external storage system
SharePoint also keeps track of externally stored files by setting the highest DocFlags bit of these files to 1. DocFlags is a column of the AllDocs table. When a user requests to download an externally stored file, SharePoint checks DocFlags and passes the Content value from the AllDocStreams table to the RetrieveBinary method of the EBS provider. In response to the RetrieveBinary call, the EBS provider must retrieve the indicated BLOB from the external storage system and return the binary content to SharePoint in form of a COM object that implements the ILockBytes interface. Note that SharePoint does not call the RetrieveBinary method for BLOBs stored directly in the content database.
Note also that the storage and retrieval processes are transparent to the user as long as the user doesn't attempt to bypass SharePoint. So, you don't need to replace built-in Web parts with custom versions that tie metadata in a list with a document stored externally; productivity applications, such as Microsoft Office, don't need to know how to store metadata in one place and then the document in another; and Search does not need to process metadata separate from documents. Moreover, and this is one of my favorite advantages of the EBS provider architecture, the user must go through SharePoint to access externally stored BLOB data. A user bypassing SharePoint and directly accessing a content database through a SQL Server connection ends up downloading BLOB IDs instead of actual file contents, as illustrated in Figure 3. You can verify this behavior if you deploy the SQL Download Web Part (which I used in the April 2009 column to demonstrate how to bypass SharePoint AD RMS protection) in a test environment. Furthermore, users don't need—and should not have—access permissions to the external BLOB store. Only SharePoint security accounts require access because SharePoint calls the EBS provider methods in the security context of the site's application pool account.
Figure 3 The EBS provider can be a roadblock to bypassing SharePoint permissions for file downloads
Keep in mind, however, that EBS providers also have drawbacks due to the complexity of maintaining integrity between metadata in the SharePoint farm's content databases and the external BLOB store. For a good discussion of pros and cons, check out the topic "Operational Limits and Trade-Off Analysis" in the WSS 3.0 SDK. Make sure you read this very important topic before implementing an EBS provider in a SharePoint environment.

Building an Unmanaged EBS Provider
Now let's tackle the challenges of building EBS providers. The ISPExternalBinaryProvider interface is well-documented in the WSS 3.0 SDK under "The BLOB Access Interface: ISPExternalBinaryProvider." However, it seems Microsoft forgot to cover the EBS provider details. After all, we are not just consuming the interface of an existing COM server. We are tasked with building that COM server ourselves and implementing the ISPExternalBinaryProvider interface. Most importantly, the WSS 3.0 SDK fails to mention the type of COM server we are supposed to build and the required threading model. A classic COM server can run out-of-process or in-process, and it can support the single-threaded apartment (STA) model, the multithreaded apartment ( MTA) model, or both, or the free-threaded model. For the EBS provider to work properly, make sure you build a thread-safe in-process COM server that supports the threading model "Both" for STAs and the MTA.
You also need to think about which programming language to use. This is important because the ISPExternalBinaryProvider interface is the lowest-level API of SharePoint. Performance issues can affect the entire SharePoint farm. For this reason, I recommend using a language that enables you to build small and fast COM objects, such as Visual C++ and Active Template Library (ATL). ATL provides helpful C++ classes to simplify the development of thread-safe COM servers in unmanaged code with the correct level of threading support.
Visual Studio also includes a variety of ATL wizards. Just create an ATL project, select Dynamic-link library ( DLL) for the server type, copy the ISPExternalBinaryProvider interface definition from the WSS 3.0 SDK into the interface definition language ( IDL) file of your ATL project, add a new class for an ATL Simple Object, select "Both" as the threading model and no aggregation, then right-click the new class, point to Add, click Implement Interface, and select ISPExternalBinaryProvider. That's it! The Implement Interface Wizard performs all necessary plumbing, so you can focus on implementing the StoreBinary and RetrieveBinary methods.
And don't let unmanaged C++ code intimidate you. If you analyze the SampleStore.cpp file in the companion material, you can see that the StoreBinary and RetrieveBinary implementations are relatively straightforward. Essentially, the sample StoreBinary method constructs a file path based on a StorePath registry value, the Site ID passed in from SharePoint, and a GUID generated for the BLOB, and then uses the Win32 WriteFile function to save the binary data obtained from the ILockBytes instance. The sample RetrieveBinary method, on the other hand, constructs the file path based on the same StorePath registry value, the Site ID, and the BLOB ID passed in from SharePoint, and then uses the Win32 ReadFile function to retrieve the unstructured data, which the EBS provider copies into a new ILockBytes instance that it then passes back to SharePoint. Figure 4 illustrates how the EBS provider constructs the file path.
Figure 4 Constructing file paths for StoreBinary and RetrieveBinary operations in the sample EBS providers

Building a Managed EBS Provider
Of course, SharePoint developers might prefer using familiar managed languages to build EBS providers, even though building managed EBS providers is not necessarily less complicated than building unmanaged providers due to the complexity of COM interoperability. Keep in mind that an application written in unmanaged code can only load one version of the common language runtime (CLR), so your code needs to work with the same version of the CLR that the rest of SharePoint is using, otherwise you might end up with unexpected behavior. Also, you still must deal with unmanaged interfaces and the corresponding marshalling of parameters and buffers. Just compare SampleStore.cpp with SampleStore.cs in the companion material. There are no gains using a managed language in terms of code structure or programming simplicity.
Moreover, be aware of 64-bit compatibility issues if you develop managed EBS providers on the x64 platform. Figure 5 shows a typical error that results from invalid COM registration settings on a development computer. If you enable the Register for COM Interop checkbox in the project properties in Visual Studio 2005 or Visual Studio 2008, you'll end up with COM registration settings for your provider in the registry under HKEY_CLASSES_ROOT\Wow6432Node\CLSID\<ProviderCLSID>. Visual Studio uses the 32-bit version of the Assembly Registration Tool (Regasm.exe) even on the x64 platform.
Figure 5 Due to invalid COM registration settings, a managed EBS provider could not be loaded
However, the 64-bit version of SharePoint cannot load a 32-bit COM server registered under the Wow6432Node, so you must manually register your managed EBS provider by using the 64-bit Regasm.exe version, located in the %WINDIR%\Microsoft.NET\Framework64\v2.0.50727 directory. For example, the command "%WINDIR%\Microsoft.NET\Framework64\v2.0.50727\Regasm.exe" ManagedProvider.dll creates the required registry settings for the managed sample provider under HKEY_CLASSES_ROOT\CLSID\<ProviderCLSID>. Another approach is to create a Setup program and mark the EBS provider for automatic COM registration.
Remember also that managed EBS providers come with significantly more overhead and performance penalties than their unmanaged ATL counterparts. You can see this if you compare the COM registration settings in the registry. As the InProcServer32 key reveals, the COM runtime loads unmanaged EBS provider DLLs directly, while managed EBS providers rely on Mscoree.dll as the in-proc server, which is the core engine of the CLR. So, for managed providers, the COM runtime loads the CLR and then the CLR loads the EBS provider assembly as registered under the Assembly key and creates a COM Callable Wrapper (CCW) proxy to handle the interaction between the unmanaged SharePoint client (Owssvr.dll) and the managed EBS provider.
Keep in mind that the unmanaged SharePoint server does not directly interact with your managed provider. It's the CCW that marshals parameters, calls the managed methods, and handles HRESULTs. This indirection is especially apparent in the different return types of managed methods in comparison to unmanaged methods. Unmanaged methods return HRESULTs to indicate success or failures while managed methods are supposed to have the void return type. So don't return explicit HRESULTs in managed code. You must raise system or user-defined exceptions in response to error conditions. If a managed method completes without an exception, the CCW automatically returns S_OK to the unmanaged client.
On the other hand, if a managed method raises an exception, the CCW maps error codes and messages to HRESULTs and error information. The CCW implements various error-handling interfaces for this purpose, such as ISupportErrorInfo and IErrorInfo, but SharePoint does not take advantage of these interfaces. EBS providers must implement their own error reporting through the Windows event log, SharePoint diagnostic logs, trace files, or other means. SharePoint only expects the HRESULT values S_OK for success and E_FAIL for any error. You can use the Marshal.ThrowExceptionForHR method to return E_FAIL to SharePoint, as demonstrated in SampleStore.cs.

Registering an EBS Provider in SharePoint
Easily the most confusing section on ISPExternalBinaryProvider in the WSS 3.0 SDK is the topic "Installing and Configuring Your BLOB Provider." At the time of this writing, this section was filled with misleading information and errors. Even the Windows PowerShell commands were incorrect. If you assign the EBS provider to $yourProviderConfig and afterwards use $providerConfig.ProviderCLSID, don't be surprised when you receive an error stating that $providerConfig doesn't exist. Of course, you won't even reach this point because the Active and ProviderCLSID properties aren't part of the ISPExternalBinaryProvider interface. These mysterious properties belong to a dual interface that is not covered in the documentation. Just for fun, I implemented a sample version in both unmanaged and managed code, but your ISPExternalBinaryProvider implementation does not require these proprietary properties at all.
The ProviderCLSID property might be handy, but the CLSID is also available in the registry if you search for the ProgID, such as UnmanagedProvider.SampleStore or ManagedProvider.SampleStore, and you can also find the CLSIDs in the code files SampleStore.rgs and SampleStore.cs. As mentioned earlier, setting the ExternalBinaryStoreClassId property of the local SPFarm object to the CLSID registers the EBS provider. Setting the ExternalBinaryStoreClassId property of the local SPFarm object to an empty GUID ("00000000-0000-0000-0000-000000000000") removes the EBS provider registration. Don't forget to call the SPFarm object's Update method to save the changes in the configuration database and restart Internet Information Services ( IIS). The following code listing illustrates how to accomplish these tasks in Windows PowerShell:
[System.Reflection.Assembly]::LoadWithPartialName('Microsoft.SharePoint')
$farm = [Microsoft.SharePoint.Administration.SPFarm]::Local

# Registering the CLSID of an EBS provider
$farm.ExternalBinaryStoreClassId = "C4A543C2-B7DB-419F-8C79-68B8842EC005"
$farm.Update()
IISRESET

# Removing the EBS provider registration
$farm.ExternalBinaryStoreClassId = "00000000-0000-0000-0000-000000000000"
$farm.Update()
IISRESET

Implementing Garbage Collection
Another section in the WSS 3.0 SDK featuring mysterious components and critical code snippets is titled "Implementing Lazy Garbage Collection." At the time of this writing, this section contained references to another mysterious Utility class with DirFromSiteId and FileFromBlobid methods as well as an incorrect assignment of Directory.GetFiles results to a FileInfo array, but let's not be too demanding on WSS 3.0 documentation quality. The DirFromSiteId and FileFromBlobid helper methods reveal their purpose through their names and the incorrect FileInfo array is easily replaced with a string array, or you can replace the Directory.GetFiles method with a call to the GetFiles method of a DirectoryInfo object. The Garbage Collector sample program in the companion material uses the DirectoryInfo approach and follows the suggested sequence of steps for garbage collection.
An important deviation of the Garbage Collector sample from the SDK explanations concerns the handling of timing conditions. This is a critical issue because timing conditions can lead to misidentification and deletion of valid files during garbage collection. Take a look at Figure 6, which illustrates the WSS 3.0 SDK–recommended approach to determine orphaned files by enumerating all BLOB files in the EBS store and then removing all those references from the BLOB list that are still in the content database as indicated through the site's ExternalBinaryIds collection. The remaining references in the BLOB list are supposed to indicate orphaned files that should be deleted.
Figure 6 Misidentification of a valid BLOB as orphaned due to a timing condition
However, the EBS provider must, of course, first finish writing BLOB data before it can return a BLOB ID to SharePoint. Depending on network bandwidth and other conditions, I/O performance can fluctuate. So, there is a chance that the EBS provider could create a new BLOB—which then appears in your BLOB list—but completes writing the BLOB data after you have determined the ExternalBinaryIds so the BLOB ID is not yet present in this collection. Accordingly, the reference to the new BLOB remains in the orphaned BLOB list and if you purge the orphaned BLOBs at this point, you accidentally delete a valid content item and lose data! In order to avoid this problem, the sample Garbage Collector checks the file creation time and adds only those items to the BLOB list that are more than one hour old.

Conclusion
By integrating an external storage solution with SharePoint, you can increase storage efficiency, system performance, and scalability of a SharePoint farm. Another advantage is that this forces users to go through SharePoint to access unstructured contents. Pulling data out of the content databases via direct SQL Server connections only yields binary BLOB identifiers instead of the actual files. However, EBS providers also have drawbacks due to the complexity of maintaining integrity between metadata in the SharePoint farm's content databases and the external BLOB store.
In order to integrate SharePoint with an external storage solution, you must build an EBS provider, which is a COM server that implements the ISPExternalBinaryProvider interface with its StoreBinary and RetrieveBinary methods. You can create unmanaged and managed EBS providers, but be aware of performance and compatibility issues if you decide to use managed code. Also keep in mind that the ISPExternalBinaryProvider interface does not include a DeleteBinary method. You must explicitly remove orphaned BLOBs through lazy garbage collection, and be careful to avoid timing conditions that can lead to the accidental deletion of valid BLOB items.

Pav Cherny is an IT expert and author specializing in Microsoft technologies for collaboration and unified communication. His publications include white papers, product manuals, and books with a focus on IT operations and system administration. Pav is President of Biblioso Corporation, a company that specializes in managed documentation and localization services.


Windows PowerShell Automating User Provisioning, Part 4
Don Jones

Contents
Deciding On Input
Populating Groups
Additional Attributes
Taking It Further
Some Final Thoughts
This is the conclusion of a four-part series in which I've shown you how to automate the process of provisioning new users in Active Directory. So far, I've demonstrated how to create new user Active Directory accounts, create their home folders, and assign permissions to these home folders. Now, I need to add the users to some domain groups and update a few directory attributes.
To quickly refresh your memory, I am using a "master function" to coordinate all these efforts, which I explained in Part 1 of this series. The master function looks like this:
Function Provision {
  PROCESS {
    CreateUser $_
    CreateHomeFolder $_
    AddToGroups $_
    UpdateAttributes $_
  }
}
In Part 2 of the series, I created the CreateUser function, and in Part 3, I covered the CreateHomeFolder function. This month, I will deal with the AddToGroups and UpdateAttributes functions.

Deciding On Input
All four of my functions get a hashtable that contains user attributes. My original example, in Part 1, read those attributes from a Comma-Separated Values (CSV) file. At the time, I didn't go into much depth on how I planned to convey what domain groups a user should be added to. There are a lot of ways I could potentially do this. For example, I could have my CSV file include a single column named Groups and populate that column with some delimited list of names, like this:
"Domain Users,Sales,Marketing,East"
Or, I could have a separate column in the CSV for each group and then on the data rows put a 1 when the user should belong to that group and a 0 for groups I don't want the user to belong to. It might look like this in the CSV:
Domain Users,Sales,Marketing,East
1,1,0,0
1,0,1,1
I suspect that the second approach might be easier for a less-technical person (perhaps someone in the Human Resources department) to work with since it looks a bit more like a checklist. This is also something you could easily create in a Microsoft Access table or a Microsoft Excel spreadsheet and then export to a CSV file, retaining this format. So for this example, I'm going to go with the second layout.
As for the "additional attributes," I'll continue to assume those are coming from specific columns in the CSV file. For example, if I want to populate the postalCode attribute, then let's assume that I've included an appropriate column in the CSV file, and that my Import function (which I discussed in Part 1 of this series) is reading that column into the hashtable along with the other columns.
That means my hashtable may contain something like $userinfo['Domain Users'], which would be 1 if I want the user to belong to that group. The hashtable might also contain $userinfo['postalCode'] with the appropriate value for each new user.

Populating Groups
My AddToGroups function needs to accommodate every possible group that might be listed in my input file. I'm basically just going to check for each group to see if this user has a 1 or 0, and if the user has a 1, then I'll add the user to the group. Since the number 0 is usually synonymous with the Boolean value False, while anything non-zero (such as 1) is synonymous with True, that will make the comparisons pretty easy. So the function looks like this:
Function AddToGroup {
  Param($userinfo)
  If ($userinfo['Sales']) { Add-QADGroupMember
    -identity 'DOMAIN\Sales' '
    -member ('DOMAIN\'+$userinfo
     ['samAccountName']) }
  If ($userinfo['East]) { Add-QADGroupMember
    -identity 'DOMAIN\East' '
    -member ('DOMAIN\'+$userinfo
     ['samAccountName']) }
  If ($userinfo['Marketing]) { Add-
    QADGroupMember
    -identity 'DOMAIN\Marketing'
    -member ('DOMAIN\'+$userinfo
     ['samAccountName']) }
}
And so forth. Keep in mind that we're still using the Quest AD Cmdlets, which are free from www.quest.com/powershell.
All that repetitive code bugs me. I think what I'd like to do instead is create an array of possible group names and then enumerate through that array. That way, adding and removing groups in the future doesn't require me to copy and paste code—I'd just have to modify the array. So now the function looks like this:
Function AddToGroup {
  Param($userinfo)
    $groups = 'Sales','Marketing','East'
  Foreach ($group in $groups) {
    If ($userinfo[$group]) {
      Add-QADGroupMember
        -identity 'DOMAIN\'+$group
        -member ('DOMAIN\'+$userinfo
         ['samAccountName']) }
}
}
That's a bit leaner and cleaner, and it will easily accommodate future changes.

Additional Attributes
Finally, I need to deal with any additional attributes that I want to set. Here I am going to use the same approach: Create an array of allowable attribute names and then enumerate through that array. In this case, I'm going to allow for a situation where one or more attributes are blank for a given user, like so:
Function UpdateAttributes {
  Param($userinfo)
    $attribs = 'postalCode','title',
      'physicalDeliveryOfficeName'
  Foreach ($attrib in $attribs) {
    If ($userinfo[$attrib] –ne '') {
      $update = @{$attrib=$userinfo
       ['attrib']}
    Set-QADUser
      -identity 'DOMAIN\'+$userinfo
       ['samAccountName']
      -objectAttributes $update
    }
  }
}
Because the Set-QADUser cmdlet actually accepts a hashtable for the –objectAttributes parameter, you might think I can just feed it the $userinfo hashtable. However, I have things in there, such as group names, that aren't Active Directory attribute names so that trick wouldn't work. Furthermore, I don't want to send any empty attributes, so I want the ability—as I've done with this function—to run through all the attributes first and then update only the attributes that actually have values in the original CSV file.

Taking It Further
There are obviously a lot of things you could do to expand this script. Here are a few ideas that come to mind:
  • Instead of retrieving information from a CSV file, you could get it from a database (such as a personnel database). You just need to replace the Import function, as discussed in Part 1 of this series.
  • You could perform other provisioning tasks. You just need to add more functions to the master Provision function. Remember to keep each major task in its own subordinate function and you'll make things easier to write, debug, and maintain.
  • You could write out a log file of provisioning activity. For example, you can use Out-File to append text to a log file within each of the subordinate functions.
  • Using the techniques covered in this series, you can write a reprovisioning script to reassign groups, permissions, and Active Directory attributes when a user moves to a different position within the company.
Windows PowerShell Q&A

QWhat Microsoft products can be managed with Windows PowerShell cmdlets?

ACurrently, the list includes Exchange Server 2007, System Center Virtual Machine Manager, System Center Operations Manager, and System Center Data Protection Manager. SQL Server 2008 also includes some Windows PowerShell cmdlets, although they're largely geared toward enabling the execution of Transact-SQL statements (Transact-SQL being the native scripting language in SQL Server).
But there's no reason to stop there. You can also manage Active Directory using a free set of cmdlets from www.quest.com/powershell; cmdlets for Internet Information Services (IIS) 7.0 are under development; and there are even cmdlets for non-Microsoft products, such as VMWare, IBM WebSphere MQ, and more. You can also use existing command-line tools to manage Windows components, such as IIS, DNS, DHCP, Windows Firewall, and more.

Some Final Thoughts
I hope you'll find the completed script and all its subordinate functions to be, at the very least, a useful starting point in creating your own provisioning scripts. By way of conclusion, I want to offer a few comments on some of the decisions I've made throughout.
I chose to use hashtables for the user information rather than custom objects because hashtables provide a somewhat easier way to enumerate through a set of properties when I don't know in advance what all those properties will be. In other words, hashtables are a bit easier to use when I know that I will need to add more attributes, and when I might not use the same attributes every time I run the script.
I chose to use Cacls.exe to modify folder permissions because, as I described in Part 3 of this series, it's just easier than the shell's *-ACL cmdlets. Also, using Cacls.exe allowed me to demonstrate one way that you can parameterize external command-line tools by using shell variables.
I chose to use Quest's Active Directory cmdlets rather than ADSI because the cmdlets offer what I feel is a more native approach to manipulating the directory in Windows PowerShell. ADSI can be a pain to work with, and it forces you to use a more procedural programming style over the command-centric style I was able to use in this example.
Whether or not you choose to make the same decisions for your scripts, I hope you'll find these techniques and different approaches useful in solving some of your administrative automation challenges. I also invite you to share your suggestions and improvements with me. On my Web site, ConcentratedTech.com, you'll find a Contact page where you're welcome to send me an e-mail. And please be sure to let me know if I can share your comments in a future issue of TechNet Magazine.

Don Jones is a co-founder of Concentrated Technology, where he blogs weekly about Windows PowerShell, SQL Server, App-V, and other topics. Contact him through his Web site.


Geek of all Trades The Benefits of Big, Cheap Disks
Greg Shields

Contents
Incorporating WSB
Multiple Backup Disks
Managing Restores
Backup on a Budget
the power of storage area networks (SANs), we could slice and dice all that capacity among any of the servers in our datacenter. Our only limit was our imagination—and the available fiber channel cabling.
Of course, at that time, a terabyte of online storage required not one but three refrigerator-size SAN enclosures. Two contained the hard drives, with the third reserved for just the routing components. It also required a ridiculous amount of power and its own Computer Room Air Conditioners, giant ethylene-glycol-filled devices lovingly nicknamed "CRAC units."
Using WSB to Back Up Hyper-V Virtual Machines
WSB can back up virtual machines hosted atop a Windows Server 2008 computer running the Hyper-V role. Virtual machines in Hyper-V that leverage high availability with Windows Failover Clustering tend to operate with a single virtual machine per volume, making WSB's volume-centric backups useful for per-VM protection.
Using WSB to back up virtual machines requires enabling the VSS Writer for Hyper-V. To do this, create a REG_SZ key named Application Identifier with a value of Hyper-V in the registry at the location
HKEY_LOCAL_MACHINE\Software\Microsoft
  \Windows NT\CurrentVersion\WindowsServerBackup\
  Application Support\{66841CD4-6DED-4F4B-8F17-
  FD23F8DDC3DE}.
Be aware that virtual machines that contain dynamic disks cannot be backed up by WSB. Virtual machines that do not include support for VSS backups—such as Windows 2000, Windows XP, or virtual machines that do not have Integration Components installed—cannot be backed up using WSB. Lastly, be cautious with the use of snapshots, as virtual machines that contain two or more snapshots will fail to restore.
You can find more information in the article "How to back up Hyper-V virtual machines from the parent partition on a Windows Server 2008–based computer by using Windows Server Backup."
I was reminded of this recently while thinking about the Microsoft Windows Server Backup (WSB) utility and how backup has changed. WSB is found in Windows Server 2008 and it finally brings reliable whole-system restores to small businesses and environments—right within the server OS. The utility introduces some new—and not always obvious—ways of architecting backups for your Windows environment.
Unfortunately, since its introduction, this backup tool has also come in for some negative press: its files are incompatible with those from its NTbackup predecessor; backups are scoped at the volume level only; system-state backups are much larger than before; and backup tapes are not supported. But these items actually reflect deliberate design decisions based on the way storage has evolved.
Today, disk-to-disk backups make far more sense than disk-to-tape for the jack-of-all-trades IT professional, due primarily to the cost and ease of use of today's USB-connected external disks. Since my first sight of that terabyte, the size and cost of disk storage has shrunk to the level of absurdity. Only a few short years later, we in IT saw terabyte capacity reduced from refrigerator-size to merely being striped across a few hard disks in a single server. Now, even consumers can buy that same terabyte, bundled into a 5" x 6" x 2" form factor for around a hundred bucks. Disks these days are indeed huge and cheap.
Which brings us to why forcing backups to occur at the volume level makes sense—it allows entire computers to be quickly backed up with a few mouse clicks. Volume Shadow Copy Service (VSS) integration with the residing OS means that no files are missed during a backup pass. Integrating backups with Microsoft's WinPE enables a dismounted OS to assuredly return a failed computer back to working order—without the multiple install-the-operating-system-and-then-restore-over-the-top steps of yesterOS. Each of these new capabilities works best when backups are accomplished from server hard drive to USB disk, rather than to traditional tape devices. And, best of all, each of these is available for no added cost with your existing server OS.

Incorporating WSB
To add WSB functionality to an existing server, you must install it through Server Manager. There are three items to install: Windows Server Backup, Command-line Tools, and Windows PowerShell, which are available in the Add Features Wizard in Server Manager.
After the installation, you will interface with WSB through Server Manager's Storage node, where you'll also find the Recover option for restoring files and a selection for configuring performance settings (see Figure 1). You can choose to perform either a one-time backup of a server or a scheduled backup and, when choosing, you'll notice two of WSB's "new" behaviors. First, if you select to Backup Once, you'll be presented with a list of volumes to back up. You'll immediately notice that WSB will not back up individual files and folders, only entire volumes.
Figure 1 The Windows Server Backup Console
If you continue through the Backup Once wizard, you'll be asked for the local storage drive or remote folder to use as a backup target, as well as an advanced option whose setting determines whether you want application settings to be retained and the VSS archive to be kept or updated. When you complete the wizard, the backup will begin, starting first with a VSS snapshot to ensure the successful backup of any locked files.
These single-instance backups are considered to be the solution for one-time needs, and thus you are allowed to target backups to a remote folder across the network. This is not possible, however, when backups are configured to operate on a schedule. In that case, you'll no longer find a GUI option for targeting backups to a remote folder (see Figure 2). You are permitted only to back up the server to an attached disk that is not already an OS or application volume.
Figure 2 Scheduled backups in WSB are scoped to the volume, not the individual file or folder
The reason for this new behavior lies in how WSB treats the disks you attach for regular backups. With scheduled backups, WSB thinks of its targets as dedicated devices for regular use. Any target disk is automatically formatted as the schedule is created. Once formatted and targeted for a scheduled backup, that disk is no longer exposed in Windows Explorer. This prevents the accidental overwrite of data by a careless administrator and effectively locks the disk for use by WSB alone. The disk is treated essentially as tape devices were, but disks offer many advantages over tapes, particularly for the small business or organization. These days, external drives are cheaper, faster, and easier to use than tape. And that's not all.
First, WSB's disk-based solution enjoys a significant reduction in complexity with incremental backups. WSB's integration with VSS means that backups start with the creation of a snapshot of the source volume. Later backups are completed through block-level incrementals that require only the processing of changes since the last backup. This is much different from tape-based backup, where incrementals are stored elsewhere on the backup tape, or on completely different tapes, requiring increasing quantities of storage as well as increasing amounts of time to complete a restore. Finally, once you set up a scheduled backup to a dedicated disk, WSB will automatically keep as many backup versions as possible for the storage and does not require you to intervene when the backup target is full. Instead, it automatically deletes the oldest backup to make space for the newer backups.

Multiple Backup Disks
WSB lets you create extra backup disks by allowing additional disks to be plugged in prior to a server's scheduled backup time. This means you can rotate one set of backup drives off-site for protective restore in the case of a disaster or electrical malfunction.
To add another dedicated disk to an existing server backup, first configure and run a scheduled backup to an attached and dedicated device. When it's done, click the Backup Schedule link again to start the wizard. When prompted, select Modify Backup and navigate to the screen titled Add or remove backup disks. Selecting Add more disks brings up the screen shown in Figure 3, where you can select the new disk on which to target the backup. As before, once the disk is targeted for a backup, it will be formatted and removed from visibility in Windows Explorer.
Figure 3 You can add a new dedicated disk to WSB for off-site rotation
Completing this step enables the secondary disk to be plugged into your server whenever you require a backup to be rotated off-site for storage. WSB will label the disk—much as tapes were traditionally labeled—with the correct markings (see Figure 4) so that the disk can be later identified. Your organization may also find reasons other than off-site storage to create backups to multiple disks. You may want one set to remain plugged into its corresponding servers to support individual file restores, or you may want one to be used for long-term archiving.
Figure 4 WSB’s Label for a Dedicated USB Disk

Managing Restores
As noted earlier, WSB does not perform backups of individual files. Whether you attempt a one-time or scheduled backup, WSB scopes that backup at the individual volume. Though this might initially seem like a limitation, it actually makes sense in terms of protecting the backup as a whole. And you can restore individual files.
Once backups have been configured for each of the servers in your environment, those backups remain online and available at all times, as long as the USB disk remains connected to the server. Restoring individual files from that USB disk's backup involves remotely connecting to that server's WSB console and selecting the Recover option. Then you simply select the date of the backup and the files to be restored. User files can be restored to their original location or alternate locations, and you can either overwrite existing files or create additional copies. Completing the wizard accomplishes the file restore from the attached USB drive.
The entire process is quick and without the traditional overhead associated with searching for the right tape, loading it into the tape device, scanning for the needed file, and ultimately restoring it. No longer will you be searching across multiple tapes, looking for the file you want to restore. Unlike tape backups, which can spread individual files across any number of tapes and require accurate cataloging and storage of the entire tape set, disk-based backups store the entire contents of the backup and all its incrementals in one place. If you've ever been unsuccessful in completing an individual file backup due to the loss or failure of a single tape in a set, the reliability of this feature will be much appreciated.
Disk-based backup and the use of VSS also improve the entire-server restoration process. Using volumes as the boundary of backups means that an entire server can always be successfully restored in the case of a loss or disaster. You can ensure this capability by making sure to back up each of the operating system and application volumes on your server. One-time backups also include a checkbox marked "Enable system recovery" that ensures all the right drives are checked.
Restoring a failed server with WSB is accomplished in a single step from the server's bare metal. It no longer requires the installation of a fresh operating system on top of which your restore will layer the backup's files. To restore a failed server, boot the server from the Windows Server 2008 DVD media and select Repair your Computer. Follow the prompts to begin a Complete PC Restore, selecting the USB disk as the source for the server's backup files. You can find the exact steps to do this at the Ask the Core Team blog.

Backup on a Budget
Admittedly, WSB may not be the right choice for large organizations. As shipped, it can't back up Exchange 2007 stores, although some third parties have developed integrations that can make this possible. While WSB can back up Hyper-V virtual machines, it can't back up connected disks within those virtual machines. Backing up just the system state of a domain controller requires a substantially larger amount of disk space than in previous versions.
On the other hand, many aspects of WSB make it extremely useful for the small-environment administrator. When used as part of a dedicated backup solution with lots of big, cheap USB disks, WSB can be an effective solution that fits the small environment budget. For more information, visit the Backup and Recovery Overview page of the Windows Server TechCenter.

Greg Shields, MVP, is a partner at Concentrated Technology. Get more of Greg's Jack-of-All-Trades tips and tricks at www.ConcentratedTech.com.


Hey, Scripting Guy! Creating A Self-Documenting Script
The Microsoft Scripting Guys

To describe the incredible depth of blue brushed broadly across the sky down here in Buenos Aires, special color names must be used. Cobalt blue, deep-sea blue, midnight blue, holy-mackerel blue—none of these even begin to portray the actual color seen in the canopy covering the old city. Wide sidewalks lined by centuries-old buildings are dotted with little tables. I sit in an empty metal chair. A perfumed breeze drifts lazily by carrying hints of cinnamon, cardamom, and thyme. Seagulls with wings outstretched surf the updrafts with enough skill, grace, and poise to engender envy in the Big Kahuna himself. The café con leche seems especially apropos and is strong enough to demand its own attention against the competition from the sights and sounds of downtown Buenos Aires. I am in town to talk to Microsoft Premier Customers about using VBScript to help manage their networks. I extract a folded map from my camera bag and proceed to plan my route from Calle Florida to the Obelisk on Nueve de Julio Avenue.
I find my location on the map. I am near the Galerias Pacifico shopping gallery. I turn the map over several times as I attempt to orient it with my location in time and space. About this time, a tall, distinguished-looking man in a three-piece dark suit appears and stands just out of the sunlight. His shadow, dancing playfully on the map, causes me to glance up. I stand as he says, "Buenos dias, se˜nor."
"Buenos dias," I dutifully reply.
"Ah, you are an American perhaps?" he replies with a slightly British accent.
I am somewhat taken aback, and hesitantly reply in the affirmative.
"Is this your first time in our fair city?" he inquires.
Once again, I am reticent but say "yes."
"Then I may offer my humble service to you. What is it that you seek?" he politely asks.
I tell him, and he proceeds to give me directions. After that we chat for a few minutes.
"Wow!" I think to myself. "A self-documenting city! No need for maps here. The people are so friendly you barely have time to look at a map before someone is there to assist you."
That was actually a few years ago. Today, at my house in North Carolina, the weather outside is so lovely it reminds me of my visit to Buenos Aires—and of how nice a self-documenting facility can be. We can have our scripts perform a similar service for us, if we take the time to plan ahead. In this article, we will see how to develop a script that will gather comments from other scripts that follow a certain pattern. We can modify the pattern to meet our own needs and can expand the technique beyond creating documentation for scripts.
Let's take a look at a script that has some documentation carefully placed in it. In the GetSetIEStartPage.ps1 script in Figure 1 (which was written for the Hey, Scripting Guy! article, "How Can I Change My Internet Explorer Home Page?", comments are placed in here-strings that are then assigned to a variable named $Comment. A here-string is delineated by a beginning tag of an at sign and a quotation mark (@" ) and a closing tag of a quotation mark and an at sign ( "@). There are three comment blocks we are interested in obtaining. The first contains normal script header information: the title, author, date, keywords, and comments on the script itself. The second and third comment blocks are specifically related to the two main functions contained in the script.
Figure 1 GetSetIEStartPage.ps1
Param([switch]$get,[switch]$set,$computer="localhost")
$Comment = @"
NAME: GetSetIEStartPage.ps1
AUTHOR: ed wilson, Microsoft
DATE: 1/5/2009

KEYWORDS: stdregprov, ie, [wmiclass] type accelerator,
Hey Scripting Guy
COMMENTS: This script uses the wmiclass type accelerator
and the stdregprov to get the ie start pages and to set the
ie start pages. Using ie 7 or better you can have multiple
start pages

"@ #end comment

Function Get-ieStartPage()
{
$Comment = @"
FUNCTION: Get-ieStartPage
Is used to retrieve the current settings for Internet Explorer 7 and greater.
The value of $hkcu is set to a constant value from the SDK that points
to the Hkey_Current_User. Two methods are used to read
from the registry because the start page is single valued and
the second start pages key is multi-valued.

"@ #end comment
 $hkcu = 2147483649
 $key = "Software\Microsoft\Internet Explorer\Main"
 $property = "Start Page"
 $property2 = "Secondary Start Pages"
 $wmi = [wmiclass]"\\$computer\root\default:stdRegProv"
 ($wmi.GetStringValue($hkcu,$key,$property)).sValue
 ($wmi.GetMultiStringValue($hkcu,$key, $property2)).sValue
} #end Get-ieStartPage

Function Set-ieStartPage()
{
$Comment = @"
FUNCTION: Set-ieStartPage
Allows you to configure one or more home pages for IE 7 and greater.
The $aryValues and the $Value variables hold the various home pages.
Specify the complete URL ex: "http://www.ScriptingGuys.Com" make sure
to include the quotation marks around each URL.

"@ #end comment
  $hkcu = 2147483649
  $key = "Software\Microsoft\Internet Explorer\Main"
  $property = "Start Page"
  $property2 = "Secondary Start Pages"
  $value = "http://www.microsoft.com/technet/scriptcenter/default.mspx"
  $aryValues = "http://social.technet.microsoft.com/Forums/en/ITCG/threads/",
  "http://www.microsoft.com/technet/scriptcenter/resources/qanda/all.mspx"
  $wmi = [wmiclass]"\\$computer\root\default:stdRegProv"
  $rtn = $wmi.SetStringValue($hkcu,$key,$property,$value)
  $rtn2 = $wmi.SetMultiStringValue($hkcu,$key,$property2,$aryValues)
  "Setting $property returned $($rtn.returnvalue)"
  "Setting $property2 returned $($rtn2.returnvalue)"
} #end Set-ieStartPage

# *** entry point to script
if($get) {Get-ieStartpage}
if($set){Set-ieStartPage}
To write the comments from the source file to another document, we need to open the original script, search for the comments, and then write the appropriate text to a new file. Sounds simple enough. Oh, yeah, we also need a name for our new script. Let's call it GetCommentsFromScript.ps1.
The GetCommentsFromScript.ps1 script, shown in Figure 2, begins with a Param statement, which is used to allow us to provide information to the script at run time. Here is the Param statement.
Figure 2 GetCommentsFromScript.ps1
Param($Script= $(throw "The path to a script is required."))
Function Get-FileName($Script)
{
 $OutPutPath = [io.path]::GetTempPath()
 Join-Path -path $OutPutPath -child "$(Split-Path $script-leaf).txt"
} #end Get-FileName

Function Remove-OutPutFile($OutPutFile)
{
  if(Test-Path -path $OutPutFile) { Remove-    Item $OutPutFile | Out-Null }
} #end Remove-OutPutFile

Function Get-Comments($Script,$OutPutFile)
{
 Get-Content -path $Script |
 Foreach-Object `
  {
    If($_ -match '^\$comment\s?=\s?@"')
     {
      $beginComment = $True
     } #end if match @"
   If($_ -match '"@')
     {
      $beginComment = $False
     } #end if match "@
   If($beginComment -AND $_ -notmatch '@"')
     {
      $_ | Out-File -FilePath $OutPutFile -append
     } # end if beginComment
  } #end Foreach
} #end Get-Comments

Function Get-OutPutFile($OutPutFile)
{
 Notepad $OutPutFile
} #end Get-OutPutFile

# *** Entry point to script ***
$OutPutFile = Get-FileName($script)
Remove-OutPutFile($OutPutFile)
Get-Comments -script $script -outputfile $OutPutFile
Get-OutPutFile($OutPutFile)vw

Param($Script= $(throw "The path to a script   is required."))
The advantage of using a command-line parameter is that we don't need to open the script and edit it to provide the path to the script whose comments we are going to copy. We are making this parameter mandatory by assigning a default value to the $Script variable. The default value uses the throw command to raise an error, and this means that the script will always raise an error when run, unless we supply a value for the –script parameter.
Let's digress for a minute and take a look at how the throw statement is used in the DemoThrow.ps1 script in Figure 3. To get past the error that is raised by the throw statement in the Set-Error function, we first need to set the $errorActionPreference variable to SilentlyContinue. This prevents the error from being displayed and lets the script continue. It is the same as the On Error Resume Next setting from VBScript. The If statement is used to evaluate the $value variable. If there is a match, the throw statement is encountered and the exception is thrown. To evaluate the error, we use the Get-ErrorDetails function. The first thing that takes place is the display of the error count, which will have been incremented by 1 because of the error raised by the throw statement. We then take the first error (the error with the index value of 0 is always the most recent one) and send the error object to the Format-List cmdlet. We choose all the properties. The invocation information, however, is returned as an object, and we therefore need to query that object directly. We do this by accessing the invocation object via the InvocationInfo property of the error object. The resulting error information is shown in Figure 4.
Figure 3 DemoThrow.ps1
Function Set-Error
{
 $errorActionPreference = "SilentlyContinue"
 "Before the throw statement: $($error.count) errors"
 $value = "bad"
 If ($value -eq "bad")
   { throw "The value is bad" }
} #end Set-Error

Function Get-ErrorDetails
{
 "After the throw statement: $($error.count) errors"
 "Error details:"
 $error[0] | Format-List -Property *
 "Invocation information:"
 $error[0].InvocationInfo
} #end Get-ErrorDetails

# *** Entry Point to Script
Set-Error

Get-ErrorDetails
Figure 4 The Throw statement is used to raise an error
Now let's get back to our main script, GetCommentsFromScript.ps1. We need a function that will create a filename for the new text document that will contain all of the comments gleaned from the script. To do this, we use the function keyword and follow it with the name for the function. We will call our function Get-FileName in keeping with the Windows PowerShell verb-noun naming convention. Get-FileName will take a single input parameter, the path to the script to be analyzed, which will be held in the $Script variable inside the function. Here's the entry to the Get-FileName function:
Function Get-FileName($Script)
{
Next we obtain the path to the temporary folder on the local computer. There are many ways to do this, including using the environmental Windows PowerShell drive. However, we decided to use the static GetTempPath method from the Io.Path .NET Framework class. The GetTempPath method returns the path to the temporary folder, which is where we will store the newly created text file. We hold the temporary folder path in the $OutPutPath variable as shown here:
$OutPutPath = [io.path]::GetTempPath()
We decide to name our new text file after the name of the script. To do this, we need to separate the script name from the path the script is stored in. We use the Split-Path cmdlet to perform this surgery. The –leaf parameter tells the cmdlet to return the script name. If we had wanted the directory path that contains the script, we would have used the –parent parameter. We put the Split-Path command inside a pair of parenthesis because we want that operation to occur first, then we place a dollar sign in front of the parentheses to create a sub-expression that will execute the code and return the name of the script. We could use.ps1 as the extension for our text file, but that could be confusing because it is the extension for a script. We therefore simply add a .txt extension to the returned filename and place the whole thing in a pair of quotation marks. Now we use the Join-Path cmdlet to create a new path to our output file. The new path is made up of the temporary folder, stored in the $OutPutPath variable, and the filename we created using Split-Path. We could have used string manipulation and concatenation to create the new file path, but it is much more reliable to use Join-Path and Split-Path to perform these kinds of operations. Here's what the code looks like:
Join-Path -path $OutPutPath -child "$(Split-Path $script-leaf).txt"
} #end Get-FileName
We now need to decide how we are going to handle duplicate files. We could prompt the user by saying a duplicate file exists, using code like this:
$Response = Read-Host -Prompt "$OutPutFile already exists. Do you wish to delete it <y / n>?"
if($Response -eq "y")
    { Remove-Item $OutPutFile | Out-Null }
ELSE { "Exiting now." ; exit }
We could implement some kind of naming algorithm that makes a backup of the existing file by renaming it with a .old extension. If we did this, the code would look something like the following:
if(Test-Path -path "$OutPutFile.old") { Remove-Item-Path "$OutPutFile.old" }
Rename-Item -path $OutPutFile -newname "$(Split-Path $OutPutFile -leaf).old"
Or we could simply delete the existing file, which is what I chose to do. The action we want to perform takes place in the Remove-OutPutFile function, which we create by using the Function keyword and specifying the name of the function. We use $OutPutFile to supply the input to the function, as shown here:
Function Remove-OutPutFile($OutPutFile)
{
To determine if the file exists, we use the Test-Path cmdlet and supply the string contained in the $OutPutFile variable to the path parameter. The Test-Path cmdlet returns only a True or a False, depending on whether a file is found. This means we can use the If statement to evaluate the existence of the file. If the file is found, we perform the action in the script block. If the file is not found, the script block is not executed. You can see here that the first command does not find the file, and False is returned. In the second command, the script block is not executed because the file can't be located:
PS C:\> Test-Path c:\missingfile.txt
False
PS C:\> if(Test-Path c:\missingfile.txt){"found file"}
PS C:\>
Inside the Remove-OutPutFile function, the If statement is used to determine if the file referenced by the $OutPutFile already exists. If it does, it is deleted by using the Remove-Item cmdlet. The information that is normally returned when a file is deleted is pipelined to the Out-Null cmdlet, providing for silent operation. The code is shown here:
if(Test-Path -path $OutPutFile) { Remove-Item $OutPutFile | Out-Null }
} #end Remove-OutPutFile
After we have created the name for the output file and deleted any previous output files that may be lying around, it is time to retrieve the comments from the script. To do this, we create the Get-Comments function and pass it both the $Script variable and the $OutPutFile variable, as shown here:
Function Get-Comments($Script,$OutPutFile)
{
Now we read the text of the script using the Get-Content cmdlet, to which we pass the path to the script. When we use Get-Content to read a file, the file is read one line at a time and each line is passed along the pipeline. If we were to store the result in a variable, we would have an array. We can treat the variable $a like any other array, including obtaining the number of elements in the array via the Length property and indexing directly into the array as shown here:
PS C:\fso> $a = Get-Content -Path C:\fso\  GetSetieStartPage.ps1
PS C:\fso> $a.Length
62
PS C:\fso> $a[32]
($wmi.GetMultiStringValue($hkcu,$key,   $property2)).sValue
Here's the line that reads the input script and sends it along the pipeline:
Get-Content -path $Script |
Next we need to look inside each line to see if it belongs to the comment block. To examine each line within a pipeline, we use the ForEach-Object cmdlet, which is similar to a ForEach…Next statement in that it lets us work with an individual object from within a collection, one at a time. The back tick character (`) is used to continue the command to the next line. The action we want to perform on each object as it comes across the pipeline is contained inside a script block, which is delineated with a set of curly brackets (also called braces). This part of the Get-Content function is seen here:
ForEach-Object `
  {
When we are inside the ForEach-Object cmdlet process block, we want to examine the line of text. To do this, we use the If statement. The $_ automatic variable is used to represent the current line that is on the pipeline. We use the –match operator to perform a regular expression pattern match against the line of text. The –match operator returns a Boolean True or False in response to the pattern The pattern begins on the right side of the –match operator. This section of the script is shown here:
PS C:\fso> '$Comment = @"' -match '^\$comment\s?=\s?@"'
True
The regular expression pattern we are using is composed of a number of special characters:
^—Match at the beginning
\—Escape character so the $ sign is treated as a literal character and not the special character used in regular expressions
$comment—Literal characters
\s?—Zero or more white space characters
=—Literal character
\s?—Zero or more white space characters
@"—Literal characters
The section of code that examines the current line of text on the pipeline is shown here:
If($_ -match '^\$comment\s?=\s?@"')
We create a variable named $beginComment that is used to mark the beginning of the comment block. If we make it past the –match statement, we have found the beginning of the comment block. We set the variable equal to $True as seen here:
{
  $beginComment = $True
} #end if match @"
Next we check to see if we are at the end of the comment block. To do this, we again use the –match operator. This time we look for the "@ character sequence, which is used to close a here-string. If we find it, we set the $beginComment variable to False:
If($_ -match '"@')
     {
      $beginComment = $False
     } #end if match "@
We have made it past the first two If statements: the first one identifies the beginning of the here-string, and the second locates the end of the here-string. Now we want to grab the text to be written to our comment file. To do this, we want the $beginComment variable to be set to True. We also want to make sure that we do not see the at sign quotation mark (@" ) character on the line because it would mean the end of the here-string. To make this determination, we use a compound If statement:
If($beginComment -AND $_ -notmatch '@"')
     {
Now we write the text to the output file. To do this, we use the $_ automatic variable, which represents the current line of text; we pipeline it to the Out-File cmdlet. The Out-File cmdlet receives the $OutPutFile variable, which contains the path to the comment file. We use the –append parameter to specify that we want to gather all the comments from the script in the comment file. If we did not use the append parameter, the text file would contain only the last comment because, by default, the Out-File cmdlet would overwrite its contents. We then close out all the curly brackets. I consider it a best practice to add a comment after each closing curly bracket that indicates the purpose of the curly bracket. This makes the script much easier to read, as well as easier to troubleshoot and to maintain:
$_ | Out-File -FilePath $OutPutFile -append
     } # end if beginComment
  } #end ForEach
} #end Get-Comments
We now create a function called Get-OutPutFile that will open the output file for us to read. Because the temporary folder is not easy to find and we had the path to the file in the $OutPutFile variable, it makes sense to use the script to open the output file. The Get-OutPutFile function receives a single input variable called $OutPutFile, which contains the path to the comment file we wish to open. When we call the Get-OutPutFile function, we will pass $OutPutFile to it. We could pass any value we wish to the Get-OutPutFile function, and inside the function the value would be referred to by the $OutPutFile variable. We can even pass a string directly (without using quotation marks around the string) to the function:
Function Get-OutPutFile($OutPutFile)
{
 Notepad $OutPutFile
} #end Get-OutPutFile

Get-OutPutFile -outputfile C:\fso\GetSetieStartPage.ps1
In general, when writing a script, if you are going to gather up something to pass to a function, it is a good idea to encase the data in the same variable name that will be used both inside and outside the function. This follows one of our best practices for script development: "Don't mess with the worker section of the script." In this example, when we call the function we are "doing work." To change this in future scripts would require that we edit the string literal value. By placing the string in a variable, we can easily edit the value of the variable. We are, in fact, set up to provide the value of the variable via the command line or by means of something done in another function. Whenever possible, you should avoid placing string literal values directly in the script. In the code that follows, we use a variable to hold the path to the file that will be passed to the Get-OutPutFile function:
Function Get-OutPutFile($OutPutFile)
{
 Notepad $OutPutFile
} #end Get-OutPutFile

$OutPutFile = "C:\fso\GetSetieStartPage.ps1"
Get-OutPutFile -outputfile $OutPutFile
The complete Get-OutPutFile function is shown here:
Function Get-OutPutFile($OutPutFile)
{
 Notepad $OutPutFile
} #end Get-OutPutFile
Instead of typing in a string literal for the path to the output file, the $OutPutFile variable receives the path that is created by the Get-FileName function. The Get-FileName function receives the path to the script that contains the comments to be extracted. The path to this script comes in via the command-line parameter. When a function has a single input parameter, you can pass it to the function by using a set of parentheses. If, on the other hand, the function uses two or more input parameters, you must use the –parameter name syntax:
$OutPutFile = Get-FileName($script)
Next we call the Remove-OutPutFile function (discussed earlier) and pass it the path to the OutPutFile that is contained in the $OutPutFile variable:
Remove-OutPutFile($OutPutFile)
When we are assured of the name of our output file, we call the Get-Comments function to retrieve comments from the script whose path is indicated by the $script variable. The comments will be written to the output file referenced by the $OutPutFile variable. This line of code is seen here:
Get-Comments -script $script -outputfile $OutPutFile
When the comments have all been written to the output file, we finally call the Get-OutPutFile function and pass it the path contained in the $OutPutFile variable. If you don't want the comment file to be opened, you can easily comment the line out of your script, or simply delete it and the Get-OutPutFile function itself from your script. If you are interested in reviewing each file before saving it, leave the line of code in place:
Get-OutPutFile($OutPutFile)
When the GetCommentsFromScript.ps1 script runs, no confirmation message is shown on your screen. The only confirmation that the script worked is the presence of the newly created text file displayed in Notepad, as shown in Figure 5.
Figure 5 The New Text File Displayed in Notepad
The GetCommentsFromScript.ps1 script can be easily adapted to your own way of writing scripts or even for gathering other kinds of documentation from text-based log files. All you need to do is modify the regular expression pattern that is used to mark the beginning and the end of the portions of text you are interested in collecting. We hope you enjoy the script, and invite you to join us on the Script Center, where we publish a new Hey, Scripting Guy! article every weekday.

Ed Wilson, a well-known scripting expert, is the author of eight books, including Windows PowerShell Scripting Guide (Microsoft Press, 2008) and Microsoft Windows PowerShell Step by Step (Microsoft Press, 2007). Ed holds more than 20 industry certifications, including Microsoft Certified Systems Engineer (MCSE) and Certified Information Systems Security Professional (CISSP). In his spare time, he enjoys woodworking, underwater photography, and scuba diving. And tea.

Craig Liebendorfer is a wordsmith and longtime Microsoft Web editor. Craig still can't believe there's a job that pays him to work with words every day. One of his favorite things is irreverent humor, so he should fit right in here. He considers his greatest accomplishment in life to be his magnificent daughter.


Security Watch Thoughts on Identity, Part 1
Jesper M. Johansson

Contents
What Is an Identity?
The Identity Problem
Authenticating Identity
Stakeholders in Identity
Using Identities
How Many Identities Do You Have?
Principles of Identity
1. The identity provider is at least as sensitive as the most sensitive relying party.
2. Permit the enterprise to protect its customer relationships and to own and control information that it deems as business confidential.
A concept many of us thought we had a good handle on keeps coming back up these days: identity. With unrelenting speed, we are seeing various new identity projects, most in the form of some kind of distributed identity systems and/or something that will finally replace passwords. These systems all have one basic goal—to replace the plethora of username-and-password combinations we all have to remember with a single identity. Instead, these new systems would offer one identity that gets us access to everything.
As I look at the 127,128 passwords I have stored in my Password Safe, I think that actually sounds quite appealing. And each time I go to any Microsoft.com property and am asked, again, for my Windows Live ID and password, I realize we also have a very long way to go before reaching a world of single sign on (SSO).
Identity, however, is a nebulous concept and an oft-misunderstood one. In this article, I lay out some thoughts on identity and principles that an identity system has to meet. I do not have enough gray hair yet to pretend that these are fundamental laws that these systems must obey. However, I do believe that any identity system that fails to abide by these principles will fail in the court of consumer opinion, as well as in the one of enterprise support and deployment. An identity system, as theoretically pleasing as it may be, must ultimately provide value to a business and to a user if it is going to succeed.

What Is an Identity?
An identity, simply put, is an abstract representation of an entity in a computer system. Philosophy defines it as the "sameness" of two things. Two (or more) things are identical if they are the same. Yes, that would qualify as a highly circular definition. So, instead, let's say that identity states that an entity is definable and recognizable. Maybe this is simpler:
(P=Q)→(P=Q)
That is the basic definition of identity in symbolic logic. P is equal to Q if P is the same thing as Q. Restated slightly differently, if Peter is Peter then clearly Peter is Peter. Now all we need is a way for Peter to prove that he (or she?) is in fact Peter.
Methods of proving identity are highly interesting in digital systems. In the digital world, we have a slightly different definition of identity from that in symbolic logic. In his seminal paper "The Laws of Identity," Kim Cameron defines digital identity as "a set of claims made by one digital subject about itself or another digital subject."
That is a different definition than the pure logical one. The pure logical definition is simply who you are. In the digital world, identity is rooted in the claims you choose to present. In other words, a digital identity is not who you are, but rather, who you choose to be. In a digital identity system, as long as Peter, or whoever claims to be Peter, can provide an acceptable proof of that claim (meaning that the can authenticate the claim), we must accept Peter's identity. The claims Cameron talks about are essentially a manifestation of identity. Those are the authenticators Peter, or whoever needs to be Peter, is presenting.
Therein lies the first identity problem. Digital identity is not the same as an ontological (in the philosophical sense) identity—a representation of a real world entity that actually exists. Digital identity is ephemeral. Digital identity is incomplete. Digital identity is optional. And, most interesting of all, a single physical entity can present many different sets of claims and, therefore, have many different digital identities. Digital identities are quite possibly most useful when they do not have a 1:1 correspondence to an ontological entity.

The Identity Problem
A claims-based digital identity concept causes many interesting problems. First is the fact that an identity has a price. You can buy a whole new identity for only a few thousand dollars. For that money you get a functioning social security number along with a valid address that you can change. Driver's licenses, credit cards, passports, and the rest can be had with that. If you want an identity with a verifiable personal history, that will probably cost you quite a bit more.
Some people, including myself, claim that this means that ontological identity is meaningless as it relates to digital identity systems. Personally, I find true identity (or at least, correctness of identity) to be irrelevant and uninteresting when dealing with it in the context of a digital system. For the vast majority of applications, identity merely means that you can lie consistently. If you are able to present the same lies to me today that you did two weeks ago, I am going to assume you are the same person/entity/computer and give you whatever access I granted to you last time.
Given that true, ontological identity is not only ephemeral, but irrelevant to most applications, I don't care about binding you to a real-world entity. I just need correlation, not true physical identity, for most purposes. Any application that relies on binding identity to a physical and immutable entity is doomed from the start, and relying on such a strong connection means you will always make mistakes and they will always be critical. Any real-world person can undermine such a system merely by lying or withholding the claims of identity or, of course, by sharing them (inadvertently or deliberately) with another entity.
Rather, by building a system that (1) is based on the assumption that any claimed identity is fake, and (2) that merely attempts to correlate identity between sessions, we can build a far more resilient, useful, and valuable system. And we waste far fewer resources attempting to solve a problem that, throughout history, nobody has been able to solve without outrageous violations of personal privacy.

Authenticating Identity
Another problem with identity is that identity by itself is not something you can use. Yet, throughout history, identities have often been taken for granted, without authentication. In Shakespeare's Henry IV, there is a passage where a character says, "I am Robert Shallow, sir, a poor esquire of this county, and one of the King's justices of the peace." Curiously, the character to whom Justice Shallow was speaking took this claim of identity as truth, with no further verification. Go back into earlier forms of literature, such as the Norse Sagas, and you will find similar claims of identities made by Thor and Odin, all of which were immediately accepted without question.
Clearly, if you have only one eye, are riding on an eight-legged steed, and claim to be Odin, one might argue that you have provided some amount of verification of identity. However, on the Internet, we cannot use the mounts we ride as verification of identity. As the old saying goes, "On the Internet nobody knows you are a dog." Therefore, unauthenticated claims of identity are hardly ever accepted. Therefore, we need to somehow authenticate identity.
"Ay, there's the rub." (Hamlet.) How do you prove your identity? After all, the proof of your identity is as important, perhaps more so, as the actual identity. So, how do we prove identities? The method we used to use was a form of shared secret authentication, known as "passwords." On February 14, 2006, Microsoft Chairman Bill Gates declared that passwords would be gone where the dinosaurs rest in three to four years.
But as I write this in March 2009, it is pretty clear that Bill was wrong. I have more passwords now than I had in February 2006. Just at work, I have my network password, my Unix password, the password for the expense reporting system, the password for the HR system, passwords for the company stock broker and benefits providers, the root passwords for my developer machine and a few servers, the admin passwords for my laptops, the PIN for my phone, and a couple of extraneous passwords to access the company products as various users. Still, despite having so many passwords, I can honestly say that I really do not wish passwords were dead. They are not actually a bad authentication claim.
What was Bill Gates thinking? He thought, three years ago, that by now InfoCard would have replaced all these passwords. InfoCard, which was eventually renamed Windows CardSpace, is a kind of authentication technology that shipped in Windows Vista and later in Windows XP SP3.
Points go to anyone who has ever created a card. I will give you extra credit if you find a single Web site that accepts a Windows CardSpace credential. If you just want to try it, you can take part in a beta test of InfoCard for Windows Live.

Stakeholders in Identity
There are, of course, other ways to authenticate an identity. Some are more useful in certain situations. To understand where, you need to understand the difference between the stakeholders in an identity system.
A provider of identity services is some entity that provides a service to the other parties. Typically, the provider holds the identity database and authenticates the end users of identities. Typically, the provider is not a person. Thus, the provider can deal with much larger volumes of structured data than a person can. For example, a provider can use a shared secret of arbitrary length to prove its own identity. Therefore, the credential used to authenticate the identity of an identity provider can often be many orders of magnitude more complex than what you and I can remember. Consequently, it is potentially many orders of magnitude more secure. I say "potentially" because whether it really is depends greatly on how the secrets are managed. It does no good to have a 4096-bit secret to prove your identity if you do not actually keep it secret.
Identity providers can very easily use constructs such as Digital Certificates to authenticate themselves. A Digital Certificate is simply a relatively sizeable chunk (a couple of thousand bytes) of structured information. A computer can easily send that as part of every transaction. You and I, however, would not be too thrilled if we had to enter that to authenticate for a transaction.
The end user is the entity that needs to have its identity asserted. Very often, the user is a person, but that is not always the case—the user can also be a computer system. Therefore, the quality of the claims the user can present varies greatly.
Finally, there's the relying party. The relying party is the entity that trusts the identity provider to verify the user before providing the user some service. The relying party is almost always a computer system. To the extent that the relying party proves its own identity to a user and the identity provider, it can do so with equally strong claims as the identity provider used.
It is important to note that a user in one scenario is often a relying party in another. If the user is a piece of software, or has the help of a piece of software, this user can easily deal with the same types of structured data as the provider. That is why distributed identity protocols, such as OAuth and the various security specifications part of WS-* focus heavily on significant quantities of structured data for authentication—they are primarily focused on services authenticating to each other. Even when those specifications deal directly with people, they tend to be brokered through services that can handle the structured data on behalf of the people. Often, the broker is just an application on the user's computer.

Using Identities
The concept that a user (a person) uses a computer system as an intermediary when accessing services from a relying party is quite important. It goes to the heart of how to make identities usable by people, as opposed to systems. Unfortunately, other than InfoCard, which so far has found little success, few systems really address that topic. To a large extent, the process flow when a physical person uses an identity service to get access to something looks something like Figure 1.
Figure 1 Process Flow of Using an Identity Service
Notice the "Then A Miracle Occurs" (TAMO) box. The sociotechnical aspects of the system—how you enable users to take full advantage of them—is where a system is truly made or broken.
There have been many attempts at resolving the TAMO issue. Most came about because of the incongruity that on the Internet we want to use strong identities, typically based on certificates, but users persistently refuse to type them in every time they need to access a service. Microsoft Passport, currently called Windows Live ID, was an early attempt. Google has entered the fray with its Google Account, and there are others, as well. Even Windows CardSpace can be said to have been an attempt at solving this problem. All are designed to provide some level of SSO. All have succeeded, largely, in doing so, within the domains owned by their various purveyors of the particular solutions. All have failed almost entirely outside of their purveyor's domains.
In the remainder of this article and Part 2, I will address this problem in more detail. Note that I do not purport to solve the problem, but merely to point out some reasons why nobody else has yet solved the problem either. Eventually, I will arrive at a set of principles that I believe any solution to this problem must follow. That, in turn, will lead to a conclusion that says, to sum it up, ontological identity may just be a pipe dream.

How Many Identities Do You Have?
The big challenge with respect to identity is not in designing an identity system that can provide SSO, even though that is where most of the technical effort is going. It's not even in making the solution smoothly functioning and usable, where, unfortunately, less effort is going. The challenge is that users today have many identities. As I mentioned above, I have well over 100. On a daily basis, I use at least 20 or 25 of those. Perhaps users have too many identities, but I would not consider that a foregone conclusion.
The purist would now say that "SSO can fix that problem." However, I don't think it is a problem. At least it is not the big problem. I like having many identities. Having many identities means I can rest assured that the various services I use cannot correlate my information. I do not have to give my e-mail provider my stock broker identity, nor do I have to give my credit card company the identity I use at my favorite online shopping site. And only I know the identity I use for the photo sharing site. Having multiple identities allows me to keep my life, and my privacy, compartmentalized.
In addition, if the credit card company manages to get itself hacked, all the other identities are unaffected. If I had a single identity, that may or may not be the case, depending on how that single identity was implemented. A properly implemented SSO system would never expose one site using the system to a failure in another, unrelated site. However, enforcing that separation is not trivial, and if the identity provider is compromised, every site that depends on it is automatically compromised.
This latter point is one reason why the digital identity systems Cameron described have, so far, failed to become the single universal identity provider—any single identity provider is an extremely sensitive entity.

Principles of Identity
There are several principles of identity that must be met in at least some way to provide a successful digital identity system. These are very different from the laws Cameron outlined in the Laws of Identity paper. Cameron's laws were design principles—use cases, more or less—that essentially define technical requirements for a trustworthy digital identity system. As such, they are a set of necessary but insufficient criteria for success. While many systems can be designed to meet or exceed Cameron's laws, I do not think any of them will be broadly successful without also taking into account the principles I define here. My principles are higher-level principles, dealing with business requirements, not with direct design points of the system.

1. The identity provider is at least as sensitive as the most sensitive relying party.
First, as I mentioned earlier, the identity provider is trusted by all parties. This is why we refer to them as "relying parties." That means that the identity provider must be at least as well protected as the most sensitive relying party requires. This single point is why many of the candidate universal systems have failed to create broad-reaching SSO systems—the trust simply is not there. Even if the trust is warranted, it is extraordinarily difficult to prove why one party should trust another.
Would you trust your e-mail provider with your bank account information and all your checking account data? If the answer is "no," you should not trust it to provide the identity you use to sign into your bank. Consumers today are deluged by unreliable software, incomprehensible pop-ups, and anti-malware vendors (legitimate and not-so-legitimate) who claim you can't possibly surf the Internet safely unless you pay them. Consequently, and quite wisely, consumers trust almost nobody. Very few organizations today are trusted enough to be candidate identity providers for other organizations. Building that kind of trust is not cheap. Keeping it is a fragile proposition. And earning it back once lost due to a breach is nearly impossible.
If we will ever see a company providing a successful universal digital identity system, it will be from a company that has earned a very high level of trust. Interestingly, many of the players in the identity provider space are not very high up in the surveys, or they do not show up at all (See the Ponemon Institute’s fifth annual survey of Most Trusted Companies for Privacy).

2. Permit the enterprise to protect its customer relationships and to own and control information that it deems as business confidential.
Anyone who has an MBA knows that there are three ways to succeed with a business: you have the most innovative product, or you have the lowest price, or you excel at customer intimacy (or some combination of all three). Interestingly, virtually every business is interested in the latter scenario: customer intimacy. Customer relationships, especially in an online business, are sacred! In a world where switching costs are next to nothing, where everyone guarantees (for some loose definition of "guarantees") the lowest price, and the same product is available everywhere, customer relationships become key. Even where innovation does happen, such as in social networking and e-mail providers, customer connection is still critical.
One primary reason why no large Web site accepts credentials from another company (or another company that did not use to own them, as in the case of Expedia) is that it dilutes its relationship with its customers. Imagine, for example, if Yahoo simply received a claim from NetIdentitiesRUs that, yes, it really is customer 923071235309342-2 that just signed in. Yahoo would no longer own the database of its customers. It would not know who the customer actually is. Yahoo could no longer manage the customer and the system the way it wanted. It could no longer do customer research by cross-checking the identity with the databases in its subsidiaries and acquisitions. It could not even sell the information to third parties for extra revenue (should it wish to and had the customers opted in, of course). Clearly, Yahoo would not be particularly interested in giving all that up and, consequently, would need some exceptional incentives to do so.
Now imagine instead that Yahoo is the identity provider. Yahoo can get information on exactly where customer 923071235309342-2 goes on the Web because it knows exactly which sites request claims for that customer. Yahoo could compile invaluable data on that customer's browsing habits, allowing extraordinarily well-targeted advertisements with astonishingly high click-through rates. For an ad-funded organization, and especially one that is actually in the ad business itself, that is a competitive advantage you just cannot give up.
Trusting identities provided by other companies would be equivalent to giving up huge opportunity. Providing your own to others would be jumping on it. In other words, there's a conflict and the result is exactly what we are seeing on the Web today. Within conglomerates, we see one universal identity, and while all of the major conglomerates are also identity providers, virtually nobody relies on them.
This analysis points to a fundamental asymmetry in identity provisioning. There is huge value in being the identity provider and no value at all in being the relying party. Unless the market can figure out ways to equalize the two, we will continue to see as many identity providers as relying parties, with virtually a 1:1 relationship between them.
Check back next month when I will look at the remaining principles in this two-part series.

Jesper M. Johansson is Principal Security Architect for a well-known Fortune 200 company, working on risk-based security vision and security strategy. He is also a contributing editor to TechNet Magazine. His work consists of ensuring security in some of the largest, most distributed systems in the world. He holds a Ph.D. in Management Information Systems, has more than 20 years experience in security, and is an MVP in Enterprise Security. His latest book is the Windows Server 2008 Security Resource Kit.


Business of IT Successful Project Management
Alessandro Muti

There’s no question that today’s technology environment presents increasing and rapidly changing challenges for IT organizations. With the proliferation of social media, smartphones, intranets, extranets, and a slew of other technologies, the job of the IT professional has evolved from the simple management of servers and infrastructure in a data center to one that approaches that of the fully fledged product software developer.
Add to this the current economic troubles that are forcing many companies to implement cost-cutting and personnel reductions, and there's more pressure than ever on IT departments. Moreover, despite all these challenges, you still have to manage the servers and the infrastructure and you still have to deliver on more projects than ever before—with fewer resources. In the face of these challenges, project management techniques and managing upwards become the necessary tools that can ensure the success of a project that might otherwise crash.
One of the problems I have experienced over and over in my career is the fact that highly technical people often have a natural distrust of management. This is usually caused by the fact that the managers seldom come from the same background and thus have different expectations with regards to the complexity of developing and delivering on a project. This in turn frustrates the development team and forces them to take, in many cases, a standoffish position or even an "I told you so" stance that will eventually bounce the blame right back to them if the project fails. The problem is essentially the inability to communicate effectively to a management chain that doesn't really speak the same language. Unfortunately, this is an all-too-common story. We think we have effectively communicated the risks and don't realize until it is too late that we have not been understood. Unfortunately, the attitude that it's not our problem can result in failures even on some very technically solid projects.
Some projects, as I'm sure you've seen, seem to be doomed from the very beginning. Even experienced folks will put in lots of late nights trying to solve bad management decisions with technical proficiencies; unfortunately, in many cases that is simply not possible. When that happens, instead of taking the approach that it is not your problem, you are much better off stopping, discovering where the real issues with the project are, and then addressing them, rather than making a heroic effort in the hope that the project will be successful. If it is not, even if you are outstanding at your job, you will be associated with that failure.
The best thing to do, in my experience, is to keep a watchful eye right from the beginning of a project. Usually, when a project kicks off, everyone is happy and energized and there seems to be agreement and understanding about the direction of the project, the cost, the deliverables, and so on. As time passes, however, problems inevitably creep in. You may have some small technical delivery problem, perhaps the performance of the database is not as expected or perhaps you can't handle the traffic. All of a sudden, something that could have been handled easily had the telltale indicators of project failure been looked at, is now compounded in a big mess that seems impossible to unravel.
So what are those indicators and how can we make sure to keep a project on track?
First of all, it is vital to maintain the vision and scope of a project. Vision and scope must be agreed on at the outset of a project, along with what is required to deliver on that vision. Without this, there will be an inevitable disconnect between what you are expected to deliver and what your team actually ends up building. Whenever reality starts to diverge from the vision, it is critical to reset it and bring it back in line so that what your manager expects, what your user base envisions, and what you end up delivering are in lockstep.
To make sure this happens, I have found it useful to have a document that is kept updated at all times that speaks to the high-level business objectives, that spells out the needs and requirements of all stakeholders and the interactions among them. By keeping this document always up-to-date and by sharing it and getting both consensus and approval across the board, you at least minimize the risk of encountering that disconnect between the requests your manager or other stakeholder will make and the expectations of your user base. This will also prevent the team from starting a project when they have only a superficial understanding of the project vision. One thing to note is that a vision document is not a scope document. The scope document is what is generated by the program manager or the team to align the vision to the details of a particular project.
This brings us to the next very important part of ensuring a successful project—understanding the scope of the project and making sure everyone else understands it as well. There should be a series of meetings of all interested parties that result in consensus regarding the scope of the project. There are lots of ways to assess the scope and I will not go into detail here, but one common mistake is to underestimate the effort required to meet the more complex technical challenges. Many of us are by nature optimistic, so try to keep that in mind when determining the effort required to build the solution.
Of course, there are lots of unknowns at this stage of the project and it will be necessary to make some assumptions. It is incredibly important to document these assumptions and to agree on the impact they will have on the project. Negotiating the details may also provide also early warnings of potential disconnects among people in interpreting the vision document. If two people, for instance, disagree about the time and effort involved, it is very likely they are reading and interpreting the vision document in completely different ways, and it is a signal that you might want to make sure the vision is clear enough that stakeholders all think of the same elements in the same terms.
This documentation of assumptions is critical because it provides a framework for discussing many of the nontechnical challenges with both upper management and stakeholders and will help make sure everyone understands the constrains needed to work within to deliver a successful project. Overall, exploit this document as a tool to spark the difficult but necessary conversations about scope with your management chain, and use it in order to translate some of the early concerns into answers that will help anchor your scope. Without it, many of the conversation will not happen.
Once that is out of the way, you'll have to choose the proper methodology for going forward with your project. There are lots of different models out there and all have pros and cons. There isn't one model that will always work, and choosing the wrong model can potentially spell doom for your project. Many companies have some accepted approach that is expected to be followed but is often resisted because it is too restrictive or is not "perfect" for the project at hand. But this model is important because, if tailored well, it will enable a set of gates and checks that will maintain the connection among all stakeholders. You will be able to keep your manager involved, thus preventing the chain of command from making uninformed decisions because they will feel they are part of the process with dedicated moments for providing feedback.
In fact, you should build into the process very well defined periodic walkthroughs designed to make sure that everyone involved has a chance to compare their expectations of the project against the reality of what is being delivered. This allows problems to surface and be resolved at regular intervals, with fewer ramifications.
Transparency is also critical in this phase. Any artifacts that have been generated should be shared and made available for everyone to review, giving everyone involved the opportunity to familiarize themselves, at their pace, with how things are progressing and to review the progress on their own terms. Transparency will build trust and trust is what will prevent anyone from trying to micromanage the project because they don't have access to what your team is doing. Lack of access tends to foster suspicion, and since many people don't like not knowing what will happen tomorrow, they will try to find that information for themselves. If the information is readily available, they will not have to come to you so often.
Remember also that every project will at some point need to undertake some course corrections. This should be done sooner rather than later because the more you build upon the wrong assumption, the more you will have to fix later and the harder that will be. As you go through these changes, remember to follow the same process that took you here. Start by asking all of the stakeholders if the vision for the project is the same or if it has changed, then correct the documents you have created, fix the estimates, and adjust the assumptions you had to make. Don't deviate from this process even if the decision or the course change seems small, because lots of small changes will eventually lead you to a completely different place and if you have not taken the time to document them and to maintain the communication among all the people involved, you will deliver something that, even if it is technically accurate, might not be perceived as a success.

Alessandro Muti is the CTO of Ascentium, a full service interactive digital agency and technology consulting company. He has been in the software industry for the past 20 years, starting his career at Microsoft in the Windows team. He now oversees the technical strategic direction for Ascentium and helps customers to formulate their technology strategies around digital presence, privacy, social media, mobility, security, online media delivery, digital brand management, and whatever else might even remotely look like technology.


Blog Tales Monitor the Default Management Pack
Jonathan Almquist

In which we search far and wide to bring you the best of the Microsoft blogs. Here's one from Jonathan Almquist on his Operations Manager blog.

Code download available at: TNonlineAprAlmquist.BlogTales2009_06.exe (153 KB)

A colleague, Kevin Holman, wrote about best practices concerning the Default Management Pack, and how to clean it if it happens to get messy, which you can see in last month's Blog Tales column. Unfortunately, it seems almost inevitable that the Default MP will become cluttered in time. This mostly comes down to educating staff on being careful not to save anything to the Default MP. Hopefully, this mistake will not be so easily made in future releases of OpsMgr.
In Kevin's post, he had an idea of monitoring the Default MP. Essentially, send a notification or raise an alert if the number of overrides stored in the Default MP should exceed a threshold. This threshold is a number you decide is acceptable. One way to determine your threshold is to check where you're at now. Run the following query against the OperationsManager DB:
select count(*) as NumberOfOverrides from AllOverrideView aov
inner join ManagementPackView mpv on aov.managementpackID = mpv.Id
where mpv.name = 'Microsoft.SystemCenter.OperationsManager.DefaultUser'
If you currently have more than 2, you may want to check what is in there and do an initial cleanup. Given an acceptable number of overrides (let's just say 2), we can set our threshold and generate alerts when this is exceeded. For brevity, I have created a sample MP (monitordefaultmp.zip) that you can download. This sample sync's at midnight, and queries the OperationsManager DB for the number of overrides in the Default MP. It compares this number with the threshold you have set in the rule parameters, and raises a Warning alert in the Operators Console. Of course, you can expand on this workflow and add a notification or change the severity if you like.

To import my sample Management Pack
If you want to get this going quickly by just importing, you can use my sample MP.
The script write action rule is not enabled by default. After importing, go into the Authoring space of the Operators Console. Click Rules > Scope to Root Management Server > search for string Default > Right-click "Monitor Default MP Write Event" rule > Overrides > Override the Rule > For all object of type: Root Management Server.
Check the Enabled parameter, and set to True.
Check the Arguments parameter, and enter your threshold, Operations Manager database server name, and Operations Manager database name.
Note: Arguments are space delimited.

To create the rules in your own custom MP using the attached script
If you want to create these rules in your custom MP, follow these instructions and use the downloaded script for your script write action rule.
First we'll create a rule that executes the query on the OpsDB, and creates an event in the Operations Manager event log for us to match and generate alerts. Create a new rule > Timed Commands > Execute a Script. Save into your custom Management Pack. Perhaps you have a MP that you've already been creating some custom monitoring or tasks.
Enter a rule name and select Root Management Server for your Rule Target.
Configure the schedule to run once a day. Synchronize the rule, so that it will always run at the same time. If a synchronize time is not specified, the rule will automatically sync to the time when the workflow became active on the agent. This means the rule will sync at the time it was initially delivered to the agent...and subsequently synchronized to the time when the agent Health Service is restarted.
Paste the MonitorDefaultMp script that you downloaded into the script pane. Keep the same file name for consistency. The filename here doesn't affect functionality of the rule, but the filename within the script is used in event data and needs to match NT Event alert expressions later in this walk-through. Click on Parameters.
Type in your determined threshold, the Operations Manager DB Server Name and the Operations Manager DB Name. The parameters are space delimited.
You're finished with this rule. Click create and let's move onto the alert generating rule.
This rule will match the event in the Operations Manager event log and generate an alert. Create a new rule > Alert Generating Rules > Event Based > NT Event Log (Alert). Save to the same MP.
Type in a rule name, select the Alert category and target Root Management Server.
Specify the Operations Manager log.
Build the expression as follows. This is where we need to match parameter 1 to our script name. This script name is actually specified in the log event method from within the script. If you take a look at the script, you'll see the script name specified. This is what's being matched. This is why it's important we keep these script names consistent throughout. Otherwise, it's more difficult to build an efficient expression.
Lastly, we'll specify the alert information. I recommend Event Description...that'll give you all the info you'll need. I specified Warning for severity, as I thought it was better suited for this type of alert.
Finally, this is the result.
This is what you'll see in the Operations Manager event log.
If you want to test this and see results quickly, specify a threshold value of 1 in the parameters, uncheck the synchronization time and configure the schedule to run every 1 minute.

Jonathan Almquist is a Premier Field Engineer, based out of Minneapolis, MN. He's been in the IT industry since 1999, with the most recent years primarily focused on the Operations Manager platform. He provides guidance to enterprise customers in the United States, as well as contributes to the Operations Manager community through his blog and other technical forums. In his free time, Jonathan enjoys spending time with his wife and son, while finding enough time to run the paths throughout the beautiful Minnesota lakes and woods.


The Cable Guy NAP on the Internet
Joseph Davies

Contents
IPsec Enforcement Overview
NAP on the Internet
NAP on the Internet, an Example
Summary
Network Access Protection (NAP) on the Internet is the natural evolution of Internet Protocol Security (IPsec) enforcement that extends health checking and ongoing remediation of system health for managed computers that are roaming on the Internet. NAP on the Internet enables a stronger, host-based security model for domain-joined computers connected to the Internet from anywhere, at any time. For example, a mobile computer connecting to a Wi-Fi network at a local coffee shop can use NAP on the Internet to evaluate and correct its system health without the user's knowledge or intervention.
Before we get into the details, let's review some of the basics and the deployment details about NAP and IPsec enforcement.

IPsec Enforcement Overview
IPsec enforcement is an extension of the domain isolation scenario that incorporates a NAP health assessment as part of the IPsec peer authentication process. In the domain isolation scenario, you configure domain members with IPsec policy settings to require that all incoming connections be authenticated and protected (encryption of traffic is optional). IPsec peers can use Kerberos or digital certificates for authentication. Domain members automatically have Kerberos credentials, and certificates can be automatically deployed to domain members with a Windows-based certificate authority (CA) and Group Policy.
For IPsec enforcement in NAP, the credential for IPsec peer authentication is a health certificate containing the System Health Authentication Enhanced Key Usage (EKU). IPsec policy for IPsec enforcement requires that all incoming connection attempts use a health certificate for authentication. This ensures that the peer initiating the communication is not only a domain member, but is also compliant with system health requirements. In full enforcement mode, if a NAP client does not have a health certificate, the IPsec peer authentication fails and the noncompliant NAP client cannot initiate communications with a compliant IPsec peer.
When a NAP client configured for IPsec enforcement starts up, it contacts a Health Registration Authority (HRA). An HRA is a computer running Windows Server 2008, Internet Information Services (IIS), and the HRA role service of the Network Policy and Access Services role. The HRA obtains an X.509 certificate from a CA on behalf of a NAP client when the NAP health policy server has determined that the client is compliant.
To obtain health certificates for compliant NAP clients, you must configure your HRAs with the locations of the NAP CAs on your intranet. For NAP clients to locate the HRAs on your intranet, you can either configure them with a trusted server group—a list of URLs for the HRAs on your intranet—or HRA discovery (see the sidebar "HRA Discovery and Internet-facing HRAs").
When NAP clients start, they locate an HRA on the intranet, submit their health status, and, if compliant, obtain a health certificate. When NAP clients initiate communications to other intranet endpoints, they attempt to negotiate IPsec protection for the traffic using the health certificate credential. If the IPsec negotiation fails, the NAP client connects without IPsec protection. If a compliant NAP client's health or network state changes, it does an additional system health check with an HRA, and if compliant, it obtains a new certificate. If there are system health conditions that need to be corrected, the NAP client will have to correct them before obtaining a health certificate.
HRA Discovery and Internet-facing HRAs
A NAP client can also use HRA discovery to automatically discover the HRAs on a network. Rather than configuring a list of URLs in a trusted server group, a NAP client that has HRA discovery enabled attempts to find an HRA by sending a DNS query for SRV records for _hra._tcp.site_name._sites.domain_name. For example, if a NAP client's primary domain is contoso.com and the client is using the default Active Directory site and has HRA discovery enabled, it will query for the SRV records for _hra._tcp.default-first-site-name._sites.contoso.com. The SRV record for HRAs contains the Fully Qualified Domain Name (FQDN) of the HRA. The DNS SRV records for HRAs must be manually configured in the appropriate zones.
For NAP on the Internet, you configure intranet HRA SRV records for the FQDNs of your intranet HRAs and Internet HRA SRV records for the FQDNs of your Internet-facing HRAs. The NAP client queries for the same name and record type, but gets different FQDNs depending on the location.
To enable HRA discovery for NAP clients that receive NAP configuration through Group Policy, set the HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\ NetworkAccessProtection\ClientConfig\Enroll\HcsGroups\ EnableDiscovery registry value (DWORD type) to 1.
For more information, see "Configure HRA Automatic Discovery."

NAP on the Internet
When a NAP client computer configured for IPsec enforcement leaves the intranet and connects to the Internet, the NAP client still attempts to locate an HRA. Because the roaming computer on the Internet can no longer contact an intranet HRA and perform system health validation, it can no longer determine whether it is compliant with system health policies and how to correct its system health. Over time, a NAP client that roams the Internet can end up with missing operating system or application updates, or the user, depending on his privilege level, may perform configuration changes that put the computer at risk, such as disabling the host firewall.
When the noncompliant computer returns to the intranet, it must first remediate its system health before it can obtain a health certificate and begin communicating. While the noncompliant computer is being remediated, there is a risk that it can infect other intranet computers that are not protected by IPsec.
By placing HRAs on the Internet and configuring NAP clients to locate them, NAP clients on the Internet can check their health on an ongoing basis as if they were connected to the intranet. Although the health certificate is not used to perform IPsec peer authentication, the NAP client is still validating its health state. If autoremediation is enabled, the NAP client will automatically attempt to correct its health state to mitigate risks to the computer while on the Internet. Autoremediation keeps the computer compliant with your organization's system health requirements without requiring user intervention. A mobile computer that is already compliant returning from the Internet greatly reduces the risk of infection to intranet resources.
Remediation of system health issues is limited to the functionality of system health agents (SHAs) and may require that you deploy additional remediation servers on the Internet.
Note that by default, NAP clients try Kerberos authentication when connecting to the Internet-facing HRAs. Because a domain controller is not available on the Internet, this authentication will fail. Therefore, you must run the following command on the Internet-facing HRAs so that NAP clients use NTLM authentication:
%windir%\system32\inetsrv\appcmd.exe set config -section:
system.webServer/security/authentication/windowsAuthentication
 /-providers.[value='Negotiate']
You can use the information logged by your NAP health policy servers to determine the overall health compliance of the NAP clients that are roaming on the Internet.
If your Internet-facing HRAs are separate from your intranet HRAs, you can configure a separate set of health requirement policies for your Internet-connected NAP clients. For example, you can configure separate sets of network policies by specifying the IP address of the HRAs as a condition. The intranet health requirement policies use all of your SHAs, such as the Windows Security Health Agent (WSHA), the Forefront SHA, and the System Center Configuration Manager (SCCM) SHA. The Internet set of health requirement policies use only the WSHA.

NAP on the Internet, an Example
Figure 1 shows a simplified illustration of how the Contoso Corporation has deployed NAP on the Internet.
Figure 1 Deploying NAP on the Internet
In this example, the Internet-facing HRAs are physically connected to both the Internet and the intranet so that they can reach the NAP CAs and the NAP health policy servers. To protect the Internet-facing HRAs, firewall settings allow only TCP port 443 traffic to and from the Internet-facing HRAs, corresponding to SSL over HTTP traffic.
Let's say the intranet HRAs have the names hra1.corp.contoso.com and hra2.corp.contoso.com. The Internet-facing HRAs have the names int_hra1.contoso.com and int_hra2.contoso.com. Of course, intranet DNS servers cannot resolve the names int_hra1.contoso.com and int_hra2.contoso.com and Internet-facing DNS servers cannot resolve the names hra1.corp.contoso.com and hra2.corp.contoso.com.
With this configuration, the network administrator for Contoso configures a trusted server group with the following list of URLs:
  • https://hra1.corp.contoso.com/domainhra/hcsrvext.dll
  • https://hra2.corp.contoso.com/domainhra/hcsrvext.dll
  • https://int_hra1.contoso.com/domainhra/hcsrvext.dll
  • https://int_hra2.contoso.com/domainhra/hcsrvext.dll
NAP clients first try the intranet HRAs and then the Internet-facing HRAs.
A NAP client on the intranet will connect to HRA1 or HRA2. A NAP client on the Internet will first attempt to resolve the names hra1.corp.contoso.com and hra2.corp.contoso.com. These two name resolution attempts quickly result in a DNS Name Not Found error. The NAP client then successfully resolves int_hra1.contoso.com or int_hra2.contoso.com and connects to INT_HRA1 or INT_HRA2.

Summary
Once the IPsec enforcement scenario has been deployed on your intranet, extending health compliance validation and correction for Internet-connected NAP clients is relatively easy, requiring some additional servers for the Internet-facing HRAs (or additional roles for existing Internet-facing servers). Moreover, depending on how your NAP clients locate HRAs, you may need to publish Internet DNS records for the Internet-facing HRAs and update your trusted server groups. You can also create an additional set of health requirement policies to specify the system health checks and autoremediation behavior for Internet-connected NAP clients.

Joseph Davies is a Principal Technical Writer on the Windows networking writing team at Microsoft. He is author or coauthor of a number of books published by Microsoft Press, including Windows Server 2008 Networking and Network Access Protection (NAP), Understanding IPv6, Second Edition, and Windows Server 2008 TCP/IP Protocols and Services.


Field Notes Building Strategic Vision into IT
Romi Mahajan

The French philosopher Auguste Comte said, "Savoir pour prévoir et prévoir pour pouvoir." Roughly translated, that means you have to know to have vision, and you have to have vision to be able. Simply put, the ability to see ahead renders one capable, even powerful. In business circles, this concept can be distilled to "strategic vision," and it allows you to envision tomorrow's world but act on it today. Great business leaders have this ability in abundance. But what about IT professionals? Do we have this capability? And if we do, can we articulate it in the right way?
At Tech Ed 2008, I had the privilege of arranging, for the fourth time, a panel discussion on the consumerization of IT. One of the major areas of debate focused on how IT organizations should react with the increasing introduction of consumer technologies into the workplace, and how they can learn to predict what's around the bend. I took the position that IT has to be more flexible than it generally is to accept the rights and desires of its users and, further, that IT organizations have to "skate to the puck" in order to be powerful.
Pretty basic stuff you say? Well, let's look at the reactions from the audience, which essentially took four forms:
  • IT organizations have to manage costs, so allowing too much freedom and spending too much time and money trying to predict what non-IT users need is not within their charter.
  • IT organizations are reactive—slaves to the pager—and therefore have no ability to evolve.
  • IT organizations aren't creative.
  • IT organizations should absolutely anticipate and support the needs of their users; where do I sign up?
Unfortunately, the last category comprised the views of perhaps 10% of the audience.
My argument is that we have to look at the picture upside-down. Let's not concentrate on what IT orgs are today, let's look at what they could be. To do this, let's step out of our designated roles for a moment and think about the knowledge and experience we ourselves have gained as consumers. This is the start of strategic vision.
What could IT organizations be? They could be entities actively engaged in meeting the future. They could invest in a lot of training in new technologies and new customer behaviors. More young people whose entire experience with computing has been in a Web 2.0 world could be hired into their ranks. They can be given leeway to spend 10–15% of their total time on non-project-based, non-reactive, new and innovative areas of technology and business. They can be organically tied to development organizations and intermingle cultures, charters, and knowledge.
How can IT benefit from IT pros' own experiences as consumers? Imagine all of the things you do today on the Web—how you arrange photos, how you keep in touch with family calendars, how you book tables at restaurants, how you buy things at good prices, how you stay in touch with people who share similar hobbies or quirks, and so on. Develop a list of all of the places you hang out on the Internet. Tell your colleagues to do the same, and make sure that you all consider that just as you do, so do all of your customers, the end users. Now you know what you are facing from your end users, and now you can also feel free to co-opt these technologies into ones with business applications.
Soon IT will be the ones sending notes to the CEO about how the industry is changing, what to concentrate on and invest in for the future, what the new marketing channels are that work with what audiences, what form-factors are key now and which are emerging, and even how to sell more widgets or services and make customers happy in the process.
Suddenly, IT folks are not limited to back-room operations. We are key to the business; and suddenly we walk with more dignity and pride and receive more acclaim. And we reclaim our rightful places at the intellectual and practical center of what our companies, organizations, and governments do.

Romi Mahajan is Chief Marketing Officer of Ascentium Corporation. Before joining Ascentium, he spent more than 7 years at Microsoft where his last role was as Director of Technical Audience and Platform Marketing. Romi is widely published in the areas of technology, politics, economics, and sociology.


Windows Confidential When Windows Audio Goes Haywire
Raymond Chen

If you look at the Description for AudioDG.exe on the Processes tab of Task Manager, you'll see that it says Windows Audio Device Graph Isolation. This is a provocative name that teases you with information without actually providing any.
The AudioDG process is the home of the Windows audio engine. This is where digital signal processing happens. The process is intentionally separated from the Windows Audio service for a variety of reasons; among these is that the AudioDG process hosts all the fancy audio effects, which go by the technical name Audio Processing Objects but which are referred to in the user interface simply as Audio Enhancements.
We were introduced to Audio Enhancements briefly back in May, when I discussed the Loudness Equalization audio enhancement, which lets you set your volume control to a comfortable level, and then you never need to touch it again. For a description of other built-in enhancements, you can read the opening pages of the document "Reusing Windows Vista Audio System Effects."
Isolating these Audio Processing Objects is important because audio hardware vendors can include new Audio Processing Objects with their drivers. By sequestering the Audio Processing Objects in a separate process, the scope of a crash in an Audio Processing Object is contained. If AudioDG crashes, you will only lose sound. The rest of your system will keep running.
Unfortunately, there are other things an Audio Processing Object can do that will cause problems without causing a crash. For example, they can hang, but again, the scope of this is limited to audio processing. What's worse is when an Audio Processing Object goes haywire and consumes exorbitant amounts of resources, sucking up all your CPU cycles or leaking memory like a sieve. Fortunately, these types of mistakes are much less common than flat-out crashes.
But if you're unlucky enough to find yourself watching the AudioDG process consuming ridiculous amounts of CPU or memory, there is something you can do:
  • Open the Sounds control panel and go to the Playback page.
  • Select your playback device (speakers, probably) and click Properties.
  • Click on the Enhancements tab. (If there isn't an Enhancements tab available, this means your playback device doesn't have Audio Processing Objects associated with it.)
  • Use the checkboxes to turn off some (or all) of the enhancements, which, as you've learned, is just a friendlier name for Audio Processing Objects. The "Disable all enhancements" checkbox acts as the master switch to turn off everything.
As you can imagine, sound card vendors really, really, really don't want you to disable the fancy features they spent a lot of money on to develop. They rely on these features to help set them apart from the competition. "Use our sound card instead of that other sound card because our sound card sounds more awesome!"
When the audio hardware vendors found out the Windows audio team was adding a way to disable these precious enhancements, the vendors became very concerned and exerted pressure via every channel available, trying to force the Windows audio team to remove the master switch that turns off everything. It was a long, hard fight, but the Windows audio team prevailed and now you have that checkbox. (And to make sure no sound card vendor tries to pull a fast one, part of the driver's WHQL test is to verify that the vendor's driver doesn't try to hide the checkbox or otherwise prevent it from working.) You should probably send them a thank you note for fighting the good fight and winning.
Okay, but why the name AudioDG? The term Device Graph is used to describe the way the various components of the audio system connect, and it is along these pathways that audio samples travel on their way to your speakers. The Windows audio team intended to name the program DeviceGraph.exe, but due to a misunderstanding, they shortened it to AudioDG. They had mistakenly thought that the name of the program could not exceed eight characters, a throwback to the old 8.3 days before long file names were in use. By the time the misunderstanding was corrected, it was too late, and the name had been chosen.
Thanks to Larry Osterman and Elliot Omiya for providing assistance in writing this article.

Raymond Chen's Web site, The Old New Thing, and identically titled book (Addison-Wesley, 2007) deal with Windows history, Win32 programming, and Krashen's Comprehensible Input Hypothesis.