 TechNet Magazine November 2007

TechNet Magazine November 2007

Windows Administration: Changes to Offline Files in Windows Vista

It's a common problem—how do you let users work with files stored on the network, when they have unreliable or no connectivity, and still avoid versioning issues? Such continuity is essential to providing a seamless experience. Discover how improvements in Windows Vista provide a better approach to working with offline files.

Jeremy Moskowitz Desktop Deployment: Simple, Scalable Deployment with BDD 2007

Don't have SMS 2003? Don't worry, there's another solution for creating simple desktop deployments. Here's an in-depth guide to using BDD 2007 and SQL Server to create scalable deployments that require little or no manual input.

Adam Shepherd Communications: Exchange Edge Transport Servers at Microsoft: Part 2

Exchange Edge Transport servers can help you create a more reliable and secure messaging environment. We showed you how to configure these servers in Part 1, now we'll take you through some detailed examples of how key features help improve your messaging environment.

Kay Unkroth Networking: Configure VPN Remote Access with ISA Server 2006

A virtual private network can provide improved connectivity and increased productivity—and it doesn't need to be difficult to implement and manage. Find out how you can use ISA Server 2006 to address two common VPN scenarios.

Alan Maddison SQL Server: Optimizing SQL Server Query Performance

When you optimize database performance, tuning individual queries is as important as tuning server hardware and software configurations. Even one runaway query can bring a database to its knees. Here's a guide to generating, analyzing, and tweaking execution plans to optimize the performance of your SQL Server queries.

Maciej Pilecki From the Editor: More Power to You

Joshua Hoffman Toolbox: New Products for IT Pros

Greg Steen Exchange Queue & A: Secure E-Mail Protocols, Mysterious Spam, and More

In this month's installment, the Exchange team discusses Transport Layer Security for SMTP, how to prevent spoofed NDRs from clogging up your queues, and how to tackle various installation challenges.

Nino Bilic and Scott Landry Utility Spotlight: SMS Cmdlets for Windows PowerShell

Find out how you can start using Windows PowerShell cmdlets to manage SMS 2003 clients from the command line.

Don Brown The Cable Guy: Wireless Single Sign-On

Single Sign On offers many advantages for both end users and administrators. Here's a look at how Single Sign On can simplify implementation of wireless authentication for your network.

Joseph Davies Research Notes: Using High-Performance Computing in the Fight against HIV

Take a look at how the eScience team in Microsoft Research is collaborating with medical researchers and using high performance computing to help find a cure for HIV.

Kristin Firth and Mia Matusow Windows PowerShell: Writing Regular Expressions

With a well-written regular expression, a Windows PowerShell script can determine whether or not data conforms to a valid format. Get an overview of how you can use Regular Expressions in Windows PowerShell.

Don Jones Hey, Scripting Guy!: Desktop Management from beyond the Grave

The Scripting Guys demonstrate how to use Windows Remote Management to retrieve the properties and instances of classes on a remote computer. Oh, and there's a cool puzzle too.

The Microsoft Scripting Guys Security Watch: Secure Access Anywhere

Explore network and security design considerations for an anywhere-access solution using Terminal Service components, and learn some practical methods and best practices for creating a solution with Windows Server 2008.

John Morello The Desktop Files: The Truth about Defragmentation

Myths about the pitfalls of defragmentation have prevented many an administrator from performing this important maintenance task. Find out which problems are real, which ones are unlikely, and how you can avoid them.

Wes Miller Field Notes: The Cod of Conduct

When community members start acting, perhaps you just need a big fish. See how social engineering helped to save Live QnA.

Betsy Aoki Windows Confidential: The File System Paradox

Ever wonder why you need a miniature file system driver built into those critical boot files?

Raymond Chen


 Windows Administration: Changes to Offline Files in Windows Vista
Windows Administration Jeremy Moskowitz

Changes to Offline Files in Windows Vista
At a Glance:
  • Online and offline continuity
  • File availability
  • How offline folders are evolving

One of the toughest problems in desktop management is letting offline users work just as if they were working online. The need is greater than you might think—users are a lot more mobile than they were even five years ago. Though there are a number of cases where you might need to work
offline, the three most common are when you disconnect your laptop from the office network and go on the road, when you connect over a slow or intermittent link, and when your branch office loses connectivity to the main office.
In these circumstances, how do you maintain a copy of a file on the offline computer while the original is on the main server? More importantly, how do you handle a conflict if someone in the home office changes the file on the server while someone else changes it on the road? Fortunately, these issues are handled by the Offline Files engine, which is built into Windows® 2000, Windows XP, and Windows Vista®.

Caching Is the Answer
The Offline Files engine is really a giant caching machine. Indeed, underneath the hood (and inside Microsoft) it's referred to as client-side caching (CSC). The Offline Files engine is configurable and flexible enough to let users decide what they want to cache as well as help you get the system to cache what you want. It allows you to access the same files offline as online—and the namespace doesn't change. That is, files are accessed using the same UNC path or drive letter whether you're working online or offline.
You can set up the files to be cached either manually or automatically. If you know you often use particular files or folders while on the road, you can simply designate a file or folder as one that will travel with you. To do this, a Windows XP user would right-click on a file that is stored locally or on a network share and select the option to make files available offline (see Figure 1). In Windows Vista the option is called Always available offline. Manually making a file available offline is sometimes called pinning a file.
Figure 1 Making a file available offline
Figure 1 Making a file available offline
Figure 1a The Offline Files Folder
Figure 1a The Offline Files Folder
The share you're using can be connected via UNC path or mapped with a drive letter. Indeed, the file doesn't even need to be one that's stored on a machine running Windows Server®. It can reside on anything that serves up the SMB (Server Message Block) protocol, such as a Samba server or NAS device (with some rare exceptions), which also adhere to the SMB protocol.
The first time you choose to pin a file, Windows XP walks you through a series of wizard screens, asking when to synchronize. If you click Next on every screen and take the defaults, synchronization happens every time you log on or log off, and also in the background when idle. When the wizard completes, you'll see an icon change indicating that the file is now available offline.
Once you disconnect from the network, all other files on the share become unavailable except those chosen for offline use. That's sometimes problematic: the icon shows which files are always available offline, but it's not immediately clear before you've disconnected that not all the other files will be available. I'll discuss this in more detail later in the article.
Making files available offline manually has its merits, but sometimes it makes sense to do it in a more automated manner. I'll explore that scenario next, using what's often called autocaching.

When Automatic Caching Beats Manual Caching
Autocaching is set up on a server on a share-by-share basis. So if you have a share called Sales and want to let salespeople take their files with them as they travel, you can instruct the CSC engine to do that. Figure 2 shows the Offline Settings dialog that appears when you click the Caching tab in Windows Server 2003. The settings are explained in Figure 3.
Figure 3 Autocaching settings (Close Figure 3)

Setting | Description
Only the files and programs that users specify will be available offline. | Allows users to manually specify the files to be available offline. This is the default.
All files and programs that users open from the share will be automatically available offline. | Allows autocaching of all files and programs that Windows Explorer touches. The Optimized for performance setting is used when caching programs rather than documents, but it typically has no effect on performance and can be safely ignored in most cases. It is completely ignored by Windows Vista.
Files or programs from the share will not be available offline. | Blocks user access to the Make available offline item when used on a Windows 2000, Windows XP, or Windows Vista client.
Figure 2 Offline Settings dialog in Windows Server 2003
Figure 2 Offline Settings dialog in Windows Server 2003
To see how autocaching works, let's assume you have a Sales share with 10 files in it—Sales01.txt through Sales10.txt—and that you've enabled the setting called All files and programs that users open from the share will be automatically available offline. The results are not immediately obvious. To the user, there's no outward sign that more files are suddenly available to them offline. Plus, only files that are opened by Windows (and hence cached) are later available to be used offline. In short, you need to actually open the files online that you want to use offline before actually disconnecting from the network. Otherwise, they won't be available offline when you need them.
Moreover, files that are autocached won't necessarily stay around forever in the cache. Windows XP and Windows Vista handle this temporary space a little differently, but the key point is that files can be ejected based on space constraints. Files are kept on an as-used basis. Those that haven't been opened in a while are ejected to make space for new files as those are opened. So, while there's a high probability that a file you recently touched is still in the cache, don't assume that autocaching guarantees that files are available. If you want to be sure a file is available, you need to pin the file. As the administrator, you can also use Group Policy to pin files with the Administratively assigned offline files setting.
Let's take a look at an actual working example. Suppose that a Windows XP user named EastSalesUser1 opened up a file named Sales05.txt and a Windows Vista user named EastSalesUser2 opened up Sales05.txt on the \\server\sales share.
Because autocaching has been set on the sales share, when the EastSalesUser1 (on Windows XP) and EastSalesUser2 (on Windows Vista) went offline, those files could still be worked on. Windows has stored a copy of those files in the Offline Files cache. But what about the other files the users haven't yet used?
Windows XP and Windows Vista have a different reaction to files left behind at the office, as you can see inFigures 4 and 5. Windows XP makes no indication that there even were files left behind uncached. When you go offline, those files just disappear in a potentially confusing manner. Windows Vista, in contrast, has an icon overlay and shows a little dimmed X against files that haven't been cached. This is a huge leap forward in showing which files are available offline. The only problem is, you only see it after disconnecting, not during an online session. So making a file's status more explicit would be a hugely welcome addition.
Figure 4 Windows XP only shows available files after going offline
Figure 4 Windows XP only shows available files after going offline
Figure 5 Windows Vista shows the unavailable files while offline
Figure 5 Windows Vista shows the unavailable files while offline

Available—or Not
Some administrators are hesitant to take advantage of Offline Files, precisely because of the potential confusion: which files are definitely available offline and which are not, and how you can tell. In Figure 5, the green circle icon over sales01.txt indicates that the file is guaranteed to be available. (Again, when files are pinned, the file is guaranteed to be available offline.)
However, as I explained, Sales05.txt, which both Windows XP and Windows Vista have made available offline via autocache, is not necessarily available offline. Remember, only the files you touch will be available offline. So it would be better if the system indicated that the files are temporarily or possibly available offline. Since there is no icon differentiation, determining which files will be available offline can be confusing. The Windows Vista UI is better, but it still takes a keen eye to spot. An Explorer property in Windows Vista titled Offline Availability does provide availability information. That property is displayed in the preview pane when an item is selected. If the item is pinned, the value is "Always available." If not pinned but cached, the value is "Available."
The idea of temporarily available offline files is similar for Windows XP and Windows Vista, but the implementations are slightly different. Figure 6 shows how Windows XP and Windows Vista handle offline files.
Figure 6 Windows XP and Windows Vista offline file areas
Figure 6 Windows XP and Windows Vista offline file areas
To put it simply, both Windows XP and Windows Vista have one place in which to hold Offline Files, but the Always Available Offline files and the Temporarily Available Offline files are stored a little differently for each operating system.
For Windows XP, files that are always available offline (designated by the icon change as seen in Figure 1) are placed where they stay forever. If you pin 10GB of files manually, thereby making them Always Available Offline, all 10GB would always be available. There is no limit to the number of files a user can pin (outside of the maximum partition size). However, Windows XP has a 2GB maximum for autocached files. The idea is that only files that are routinely opened are stored here while those that haven't been used for a while are kicked out when necessary. By default, the Temporary Available Cache is allocated 10 percent of the free space on the hard drive, as shown in Figure 7. This setting is only available to administrators on Windows XP.
Figure 7 Setting disk space for offline files in Windows XP
Figure 7 Setting disk space for offline files in Windows XP
Windows Vista can also store files permanently or temporarily. However, there's a difference. The Temporary Files Cache is contained within the overall cache. By default, the entire cache (both Temporary Files and Always Available Offline files) is limited to 25 percent of the free space on the hard drive. The reason this was changed is because in Windows XP a user could keep manually caching files (perhaps multiple tens of gigabytes)—and run right out of hard drive space. With the new scheme, the administrator sets aside a specific amount of space for the entire cache so that can't happen. When you look at Figure 8, note that the 25 percent of the free space on the drive is confusingly shown as 15.2 percent of the whole drive. Note also that the second slider, which sets the temporary space, can never be set higher than the first slider, which sets maximum space of all offline files. These Offline Files Disk Usage Limits sliders are accessed by clicking the Change Limits option, but be aware that you need administrator credentials to do this.
Figure 8 Space available for offline files in Windows Vista
Figure 8 Space available for offline files in Windows Vista

Preventing Too Much Caching
Folder Redirection is another Windows feature that lets you ensure that certain key folders normally local on the machine are actually stored on servers. The idea is that you don't have to worry if that local machine goes belly-up; you've got a copy of important files on a network share.
Suppose you have a redirected My Documents folder (for Windows XP) or Documents folder (for Windows Vista). In this case (in fact when any folder is redirected), the operating system assumes you always want these files available offline and every file within the redirected folder will receive the symbol shown in Figure 1. The downside is that if you're on a well-connected, always-online network and you roam to a new machine, you're caching the entire contents of the redirected folder on the new machine—even if you only plan to use it for one session. Not only is this a waste of space, it's a security risk if not properly handled.
Another case where files are automatically and unknowingly cached is when you use Windows Explorer to browse the files on a share. If you merely look at a gaggle of, say, graphic files on a network share with Explorer and autocache is turned on, it's very likely those files will be automatically downloaded and placed into your cache. That's because using Explorer to touch a file is pretty much all it takes to put it into the temporarily offline cache. Often every file in the share is touched to get metadata such as file size or a preview of an image. When this happens over a fast, well-connected link, it's no big deal. Windows quickly and quietly caches the file for later use. But when you're connected to that network share with autocache over a slow link, things can really get bogged down. The situation is basically unchanged by default for Windows XP Explorer and Windows Vista Explorer. The good news is that the Windows Vista Offline Caching engine can be trained to better recognize a slow link.

Offline Files with Group Policy
Top Changes to Windows Vista Offline Files
The Offline Files team members want you to know about their favorite changes, but there wasn't room to explore all of them in the article. So, in a nutshell, here are the major changes in Offline Files:
  1. Seamless offline-to-online transitions means that open handles no longer block transition.
  2. Per-file offline granularity so one offline share no longer takes the entire server offline.
  3. Forced offline mode via a Work online\offline button in Windows Explorer.
  4. Per-user encryption of cached content.
  5. New synchronization engine that's much faster at detecting and analyzing changes.
  6. Integration with Windows Search so that redirected folders can be indexed.
  7. Local-to-remote synchronization can leverage Differential Bitmap Transfer feature.
  8. Deferred conflict resolution via Sync Center.
  9. Notification area balloons are gone; no longer bother user.
  10. New comprehensive COM API and WMI support for management; documented on MSDN® at msdn2.microsoft.com/bb530662.
  11. Script support through a WMI provider (which will be documented on MSDN shortly).
  12. New shell item properties Offline Status and Offline Availability.
  13. Ghosting to represent unavailable namespace elements while offline. Explorer displays these as dimmed items with a gray X overlay, helping to avoid drastic shell folder view changes when transitioning from online to offline.
  14. New Control Panel applet.
You'll find lots of information on these in my book and in Microsoft documentation, if you decide you want to learn more.

As we noted, whenever a user (or Explorer) touches a file, the whole file is brought into the cache. And if that file is an 80MB Visio® or Word document, so be it. Both Windows XP and Windows Vista understand what a slow link is. Indeed, by default, Windows XP thinks a slow link is 64Kbps. If the connection speed is greater than this and the client is Windows XP, that file is brought down through that slow pipeline at logon time, much to the dismay of the user. So it's best to configure Windows to better manage what to do over that slow link. As you'll see in a moment, you can accomplish this via Group Policy by creating a Group Policy Object (GPO) to be linked to an organizational unit (OU) that contains the target accounts—Windows XP or Windows Vista.
When the user connects over a slow link, Windows XP tries to use the locally cached files before attempting to use the network copy. This makes sense. But Windows XP still does something you might not expect: it downloads big files the user does not yet have in the cache if the user opens them from the network or Explorer touches them. Ouch. And, in my testing, if synchronization is scheduled to happen at logoff (the default), the remainder of those files are then downloaded over the slow link and put into the cache. (Double ouch.)
Windows Vista behavior is different. It doesn't recognize any situation as slow by default. Indeed, if you connect over a tin can, getting 1-bit per second, Windows Vista treats that connection as fast and uses the copy of the files on the network. But Windows Vista can be trained to understand what shares you want to designate to transition when the connection is slow. When such a scenario occurs, things get a little better. The Windows Vista behavior (once configured using Group Policy) is such that the contents of slow-link-defined shares are simply not available over a slow link using normal file and Explorer operations.
The background synchronization performed by the Offline Files service does not occur over a slow connection. However, it should be noted that manual synchronization can be performed via Sync Center.
Configuring Windows Vista to modify its handling of slow links is done via the Group Policy setting Configure Slow-link Mode (not to be confused with the similarly named Windows XP setting Configure Slow Link Speed). Configure Slow-link Mode takes the following values: the names of servers and shares you want to automatically transition to offline when the connection is slow, and how slow that connection needs to be before that server\share combination goes offline. The server\share combination can be any combination of \\server\share or even just a single star (*) to indicate all shares on all servers. The second set of values is a declaration of the throughput (in kbps) and latency (in milliseconds) that has to be crossed in order to be declared slow. Here's the scoop, though: the more useful of the pair (throughput) does not get checked when downloading files. So if you have 10GB of files in a redirected Documents folder and you're on a slow link on a laptop that doesn't yet contain any of those files—be prepared for a long wait. However, if you pair it up with the latency check, and the latency dips below the prescribed value, the share will transition to offline, sparing the user the painful delay of waiting for those files.
The Windows Vista hotfix in Knowledge Base article 934202 (a change that's slated for inclusion with Windows Vista Service Pack 1) should improve performance over certain slow VPN conditions. (This article is available online at support.microsoft.com/kb/934202.) In fact, this hotfix should enable the throughput value to be checked when downloading occurs over a slow link.

Synchronization Conflicts
You're probably wondering what happens if someone changed a file offline (say, while on an airplane), while another person changed it online (hardwired at the main-office). This situation is handled at the client level. When you transition from offline to online, the Windows Vista Offline Files background synchronization occurs immediately. Then you're prompted about what to do (through an entry in the Sync Center after an icon appears in the taskbar notification area). The user interface has been redesigned to show that one file seems to be newer and to show more information about the collision. A user can preserve files with collisions by choosing to rename the file that's currently only on his machine when saving it back on the server, as in Figure 9.
Figure 9 Windows Vista file change conflict notice
Figure 9 Windows Vista file change conflict notice

Want to Learn More?
Offline Files is a nice feature for both you and your users. Indeed, if you're using Folder Redirection, you're already using it, so getting smarter about it is in your best interest. I've mentioned a number of changes to Offline Files in Windows Vista, but not all of them. For more on this, see the sidebar "Top Changes to Windows Vista Offline Files."
You'll find a lot more information about Offline Files online at go.microsoft.com/fwlink/?LinkId=98141 for Windows XP and go.microsoft.com/fwlink/?LinkId=98138 for Windows Vista.
There's also a huge section dedicated to Offline Files—including the ins and outs of every Group Policy setting—in my book (see the bio at the end of the article). Finally, for even more information and to talk with others about your Offline Files experiences, spend some time at GPanswers.com in the community forum where there's a section dedicated to just this topic.

Jeremy Moskowitz MCSE and MVP in Group Policy, runs GPanswers.com, a community forum on Group Policy. He also runs a series of Group Policy intensive training workshops. His latest book is Group Policy: Management, Troubleshooting, and Security (Sybex, 2007). Contact Jeremy and learn more at www.GPanswers.com.
© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


 Desktop Deployment: Simple, Scalable Deployment with BDD 2007
Desktop Deployment Adam Shepherd

Simple, Scalable Deployment with BDD 2007
At a Glance:
  • Core capabilities of BDD 2007
  • Integrating with SQL Server
  • Building a scalable deployment solution

Most people involved in deploying Windows have heard of the Microsoft Solution Accelerator for Business Desktop Deployment. Also known as BDD, it is a set of best practices guidance and
tools developed to help you deploy Windows® to client desktops. Windows Vista® includes an update to the Solution Accelerator in the form of BDD 2007. This version contains a new Microsoft® Management Console (MMC) and a task sequencer that originates from System Center Configuration Manager 2007, the successor to Systems Management Server (SMS) 2003.
One of the less understood features of BDD 2007 is its ability to function as a full deployment solution. In previous versions, BDD could help you create and maintain desktop images in a manageable and repeatable way. BDD 2007 now gives you the additional capability to build a scalable deployment solution by combining BDD with Windows Deployment Services (WDS), SQL ServerTM, and the Windows Server® 2003 Distributed File System Replication (DFS-R) .
But wait ... isn't using SMS 2003 with the Operating System Deployment (OSD) Feature Pack the recommended practice for Windows client deployment?
Yes, if you already have an SMS 2003 infrastructure, you should work towards using BDD 2007, SMS 2003, and the OSD Feature Pack for deployment. This is currently the most comprehensive way to achieve zero-touch deployment, and it can really drive down the costs of configuring, installing, and managing Windows in your organization.
However, many customers don't have SMS 2003 or any equivalent software deployment solution. For these customers, there is another option that I'll focus on in this article.

BDD 2007 Core Concepts
BDD 2007 offers two main methods of deployment. The first method, Lite Touch Installation, uses just BDD 2007 for deployment of the client. Out of the box this presents a set of wizards at build time to capture information about the deployment, such as what computer name should be assigned to the client (see Figure 1), what keyboard layout and time zone should be used, and so on. This method is referred to as "Lite Touch" because there's usually a minimal amount of manual input required to collect this information before the build process starts.
Figure 1 BDD 2007 Lite Touch Deployment Wizard
Figure 1 BDD 2007 Lite Touch Deployment Wizard
The second method, Zero Touch Installation, builds on the capabilities of SMS 2003 with the OSD Feature Pack by integrating with BDD 2007. This configuration can provide a completely automated, scalable, and manageable hands-off deployment solution.
At build time, in one of the first steps of a client OS deployment, BDD 2007 gathers information from a variety of different sources. These sources include WMI calls and the BDD 2007 configuration files—Bootstrap.ini and CustomSettings.ini. The information is collected and stored as variables that are used throughout the deployment. For a Lite Touch scenario, the Bootstrap.ini and CustomSettings.ini files provide the ability to predefine all of the information that would otherwise be collected by manual input into the deployment wizards.
The ability to specify in advance all of the information that the Lite Touch wizards would otherwise need to capture at build time allows you to automate the entire Lite Touch process and move towards a Zero Touch solution in those environments not supported by SMS 2003. Examples of the information you might choose to predefine include ComputerName=RDG0001VST to assign the computer a name of RDG0001VDT or TimeZoneName=GMT Standard Time to ensure the computer time zone is set to Greenwich Mean Time. All of the available properties that can be used during a deployment with BDD are well documented in a configuration reference available at tech-net.microsoft.com/library/bb490302.aspx.
For more on setting up and using BDD 2007, see the article "Deploying Windows Vista with BDD 2007" in the September 2007 issue of TechNet Magazine (technetmagazine.com/issues/2007/09/BDD).

BDD and SQL Server
Why use a database with BDD 2007? The answer lies in how you dynamically provide details during a deployment that you would otherwise have to manually input through the Lite Touch Installation wizards. As I noted, this can be achieved using the CustomSettings.ini file. The problem is that in a real-world deployment these details are going to change depending on which machine you're deploying to. Also, the CustomSettings.ini file can quickly become unmanageable if you're trying to define specific options and settings for 500 different machines. By attaching a back-end database to the BDD console, you add a much more manageable solution that's both dynamic and extensible.
Let's talk a little bit more about how SQL Server integrates with BDD 2007. First, you need to make sure SQL Server (preferably SQL Server 2005 or later) is installed on your deployment server. Thankfully, the steps to create the database in BDD 2007 are easy: you simply right-click on the Database node in the BDD 2007 Deployment Workbench, click New, and follow the instructions (see Figure 2). The BDD team has done a great job providing wizards that allow you to quickly and easily get up and running.
Figure 2 Deployment database configuration for BDD 2007
Figure 2 Deployment database configuration for BDD 2007
Once the database has been created, you can start exploring its capabilities. Entries in the database are split into four main categories: Computer, Role, Location, and Make and Model. Each category lets you define entries and, for each entry you define, populate with BDD 2007 variables, assign applications, and supply other key settings (see Figure 3).
Figure 3 Assigning deployment properties
Figure 3 Assigning deployment properties
It's useful to look at these four categories in two distinct groups. The Computer, Location, and Make and Model categories give you different methods of identifying a computer at build time. Each identified computer can then be associated with a specific Role in the database so it is provisioned for a specific purpose, such as a sales, marketing or finance, with each role ensuring the relevant line of business applications are installed.
Use the Computer section to identify machines by using a MAC address, asset tag, universal unique identifier (UUID), or serial number. Because of the large number of entries likely to be created in this section, one for each computer in your organization, only add very specific properties such as ComputerName into each entry.
Use the Location section to create entries for specific geographical locations around your organization. Each entry is identified by the default gateway at that location. This can be especially useful if you have location-based organizational units (OUs) in Active Directory® and you want computers built at a location to be added to the domain in the OU that corresponds with that location.
Use the Make and Model section to create entries for each of the hardware types that your deployment must support. BDD 2007 matches these with information discovered through WMI calls. A common usage of the Make and Model section in development environments is to create an entry that looks to see whether the machine is a Virtual PC 2007 or Virtual Server 2005 virtual machine by checking for "Microsoft Corporation" and "Virtual Machine" Make and Model properties, respectively. If at build time you match these properties, you can tell BDD that it needs to install the Virtual Machine Additions application at deployment time simply by adding the application to the Make and Model entry.
Now that you've provided methods of identifying the machines, use the Roles section to create the entries for each deployment role you want to provision. Examples might include an entry for a Windows XP role and a Windows Vista role configured by adding the BuildID property that corresponds to the Windows XP or Windows Vista build defined in BDD to each entry. An alternative approach is to specify departmental roles such as Accounting, Sales, or Finance. I tend to populate the majority of the BDD properties in this section (time zone, regional settings, organizational information, and so on). This offers the minimum administration of the database because of the fact that the entries are not tied to any specific machine, hardware, or location.
We need to tell BDD 2007 that it should search the database at deployment time for any entries that match the computer it's currently building. To configure BDD to use the database during a deployment, right-click on your deployment point and click Configure DB. This takes you through a series of wizards that populate the CustomSettings.ini file with the entries that cause the Lite Touch Installation process to query the database for information.

Making Your Deployment Scalable
Now you have a deployment server that can automate the Lite Touch Installation and dynamically provision each machine to function in a specific business role. But how can you scale this solution?
The architecture of the solution is based around a hub-and-spoke topology as illustrated in Figure 4. The first deployment server that you set up will be the hub and the parent deployment server. Each of the child deployment servers will act as spokes.
Figure 4 Hub-and-spoke deployment architecture
Figure 4 Hub-and-spoke deployment architecture
To enable this architecture, you need to use DFS-R to replicate the Distribution share to each of your deployment servers. Then use SQL Server snapshot replication to provision a copy of the BDD deployment database to each of the child deployment servers. The great thing about this solution is that the child deployment servers have minimal requirements—you only need SQL Server Express, WDS, and DFS-R installed on each machine to enable deployment.
Because of the large amounts of data that can be stored in the BDD 2007 Distribution share, I stipulate using DFS-R in Windows Server 2003 R2 instead of the File Replication Services in earlier version of Windows Server. DFS-R uses Remote Differential Compression (RDC) to only replicate differential (delta) changes of files between members of the replication group. This can have a big impact on the amount of replication traffic when a small change such as a new driver is made to your custom image file. With DFS-R, the replication traffic equates only to the size of the changes made instead of redistributing the entire image file.
There's a wealth of information about DFS on the Microsoft Web site—start by checking out microsoft.com/windowsserver2003/technologies/storage/dfs—but I'll cover the basic setup and configuration steps required to give an overview of the process in the context of BDD-based deployment.
Note that if your servers have been running Active Directory based on an installation of Windows Server 2003 prior to R2, the Active Directory schema may need to be updated to allow for DFS-R as the replication service requires new object classes. For full details on extending the Active Directory schema in Windows Server 2003 R2 see go.microsoft.com/fwlink/?LinkId=99936.
The first step is to install the DFS components onto your Windows Server 2003 R2 deployment server. You can do this in multiple ways, one of the simplest being the standard Add/Remove Windows Components wizard. Once that's done, you need to set up a replication group using the DFS Management console shown in Figure 5.
Figure 5 DFS Management console
Figure 5 DFS Management console
To scale out your BDD 2007 deployment solution, there are two specific folders that need to be replicated. The first of these is the BDD 2007 distribution share that holds all of the BDD source and configuration files. These files need to be available on each deployment server.
The second folder to replicate is the Boot folder that WDS uses to hold the LiteTouch_x86.wim file. This is the BDD boot environment that is delivered by WDS to your desktop client machines to start the deployment process. You need to replicate this file so that when a change is made to the boot environment on your parent deployment server, the changes get replicated around to the rest of the deployment infrastructure.
The paths to be replicated are X:\Distribution (where X is the drive letter you specify at BDD 2007 install time) and Y:\RemoteInstall\boot (where Y is the drive letter of the volume holding the WDS RemoteInstall share).
Replication in DFS-R is multi-master and therefore it is not possible to create a one-way replication topology. For this reason, to allow for centralized management of the distribution share, configure the distribution folder on each of your child deployment servers to be read-only. This allows only DFS-R replication to write into the folder (as it uses backup and restore rights). The client machines building from these child servers will never need to write into this folder, so there's no need to grant any accounts higher than read permissions.
The final configuration step to complete replication of data using DFS-R is to set the Boot Configuration Data (BCD) store refresh policy in WDS. This ensures that any changes made to your boot environment get reflected on each of the child deployment servers. This configuration change has to be set on all WDS servers, and I recommend building this into your build and configuration process for the provisioning of a deployment server.
At each deployment server, run the following command:
WDSUTIL /set-server /BCDRefreshPolicy /Enabled:yes /RefreshPeriod:<time in minutes>
The time refresh period depends on how often you are refreshing the data on the deployment servers. If you're configuring DFS-R to replicate every hour, it makes sense to configure the BCD store refresh policy to refresh every 60 minutes.

SQL Server Replication
So far you've scaled out BDD 2007 to replicate the BDD distribution share and the WDS boot image from your primary deployment server to each of your child deployment servers. You now need to complete the process and replicate the BDD 2007 Deployment Workbench database so that it is available locally at each deployment server.
SQL Server uses a replication methodology that the product documentation describes as not dissimilar to the way magazines are published. With a magazine, there is a publisher who produces the magazine, distributors that distribute the magazine for the publisher, and subscribers who subscribe to and receive the magazine. SQL Server uses this same terminology for its built-in replication capabilities.
Note that, in order to be a SQL Server database publisher, the server cannot be running SQL Server Express Edition; it must be using a full version of SQL Server. For the primary deployment server, I use SQL Server 2005 (though SQL Server 2000 is supported, too). For each child deployment server, you can use either SQL Server 2005 or SQL Server 2005 Express.
Before you start configuring SQL Server replication, there are a few steps you should complete to ensure SQL Server is set up correctly to support replication. When provisioning your deployment servers, make sure that the Replication Components are included during the installation of SQL Server 2005 or SQL Server 2005 Express Edition. By default, SQL Server Express does not install Replication Components.
Next, allowing the Lite Touch boot environment to connect to SQL Server remotely requires that remote connections are enabled in SQL Server. To configure SQL Server for remote connections, launch the SQL Server Surface Area Configuration tool, select Surface Area Configuration for Services and Connections, and configure the server to accept both local and remote connections from TCP/IP and named pipes.
Then, on the primary deployment server, create a shared folder to hold the replication snapshot data that the replication agents on each child deployment server will read from. I'll be referring to this shared folder later on. I usually put this folder on the same volume as the rest of my deployment content.
The final setup steps are required only for SQL Server Express Edition on your child deployment servers. By default, the SQL Server Browser service is disabled. To allow replication, this service must be set to automatic and started. Use the SQL Server Configuration Manager tool to configure this service. You also need to create a database on the child servers that you will replicate the content into. Keep this database name the same as the name of your BDD 2007 database on the primary deployment server as it minimizes the additional configuration required.

Setting Up Replication
You're now ready to configure SQL Server replication. To get started, on the master deployment server, which should be running the full version of SQL Server 2005, launch SQL Server Management Studio. The first step is to create and configure a distribution. To do this, navigate to the Replication folder of the management console as shown in Figure 6. Right-click the Replication folder and select Configure Distribution to launch the distribution wizard.
Figure 6 SQL Server Management Server: Replication
Figure 6 SQL Server Management Server: Replication
In the wizard, confirm that the primary deployment server should act as its own distributor. Set the root snapshot folder to be the UNC path of the SQL Server replication share that was created earlier. Finally, set the primary deployment server to be configured as a publisher.
When this wizard has successfully configured the server as a publisher and distributor, you need to tell SQL Server which database you want to be published. To do this, right-click the Replication folder and click Publisher Properties. In the properties dialog box, select Publication Databases and choose your BDD 2007 database for transactional replication as shown in Figure 7. Now you're not actually using transactional replication, but this same option is selected whether your intention is snapshot or transactional replication.
Figure 7 Publishing your database
Figure 7 Publishing your database
You now need to create a publication that your child deployment servers can subscribe to. In the Replication Folder, right-click on Local Publications and click New publication to launch the publication wizard. Select the BDD database as the database to be published. Select Snapshot replication as the publication type, and specify all tables, stored procedures, and views to be replicated. You should elect to have an initial snapshot created immediately. For scheduling the snapshot agent, it should be more than sufficient for this to run once a day by default unless you anticipate making frequent changes to the database.
The final step is to subscribe each of your child deployment servers to this publication. This tells SQL Server to push a copy of the database out at specified intervals to each child deployment server. In the Replication folder, under the Local Publications subfolder, locate the publication that was just created, right-click on it, and select New subscriptions to launch the subscriptions wizard. In this wizard, select the BDD publication created in the previous steps as the chosen publication. Choose to run all agents at the distribution, enabling the replication topology as a push subscription. Then add each child deployment server as a subscriber and nominate the database that you have created on each server to receive the copy of the replicated data (remember, this database should be named the same as the primary deployment database). Finally, configure an account to be used for the connections and define the schedule to be used for the replication. This should mirror the schedule you've chosen for the snapshot agent. That's a whirlwind tour of what you need to get up and running for SQL Server replication. You should start to see your data being replicated around in no time at all!

Configuring BDD 2007
So far you've configured BDD 2007 to use a database and set up replication of the database and BDD 2007 distribution share to your child deployment servers. To complete the deployment solution you need to configure BDD 2007 so that, when a client downloads the BDD boot environment, it automatically connects to whichever server is its local deployment server.
When you boot into the Lite Touch Windows PE environment, if the client has booted from a WDS server, a registry value is set in Windows PE that stores the name of the server that the client has just downloaded the boot environment from. The initial BDD 2007 deployment scripts pick up this value and store it in an environment variable called %WDSServer%.
Note that, if you're using the initial BDD 2007 release, you may find that this value isn't populated properly in the Windows PE registry. An update to fix this behavior is available from support.microsoft.com/kb/937191.
To configure BDD 2007, just edit the BootStrap.ini and CustomSettings.ini configuration files so that any references to the deployment server are replaced with %WDSServer%. This usually involves replacing the values for the SQL Server instance name along with the value of DeploymentRoot, the server that holds the distribution share. A sample Bootstrap.ini and an excerpt from a CustomSettings.ini file are shown in Figure 8 and Figure 9.
Figure 9 CustomSettings.ini (Close Figure 9)
[Settings]
Priority=MMSettings, MMApps, CSettings, CRoles, RSettings, RApps, Default
Properties=Manufacturer

[Default]
OSInstall=Y
DeployRoot=\\%WDSServer%\Distribution$
UserDomain=BUILDLAB
UserID=BddAdmin
UserPassword=Password1234

[CSettings]
SQLServer=%WDSServer%
Database=BDD_DB1
Netlib=DBNMPNTW
Table=ComputerSettings
Parameters=UUID, AssetTag, SerialNumber, MacAddress
ParameterCondition=OR

[CRoles]
SQLServer=%WDSServer%
Database=BDD_DB1
Netlib=DBNMPNTW
Table=ComputerRoles
Parameters=UUID, AssetTag, SerialNumber, MacAddress
ParameterCondition=OR
Figure 8 Bootstrap.ini (Close Figure 8)
[Settings]
Priority=Default

[Default]
DeployRoot=\\%WDSServer%\Distribution$
UserDomain=BUILDLAB
UserID=BddAdmin
UserPassword=Password1234
SkipBDDWelcome=YES
Note that this solution stipulates having WDS on the same server that stores the BDD 2007 replicated database and the BDD 2007 replicated distribution share. The reasons should be apparent: it enables you to use the %WDSServer% variable to provide a simple failsafe method to tell your client machine where its local deployment server is and prevent it from talking back across the WAN.

Additional Considerations
The great thing about BDD 2007 is that it's extensible, so there are many more aspects that can be considered for extending this solution. For example, you could use the built-in capabilities for monitoring SQL Server and DFS-R, designing methods for populating the computers in your environment with information from an asset management database, or even using a SQL stored procedure to retrieve such information. Unfortunately, I don't have the space to cover these here, but hopefully I've given you a glimpse of some ways you can harness BDD's capabilities inside your own organization.

Adam Shepherd is a consultant working for Microsoft Consulting Services in the UK. Adam's technical focus is on deploying, maintaining, and managing the Windows Client. You can reach Adam at adamshep@microsoft.com.


 Communications: Exchange Edge Transport Servers at Microsoft: Part 2
Communications Kay Unkroth

Exchange Edge Transport Servers at Microsoft: Part 2
At a Glance:
  • Using Edge Transport logs and traces
  • Anti-spam configuration with scripts
  • Testing typical spam scenarios
Download the code for this article: Edge2007_11.exe (161KB)

Faced with a flood of spam and other malicious content flowing across the Internet, how do you ensure reliable and safe messaging to users within your organization? One way, which I started to describe last month in the first part of this series, is to deploy
Exchange Server 2007 Edge Transport servers and Forefront Security for Exchange Server in a perimeter network between the Internet and your production environment.
In this final installment of my two-part series, I will explain how to analyze Edge Transport agents in action by using standard agent logging and tracing features of Exchange Server 2007. I will also use a custom method to dump all event objects from the transport pipeline into XML-based files. I will then illustrate how to apply anti-spam settings to an Edge Transport test lab according to the configuration that Microsoft uses in its own corporate production environment. Finally, I'll conclude with a few realistic and interesting test scenarios to see how Edge Transport agents respond to various spam situations.
For more information on the installation of the Edge Transport test lab and the general Edge Transport architecture, see the first article of this series in the October 2007 issue of TechNet Magazine, available at technetmagazine.com/issues/2007/10/Edge.
I make heavy use of scripts and batch files to automate the most important configuration tasks, and you can find these scripts in the code download that accompanies this article (go to technetmagazine.com/code07.aspx). Furthermore, for more detailed background information regarding the Microsoft IT configuration of Edge Transport agents, I recommend the technical white paper "Microsoft® Exchange Server 2007 Edge Transport and Messaging Protection," produced for Microsoft IT Showcase. You can download this IT Showcase white paper from microsoft.com/technet/itshowcase/content/edgetransport.mspx.

Logging and Tracing
Exchange Server 2007 includes a number of logging and tracing features, including protocol logging, agent logging, and pipeline tracing, that facilitate diagnosing and troubleshooting transport agent problems. Although I am not exactly troubleshooting transport agents in this article, the logging and tracing information is useful for examining how Edge Transport agents work.
If you want to track information about SMTP send and receive activities, you should enable protocol logging for send and receive connectors. Using the test environment I set up in the first part of this series (see Figure 1), I enabled protocol logging by means of the following setting on the New-SendConnector and New-ReceiveConnector cmdlets:
Figure 1 The test environment
Figure 1 The test environment
–ProtocolLoggingLevel: Verbose
You can find the resulting SMTP log files by default in subfolders under %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog.
Protocol logging is useful if you want to analyze SMTP conversations, yet anti-spam agents do not always interfere with the SMTP session. For example, the Content Filter agent might only stamp inbound messages with a Spam Confidence Level (SCL) value. As long as the SCL rating is below a specified threshold (the default is 7), the Content Filter agent does not cause the Edge Transport process to reject the message. Therefore the SMTP conversation progresses unaffected.
To analyze the actions that anti-spam agents (Connection Filter, Content Filter, Edge Rules, Recipient Filter, Sender Filter, and Sender ID) perform on messages that pass through the transport pipeline, you need to check the agent logs instead of the protocol logs. Agent logging is typically enabled according to the AgentLogEnabled parameter in the EdgeTransport.exe.config file. By default, agent logs are located in the %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\AgentLog folder. You can open the logs in Notepad, though the Get-AgentLog cmdlet displays the agent log information more intuitively in the Exchange Management Shell.
Another useful feature is pipeline tracing, which enables you to capture snapshots of messages from a particular sender as the messages pass through the transport pipeline. The idea is relatively straightforward; the Edge Transport process creates a snapshot of each message before and after invoking SMTP receive and routing agents registered for the OnEndOfHeaders, OnEndofData, OnSubmittedMessage, and OnRoutedMessage events. By comparing the message snapshots, you can see how individual agents modify each message. Of course, it is imperative that you enable this valuable feature in the test environment.
The following usage of the Set-TransportServer cmdlet activates pipeline tracing for a test user. The PipelineTracingEnable parameter set to $true enables pipeline tracing, as you would expect. The test user's e-mail address is set in the PipelineTracingSenderAddress parameter:
Set-TransportServer –Identity EDGE01
–PipelineTracingEnabled $true
-PipelineTracingSenderAddress Contoso.User@contoso.com
With pipeline tracing enabled, you can find message snapshot files for each message from the specified sender (in this case, it is Contoso.User@contoso.com) in the %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\PipelineTracing\MessageSnapshots folder. This is the default folder. For each message, the snapshot files are located in a subdirectory named according to a GUID assigned to the message. Note that you can find the original message in a file called Original.eml and copies of the message in .eml files that start with SmtpReceive or Routing according to the encountered events and installed agents. To analyze the processing of anti-spam agents registered for the SMTP receive events OnEndofData and OnEndOfHeaders, check out the SmtpReceive*.eml files. To analyze the processing of antivirus and other agents registered for the routing events OnSubmittedMessage and OnRoutedMessage, analyze the Routing*.eml files instead.

Custom Pipeline Dumping
The standard agent logging and tracing features of Exchange Server 2007 address all troubleshooting needs despite the fact that the standard features do not cover all transport events. For example, pipeline tracing cannot capture information about OnConnectEvent, OnEhloCommand, OnMailCommand, OnRcptCommand, OnDataCommand, and similar transport events because the sending host has not yet transferred any messages that could be written to the %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\PipelineTracing folder. If you want to capture detailed information about these events, you need to implement a custom agent that dumps the event data into files on the file system.
As discussed in my previous article, it is not complicated to create custom agents, and so I could not resist building an SMTP receive agent and a routing agent to dump out all event objects from the transport pipeline into XML files. You can find a corresponding Microsoft Visual Studio® 2005 project, called PipelineXMLDump, in the download that is available for this article. The two important files are DumpAgents.cs, which implements the agent factories and agents, and XMLDump.cs, which implements a helper class to write the fields of event source and event argument objects by means of System.Re- flection methods into XML files. If you compile the source code and copy the resulting PipelineXMLDump.dll file into a folder called C:\PipelineXMLDump on the Edge Transport test server, you can register and enable the custom agents by using the scripts RegisterPipelineXMLDump.ps1 and EnableDumpAgents.ps1. The scripts are included in the PipelineXMLDump project folder in the download. Of course, my custom agents are for illustrative purposes only, have not been tested sufficiently, and should never be installed on a production server. Use these agents at your own risk.
DumpAgents.cs shows you how to implement all SMTP receive and routing event handlers that the transport pipeline supports. This can be a good starting point if you want to implement your own custom agents. My dump agents write XML files in subfolders under %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\PipelineXMLDump that correspond to the identifier of the SMTP session (SMTP receive agent) or the message identifier (routing agent). Figure 2 shows an example of event argument information passed to the SMTP receive agent in the OnConnectEvent handler. The example reveals the local IP endpoint (IP address and port) that accepted an incoming connection request.
Figure 2 Argument information dumped during the OnConnect event
Figure 2 Argument information dumped during the OnConnect event

Preparing the Test Runs
Enough theory about logging, tracing, and dumping. Let's get ready for a few test runs that show anti-spam agents in action. The following is a brief rundown that mirrors the Microsoft IT configuration as closely as is possible in your test environment. As mentioned earlier, background information is available in the technical white paper "Microsoft Exchange Server 2007 Edge Transport and Messaging Protection."
Start by deactivating the Address Rewriting Inbound, Attachment Filter, and Address Rewriting Outbound agents by running the EDGE01_disable_agents.ps1 script from the accompanying download. Address rewriting is not necessary because recipients use the same e-mail addresses internally and externally. I do not use the Attachment Filter agent because of the fact that superseding attachment filtering capabilities are available in Forefront Security.
The test environment is not connected to the Internet. This makes it difficult to test the Connection Filter agent. To remedy the issue, you can create an IP block list on the Internet host by running the INTERNET01_create_IP_block_list.bat file. Make sure you have Windows® Support Tools installed, as these include the DNS command-line tool (dnscmd.exe). Otherwise, you have to spend time creating the DNS records manually using the DNS console.
Now you can configure the Connection Filter agent by running the EDGE01_add_block_list_provider.ps1 script. This script adds an IP Block List provider for ip-bl.consolidatedmessenger.com. This is the IP Block list of Consolidated Messenger, already created in the test environment by running the INTERNET01_create_IP_block_list.bat file on the Internet host, as mentioned previously. Microsoft IT also uses the local IP Block list and IP Allow list, but these features do not require manual configuration. It's important to note that Microsoft IT does not use IP Allow List providers.
Next, configure the Recipient Filter agent by running the EDGE01_recipient_filter.ps1 script to block messages to non-existent recipients as well as messages to mailboxes and global distribution lists that are for internal or outbound use only.
The Sender Filter agent also needs a configuration update. Microsoft IT uses sender filtering as a countermeasure against mail flooding attacks from specific sender addresses and to block certain list servers and similar sources that are deemed not relevant for business. Microsoft IT furthermore uses sender filtering to block messages with invalid or missing sender information. Let's apply this configuration to the test environment by running the EDGE01_sender_filter.ps1 script on the Edge Transport server.
You also need to configure Sender Policy Framework (SPF) records in DNS if you want to mirror the Microsoft IT configuration. To generate an SPF record that attests the authority of your Edge Transport server for messages from your domain, adventure-works.com, you can use the Microsoft Sender ID Framework SPF Record Wizard available at microsoft.com/mscorp/safety/content/technologies/senderid/wizard. Give it a try and specify an existing Internet domain name. In the test environment, you can run the INTERNET01_create_SPF_record.bat file on the Internet host.
That's it! No further parameters need to be configured because most of the agents come with reasonable default settings. To perform a quick check of the most important default settings for the Edge Rules, Sender ID, Protocol Analysis, and Content Filter agents, run the EDGE01_check_defaults.ps1 script on the Edge Transport server.

Edge Transport Agents in Action
Now, let's put the Edge Transport agents to work in a few realistic messaging scenarios. To send test messages to the Edge Transport server, I use the SMTP and POP3 service included in Windows Server 2003 and Outlook® Express as the messaging client, using the default settings most of the time, but making sure logging is enabled.
But first, a warning: do not follow any of the steps or run the scripts described in this section if your test environment has any connections to a production system or the Internet. You are testing at your own risk.
The first agent I am going to test is the Connection Filter agent. It deserves to be tested first because it is the hardest working agent on an Edge Transport server. At Microsoft, the Connection Filter agent blocks about 88 percent of all Internet message submissions. Out of 13 million submission attempts on a regular business day, only about 1.5 million are from legitimate senders.
So let's send an e-mail as Fabrikam.User@Fabrikam.com to Administrator@adventure-works.com. If things work properly, Administrator should not receive the message. Instead, Fabrikam.User should receive a delivery status notification (DSN) stating that delivery to the recipient has failed. Fabri- kam.User verifies that the e-mail address is correct, sends the same message again, and again Fabrikam.User receives a DSN. Then Fabrikam.User calls the help desk and reports the issue. She is not the first. In fact, nobody at Fabrikam can communicate with Adventure Works anymore. Something has happened. The issue is urgent. Trying to resolve it quickly, you check the SMTP logs in the %systemroot%\system32\Logfiles\SMTPSVC1 folder on INTERNET01, and as shown by the red text in Figure 3, there it is!
Figure 3 SMTP logs show the problem (Close Figure 3)
... 220 mail.adventure-works.com Microsoft ESMTP MAIL Service ready at Thu,
31 May 2007 12:21:33 -0700,
... EHLO, -, INTERNET01,
... 250-mail.adventure-works.com Hello [192.168.1.100],
... MAIL, -, FROM:<Fabrikam.User@fabrikam.com> SIZE=840,
... 250 2.1.0 Sender OK,
... RCPT, -, TO:<Administrator@adventure-works.com>,
... 550 5.7.1 E-mail rejected because your IP address is listed by
ip-bl.consolidatedmessenger.com. Please visit
http://www.consolidatedmessenger.com/ip-bl for more information.
If you still need assistance, contact cmipbl@adventure-works.com.,
... RSET, -, -,
... 250 2.0.0 Resetting,
... QUIT, -, -,
... 221 2.0.0 Service closing transmission channel,
The 550 error response tells you that Consolidated Messenger put your Internet host's IP address on an IP Block list. Fair enough. You contact Consolidated Messenger to get removed from that list. As it turns out, it is easier to get on the Consolidated Messenger block list than to be removed from it. It will take days if not weeks to clean this up.
You need help more quickly, and so you contact Adventure Works through the e-mail address cmipbl@adventure-works.com. You send an e-mail as Fabrikam.Admin@Fabrikam.com, describing the situation and asking to unblock your host's Internet IP address (in this case 192.168.1.100). After verifying your identity, Adventure Works unblocks the address by using the following command to grant a temporary exception that automatically expires after 30 days:
Add-IPAllowListEntry -IPAddress 192.168.1.100
-ExpirationTime (get-date).AddDays(30)
Thirty days should give you enough time to sort things out with Consolidated Messenger, and Adventure Works does not need to spend time maintaining the exception manually. By granting self-expiring exceptions, Adventure Works keeps the administrative overhead associated with maintaining IP Allow lists at a minimum.

Safe Sender
Next, let's test the safe sender awareness of content filtering, which is a new and interesting feature that increases the precision of spam filtering. EdgeSync propagates safe sender information from the internal environment in hashed and encrypted form to Edge Transport servers. The Content Filter agent uses this information during the filtering process to make informed decisions. You only need to get the safe sender information (that is, Safe Senders list, Safe Recipients list, Blocked Senders list, and external contacts) into Active Directory® by using the Update-SafeList cmdlet and then replicating the information to the Edge Transport server by using the Start-EdgeSynchronization cmdlet. Microsoft IT runs the Update-SafeList cmdlet on a scheduled basis during non-peak hours. In our test environment it is sufficient to run a corresponding script (HUB-MBX-01_update_safelist.ps1) manually.
First, clean up the test environment by running the EDGE01_clean_up.ps1 script on the Edge Transport server to remove the previously created IP Allow list entry. This is necessary because the Content Filter agent does not act on messages from sources that are on this list. Next, remove the IP Block list entry for 100.1.168.192.ip-bl.consolidatedmessenger.com from the Consolidated Messenger block list by running the INTERNET01_clean_up.bat file on the Internet host. Otherwise, the simulated Internet host will not be able to send messages, as demonstrated earlier.
Finally, send a message that the Content Filter agent almost certainly considers to be spam. This is tricky because the Content Filter agent uses the default SCL reject threshold to deliver most messages to the users to minimize false positives. Yet after some testing, I found one message version that brings the SCL rating up to the reject level. To send the message as Contoso.User@Contoso.com, run the INTERNET01_contoso_msg.vbs script on the Internet host.
My script uses an anonymous SMTP connection to submit the message to the SMTP service on INTERNET01. For it to work, you need to configure the SMTP service to relay messages for anonymous users. In IIS Manager, open the properties for the Default SMTP Virtual Server. On the Access tab, click on the Relay button, and then select All except the list below. I will cover the implications of this configuration in a moment.
When you run the script to send this message, Contoso.User should receive a DSN stating that delivery failed with: Diagnostic-Code: smtp;550 5.7.1 Message rejected due to content restrictions. You can also see this response in the SMTP log on the Internet host, and if you run the command Get-AgentLog | Select-Object -Last 1 on the Edge Transport server to display the last entry from the agent log, you should be able to see that the message was rejected because its SCL rating was 7, as shown in Figure 4.
Figure 4 Message blocked due to content restrictions
Figure 4 Message blocked due to content restrictions
Now, Contoso.User is not a spammer. It is unfortunate that her message did not go through in this case, but Contoso.User can usually communicate with recipients at Adventure Works. So Contoso.User informs Administrator in another e-mail that her original message was mistakenly blocked. Administrator knows Contoso.User and, to ensure that spam filters do not block her messages in the future anymore, Administrator adds Contoso.User to the Safe Senders List. In order to do this in Office Outlook 2007, right-click on a received message from Contoso.User, point to Junk E-mail, and click on Add Sender to Safe Senders List.
At the usual interval, the Update-SafeList cmdlet runs to refresh the safelist information in Active Directory and after the next Edge Synchronization, Contoso.User will be able to communicate with Administrator without false positives. To simulate this procedure, run the HUB-MBX-01_update_safelist.ps1 script.
Now run the INTERNET01_contoso_msg.vbs script to send the Contoso.User message again. Check the agent log on the Edge Transport server afterward by using the command Get-AgentLog | Select-Object -Last 1, and you can see under ReasonData that content filtering was now bypassed. No more false positives for Contoso.User!

Spam Attack
To conclude the test runs, let's go overboard. As you no doubt have noticed, I configured the Internet host during the previous test as an open relay, which opens the door to spammers. So now let's see what happens if Spam.Sender@cohovineyards.com notices this weak configuration and abuses the host to send thousands of spam messages to Adventure Works. The INTERNET01_spam_msg.vbs script only submits one spam message, but you can edit the script and change the max_loop value to generate more.
To simulate a spam attack, I submitted 1,000 messages to my compromised host for relay to the Edge Transport server. The SMTP service limits the number of messages per outbound connection to 20, so it took some time to relay a sufficient number of spam messages before the Protocol Analysis agent on the Edge Transport server had enough and kicked in.
The Protocol Analysis agent assists the Connection Filter agent by automatically maintaining IP Block list entries based on SMTP protocol analysis and IP reputation updates from Microsoft Update. In this case, the SMTP protocol analysis caught the spam e-mail messages. The Protocol Analysis agent keeps track of each message's SCL rating to calculate an average SCL value that it uses to determine a sender reputation level (SRL). As more and more illegitimate messages arrived during my simulated spam campaign, the Internet host's SRL increased until it exceeded the SRL block threshold. As a result, the host landed for 24 hours on the IP Block list. No more spam whatsoever from this host! It happened automatically. No administrator intervention was necessary.
Figure 5 shows the resulting IP Block list entry and corresponding changes in message handling. Prior to blocking the host, the Edge Transport server rejected each spam message during the OnEndOfData event because it was the Content Filter agent that acted upon each message based on the SCL threshold (for reference, see Figure 4). The message was submitted, although unsuccessfully. After blocking my host, the connection was rejected during the OnMailCommand event by the Connection Filter agent. Messages are no longer submitted. It saves network bandwidth and processing resources to disconnect spammers before message submission.
Figure 5 Stopping an offending Internet host
Figure 5 Stopping an offending Internet host

Further Testing
Of course, there are many more agent features and scenarios that you can test. For example, you can test sender filtering by sending messages as Blocked.User@Contoso.com or Blocked.User@Fabrikam.com. You can test recipient filtering by sending messages to Blocked.User@adventure-works.com. You can also use Telnet to simulate directory harvesting attacks with varying tarpitting intervals. Microsoft IT uses the default interval of five seconds for SMTP 5yz responses on the Internet-facing receive connector, which is a sufficient amount of time for Microsoft IT to render directory-harvesting attacks impractical for spammers. But you can apply different values using the Set-ReceiveConnector cmdlet, as demonstrated in the EDGE01_tarpitting.ps1 script.
You can also go deeper and check out the snapshot files that pipeline tracing creates for the messages from Contoso.User@Contoso.com. Examine the antivirus stamps, X-MS-Exchange-Organization-AVStamp-Mailbox: MSFTFF;1;0;0 0 0, which Forefront Security uses to mark messages as scanned. When Forefront Security on the Hub Transport server finds this stamp with recent version information, it does not have to scan the message a second time. If you try to fool Forefront Security by inserting fake antivirus stamps into your test messages, you can see the header firewall in action, which removes these stamps right after the OnDataCommand event, long before the OnSubmittedMessage event is triggered that invokes the FSE Routing Agent.
While you are looking at the message headers in the snapshot files, perhaps you might want to create an SPF test record for Contoso.com that identifies a wrong IP address as the legitimate sender. One way to accomplish this is by running the INTERNET01_wrong_SPF_record.bat file on the Internet host. Then you should send a test message as Contoso.User@Contoso.com and look at the X-MS-Exchange-Organization-SenderIdResult and Received-SPF header:
X-MS-Exchange-Organization-SenderIdResult: SoftFail
Received-SPF: SoftFail (EDGE01.extranet.adventure-works.com: domain of transitioning
Contoso.User@Contoso.com discourages use of 192.168.1.100 as permitted sender)

Wrapping Up
As you've seen in this and my previous article, Exchange Server 2007 Edge Transport servers come with a comprehensive set of transport agents to implement reliable, high-precision anti-spam capabilities, including connection filtering, protocol analysis, and content filtering. The transport architecture is extensible and supports the implementation of further agents for additional functionality. The FSE Routing agent is a good example of an additional agent that integrates Forefront Security for Exchange Server into the transport architecture.
Combined with additional features, such as connection tarpitting, header firewall, TLS, and authentication, Edge Transport servers provide the necessary means to achieve a very high level of messaging protection at multiple levels and without single points of failure. By deploying Edge Transport servers and Forefront Security in a perimeter network, strictly separated from your internal Exchange Server organization, you can keep the flood of illegitimate and malicious content away from your messaging environment while delivering legitimate messages with very low false positive rates to your users.

Kay Unkroth is an entrepreneur who has worked as a support engineer, system developer, consultant, trainer, and author focused on Microsoft server technologies for more than 15 years. Kay is also co-founder and chairman of Biblioso Corporation, a company that specializes in managed documentation and localization services.


 Networking: Configure VPN Remote Access with ISA Server 2006
Networking Alan Maddison

Configure VPN Remote Access with ISA Server 2006
At a Glance:
  • VPN protocols
  • Configuring a site-to-site VPN connection
  • Configuring a remote access VPN connection
  • New features in ISA Server 2006

Virtual Private Networks (VPNs) provide a high level of adaptability, scalability, and control over network connectivity. In addition to enhancing security, they can deliver significant cost savings
over traditional dedicated point-to-point connections. And they're flexible.
There are many types of VPNs. Some are used to connect mobile users to the corporate network; some are used to connect two networks that are geographically separated. The protocols these different VPNs use and the features they offer vary. Some will provide security capabilities, such as authentication and encryption, while others might offer no built-in security features at all. And some, obviously, are easier than others to set up and manage.
In this article, I will look at how you can implement a VPN using Internet Security and Acceleration (ISA) Server 2006. In particular, I will focus on how the ISA VPN functionality can be used to address two common scenarios: as a way to provide VPN remote access connections for users and as a way to provide site-to-site VPN connections. Both implementations include security features to ensure data privacy and to provide protection for internal network resources.
In the first scenario—a remote access VPN connection—a remote client initiates a VPN connection to your ISA Server over the Internet. ISA Server then connects the remote client to your internal network, providing the user with seamless access to internal network resources, including data and applications. The second scenario, a site-to-site VPN connection, is a little more complex because of its use of a router, which may be ISA itself. But this connection type provides the ability to seamlessly connect different office or branch locations to one another or to a central office.
There are several additional benefits gained by using ISA Server 2006 to implement a VPN. One of the most significant is due to the integrated nature of ISA Server, which means that the VPN functionality and the firewall functionality work hand in hand. In addition, ISA Server provides a VPN quarantine feature that utilizes the Network Access Quarantine Control feature of Windows Server® 2003 to quarantine a remote access computer until its configuration has been validated by a server-side script. This adds another layer of protection by providing a means to check such things as antivirus definition status and local firewall policy on the remote computer before you allow it access to your internal network resources.
The principal administrative benefits offered by ISA Server as a VPN solution include centralized management of policies, monitoring, logging, and reporting. For your day-to-day administrative responsibilities, these capabilities can prove to be invaluable. In particular, the real-time monitoring and log filtering capabilities give you key insight into network traffic and connections passing through ISA Server.

VPN Protocols
The core tunneling protocols used in an ISA VPN connection provide the first line of defense for securing connections. ISA Server supports three protocols—Layer 2 Tunneling Protocol (L2TP) over IPsec, Point-to-Point Tunneling Protocol (PPTP), and IPsec tunnel mode. This last protocol is supported only for site-to-site connections and is primarily used for interoperability with routers and other operating systems that do not support L2TP or PPTP.
L2TP is paired with IPsec because L2TP has no mechanism of its own to provide data privacy. When teamed with IPsec, which offers robust authentication and encryption methods, the combination provides a compelling solution. PPTP uses either Microsoft® Challenge Handshake Authentication Protocol (MS CHAP) version 2 or Extensible Authentication Protocol-Transport Layer Security (EAP-TLS) for authentication and Microsoft Point-to-Point Encryption (MPPE) for, as you can guess, encryption.
Whether an organization should choose L2TP over IPsec or PPTP often depends on factors that are specific to that organization. L2TP over IPsec will let you use more complex and sophisticated encryption and supports more types of networks (including IP, X.25, Frame Relay, and ATM). However, PPTP is easier to implement and typically has fewer requirements. That said, in situations where there are no organizational constraints, it's considered a best practice to implement L2TP over IPsec.

Site-to-Site VPN Connection
ISA Server 2006 has made some tremendous progress in simplifying the setup of a site-to-site VPN. In earlier versions, there were many more steps involved. For this example, I'll discuss a single remote site that is connecting to a central office using L2TP over IPsec with an ISA Server in both locations. The process requires that you configure the central office ISA Server before you configure the remote site.
The first step in the process is to configure the local user accounts that are required on both ISA Servers. This user identity will provide the security context under which the connection to the remote or main ISA Server will be made. The name of this account should match the name of the VPN connection you create within the ISA administration console. For instance, a good naming convention would be to set up the user name "Site VPN" on the ISA Server located at the headquarters (pointing to the remote site) and "HQ VPN" on the remote ISA Server. Once you've created these accounts, you need to access the account properties, open the Dial In tab, and make sure the Allow Access option is selected.
After you have created the local user account, the next step is to create the public key infrastructure (PKI) certificate that will be used for authentication between the two sites. You have the option of using a pre-shared key, but use of a certificate is always a preferable method. The simplest way to install a certificate for a VPN connection is to connect directly to your own enterprise certificate authority and request a certificate through the certificate server's Web site. But to do this, you may need to create an access rule to allow your ISA Server to connect to the certificate server via HTTP.
If you are familiar with earlier versions of ISA Server, you'll notice that the ISA Server 2006 interface, shown in Figure 1, is far more intuitive. When you select VPN in the left-hand pane, context-sensitive configuration and task options are displayed in the center and right panes.
Figure 1 ISA Server 2006 has a more intuitive interface
Figure 1 ISA Server 2006 has a more intuitive interface
Launch the site-to-site VPN wizard by clicking on the Create VPN Site-to-Site Connection task in the right-hand pane. When the wizard launches, it asks for a site-to-site network name; this name should match the name of the user account that you created earlier. After entering the name, press the Next button. On the next screen (see Figure 2), select the VPN protocol to be used—in my example, this is L2TP over IPsec. Press Next and the wizard helpfully warns you that a user account matching the network name must be available—since you were careful and did this, you can just press OK to get to the next screen.
Figure 2 Select the VPN protocol you want to use
Figure 2 Select the VPN protocol you want to use
Now you have to create an IP address pool. You have two choices here: you can either create a static address pool on the ISA Server or you can use your existing DHCP server to handle address assignment. Since both methods are acceptable, your decision should be guided by your company's procedures as they relate to this particular option. The screen now asks you to input the remote server name. Enter the fully qualified domain name (FQDN) of the remote server and then enter the user credentials for the connection. Remember that you have to enter the user account information you created on the remote ISA Server. In the example here, this would be the user account HQ VPN, as shown in Figure 3.
Figure 3 Enter the FQDN of the remote server
Figure 3 Enter the FQDN of the remote server
On the next screen, select the outgoing authentication method. (As I mentioned earlier, it is always preferable to use a certificate.) Then, input the IP address range being used on the internal network at the remote site.
After this, if you are using ISA Server 2006 Enterprise Edition, the wizard will allow you to configure the Network Load Balancing dedicated IP addresses of the remote site. If you are using ISA Server 2006 Standard Edition, skip the load-balancing step and go directly to the next screen (see Figure 4), which is where you create the network rule used to route traffic from the remote site to the local network.
Figure 4 Create the network rule that routes traffic
Figure 4 Create the network rule that routes traffic
You also have to create an access rule, which is the next step in the process. When you configure the access rule, there are three options for allowing protocols to be used—you can choose to allow all outbound traffic, allow all outbound traffic with the exception of certain protocols, or allow only certain protocols. In most situations you'll want to select the all outbound traffic option.
You're nearly finished. At this point, the wizard will present a summary of your configuration and once you press the Finish button, the Site-to-Site VPN connection will be created. You will receive a dialog indicating that a system policy rule needs to be enabled for Certificate Revocation List downloads—press Yes to enable this rule. The wizard will then provide information about any additional configuration steps that may be needed. After completing the configuration of ISA Server at the main site, you need to follow all the same steps to configure the remote site. Once both steps have been completed, users at both sites will be able to communicate across the VPN.

Remote Access VPN Connection
Configuring a remote access VPN connection for client access is far more straightforward (see Figure 5). The first step is to select VPN in the left-hand pane of the ISA Server administration console. Then select the Enable VPN Client Access task in the right-hand pane. You will receive a warning that this may cause the Routing and Remote Access service to restart. (Since this can cause connection issues, you may want to implement this change during a scheduled maintenance window.) Click OK to proceed past this warning; ISA Server enables a system policy that allows VPN client traffic to access the ISA Server. You can view this rule by selecting Firewall Policy in the ISA Server administration console and then selecting the Show System Policy Rules task.
Figure 5 Configure a remote access VPN connection
Figure 5 Configure a remote access VPN connection
A default network rule is also enabled to allow routing between the internal network and the two VPN networks (VPN Clients and Quarantined VPN Clients). This network rule can be viewed or edited by selecting Networks in the left pane and then opening the Network Rules tab in the center pane.
Now you have to configure VPN client access. There are three additional parameters that need to be configured: the Windows Security Groups that are permitted access, the protocols that may be used by the clients, and user mapping. The dialog for configuring these parameters is shown in Figure 6.
Figure 6 Configure VPN client access
Figure 6 Configure VPN client access
The Groups tab simply requires that you select which Windows groups should be allowed remote access via VPN. From an administration perspective, I think it's a good idea to create a single group to which permission is given. This makes it very easy to manage remote access.
The Protocols tab shows that only PPTP is enabled by default. You should definitely enable L2TP over IPsec if this is at all feasible in your environment.
User Mapping lets you map user accounts from namespaces, such as Remote Authentication Dial-In User Service (RADIUS), to Windows accounts to ensure that access policies are applied correctly. Once you have completed these configuration options and applied the changes to ISA Server you are finished. Clients should now be able to seamlessly access your internal network data and applications using a VPN connection.

Enhancements to ISA Server 2006
ISA Server 2006 introduces a number of enhancements that increase its power and performance over previous versions. These features—which include HTTP compression, Background Intelligent Transfer Service (BITS) support, and Quality of Service (QoS)—provide various techniques for optimizing network usage. Of course, even with these techniques, you should also continue to use common bandwidth optimization technologies that are core to ISA Server, including caching.
HTTP compression has been supported in various Microsoft products for some time now—it's been in Internet Explorer® since version 4 and Windows Server since Windows® 2000. However, this is the first version of ISA Server to support HTTP compression as well as the industry standard GZIP and Deflate algorithms.
What does this mean? With support for HTTP compression, a Web browser that complies with HTTP 1.1 can request compressed content from any Web site. Remember, though, that only the responses are compressed, not the initial outgoing connections from the client.
HTTP compression is a global HTTP policy setting that applies to all HTTP traffic that passes through ISA Server, as opposed to being associated with a specific network rule. You do, however, have the option of implementing HTTP compression on a per-listener basis; this is configured through the Web Listener wizard.
HTTP compression, which can be accessed through the General option in the left-hand pane, is enabled by default. But as Figure 7 shows, you have to configure the network elements where compression should be used. Continuing with the site-to-site VPN example, you need to select the Return Compressed Data tab and add the Site VPN network element that you created earlier. At this point you also have the option of configuring which content types should be compressed. ISA Server 2006 comes with a list of standard content types, but you can add custom content types through the Firewall Policy task pane on the Toolbox tab. Remember that the Return Compressed Data tab refers to ISA Server compressing the data before it is returned to the client. In order to have ISA Server request that a Web server handle the compression of data before the data is sent to ISA Server, you need to use the Request Compressed Data tab.
Figure 7 Configure HTTP compression
Figure 7 Configure HTTP compression
Background Intelligent Transfer System is an asynchronous file transfer service for HTTP and HTTPS traffic. Windows Server 2003 includes BITS version 1.5, which supports downloads and uploads, though uploads also require Internet Information Services (IIS) version 5.0 or higher. As an intelligent file transfer service, BITS has the ability to examine network traffic and use only the idle portion of your network bandwidth. In addition, the file transfer is dynamic and BITS will respond to spikes in normal network traffic by throttling back its network usage. The advantage here is that BITS ensures that downloads and uploads of large files will not negatively impact other network usage. From an administrator's perspective, this means you should receive far fewer calls complaining about poor network performance. Good news!
BITS offers other benefits that can improve the user experience and make for better network performance. For instance, a file transfer that uses BITS is binary, meaning BITS can resume file transfers (without starting from scratch) that have been interrupted by such factors as a network outage. And ISA Server 2006 includes built-in support for a Microsoft Update caching feature that uses BITS caching to optimize updates.
The Differentiated Services (DiffServ) protocol enables packet prioritization (or QoS) for HTTP and HTTPS traffic. In other words, depending on your ISA Server configuration, certain traffic will receive priority. This is useful on networks that have limited bandwidth, either due to traffic volume or network connection speed. According to the Internet Engineering Task Force (IETF), DiffServ is a class-based traffic management mechanism. Thus, DiffServ can differentiate between types of traffic and manage them accordingly, ensuring that high-priority traffic gets preference.
To provide this feature, ISA Server works in conjunction with routers that support QoS. It is important to ensure that ISA Server DiffServ bits match the priorities on your routers if you want the packets to be routed with the same prioritization.
As implemented in ISA Server, this type of packet prioritization is an HTTP policy setting that is global and is applied to all HTTP traffic that passes through ISA Server 2006 using a DiffServ Web filter. This means that ISA Server does not support DiffServ for other protocols and may, in fact, drop existing DiffServ bits on non-HTTP traffic.
As shown in Figure 8, DiffServ is implemented as a Web filter and can be accessed by selecting Add Ins in the left pane. It is important that you not change the default settings or the priority order of the DiffServ filter due to the need for ISA Server to inspect the size of request/response at the time it is processed.
Figure 8 View the DiffServ Web filter
Figure 8 View the DiffServ Web filter
If you take a close look at Figure 8, you might notice that the DiffServ filter is not enabled by default. To enable the filter, simply select the Enable Selected Filters in the Tasks pane and then configure the filter. Since the DiffServ packet prioritization in ISA Server is based on either specific URLs or domains, you need to add this information to the DiffServ preferences. To do this, select General in the left pane of the management console and then, under Global HTTP Policy Settings, select Specify DiffServ Preferences. When you specify these preferences, you must define the priorities and the URLs and domains that should be prioritized. ISA Server 2006 provides powerful new capabilities for setting up remote access for VPN clients or creating site-to-site VPNs. It should be at the top of your list when considering which VPN solution to implement.

Alan Maddison is a Senior Consultant in the Microsoft technology practice of MTI Technology Corporation. His primary focus is on Active Directory, Exchange Server, and virtualization.


 SQL Server: Optimizing SQL Server Query Performance
SQL Server Maciej Pilecki

Optimizing SQL Server Query Performance
At a Glance:
  • Analyzing execution plans
  • Optimizing queries
  • Identifying queries to tune

When optimizing your database server, you need to tune the performance of individual queries. This is as important as—perhaps even more important than—tuning other aspects of your server installation, that affect performance, such as hardware and software configurations.
Even if your database server runs on the most powerful hardware available, its performance can be negatively affected by a handful of misbehaving queries. In fact, even one bad query, sometimes called a "runaway query," can cause serious performance issues for your database.
Conversely, the overall performance of your database can be greatly improved by tuning a set of most expensive or most often executed queries. In this article, I will look at some of the techniques you can employ to identify and tune the most expensive and worst performing queries on your server.

Analyzing Execution Plans
When tuning an individual query, you usually start by looking at the execution plan of that query. The execution plan describes the sequence of operations, physical and logical, that SQL ServerTM will perform in order to fulfill the query and produce the desired resultset. The execution plan is produced by a database engine component called Query Optimizer during the optimization phase of query processing—this takes into account many different factors, such as the search predicates used in the query, the tables involved and their join conditions, the list of columns returned, and the presence of useful indexes that can serve as efficient access paths to the data.
For complex queries, the number of all possible permutations can be huge, so the query optimizer does not evaluate all possibilities but instead tries to find a plan that is "good enough" for a given query. This is because finding a perfect plan may not always be possible; and even when it is possible, the cost of evaluating all the possibilities to find the perfect plan could easily outweigh any performance gains. From a DBA point of view, it is important to understand the process and its limitations.
There are a number of ways to retrieve an execution plan for a query:
  • Management Studio provides Display Actual Execution Plan and Display Estimated Execution Plan features, which present the plan in a graphical way. These features offer the most suitable solution for direct examination and are by far the most often-used approach to display and analyze execution plans. (In this article, I will use graphical plans generated in this way to illustrate my examples.)
  • Various SET options, such as SHOWPLAN_XML and SHOWPLAN_ALL, return the execution plan as either an XML document describing the plan using a special schema or a rowset with textual description of each of the operations in the execution plan.
  • SQL Server Profiler event classes, such as Showplan XML, allow you to gather execution plans of statements collected by a trace.
While an XML representation of the execution plan may not be the easiest format for a human to read, this option does allow you to write procedures and utilities that can analyze your execution plans, looking for signs of performance problems and suboptimal plans. An XML-based representation can also be saved to a file with the .sqlplan extension and then opened in the Management Studio to produce a graphical representation. These files can also be saved for later analysis, eliminating, of course, the need to reproduce the execution plan each time you want to analyze it. This is especially useful when you want to compare your plans to see how they change over time.

Estimated Cost of Execution
The first thing you have to understand about execution plans is how they are generated. SQL Server uses a cost-based query optimizer—that is, it tries to generate an execution plan with the lowest estimated cost. The estimate is based on the data distribution statistics that are available to the optimizer when it evaluates each table involved in the query. If those statistics are missing or outdated, the query optimizer will lack vital information that it needs for the query optimization process and therefore its estimates will likely be off the mark. In such cases, the optimizer will choose a less than optimal plan by either overestimating or underestimating the execution costs of different plans.
There are some common false assumptions about the estimated execution cost. In particular, people often assume that the estimated execution cost is a good indicator of how much time the query will take to execute and that this estimate allows you to distinguish good plans from bad plans. This is not true. First, it is sufficiently documented what the units for expressing the estimated cost are and if they have any direct relation to the time of execution. Second, since this is an estimate and can be wrong, plans with higher estimated costs can sometimes be much more efficient in terms of CPU, I/O, and time of execution, despite their higher estimates. This is often the case with queries that involve table variables—since no statistics are available for table variables, query optimizer always assumes that a table variable contains just one row, even if it contains many times that. Thus, query optimizer will choose a plan based on an inaccurate estimate. Therefore, when comparing the execution plans of your queries, you should not rely on estimated query cost alone. Instead, include the output of the STATISTICS I/O and STATISTICS TIME options in the analysis to understand of what the real cost execution is in terms of I/O and CPU time.
One special type of execution plan, called a parallel plan, is worth mentioning here. If you are running your query on a server with more than one CPU and your query is eligible for parallelization, a parallel plan may be chosen. (Generally, the query optimizer will only consider a parallel plan for a query that has a cost exceeding a certain configurable threshold.) Due to the overhead of managing multiple parallel threads of execution (meaning distributing work across threads, performing synchronization, and gathering results), parallel plans cost more to execute and that is reflected in their estimated cost. So why are they preferred over cheaper, non-parallel plans? Thanks to using the processing power of multiple CPUs, parallel plans tend to yield results faster than standard plans. Depending on your specific scenario, including such variables as available resources and concurrent load from other queries, this situation may be desirable for your setup. If so, you should control which of your queries can produce parallel plans and how many CPUs each can utilize. You do this by setting the max degree of parallelism option on the server level and overriding it on individual query level with OPTION (MAXDOP n) as needed.

Analyzing an Execution Plan
Now I'll look at a simple query, its execution plan, and some ways to improve its performance. Say I execute this query using Management Studio with the Include Actual Execution Plan option turned on in the Adventure Works sample database on SQL Server 2005:
SELECT c.CustomerID, SUM(LineTotal)
FROM Sales.SalesOrderDetail od
JOIN Sales.SalesOrderHeader oh
ON od.SalesOrderID=oh.SalesOrderID
JOIN Sales.Customer c ON oh.CustomerID=c.CustomerID
GROUP BY c.CustomerID
As a result, I see the execution plan depicted in Figure 1. This simple query calculates the total amount of orders placed by each customer of Adventure Works. Looking at the execution plan, you can see how the database engine processes the query and produces the result. Graphical execution plans should be read from top to bottom and from right to left. Each icon represents a logical and physical operation performed, and arrows show data flow between operations. The thickness of the arrows represents the number of rows being passed between operations—the thicker the arrow, the more rows involved. If you place your pointer over one of the operator icons, a yellow ToolTip (like the one shown in Figure 2) will display details of that particular operation.
Figure 1 Sample execution plan
Figure 1 Sample execution plan
Figure 2 Details about an operation
Figure 2 Details about an operation
Looking at each of the operators, you can analyze the sequence of steps performed:
  1. The database engine performs a Clustered Index Scan operation on the Sales.Customer table and returns the CustomerID column for all the rows in that table.
  2. It then performs an Index Scan (meaning a non-clustered index scan) on one of the indexes in the Sales.SalesOrderHeader table. This is an index on the CustomerID column, but it also implicitly includes the SalesOrderID column (the table clustering key). The values from both of those columns are returned by the scan.
  3. Output from both scans is joined on the CustomerID column using the Merge Join physical operator. (This is one of three possible physical ways of performing a logical join operation. It's fast but requires both inputs to be sorted on a joined column. In this case, both scan operations have already returned the rows sorted on CustomerID so there is no need to perform the additional sort operation).
  4. Next, the database engine performs a scan of the clustered index on the table Sales.SalesOrderDetail, retrieving the values of four columns (SalesOrderID, OrderQty, UnitPrice, and UnitPriceDiscount) from all rows in this table. (There were 123,317 rows estimated to be returned by this operation and that number was actually returned, as you can see from the Estimated Number of Rows and Actual Number of Rows properties in Figure 2—so the estimate was very accurate.)
  5. Rows produced by the clustered index scan are passed to the first Compute Scalar operator so that the value of the computed column LineTotal can be calculated for each row, based on the OrderQty, UnitPrice, and UnitPriceDiscount columns involved in the formula.
  6. The second Compute Scalar operator applies the ISNULL function to the result of the previous calculation, as required by the computed column formula. This completes the calculation of the LineTotal column and returns it, along with column SalesOrderID, to the next operator.
  7. The output of the Merge Join operator in Step 3 is joined with the output of the Compute Scalar operator from Step 6, using the Hash Match physical operator.
  8. Another Hash Match operator is then applied to group rows returned from Merge Join by the CustomerID column value and calculated SUM aggregate of the LineTotal column.
  9. The last node, SELECT, is not a physical or logical operator but rather a place holder that represents the overall query results and cost.
On my laptop, this execution plan had an estimated cost of 3,31365 (as shown in Figure 3). When executed with STATISTICS I/O ON, the query reported performing a total of 1,388 logical read operations across the three tables involved. The percentages displayed under each of the operators represent the cost of each individual operator relative to the overall estimated cost of the whole execution plan. Looking at the plan in Figure 1, you can see that most of the total cost of the entire execution plan is associated with the following three operators: the Clustered Index Scan of the Sales.SalesOrderDetail table and the two Hash Match operators. But before I move to optimize those, I would like to point out a very simple change in my query that will allow me to eliminate two of the operators altogether.
Figure 3 Total estimated execution cost of the query
Figure 3 Total estimated execution cost of the query
Since the only column I am returning from the Sales.Customer table is CustomerID, and this column is also included as a foreign key in the Sales.SalesOrderHeaderTable, I can completely eliminate the Customer table from the query without changing the logical meaning or the result produced by our query by using this code:
SELECT oh.CustomerID, SUM(LineTotal)
FROM Sales.SalesOrderDetail od JOIN Sales.SalesOrderHeader oh
ON od.SalesOrderID=oh.SalesOrderID
GROUP BY oh.CustomerID
This results in a different execution plan, which is shown in Figure 4.
Figure 4 The execution plan after eliminating Customer table from the query
Figure 4 The execution plan after eliminating Customer table from the query
Two operations were eliminated entirely—the Clustered Index Scan on Customer table and the Merge Join between Customer and SalesOrderHeader—and the Hash Match join was replaced by the much more efficient Merge Join. However, in order to use Merge Join between the SalesOrderHeader and SalesOrderDetail tables, the rows from both tables had to be returned sorted by the join column SalesOrderID. To achieve this, the query optimizer decided to perform a Clustered Index Scan on the SalesOrderHeader table, instead of using the Non-Clustered Index Scan, which would have been cheaper in terms of the I/O involved. This is a good example of how the query optimizer works in practice—since the cost savings from changing the physical way of performing the join operation were more than the additional I/O cost generated by the Clustered Index Scan, the query optimizer chose the resulting combination of operators because it gave the lowest total estimated execution cost possible. On my laptop, even though the number of logical reads went up (to 1,941), the CPU time being consumed was actually smaller and the estimated execution cost of this query was down about 13 percent (2,89548).
Say I want to further improve the performance of this query. Now I look at the Clustered Index Scan of the SalesOrderHeader table, which has become the most expensive operator in this execution plan. Since I only need two columns from this table to fulfill the query, I can create a non-clustered index that contains just those two columns—replacing the scan of the whole table with a scan of the much smaller non-clustered index. The index definition might look something like this:
CREATE INDEX IDX_OrderDetail_OrderID_TotalLine
ON Sales.SalesOrderDetail (SalesOrderID) INCLUDE (LineTotal)
Notice that the index I've created includes a computed column. This is not always possible, depending of the definition of the computed column.
After creating this index and executing the same query, I get the new execution plan shown in Figure 5.
Figure 5 Optimized execution plan
Figure 5 Optimized execution plan
The clustered index scan on the SalesOrderDetail table has been replaced by a non-clustered index scan with a significantly lower I/O cost. I have also eliminated one of the Compute Scalar operators, as my index includes an already calculated value of the LineTotal column. The estimated execution plan cost is now 2,28112 and the query performs 1,125 logical reads when executed.

The Covering Index
Customer Order Query Exercise
Q Here's your customer order query exercise: try figuring out the index definition—which columns it should contain to become covering index for this query and if the order of columns in the index definition would make a difference in performance.
A I challenged you to figure out the most optimal covering index to create on Sales.SalesOrderHeader table for the example query in my article. In order to achieve this, the first thing you should note is that the query uses only two columns from the table: CustomerID and SalesOrderID. If you have read my article carefully, you will have noticed that in the case of the SalesOrderHeader table, there is already an existing index that covers this query—it's the index on CustomerID and it implicitly contains the SalesOrderID column, which is the clustering key of the table.
Of course, I also explained why the query optimizer decided not to use this index. Yes, you could force the query optimizer to use this index, but the solution would be less efficient than the existing plan that uses Clustered Index Scan and Merge Join operators. This is because you would be forcing the query optimizer to make a choice between either performing an additional Sort operation in order to be able to still use Merge Join or falling back to using a less efficient Hash Join. Both options have a higher estimated execution cost than the existing plan (the version with Sort operation would perform particularly poorly), so the query optimizer will not use them unless it is forced to do so. So, in this situation, the only index that will perform better than the Clustered Index Scan is a non-clustered index on SalesOrderID, CustomerID. But it is important to note that the columns must be in that exact order:
CREATE INDEX IDX_OrderHeader_SalesOrderID_CustomerID
ON Sales.SalesOrderHeader (SalesOrderID, CustomerID)
If you create this index, the execution plan will contain the Index Scan rather than the Clustered Index Scan operator. This is a significant difference. In this case, the non-clustered index containing just two columns is much smaller than the whole table in the form of clustered index. Therefore, it will require less I/O to read the necessary data.
This example also shows how the order of columns in your index can have a significant impact on how useful it is for the query optimizer. Be sure to keep this in mind when you design multi-column indexes

The index I have created on SalesOrderDetail is an example of the so-called "covering index". It is a non-clustered index that contains all the columns needed to fulfill the query, eliminating the need to scan the whole table using either the Table Scan or Clustered Index Scan operators. The index is in essence a smaller copy of the table, containing a subset of columns from the table. Only those columns needed to answer the query (or queries) are included in the index—in other words, the index contains just what it needs to "cover" the query.
Creating covering indexes for the most frequently executed queries is one of the easiest and most common techniques used in query tuning. It works particularly well in situations where the table contains a number of columns but only a few of them are frequently referenced by queries. By creating one or more covering indexes, you can greatly improve the performance of the affected queries, as they will be accessing a much smaller amount of data and, in turn, incurring less I/O. There is, however, a hidden cost of maintaining additional indexes during data modification operations (INSERT, UPDATE, and DELETE). Depending on your environment and the ratio between SELECT queries and data modifications, you should carefully judge whether this additional index maintenance overhead is justified by query performance improvements.
Don't be afraid of creating multicolumn indexes, as opposed to single-column indexes. These tend to be much more useful than single-column indexes and the query optimizer is more likely to use them to cover the query. Most covering indexes are multicolumn indexes.
In the case of my sample query, there is still some room for improvement, and this query could be further optimized by placing a covering index on the SalesOrderHeader table. This eliminates the Clustered Index Scan in favor of a non-clustered Index Scan. I will leave this as an exercise for you. Try figuring out the index definition—which columns it should contain to become a covering index for this query and if the order of columns in the index definition would make a difference in performance. See the "Customer Order Query Exercise" sidebar for the solution.

Indexed Views
If the performance of my example query is very important, I can go one step further and create an indexed view that physically stores the materialized results of the query. There are certain prerequisites and limitations of indexed views, but if one can be used, it can improve performance dramatically. Keep in mind that indexed views carry a higher maintenance cost than standard indexes. As a result, you should be careful about when you use them. In this case, the index definition looks like this:
CREATE VIEW vTotalCustomerOrders
WITH SCHEMABINDING
AS
SELECT oh.CustomerID, SUM(LineTotal) AS OrdersTotalAmt, COUNT_BIG(*) AS TotalOrderLines
FROM Sales.SalesOrderDetail od
JOIN Sales.SalesOrderHeader oh
ON od.SalesOrderID=oh.SalesOrderID
GROUP BY oh.CustomerID
Note the WITH SCHEMABINDING option, which is a prerequisite for creating an index on such a view, and the COUNT_BIG(*) function, which is necessary if our index definition contains an aggregate function (in this example, SUM). After I create this view, I can create an index on it, like so:
CREATE UNIQUE CLUSTERED INDEX CIX_vTotalCustomerOrders_CustomerID
ON vTotalCustomerOrders(CustomerID)
When I create this index, the result of the query included in the view definition is materialized and physically stored on disk in the index. Note that all data modification operations on the base tables then automatically update the values in the view according to its definition.
If I now rerun the query, what will happen depends on which edition of SQL Server I am running. On the Enterprise or Developer editions, the query optimizer will automatically match this query with the indexed view definition and use the indexed view instead of querying the base tables involved. Figure 6 shows the execution plan produced in this case. It consists of only one operation—a Clustered Index Scan of the index that I created on the view. The estimated execution cost is only 0,09023 and it performs only 92 logical reads.
Figure 6 Execution plan when using indexed view
Figure 6 Execution plan when using indexed view
You can still create and use this indexed view on other editions of SQL Server, but to achieve the same effect you have to change the query to reference the view directly using the NOEXPAND hint, like this:
SELECT CustomerID, OrdersTotalAmt
FROM vTotalCustomerOrders WITH (NOEXPAND)
As you can see, indexed views can be a very powerful feature if used properly. They are most useful for optimizing queries that perform aggregations over large amounts of data. If used on Enterprise edition, they can benefit many queries without requiring changes to your code.

Identifying Which Queries to Tune
How do I identify the queries that are worth tuning? I look for the queries that are executed frequently—they may not have a very high cost of single execution, but the aggregated cost of execution can be much higher than that of a large query that is rarely executed. I am not saying you shouldn't tune the big ones, I just believe that you should first concentrate on the most frequently executed queries. So how do you identify them?
Unfortunately, the most reliable method is somewhat complicated and involves performing a trace of all queries executed against your server and then grouping them by their signatures. (That is, query text with the actual parameter values replaced by placeholders in order to identify the same query type, even if it was executed with different parameter values.) This is a complicated process, as query signatures are difficult to generate. Itzik Ben-Gan describes a solution using CLR user-defined functions and regular expressions in his book Inside Microsoft SQL Server 2005: T-SQL Querying.
There is another much simpler, though somewhat less reliable, method. You can rely on statistics that are kept for all queries in the execution plan cache and query them using dynamic management views. Figure 7 contains a sample query that shows you both the text and the execution plan of the 20 queries in your cache with the highest accumulated number of logical reads. This query is very handy for quickly identifying the queries that are generating the highest number of logical reads—but it has limitations. That is, it will only show the queries that have their plans cached at the time you run the query. If something is not cached, you will miss it.
Figure 7 Identifying top 20 most expensive queries in terms of read I/O (Close Figure 7)
SELECT TOP 20 SUBSTRING(qt.text, (qs.statement_start_offset/2)+1,
        ((CASE qs.statement_end_offset
          WHEN -1 THEN DATALENGTH(qt.text)
         ELSE qs.statement_end_offset
         END - qs.statement_start_offset)/2)+1),
qs.execution_count,
qs.total_logical_reads, qs.last_logical_reads,
qs.min_logical_reads, qs.max_logical_reads,
qs.total_elapsed_time, qs.last_elapsed_time,
qs.min_elapsed_time, qs.max_elapsed_time,
qs.last_execution_time,
qp.query_plan
FROM sys.dm_exec_query_stats qs
CROSS APPLY sys.dm_exec_sql_text(qs.sql_handle) qt
CROSS APPLY sys.dm_exec_query_plan(qs.plan_handle) qp
WHERE qt.encrypted=0
ORDER BY qs.total_logical_reads DESC
Once you have identified these poor performers, take a look at their query plans and search for ways to improve their performance by using some of the indexing techniques I've described in this article. If you succeed, it will be time well invested.
Happy tuning!

Maciej Pilecki is an Associate Mentor with Solid Quality Mentors, a global organization that specializes in training, mentoring, and consulting. He is a Microsoft Certified Trainer (MCT) and SQL Server Most Valuable Professional (MVP), frequently delivering courses and conference speeches on many aspects of SQL Server and application development.


 From the Editor: More Power to You
From the Editor
More Power to You

Last month's issue kicked off what I hope will be an ongoing dialog on the topic of Green Computing. True, this topic is still not well-defined. (What is Green? If my car gets 30 miles to the gallon instead of 25, is that Green? Not that I have a car, mind you. I live in New York. We walk.)
It also seems as though every company worth its salt is trying to put its own Green stamp on its products or services. There's a good reason for this though, and it's why the topic appeared on our own cover last month—the potential is amazing.
We all know it's as much about generating revenue as helping the environment. The fact is, being eco-friendly has become a selling point that will yield real results. ("It's the economy, stupid!") If a car company can sell more cars by making them more efficient, great. In the end, the hope is that we can have a healthy economy built on a healthy and sustainable environment.
So how does this impact the IT professional? Dr. Jonathan Koomey, a staff scientist at the Lawrence Berkeley National Laboratory and consulting professor at Stanford University, recently released a study estimating total power consumption by servers in the U.S. and worldwide. In 2005, datacenters (servers along with their supporting infrastructure, such as cooling systems) were responsible for 1.2 percent of the total power consumption in the U.S.—that's the full output of five 1000-megawatt (MW) power plants, at a cost of $2.7 billion. Worldwide, that cost soars to $7.2 billion (the full output of 14 1000-MW power plants). There is serious money—and power—to be saved, and much of that work can start inside the datacenter.
How do you plan datacenter capacity? How can capacity be adjusted to accommodate demand, without having to run at 100 percent at all times? (Hint: virtualization.) We monitor servers for network connectivity, memory usage, and so on, but who pays attention to power consumption? Are older servers consuming drastically more power than new machines? Is that increased cost greater than the cost of a replacement machine?
In the coming months, one of our resident experts, Dave Ohara (author of last month's article, which is now online at technetmagazine.com), will show you the work that is going on here at Microsoft with the construction of our own Green datacenters in Quincy, WA and San Antonio, TX, and how some of those questions are being answered. We'd like to hear how you're tackling these issues as well (or if you're choosing not to, tell us why). E-mail us at tnmag@microsoft.com, or visit our blog at blogs.technet.com/tnmag. —Joshua Hoffman

Thank you to the following Microsoft technical experts: Tim Carroll, Omesh Desai, Steve Jenness, Sam Kroonenburg, Mutlu Kurtoglu, KC Lemson, Tim Mintner, Jim Truher, Buck Woody, and Sudarshan Yadav.


 Toolbox: New Products for IT Pros
Toolbox
New Products for IT Pros
Greg Steen

Video Screen Capture
My Screen Recorder Pro
deskshare.com/msrpro.aspx
Suppose you have a meeting in 20 minutes and still need to put together a presentation on the new system configuration. Or perhaps you find yourself going through the same application tutorial almost every day with different users. If this sounds familiar, here's a tip: check out My Screen Recorder Pro from DeskShare.
As the name indicates, this app lets you record the activity taking place on your desktop as a video, offering a great way to put together an impromptu presentation or reusable tutorial for your clients. My Screen Recorder Pro also supports audio recording, so you can provide narration along the way. (Hint: create a script or outline before you start recording.)
Creating a new video is straightforward. Select the region you would like to record—your full desktop, a customizable area of the desktop, a fixed portion of the desktop, or a particular application window—then click the Record button.
The application can record to AVI, Windows Media® Video (WMV), or Flash, so you will be able to distribute your results relatively easily and even embed your videos in Web pages or stream them for broader public use. You can also create an executable for your video for playing it locally on most Windows® PCs. The video is embedded within the executable, simplifying distribution and making playback extremely intuitive for the end user.
If your tutorial or presentation lends itself to multiple segments, you can stitch together several takes into one video. Or, if you need to do a bit of editing on the video, you can split or cut the recording to meet your needs.
Though the default configuration is generally usable, the application lets you tweak the frame rate, audio quality, and compression settings to best suit your environment. This is useful for addressing any size or bandwidth constraints you may run into. You can also create a recording using a simple time-lapse mode, which in effect allows you to compress long-running portions of your recording (such as a slow progress bar). Another nice feature is the option to change the mouse cursor for the recording and add mouse pointer effects to draw attention to the actions occurring while the recording session is underway. For example, you can enlarge the pointer or make rings appear around it whenever you click the mouse button. In addition, you can add a caption and timestamp to the recording for archival purposes.
For the sneaky, paranoid, or extra-cautious folks out there, My Screen Recorder Pro lets you schedule recordings, and the program can be run from the command line, giving you the option to discreetly monitor activities on a workstation. Finally, the application has built-in features for publishing your videos to an FTP site. When combined with the scheduler, this option can be useful for auditing purposes.

Price: $99 (direct) for a single license.
Create a presentation by recording what's happening on your desktop
Create a presentation by recording what's happening on your desktop

Disk Defrag
PerfectDisk 8
raxco.com/products/PerfectDisk2k
Hard disk defragmentation is essential for any Windows system. If you're like me, you are constantly adding and removing applications, features, and software updates. There are any number of disk defragmentation utilities to choose from—in fact, I covered one just last month. But defrag is critical and PerfectDisk 8 from Raxco Software is definitely worth discussing. Even though all flavors of Windows include a defrag tool, sometimes you need a utility that offers more precise control over the defrag operation. If so, take a look at PerfectDisk.
As you'd expect from any Windows defrag tool, PerfectDisk 8 can defrag FAT16, FAT32, and NTFS drives, and it also lets you defrag RAID drives. Available in desktop and server versions, PerfectDisk 8 is certified for Windows Vista® and supports other versions of Windows, including Windows Server® 2003 and Windows 2000. There's even a version for Exchange.
The company touts its SMARTPlacement technology, which attempts to organize the file structure on the disk for faster access speeds. For instance, on Windows XP systems, the SMARTPlacement technology takes responsibility for managing your boot and layout.ini files, ordering them in such a way as to decrease disk seek times on system startup. In addition to defragmenting an entire drive, you can also choose to defrag a single file, which is useful after downloading a large ISO file, for instance.
To tackle those stubborn system files you usually can't get your hands on while the OS is running, the application uses an offline mode that can run at the next system startup or, under the right conditions, the application can try to unmount the disk to run the defrag. Another cool feature is PerfectDisk's screen saver defragmentation mode, which allows you to automatically start defragmenting when the screen saver runs.
PerfectDisk also includes a command-line version that lets you easily schedule defragmentation through your preferred task scheduler, and enable remote execution through any system management agents you may have running on the system. For larger deployments, you should also go ahead and download the free PerfectDisk Command Center, which lets you remotely deploy, update, schedule, and configure the workstation application across the enterprise.

Price: PerfectDisk 8 Professional (for the desktop) starts at $39.99 for a single license. PerfectDisk 8 Server starts at $239.99 for a single license.
PerfectDisk offers a number of configuration options
PerfectDisk offers a number of configuration options

Book Review
Network Warrior
oreilly.com
If your shop is running Cisco gear, you might want to pick up a copy of Gary Donahue's Network Warrior (O'Reilly, 2007). This book can be a valuable resource for a wide spectrum of readers, whether you're just getting started in the field of networking, you've recently received your CCNA certification, or you are an accomplished systems administrator looking to refresh your networking skills.
I particularly like that this book it is based on the real-world application of networking technology and that it gives you a quick reference to many of the common tasks you encounter in your day-to-day activities as a network professional. The book starts out on the ground level, defining exactly what comprises a network at its most basic. And it progresses into an overview of hubs, switches, VLANs, trunking, and the VLAN Trunk Protocol (VTP), as well as spanning trees with Spanning Tree Protocol (STP) and Cisco System's EtherChannel technology.
Throughout Network Warrior, you get basic examples using Cisco's Internetwork Operating System (IOS) and CatOS command-line interface (CLI) commands paired with the explanations of the methods and technologies. I like how the book gives a little history behind the current methodologies, offering some nice perspective along the way (such as why there is still a CatOS rather than just the IOS, for example).
The book includes a discussion of routers and routing in general where you learn about the fundamentals of routing from metrics to the administrative distances of the different routing protocols. You also get overviews of protocols, such as Open Shortest Path First (OSPF), Border Gateway Protocol (BGP), and Routing Information Protocol (RIP).
In the section on Cisco's Hot Standby Router Protocol (HSRP), the author delves into addressing availability and offers some basic configurations for protecting against downtime in your network infrastructure. That's followed by a review of the features and functionality of some of Cisco's multilayer switches, including the Cisco 6500 and the Catalyst 3750, and a discussion of some of the basic configurations you might encounter.
I found the information pertaining to the world of telecom useful. The book teaches you some of the lingo so you can avoid the smile-and-nod routine the next time they come to wire you up. You'll learn enough about Channel Service Unit/Data Service Unit (CSU/DSU), configuring DS3 connections, and multi-node frame relays that you can actually converse with the telecom techs.
There is a good section on security and firewalls, which shows you how to design and configure access control lists (ACLs), and covers authentication, DMZs, and specific PIX firewall configuration options. I like the succinct segment on firewall theory that emphasizes network mantras to work by such as "deny everything; permit what you need" and "security is a balance between convenience and paranoia."
The last section of Network Warrior provides insight into network design, covering the basics of common network infrastructure designs, such as a three-tiered e-commerce configuration and a collapsed-core network without a distribution layer, underscoring the importance of documentation and naming conventions.

Price: $44.99 list.

Manage File Locks
WhoLockMe
dr-hoiby.com/WhoLockMe
It would be nice if every application closed files correctly. Unfortunately, they don't. Sharing violations can be a real nuisance and finding out which process is currently locking a file can be even more troublesome. I don't know about you, but I hate closing down applications one by one, just to get control of a file. Fortunately, there is a free, simple Windows Explorer extension called WhoLockMe that can help.
Available for Windows NT®, Windows 2000, Windows Server 2003, and Windows XP, this app lets you know which process is currently locking the file of interest. All it takes is a simple right-click and a selection from within your Windows Explorer context menu.
WhoLockMe shows you the name of the process that is locking the file, its process identifier (PID), the path to the opened file, the domain and user of the process's execution context, and the full path to the offender. If you're so bold, you can choose to kill the process directly from the UI.
Unfortunately, WhoLockMe hasn't been updated in a while and it won't work on Windows Vista. Nonetheless, if you are running an earlier version of Windows, this app is a quick and effective tool for finding out who or what has a lock on the file you need to move, delete, or edit.

Price: Free

DNS Tools
DNSStuff.com
www.dnsstuff.com
If you deal with Internet-facing applications or networks, you undoubtedly depend on DNS to help you manage them, keep things running smoothly, and allow outsiders to know where you and your apps reside. One toolset I find myself relying on when there's a DNS issue comes from DNSstuff.com. This collection of Web-based tools can be a great resource for troubleshooting and verifying your infrastructure configurations.
The site provides tools for domain, IP, hostname, e-mail, Web site, and general network issues—more than 45 different query tools that can help you do everything from validate your mail server configuration to check the speed of your DNS hosting. Sure, you can probably find the equivalents of most of these tools within your infrastructure already, but external verification can offer key information for troubleshooting. And you won't have to bug your buddy at the company next door to do a reverse DNS lookup for you.
One of the most powerful and comprehensive tools on the site is the DNSreport. Simply enter a domain name, click the button, and you'll retrieve a wealth of information regarding your setup. Each test line item displays a color-coded status, which is great for quickly scanning through the test results to check for potential problems. The report checks the domain's parent zones, showing you details like name server (NS) records and whether or not they send out the IP address as well as the host name. The report also checks potential security holes, verifying that your DNS servers are not open servers, that they all have identical NS records, and that they don't leak any "stealth" NS records in non-NS requests.
You also get some insight into your setup's availability as DNSreport tests for multiple name servers, public IP bindings, and missing name servers while checking for single points of failure. In addition, DNSreport checks your Start Of Authority (SOA) details, returning the SOA serial number, refresh, expire, retry, and minimum time-to-live (TTL) values, as well as performing a check on your MNAME and RNAME records.
The report's Mail Exchange (MX) record-checks validate reverse DNS entries and ensures that you do not have any duplicate records and that your records have valid hostnames. Further, the tool tries to connect to your mail server and verify items such as whether your mail server's hostname is in the greeting, whether it accepts abuse@yourdomain messages (as specified in various RFCs), and whether the SMTP server is currently configured as an open relay (and thus a magnet for spammers around the globe). Finally, DNSreport returns your www A record, verifies that all the IPs are public, and determines whether your CNAME configuration will cause an extra DNS lookup.
If DNSreport isn't enough, the site also lets you use one lookup tool to check whether a mail server is listed in a set of well known spam databases and another to find the abuse contact for a domain. You can do IP Whois lookups, traceroutes, and pings, as well as general DNS lookups and cached DNS lookups at major ISPs. You can even do NETGEO IP lookups, MAC address checks, and complete zone file dumps if you subscribe. All in all, this is quite a valuable set of tools.

Price: $3 per month for a single user membership (full access and unlimited lookups).
DNSstuff.com provides a whole set of handy monitoring tools
DNSstuff.com provides a whole set of handy monitoring tools

Greg Steen is a technology professional and enthusiast. He is always on the hunt for new tools to help make operations, qa, and development easier for the IT professional.


 Exchange Queue &amp; A: Secure E-Mail Protocols, Mysterious Spam, and More
Exchange Queue & A
Secure E-Mail Protocols, Mysterious Spam, and More
Nino Bilic and Scott Landry
This column is based in part on a prerelease version of Windows Server 2008. Details herein are subject to change.

Q I want to use secure SMTP—how do I get Exchange Server to listen for SMTP on port 465?
A I'm sorry, but you can't do this. Yes, you can make any SMTP virtual server or Receive connector listen on port 465, but that will not achieve your goal of secure SMTP (SMTPS).
Why? Well, let's back up a little and see. There are two types of SSL: explicit and implicit. Initially, most of the SSL was implicit, meaning that a dedicated port for SSL was used. For example, HTTP is on port 80 by default, but HTTPS (HTTP with SSL) is on port 443. Several years ago, the Internet community decided that a dedicated port should not be required for SSL. Thus, explicit SSL was born.
Netscape had already chosen 465 as the SMTPS port, but Exchange Server had no SSL functionality in SMTP. However, the Exchange team saw the advantage of explicit SSL—that it could be used equally by clients and servers—and chose to support explicit SSL for SMTP.
In the case of SMTP, explicit SSL uses the STARTTLS ESMTP command to signal that the existing socket is about to be secured. Most other SMTP server and client vendors have implemented the STARTTLS command as well, so there never was much need to support port 465, which wasn't an official Internet standard anyway.
To this day, no version of Exchange Server supports implicit SSL for SMTP. Telling an Exchange Receive connector or SMTP virtual server to listen on port 465 does not change this fact. Therefore, you need to use a client that supports STARTTLS on port 25. If you can't use port 25, the next logical choice is 587, which is the standard port for client SMTP submissions. There aren't many modern clients that don't support STARTTLS on 25, so adding support for implicit SSL has not been necessary.
By the way, Exchange POP3 and IMAP4 protocols have always supported implicit SSL. But in Exchange Server 2007, there is now added support for explicit SSL for these two protocols as well. But since not many clients support this newer standard yet, implicit SSL is going to be around for the foreseeable future.

Q I have a great deal of mail queued to a number of domains—and none of my users sent any of the mail. What's going on, and how do I prevent it?
A You're not alone. Anyone who has a server on the Internet may run into this problem. Basically, there are two possible causes. The first is that you somehow opened yourself up for relay (see support.microsoft.com/kb/304897). But of course you wouldn't do that, would you? (Open relays have been disabled by default since Exchange Server 2000.) So it's more likely you're seeing what's called non-delivery report (NDR) spam. In the process of sending unsolicited commercial e-mail (UCE), spammers often send to nonexistent addresses at your domain. Your server tries to let the spammer know that the users don't exist, but of course, the spammer has spoofed the return address. The spammer may be spoofing an invalid address (in which case the NDR hangs out for a while until it times out), or he may be attempting to have your server send spam to another domain on his behalf, as an attachment of the NDR that your server has generated.
You could disable NDRs, but then if a legitimate user mistypes an address by mistake, your server will never let him know that the e-mail did not go through and critical messages could be lost. Here's a better solution.
First, make sure you are not open for relay. (I just had to say it.) Next, turn on some sort of anti-spam filtering, such as the intelligent message filter (IMF) or the Exchange Server 2007 content filter, as well as a few Realtime Block Lists (RBLs). This can be in either the Edge Transport role or the Hub Transport role, but it should be done at the very first hop because over 90 percent of mail volume tends to be spam and you don't want to keep your servers busy with all that junk).
Finally, enable recipient filtering on the first Exchange Server that accepts mail into your environment. This allows your server to reject a message before it comes into your network. Legitimate address mistypes will still get the NDR, but the NDR will be generated by the sender's server.

Q I had one server running Exchange Server 2000 and one running Exchange Server 2003, each sending mail successfully to the Internet. Then I installed Exchange Server 2007 and now mailboxes on each server cannot send mail.
A If you have had only one Exchange Server in the past, you might not be very familiar with the concept of connectors. Exchange connectors are logical routing configuration objects that tell Exchange where to direct e-mail. When you introduce Exchange Server 2007 into an existing organization, in order to route mail you absolutely must have routing group connectors and an SMTP connector.
You'll need two routing group connectors, one going from the Exchange Server 2007 routing group to the Exchange Server 2003 routing group, and vice versa. You can set this up as part of the installation process, but if you missed it or you're not sure, you can check using the Exchange Management Shell and correct the problem there. If you don't, you will not be able to send mail between your servers. Messages will end up in unreachable destination queues.
To route Internet mail you only need one SMTP connector, also known as a Send connector in Exchange Server 2007. You should have one in Exchange Server 2000 and Exchange Server 2003, but you may have gotten by without it. The address space should be SMTP:* for all domains, and you can specify either the use of a smart host or DNS for mail delivery. You choose whether you want the Exchange Server 2007 server or the older server to handle the outgoing Internet mail, or you can create one on both routing groups if you want each server to handle its own. You can also create one of these as part of the Edgesync process if you have installed an Edge Transport server role.
If you previously put a smart host on the SMTP virtual server, now is a good time to remove it. It should only be on the SMTP connector, never on the virtual server as that will break the routing group connector.
You should also be aware that inbound e-mail is controlled by your MX record or the IP the firewall is forwarding to. There isn't much to configure on the Exchange side, but this document should help if you're still having trouble: msexchangeteam.com/archive/2006/11/17/431555.aspx.

Q Why do I get multiple journal reports for the same message in Exchange Server 2007?
A The Exchange Server 2007 transport journaling agent will journal messages after categorization. The categorizer has many reasons why it bifurcates a message (that is, copies the message body and puts different envelope recipients on the different copies). Here's an example: because the journaling now has the ability to tell you what the membership of a distribution group was at the time the message was sent, one case where you might get multiple reports is for a nested distribution group.
This additional richness in the reporting means that you may get a few copies of the same message, each with a unique report. There isn't a guaranteed way to know if all the reports have arrived for a message, but if you're doing archiving, you'll want to work with your archive vendor to make sure it's aware of the changes.

Q Where did the feature for forwarding unresolved messages to the host go in Exchange Server 2007? What do I do now?
A The dog ate it.
Actually, this specific feature never worked very well in situations where you had more than one Exchange Server. Exchange already had another way to accomplish the same thing, however, and that method was much more powerful. Specifically, you have the ability to share individual SMTP domains with other systems. So the "forward unresolved" feature was dropped and the shared domain concept was carried forward and simplified. In Exchange Server 2007, just go to Organization | Hub Transport | Accepted Domains and change the domain type from Authoritative to Internal Relay. It's actually a little more complex than this for some environments, and we're working to update some of the documentation. But this should help in the meantime.

Q I am trying to prepare my root domain for Exchange Server 2007 installation. The server I am trying to run Exchange Server 2007 setup from has Exchange Server 2003 Exchange System Manager (ESM) installed and the setup is failing. What's the problem here?
A Simply put, running any part of Exchange Server 2007 setup on a machine that has any Exchange Server 2000 or 2003 components installed is not supported. Because Exchange Server 2003 ESM is installed, the Exchange Server 2007 setup will see this and a setup prerequisite check will fail and tell you, "A previous version of Exchange Server is already installed on this computer. Run Exchange Server 2007 Setup from a different computer or remove the previous version of Exchange Server."
Probably the easiest way to work around this problem is to simply run the Exchange Server 2007 setup from another server in the root domain and prepare your domain that way. If that's not feasible, the Exchange Server 2003 component will have to be uninstalled before you can continue with the Exchange Server 2007 setup.
Remember that you can use the 32-bit version of Exchange Server 2007 to prepare the domain, so any 32-bit server in the root domain will usually do. For more information on this subject, please see technet.microsoft.com/library/bb232170.aspx.
By the way, this means that you can't install Exchange Server 2003 ESM and Exchange Server 2007 Exchange Management Console on the same machine as the coexistence of Exchange Server 2003 and Exchange Server 2007 management tools on the same machine is not supported. Exchange Server 2007 will block setup if you attempt to install it on a machine that has any Exchange Server 2000 or Exchange Server 2003 component installed.
Finally, note that you should not attempt to install Exchange Server 2007 management tools first and then follow with Exchange Server 2003 tools on the same machine. This approach will put you in a configuration that has not been tested and might give you unexpected results when trying to manage your servers. If you need to manage both server versions from a single machine, you could use remote desktop to connect to one version, or use a virtual machine to host a different version of the management tools in an isolated environment.

Q When, oh when, will I finally be able to run Exchange Server 2007 Management Tools on my Windows Vista® workstation?
A The official support for Exchange Server 2007 Management Tools on Windows Vista is coming with the release of Exchange Server 2007 SP1. A package containing the Exchange Server 2007 SP1 management tools will be available for download once Exchange Server 2007 SP1 is released.

Q What about Exchange Server 2003 ESM on Windows Vista or Windows Server 2008? Will I be able to run that too?
A No, unfortunately this will not work. Management tools for any version of Exchange Server prior to Exchange Server 2007 SP1 will not be supported on either Windows Vista or Windows Server 2008. That means that even after Windows Server 2008 releases, installing Exchange Server 2003 ESM on it will not be supported. Management of Exchange Server 2003 servers will have to be done either from Windows Server 2003 or Windows XP workstations; alternatively, you can use the Remote Desktop connection from any OS.

Q I have several Exchange Server 2003 servers running in my domain. Will I be able to upgrade my domain controllers to Windows Server 2008 domain controllers?
A Yes indeed, running Exchange Server 2003 SP2 in the domain that has Windows Server 2008 domain controllers is supported. Please note that Exchange Server cannot use Windows Server 2008 read-only domain controllers (RODCs) or read-only global catalog servers (ROGCs). Manually specifying (hardcoding) Exchange Server to use the Windows Server 2008 RODC/ROGC might result in unexpected behavior.

Nino Bilic a Supportability Program Manager for Exchange Server, spends his free time discovering the beauty of server-to-server communication by reading a ton of Netmon traces before going to sleep at night. Scott Landry, a Support Escalation Engineer for Exchange Server, does not go anywhere without his towel, copy of the Guide, and his trusty Windows Mobile phone.
Scott Landry a Support Escalation Engineer for Exchange Server, does not go anywhere without his towel, copy of the Guide, and his trusty Windows Mobile phone.


 Utility Spotlight: SMS Cmdlets for Windows PowerShell
Utility Spotlight
SMS Cmdlets for Windows PowerShell
Don Brown
Download the code for this article: Utility2007_11.exe (1211KB)

There used to be no way to manage your Microsoft System Management Server (SMS) 2003 clients from the command line. Luckily, among the many great technologies that have come along lately to help administrators manage complexity, change, and configuration, Windows PowerShell arrived to solve that problem.
To take advantage of it, I wrote a small "utility," SMS2003PowerShellSnapinSample (which you can find in the code download that accompanies this column, available online at technetmagazine.com/code07.aspx). The utility really boils down to a collection of six cmdlets assembled into a single Windows PowerShellTM snap-in. Using these cmdlets, you can configure SMS client local policy from Windows PowerShell for the local machine or groups of remote machines.
The more I played with SMS client local policy, the more this made sense. As a side benefit, I also came to understand more about how SMS works. I learned that by careful application of SMS client local policy, the SMS client could be controlled in a whole new way: machines reporting to the same site did not all have to be configured the same. Ideas started to flow—from setting certain machines so that permission isn't required to remotely control the machine, to changing how often the client checks for new policy, to disabling specific components at various times of the day—the sky is the limit using SMS client local policy. Let's take a closer look.

About SMS Advanced Client Local Policy
Programming Resources
  • Introducing Windows PowerShell A great primer for anyone looking to start using Windows PowerShell
  • Customize SMS Using Local Policies TechNet Forum MOF-based application of SMS Advanced Client local policy
  • Creating Your First Cmdlet
  • How to Create a Windows PowerShell Snap-In
  • Network Access Protection (NAP) for Windows Server 2008
  • Windows PowerShell Programmer's Guide For those who want more information on writing cmdlets, snap-ins, hosts, and other programming help for Windows PowerShell
  • Windows Server 2008 Beta 3 Step-by-Step Guides

All functional SMS 2003 Advanced Clients have a configuration policy, which is basically a list of settings that drive their various components. All client, inventory collection, and software distribution settings (among others) are included in these configuration policies. The policies themselves are created on the site server and handed down to the Advanced Client through the SMS management point.
The actual body of an SMS policy is similar to a Managed Object Format (MOF) file in that it contains a set of instances that will be compiled into the \\.\root\CCM WMI (Windows® Management Instrumentation) namespaces on the Advanced Client. The various other agents will then read these settings, which are located in the \\.\root\ccm\policy\machine\requestedconfig namespace. When applying local policy via MOF, however, you can only compile a MOF on a single machine and it must be run locally (or rather, when you are logged on directly to the machine). But WMI, because of its distributed nature, is accessible both locally and remotely, which yields a greater array of options to assist SMS administrators. This means that you can connect to WMI on a remote machine as easily as you can connect to WMI on your local machine, given proper administrative rights.
SMS client policy consists of configuration settings for the various client components, but it may also include instructions for executing software package content. As with Active Directory® Group Policy, SMS 2003 Advanced Client Policy obtained from an SMS management point may be overridden by SMS client local policy. You can't override all parts of a policy, but some really interesting properties can be overridden. This gives SMS administrators a greater degree of control over the configuration and operations of the SMS Client because it allows exceptions to the standard configuration applied at the SMS site.
Take the example of a secure and consolidated environment, where SMS administrators manage both servers and workstations as clients of a single SMS primary site. In this imaginary environment, a security policy might state that a user must be prompted for permission before help desk technicians can take remote control over their machine. That's obviously a problem if the help desk technicians want to use remote control to attach to the servers: typically, no users are logged on to the servers, so there's nobody to grant remote control permission. By careful application of local policy, though, the requirement for user permission may be overridden on specific clients. You can probably imagine dozens of other circumstances where SMS local policy may be useful in granting an exception to configuration of a particular SMS client agent.

Inside the Windows PowerShell Snap-In
Once you have some familiarity with the basics of Windows PowerShell, you can add to the sample source code provided with this column to expand what you can do to SMS client local policy from the command line. There are specific attributes that should be applied to a class before it is exposed to Windows PowerShell. You need to decide what the cmdlet is about and what action should be carried out. This is known as the "Verb-Noun" pairing. Some common verbs you'll see in Windows PowerShell are Add, Get, and Set. The noun portion describes the object you want to act upon. Cmdlets usually have parameters, which are declared in a Cmdlet class as public properties of various types. Lastly, the ProcessRecord function is the main place where you will do most of your work. Figure 1 shows a generic template you can use for cmdlets of your own.
Figure 1 Cmdlet template (Close Figure 1)
[Cmdlet( "Verb", "Noun", SupportsShouldProcess = true )]
public class Verb_Noun : PSCmdlet
{
    [Parameter( ValueFromPipeline = true, ValueFromPipelineByPropertyName = true,
        HelpMessage = "Parameter" )]
    [ValidateNotNullOrEmpty]
    [Alias( "param" )]
    public string Parameter
    {
        get { return MyParameter; }
        set { MyParameter = value; }
    }
    private string MyParameter;

    protected override void ProcessRecord( )
    {
        //Do your stuff here!
    }
}
Once you've compiled your snap-in, you register it with Windows PowerShell by including a class in your project that has the RunInstaller attribute set and a few specific properties defined. See the source code accompanying this article for specifics. To register your snap-in, you will need to use the InstallUtil.exe tool included with the Microsoft® .NET Framework. Figure 2 shows the syntax you use to register your snap-in. Note that in Windows Vista®, this is an operation that requires elevation of privileges, so you will want to open Windows PowerShell with the option of "Open as administrator," or use the Script Elevation PowerToys for Windows Vista (which you can download at technetmagazine.com/issues/2007/06/UtilitySpotlight).
Figure 2 Installing the snap-in from Windows PowerShell (Close Figure 2)
PS> set-alias installutil $env:windir\Microsoft.NET\Framework\v2.0.50727\installutil
PS> installutil C:\MySMSTools\SMS2003PowerShellSnapinSample.dll
Microsoft (R) .NET Framework Installation utility Version 2.0.50727.42
Copyright (C) Microsoft Corporation. All rights reserved.
Running a transacted installation.
...
The transacted install has completed.
The next step is to find out if your snap-in is acknowledged by Windows PowerShell. Use the Get-PSsnapin cmdlet with the Registered parameter and Windows PowerShell summarizes the currently loaded snap-ins, followed by a list of registered snap-ins to be added. Your snap-in should be found in this list:
PS> Get-PSsnapin -registered
Now you can add the Windows PowerShell snap-in to the shell using the Add-PSsnapin cmdlet, as follows:
PS> add-pssnapin SMS2003PowerShellSnapinSample
If the operation is successful, you should be able to execute the cmdlets in your Windows PowerShell snap-in.
To see all the cmdlets that are made available by any Windows PowerShell snap-in, just specify the particular snap-in with the Get-Command cmdlet, passing the name of your snap-in as the value for the PSsnapin parameter. The sample snap-in accompanying this column displays six cmdlets, including the Verb-Noun Cmdlet template, as Figure 3 shows.
Figure 3 Displaying the cmdlets in a snap-in
Figure 3 Displaying the cmdlets in a snap-in
In the sample source code, I've written help only for the Get-SMSServerConnection cmdlet to demonstrate how it is constructed. If you want to expand the help, you will have an example to model after in the XML file. Figure 4 shows the help output.
Figure 4 Sample Get-Help output (Close Figure 4)
PS > get-help Get-SMSServerConnection

NAME
    Get-SMSServerConnection

SYNOPSIS
    This cmdlet establishes a connection to the specified SMS primary site server using your current credentials. An object of type "SMSProvider" is returned through the pipeline.

SYNTAX
    Get-SMSServerConnection [-SMSServerName] [<string>] [<CommonParameters>]

DETAILED DESCRIPTION
    This Cmdlet makes a connection to the specified SMS primary site server.  An object of type "SMSProvider" is returned. The "SMSProvider" object is not serializable and is used only to forward on through the pipeline to other cmdlets.

RELATED LINKS
Now that you've seen how to install the Windows PowerShell snap-in, how to find out what commands are present in the snap-in, and how to get help on a typical cmdlet, let's start using the cmdlets. To show a list of all collections at an SMS primary site server, you can use the following command:
PS > Get-SMSServerConnection -server MYSMSSERVER | Get-Collections | Format-Table Name
Note the pipeline. This is a core part of how Windows PowerShell works, and it's a very powerful tool. It may be possible to do just about anything in Windows PowerShell on one line. The Get-SMSServerConnection cmdlet establishes a connection with the SMS server. Since the Get-Collections cmdlet has a single input parameter of the type that is returned by Get-SMSServerConnection, you can simply pass the output from the Get-SMSServerConnection cmdlet to the Get-Collections cmdlet. This is how the pipeline works, and it's a good example of how you can pass complex objects from one cmdlet to another. You can also store objects in variables, and if you were doing this in a Windows PowerShell script, it might look something like this:
$SMS = Get-SMSServerConnection
     -server MYSMSSERVER
Get-Collections -SMSServerProvider $SMS
The output might be a little bit hard to read without some nice formatting of the output, because Get-Collections returns an object of type SMSCollections (which is basically an array of objects of type SMSCollection). In short, not a very pretty display, until you either filter out collections you aren't interested in seeing or format the display in a table of only those properties you care about. You do this by passing the output from the Get-Collections cmdlet through the pipeline to the Format-Table cmdlet, which can display only those properties that you specify. For example, you may append | Format-Table Name, Members.
Ah, but there's a better way to show all the members of a particular collection, and in this sample it is Get-CollectionMembers. This cmdlet returns an array of strings, representing the name of each member of the collection. As you might have guessed, this is a great candidate for passing down the pipeline to the next cmdlet.
Up to this point, there has been no SMS Advanced Client local policy—just connections to the SMS primary site server and enumerating collections and collection members. Included in this sample Windows PowerShell snap-in are two last cmdlets, named Enable-SoftwareDistribution and Disable-SoftwareDistribution respectively. This is where the SMS client local policy comes into play. These last two cmdlets manipulate the SMS client local policy for the Software Distribution Client Agent component on the SMS Advanced Client. As you might have guessed by the verb part of the cmdlet, Disable places a client local policy override that specifies the Software Distribution Client Agent should be disabled as well. In the same way, the verb Enable removes all client local policy overrides, returning the Software Distribution Client Agent back to its normal state, which is defined by the SMS policy handed down by the SMS management point. Figure 5 shows an example of what a Windows PowerShell one-liner might look like that places an SMS client local policy override on the Software Distribution Client Agent that forces it to be disabled for all members of the Windows Server 2003 Systems custom collection. The one-liner to remove all SMS client local policy overrides for the Software Distribution Client Agent on that same collection is also listed. The Disable-SoftwareDistribution and Enable-SoftwareDistribution cmdlets can also be used on their own, if you only have a few machines on which you want to manipulate SMS client local policy.
Figure 5 Assembling the cmdlets—examples (Close Figure 5)
PS >Get-SMSServerConnection -server MYSMSSERVER | Get-Collections | where-object {$_.Name -eq "Windows Server 2003 Systems"} | Get-CollectionMembers | Disable-SoftwareDistribution
PS >
PS > Get-SMSServerConnection -server MYSMSSERVER | Get-Collections | where-object {$_.Name -eq "Windows Server 2003 Systems"} | Get-CollectionMembers | Enable-SoftwareDistribution
PS >
PS >Disable-SoftwareDistribution –hosts SMSCLIENT1, SMSCLIENT2, SMSCLIENT3
PS >
PS >Enable-SoftwareDistribution –hosts SMSCLIENT1, SMSCLIENT2, SMSCLIENT3

Next Steps
The sample source code that comes with this article can provide you a good starting point in applying SMS client local policy overrides to groups of computers. The sample here only touches on the SMS Software Distribution Client Agent, and then only the Enabled property. The other client agents have many more properties you can apply overrides to in the same way. Now that you have a common starting point, the next logical step is to look at other parts of the SMS client local policy. Think about groups of managed systems that might be good candidates for SMS client local policy overrides and what local policy overrides make sense for you to use. One final note: it's good practice to know what you might be getting yourself into, so always test everything thoroughly in a lab environment.

Don Brown is a Senior Premier Field Engineer at Microsoft and has been working and supporting with SMS (ahem, SCCM) for as long as he can remember. Contact him at donbrown@microsoft.com.


 The Cable Guy: Wireless Single Sign-On
The Cable Guy
Wireless Single Sign-On
Joseph Davies

Wireless connections that are established and then have to authenticate with a private network can add an extra layer of challenge for user and administrator alike. A Windows-based wireless client that is a member of an Active Directory domain must perform an IEEE 802.1X-based authentication to obtain a protected
wireless connection and a domain logon. Because authentication can involve either user or computer credentials and because domain logons are dependent on network connectivity, issues arise when configuring the order in which the authentications will be performed and when specifying which credentials to use. All this can impact domain logons and cause a user to be prompted for credentials multiple times.
Fortunately, Wireless Single Sign-On in Windows Vista® lets you specify that IEEE 802.1X authentication for Wi-Fi Protected Access 2 (WPA2)-Enterprise, WPA-Enterprise, and 802.1X authentication with Wireless Equivalency Privacy (WEP) occur before the user logon process. This allows you to solve domain logon issues in specific configurations. Wireless Single Sign-On also seamlessly integrates wireless authentication with the Windows® logon experience, resulting in happier users overall. For these reasons, I want to take a look this month at the features of Wireless Single Sign-On, how to configure it, and how it works during the user logon process.

Single Sign-On Essentials
A Windows wireless client can perform authentication to the wireless network using only computer or user credentials, or a combination of both. When using only computer credentials, Windows performs 802.1X authentication before displaying the Windows logon screen. This gives the wireless client computer early access to networking resources such as Active Directory® domain controllers.
When using only user credentials, Windows without Single Sign-On performs 802.1X authentication after the user logon process has completed. Figure 1 shows the startup and logon timeline for this scenario.
Figure 1 Timeline when using only user credentials for wireless authentication
Figure 1 Timeline when using only user credentials for wireless authentication
The consequences of using only user credentials for wireless authentication are that a user cannot do an initial domain logon to a computer because there are no locally cached credentials for his or her user account and there is no connectivity to the domain controller to authenticate new logon credentials. Moreover, some domain logon operations will fail because there is no connectivity to the domain controllers of the Active Directory domain at this time. Logon scripts, Group Policy updates, and user profile updates fail, resulting in Windows event log errors.
When using a combination of user and computer credentials, Windows performs an 802.1X authentication with computer credentials before displaying the Windows logon screen. After the user has logged on, Windows performs another 802.1X authentication with user credentials. Some network infrastructures use different virtual LANs (VLANs) for computers that have authenticated with computer credentials and those that have used user credentials. If the authentication to the wireless network using user credentials and the switch to the user-authenticated VLAN occur after the user logon process, the Windows wireless client will not have access to networking resources on the user-authenticated VLAN, such as Active Directory domain controllers, during the user logon process. Once again, this leads to failed domain logon operations.
Another problem occurs when the wireless authentication with user credentials uses a different set of security credentials than the user logon credentials. For such configurations, Windows prompts for additional user credentials when wireless authentication begins, which can be confusing.
The Single Sign-On feature in Windows Vista was developed to address the issues of domain logons, separate computer- and user-authenticated VLANs, and prompting the user for different credentials at different times. With Single Sign-On, network administrators can specify that wireless authentication with user credentials occur before the user logon process. When the user credentials needed for user logon and for wireless authentication are different, all of the required credentials are requested and collected on the Windows logon screen.
With Single Sign-On, a Windows Vista wireless client can be configured to perform wireless network authentication with user credentials before the user logon process. Figure 2 shows the new startup and logon timeline.
Figure 2 Timeline when wireless network authentication with user credentials occurs before user logon
Figure 2 Timeline when wireless network authentication with user credentials occurs before user logon
This allows for configurations that use only user credentials or a combination of user and computer credentials with separate VLANs with no loss of functionality. Because the wireless client has network connectivity or connectivity to the user-authenticated VLAN before the user logon process starts, domain logon operations can complete successfully. For an example, see "Joining a Windows Vista Wireless Client to a Domain" at microsoft.com/technet/network/wifi/vista_bootstrap_wireless.mspx.
Based on the configuration of the wireless profile, Single Sign-On consolidates the input fields for the user credentials for wireless authentication and adds them to the Windows logon screen. Therefore, all of the user credentials for user logon and wireless authentication using user credentials are supplied by the user at the same time.
Extensible Authentication Protocol (EAP) methods written for the new EAPHost architecture in Windows Vista can use the EAP Pre-Logon Authentication Provider (PLAP) Support API to include input fields needed for authentication on the Windows logon screen. For more information, see the SSO and PLAP API at msdn2.microsoft.com/bb530584.
As an example of a Single Sign-On session, consider a Windows Vista wireless client that is configured to perform authentication using only user credentials and a user name and password for both domain logon and 802.1X authentication. When the user presses Ctrl+Alt+Del at the initial Windows Vista screen, Single Sign-On determines that the 802.1X authentication must occur prior to the user logon. When the user types his or her user name and password, Single Sign-On performs user-based wireless network authentication first and displays a message indicating that it is connecting to the wireless network by its name. After wireless authentication, Windows Vista performs user logon and displays the user's desktop.

Configuring Single Sign-On
Behind the Scenes
To add to your understanding of the wireless authentication process, let's take a more detailed look at what happens when the user actively tries to log on. When a user hits Ctrl+Alt+Del for user logon on a wireless client running Windows Vista, the following steps are executed:
  1. Wireless Auto Configuration determines the wireless network profile to use. If the wireless client has already successfully performed authentication with computer credentials, the currently connected wireless network profile is used. If the determined wireless profile is configured to perform authentication with only computer credentials, no additional wireless authentication with user credentials is needed. Steps 2 through 6 assume that the selected wireless profile is configured to perform authentication with user credentials, has Single Sign-On enabled, and the setting Perform immediately before User Logon is selected.
  2. The user types his or her credentials for user logon and wireless authentication (if needed) and initiates the user logon.
  3. Windows displays a message to the user that it is attempting to connect to the wireless network name.
  4. Windows attempts to perform a wireless network authentication with user credentials. If the Allow additional dialogs to be displayed during Single Sign-On setting is enabled and the EAP type needs to display additional dialog boxes to the user, Windows displays the dialog boxes. If wireless authentication does not successfully occur within the Max delay for connectivity (in seconds), wireless authentication is abandoned. If the setting This network uses different VLAN for authentication with machine and user credentials is selected, Windows performs a DHCP renewal of the IP address configuration of the wireless network adapter when the wireless authentication is successful.
  5. Windows performs the user logon process.
  6. Windows displays the desktop.
If the Perform immediately after User Logon setting is selected, Windows performs the steps in the following order: 1, 2, 5, 3, 4, 6.

To enable and configure Single Sign-On, you can use the Wireless Network (IEEE 802.11) Policies Group Policy extension and configure the advanced security settings for a wireless profile of a Windows Vista policy. For more information on this, see "Wireless Group Policy Settings for Windows Vista" at technetmagazine.com/issues/2007/04/CableGuy.
On the Advanced security settings dialog box, select Enable Single Sign-On for this network and configure Single Sign-On options as needed. Figure 3 shows an example of Single Sign-On enabled with the default settings.
Figure 3 Default Single Sign-On settings
Figure 3 Default Single Sign-On settings
There are Single Sign-On settings to perform 802.1X wireless authentication immediately before or after the user logon process and to specify the time delay for 802.1X authentication to complete before starting the user logon process. You can also indicate whether to display dialog boxes beyond the consolidation of input fields on the Windows logon screen. For example, if an EAP type wants the user to confirm the certificate sent from the Remote Authentication Dial-in User Service (RADIUS) server during authentication, the EAP type can display the dialog box.
You can also specify that the wireless client should initiate a Dynamic Host Configuration Protocol (DHCP) renewal of the TCP/IP configuration of the wireless adapter after performing user-based authentication. Select this option if there are separate VLANs for computer- and user-based authenticated wireless clients and those VLANs are different IPv4 or IPv6 subnets.
After you create the wireless profile containing the Single Sign-On settings within the policy, you can also configure Windows Vista wireless clients by exporting the profile as an XML file and then importing the XML profile on Windows Vista wireless clients.
To create a Single Sign-On XML wireless profile, create a wireless profile with the appropriate Single Sign-On settings. On the General tab of the Windows Vista wireless policy, click the wireless profile with the Single Sign-On settings, and then click Export. When prompted, specify the profile XML file name and its location.
To use the Single Sign-On XML profile to configure another Windows Vista wireless client, import the XML profile with the command:
netsh wlan add profile filename=
    "[FileName].xml"
To determine if a wireless profile has Single Sign-On enabled, use:
netsh wlan show profile=[ProfileName]
Single Sign-On settings are listed in the "Security settings" section of the netsh wlan show profile command display and in the text of wireless connection events in the Windows event log. You can also use the Event Viewer snap-in to view the events in the Microsoft\Windows\WLAN-AutoConfig folder of the Applications and Services logs with the source of "WLAN-AutoConfig" and event IDs 13001, 13002, 13003, and 13004.
You'll find more information at the Wireless Networking Web page at microsoft.com/wifi. And see the "Behind the Scenes" sidebar for more details on the Single Sign-On process.

Joseph Davies is a technical writer with Microsoft and has been teaching and writing about Windows networking topics since 1992. He has written five books for Microsoft Press and is the author of the monthly TechNet Magazine Cable Guy column.


 Research Notes: Using High-Performance Computing in the Fight against HIV
Research Notes
Using High-Performance Computing in the Fight against HIV
Kristin Firth and Mia Matusow

When you think of people conducting medical research, do you envision men and women in white lab coats working with test tubes and microscopes? Perhaps a team of chemists at a university or pharmaceutical company? These are common and certainly real images of medical research, but they represent just
one aspect of the field. In reality, some medical research is performed in very different settings.
You might be surprised to learn that Microsoft Research is playing a key role in the effort to develop a vaccine for the human immunodeficiency virus (HIV). What's more, Microsoft is doing this without a single Bunsen burner in sight. The setting looks like any other office space, furnished with regular desks and lots of PCs.
So what role does Microsoft play in all this research? Those people in white lab coats—the scientists at various universities and research centers, including Mass General in Boston, the University of British Columbia, the University of Washington, the Fred Hutchinson Cancer Research Center, and Murdoch University in Australia—gather vast amounts of data in their projects. The eScience group at Microsoft Research, which consists of about half a dozen people mostly located in Redmond, WA (and one person in New Mexico), helps those researchers process and analyze massive amounts of data. Microsoft Research first got involved in this project back in 2003, and today the eScience group collaborates with scientists on a variety of projects dealing with issues that affect society. The team works closely with external scientists, building custom solutions for projects, crunching numbers, and analyzing results.
In one high-priority project, Microsoft is helping analyze data related to HIV mutation patterns. David Heckerman, MD, PhD, Senior Researcher for Microsoft, explains that HIV mutates rapidly when attacked by an infected person's immune system. "We're working on a study that is attempting to determine how HIV mutates in response to the host's immune system. To do so, we are looking for correlations among an individual's immune system type and the HIV protein sequences that infect him."
Scientists use samples from people infected with HIV to determine their immune system types and HIV sequences. That's where the number crunching comes in. The study searches for correlations between 3,000 HIV amino acids and hundreds of immune system types across many hundreds of individuals. "We've devised statistical tests that produce more trustworthy correlations, with a reduced number of false positives and negatives," says Carl Kadie, PhD, Principal Research Software Development Engineer for Microsoft. "However, those tests demand a great deal of compute power, and the more subjects we include, the better. The idea is to run millions of simulations to reveal the most trustworthy correlations."

Using HPC to Fight HIV
Just a few years back, processing limitations hampered research efforts. Armed with only half a dozen computers, the Microsoft researchers involved in this project didn't have enough processing power to perform analysis in an adequate amount of time. Looking across every position along the genome of the vaccine and the different immune types would have taken an entire year for just 200 subjects. Even if the researchers had 20 computers to dedicate to this analysis, they still would have faced the fundamental problem of running the tests manually on 20 separate computers, receiving 20 individual sets of results, and needing another program (as well as more time) to tabulate the separate sets of results. Managing multiple jobs, gathering partial output, and all the other individual tasks involved would have taken too much time. So they turned to the High-Performance Computing (HPC) group at Microsoft.
In 2006, the researchers implemented Windows® Compute Cluster Server 2003. This HPC solution offered a straightforward way for the researchers to harness the power of many computers working together. Essentially, Windows Compute Cluster Server allows the work to be distributed across server nodes running in parallel. The task of distributing data among nodes, managing the data, and combining the results is all automated. The solution includes setup procedures, a suite of management tools, and an integrated job scheduler.
Finally, with this solution, the most significant technical roadblocks were overcome: compute power was no longer limited and the process of distributing work and managing the data was fully automated.

The Setup
For Microsoft Research, the compute cluster approach was a great fit. Windows Compute Cluster Server 2003 is well suited for projects that involve applications performing the same operation over and over—for instance, when problem-solving and analysis can be accelerated by running tasks in parallel. However, deploying this solution should not be undertaken lightly, since high-performance computing can be capital-intensive—especially in terms of power and cooling. It involves multiple servers that often run at 100 percent CPU utilization for weeks on end.
The next steps are to determine how big a cluster the organization can afford and where it can be stored. If an environment is already Windows-based with Active Directory® in place, administrators have the infrastructure they need to install, deploy, and support a cluster. In environments where Active Directory isn't already in use, some additional configuration steps are required.
You should also decide how to provision nodes within the compute cluster, which means choosing whether to use the tools that come with Windows Compute Cluster Server 2003 or using your own internal deployment techniques. You need to deploy Windows Compute Cluster Server 2003 itself along with the software applications that you want to run on the cluster. User access to the cluster must also be established so users have the ability to connect to the cluster and submit jobs, which can be done through the included graphical UI or through command-line interfaces.
Windows Compute Cluster Server 2003 is a 64-bit operating system. (The typical architecture of a Windows Compute Cluster Server 2003 environment is illustrated in Figure 1.) At the time of writing this column (June 2007), Microsoft Research was running a variety of different applications on a cluster built upon 25 IBM eServer 326 servers. Each of these servers features two AMD Opteron processors running at 2.6 GHz.
Figure 1 Harnessing power with Windows Compute Cluster Server 2003
Figure 1 Harnessing power with Windows Compute Cluster Server 2003
After deploying Windows Compute Cluster Server, the group updated the application that it uses to perform genetic correlation to run on the cluster. The complexity of deploying applications for compute clusters can vary. The applications themselves influence how much programming is necessary.
At first, the group used the tools that were built into Windows Compute Cluster Server to quickly establish a generic UI. That was just a quick fix, however, and the group soon created its own custom Web application, which provides greater flexibility and supports the option of exposing some of the cluster nodes to scientists located outside of Microsoft. In fact, Microsoft is part of the Open Grid Forum and some of the Microsoft Research clusters are made available to users from other universities around the world, enabling researchers to collaborate and share the work load.
In addition to its graphical UIs, Windows Compute Cluster Server also supports command-line operations, allowing users to write scripts. And it provides rich APIs that can be used to write programs that interact directly with the Windows Compute Cluster Job Scheduler—a technique that Microsoft Research chose to utilize.

Getting Results
Thanks to its use of high-performance computing, the eScience group at Microsoft Research is making enormous strides in the race to develop an HIV vaccine. "With high-performance computing, we've been able to accelerate our time to insight," says Heckerman. "Several of the external groups with which we're collaborating now use our statistical techniques and share their findings. And as a result, scientists are already coming up with new hypotheses for us to test. Before, getting results would have taken a year for every step of the process. Now it takes just a single day."
Before Windows Compute Cluster Server, the methods used by Microsoft Research would have been impractical due to the time required to perform analysis. "With Windows Compute Cluster Server," says Kadie, "we can run 50 jobs of 200,000 work items each in the same amount of time that it used to take to run 1 job."
Now that the eScience group has massive computation capabilities at hand, it can conduct tests with many runs on simulated data. The runs on simulated data are critical to determining which results on real data are interesting. The more simulations performed, the more reliable the results.

Lessons Learned
Microsoft Research has obviously benefited from the use of Windows Compute Cluster Server, but the relationship is symbiotic in nature. Developers in the High-Performance Computing group, which is working on a second version of Windows Compute Cluster Server, continue to gain insight from the feedback the eScience group provides.
In particular, the HPC group has learned a great deal about resource allocation by monitoring and analyzing the behavior of the Microsoft Research cluster to determine the best way to balance the cluster's resources among multiple users. For instance, on this project, a user typically creates one job at a time with an average of 50 tasks associated with each job. The user submits the job and the Windows Compute Cluster Job Manager allocates enough resources to manage all 50 tasks. To do this, it immediately claims all available nodes on the cluster to work on the tasks. This scenario would not be problematic in a single-user environment, but it becomes an issue in a multi-user setting where it is very important that projects progress simultaneously.
Currently, when 10 of the job's 50 tasks are complete, 10 servers are not freed up. The Job Manager waits until all 50 tasks within the job have been completed before freeing up the servers to tackle another job. HPC developers are now working on a way to reallocate resources upon completion of each task—not the overall job.
Microsoft Research also looked to the HPC group to help resolve specific issues that pertained to the way that the researchers wanted to use Windows Compute Cluster Server. Notably, the researchers wanted to set up security so that user credentials would automatically propagate from the Web front end to the compute cluster. The solution was to work within the Microsoft® .NET Framework and extend the ASP.NET forms authentication so that the Web application would be able to supply the cluster nodes with full user credentials every time a user submits a job.
Now the eScience team members at Microsoft Research and their colleagues from around the world can focus on the most important parts of their jobs and leave all the tasks involved in processing and managing data to the computers. "As a result, we're able to go further as a team," says Heckerman. "We're pushing out the boundaries of what is known about HIV, making good progress in the fight against the disease."

Kristin Firth and Mia Matusow, both of Blue Line Writing & Editing, have spent the past decade creating strategic content for enterprise organizations in the public and private sectors on three continents.


 Windows PowerShell: Writing Regular Expressions
Windows PowerShell
Writing Regular Expressions
Don Jones

192.168.4.5. \\Server57\Share. johnd@contoso.com. You no doubt recognize these three items as an IP address, a Universal Naming Convention (UNC) path, and an e-mail address. Your brain recognizes their formats. Four groupings of digits, backslashes, the @ symbol, and other cues indicate what types of data these strings of
characters represent. With little thought, you can quickly recognize that 192.168 on its own isn't a valid IP address, that 7\\Server2\\Share isn't a valid UNC, and that joe@contoso isn't a valid e-mail address.
Computers, unfortunately, have to work a bit harder in order to "understand" complicated formats like these. That's where regular expressions come into play. A regular expression is a string, written using a special regular expression language, that helps a computer identify strings that are of a particular format—such as an IP address, a UNC, or an e-mail address. A well-written regular expression has the ability to allow a Windows PowerShellTM script to accept as valid or reject as invalid data that does not conform to the format you've specified.

Making a Simple Match
The Windows PowerShell –match operator compares a string to a regular expression, or regex, and then returns either True or False depending on whether the string matches the regex. A very simple regex doesn't even need to contain any special syntax—literal characters will suffice. For example:
"Microsoft" –match "soft"
"Software" –match "soft"
"Computers" –match "soft"
When run in Windows PowerShell, the first two expressions return True while the third returns False. In each, a string is followed by the –match operator, which is followed by a regex. By default, a regex will float across a string to find a match. The characters "soft" can be found within both Software and Microsoft, but at different positions. Also notice that, by default, a regex is case-insensitive—"soft" is found in "Software" despite its capital S.
But if necessary, a different operator, –cmatch, offers case-sensitive regex comparison, like so:
"Software" –cmatch "soft"
This expression returns False since the string "soft" doesn't match "Software" in a case-sensitive comparison. Note that the –imatch operator is also available as an explicit case-insensitive option, despite that being the default behavior of –match.

Wildcards and Repeaters
A regex can contain a few wildcard characters. A period, for example, matches one instance of any character. A question mark matches zero or one instance of any character. Here are some examples to illustrate:
"Don" –match "D.n" (True)
"Dn" –match "D.n" (False)
"Don" –match "D?n" (True)
"Dn" –match "D?n" (True)
In the first expression, the period stands in for exactly one character, so the match is True. In the second expression, the period doesn't find the one character that it requires to be included, and so the match is False. The question mark, as shown in the third and fourth expressions, can match a single unknown character or no character at all. Finally, in the fourth example, the match is True because both "D" and "n" are found without a character between them. Thus, the question mark can be thought of as standing for an optional character, so the match is still True even if no character appears in that position.
A regex also recognizes the * and + symbols as repeaters. These need to follow some character or characters. The * matches zero or more of the specified characters, while the + matches one or more of the specified characters. Here are some examples:
"DoDon" –match "Do*n" (True)
"Dn" -match "Do*n" (True)
"DoDon" -match "Do+n" (True)
"Dn" -match "Do+n" (False)
Notice that both * and + are matching "Do", not just the "o". That's because these repeaters are designed to match a series of characters, not just one character.
What if you need to match the period, *, ?, or + symbols themselves? You simply precede them with a backslash, which is the regex escape character:
"D.n" -match "D\.n" (True)
Notice that this is different from the Windows PowerShell escape character (the backward apostrophe), but it follows industry-standard regex syntax.

Character Classes
A character class is a broader form of wildcard, representing an entire group of characters. Windows PowerShell recognizes quite a few character classes. For instance:
  • \w matches any word character, meaning letters and numbers.
  • \s matches any white space character, such as tabs, spaces, and so forth.
  • \d matches any digit character.
There are also negative character classes: \W matches any non-word character, \S matches non-white space characters, and \D matches non-digits. These classes can be followed by * or + to indicate that multiple matches are acceptable. Here are some examples:
"Shell" -match "\w" (True)
"Shell" -match "\w*" (True)
Cmdlet of the Month
The Write-Debug cmdlet is very handy for writing objects (such as text strings) to the Debug pipeline. Trying this cmdlet in the shell can be somewhat disappointing, though, because it doesn't look like the cmdlet is doing anything.
The trick is that the Debug pipeline is shut off by default—the $DebugPreference variable is set to "SilentlyContinue." Set it to "Continue," however, and everything you send with Write-Debug will appear at the console in yellow text. This is a perfect way to add trace code to your scripts, allowing you to follow the execution of a complex script. The yellow color helps you distinguish between trace about and the script's normal output, and you can shut off the debug messages at any time without having to remove all the Write-Debug statements. Simply set $DebugPreference = "SilentlyContinue" again and the debug text will be suppressed.

Though both expressions return True, they're matching significantly different things. Fortunately, there's a way to see what the –match operator is thinking under the hood: each time a match is made, a special variable called $matches is populated with the results of the match—that is, whatever characters in the string the operator matched against your regex. The $matches variable retains its results until another positive match is made using the –match operator. Figure 1 shows the difference between the two expressions I just showed you. As you can see, \w matched the "S" in "Shell", while the repeating \w* matched the entire word.
Figure 1 What a difference a * can make
Figure 1 What a difference a * can make

Character Groups, Ranges, and Sizes
A regex can also contain groups or ranges of characters, enclosed in square brackets. For example, [aeiou] means that any one of the included characters—a, e, i, o, or u—is an acceptable match. [a-zA-Z] indicates that any letter in the range a-z or A-Z is acceptable (although if you're using the non-case-sensitive –match operator, just a-z or A-Z on its own would be sufficient). Here's an example:
"Jeff" -match "J[aeiou]ff" (True)
"Jeeeeeeeeeeff" -match "J[aeiou]ff" (False)
You can also specify a minimum and maximum number of characters using curly braces. {3} indicates that you want exactly three of the specified character, {3,} means that you want at least three or more, and {3,4} indicates that you want at least three but no more than four. This is an ideal way to create a regex for IP addresses:
"192.168.15.20" -match "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" (True)
This regex wants four groups of digits with one to three digits each, all of which are separated by a literal period. But consider this example:
"300.168.15.20" -match "\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}" (True)
This shows the limitations of a regex. While the formatting of this string looks like an IP address, it obviously isn't a valid IP address. A regex can't determine whether data is actually valid; it can only determine whether the data looks right.

Stop the Float
Troubleshooting a regex can be tricky. For example, here's a regex that tests for a UNC path in the format \\Server2\Share:
"\\Server2\Share" -match "\\\\\w+\\\w+" (True)
Here, the regex itself is difficult to read because every literal backslash I want to test for has to be escaped with a second backslash. Though this seems to work fine, it really doesn't:
"57\\Server2\Share" -match "\\\\\w+\\\w+" (True)
This second example is clearly (to you and me, at least) not a valid UNC path, but the regex gave it the all clear. Why? Remember that a regex will float by default. This regex merely looks for two backslashes, one or more letters and numbers, another backslash, and more letters and numbers. That pattern exists in the string—along with the extra digits at the start, which make it an invalid UNC. The trick is to tell the regex to start matching at the beginning of the string, without floating. I can do that like this:
"57\\Server2\Share" -match "^\\\\\w+\\\w+" (False)
The ^ character indicates that this is the location where the string begins. With that addition, the invalid UNC path fails because the regex is looking for the first two characters to be backslashes, and in this case they aren't.
Similarly, the $ symbol can be used to indicate the end of a string. This wouldn't be very useful in the case of a UNC path since a UNC path can contain additional path segments, such as \\Server2\Share\Folder\File, for example. However, I'm sure there are many cases where you would want to specify the end of a string.

Help with Regular Expressions
In Windows PowerShell, the about_regular_expressions help topic provides basic syntax assistance for the regex language, but online resources can provide even more information. For instance, one of my favorite Web sites, www.RegExLib.com, offers a free library of regular expressions that have been written for various purposes and contributed to by the public. You can search on keywords, such as "e-mail" or "UNC," to quickly locate a regex that suits your need—or at least provides a good starting point. If you manage to create a great regex, you can contribute it to the library so others can make use of it, too.
I also like RegexBuddy (www.RegexBuddy.com). This is an inexpensive tool that provides a graphical regex editor. RegexBuddy makes it easier to assemble a complex regex, and this tool also makes it easier to test a regex to ensure that it is properly accepting valid strings and rejecting invalid ones. A number of other software developers have also created free, shareware, and commercial regex editors and testers that users will surely find to be useful.

Using Regular Expressions
You may be wondering why you would use a regex in real life. Imagine you're reading information from a CSV file and using the information to create new users in Active Directory®. If the CSV file is generated by someone else, you'll want to validate that the data in it looks right. A regex is perfect for this task. A simple regex like \w+, for example, can confirm that first and last names don't contain any special characters or symbols, while something more complicated can confirm that the e-mail addresses conform to your corporate standard. For example, you could use this:
"^[a-z]+\.[a-z]+@contoso.com$"
This regex requires an e-mail address in the form don.jones@contoso.com, where the first and last names can only contain letters, and where they must be separated by a period. E-mail addresses, by the way, are the trickiest strings to write a regex for. If you can narrow your scope down to a specific corporate standard, you'll have an easier time of it.
Don't forget those start and end anchors (^ and $), which ensure that nothing follows contoso.com and also that nothing precedes the characters that make up the user's first name.
Actually using this regex in Windows PowerShell is pretty easy. Assuming the variable $email contains the e-mail address you read from the CSV file, something like this will check to see whether it's valid or not:
$regex = "^[a-z]+\.[a-z]+@contoso.com$"
If ($email –notmatch $regex) {
  Write-Error "Invalid e-mail address $email"
}
And in this example you've learned a new operator. -notmatch returns True if the string doesn't match the provided regex. (There is also a –cnotmatch for case-sensitive comparisons.)
There's a lot more about regular expressions I haven't covered here—additional character classes, more advanced operations, and even an operator or two. And then there's the [regex] object type that Windows PowerShell supports. However, what I have covered in this quick overview of regex syntax should be enough to get you started. Feel free to visit me anytime at www.ScriptingAnswers.com if you need help puzzling through an especially tricky regex.

Don Jones is a contributing editor for TechNet Magazine and is the coauthor of Windows PowerShell: TFM (SAPIEN Press). He teaches Windows PowerShell (www.ScriptingTraining.com) and can be reached through the ScriptingAnswers.com Web site.


 Hey, Scripting Guy!: Desktop Management from beyond the Grave
Hey, Scripting Guy!
Desktop Management from beyond the Grave
The Microsoft Scripting Guys
Download the code for this article: HeyScriptingGuy2007_11.exe (151KB)

Due to overwhelming popular demand, we thought we'd do something a little different this month: rather than start out by talking about system administration scripting, we're going to start out by—cue the ominous music—telling a ghost story instead!
Note: OK, so technically, if we really wanted to do something a little different this month we'd start out by actually talking about system administration scripting for a change. But just play along with us, OK? Thanks!
Many years ago, one of the Scripting Great-Great-Great Grandmothers passed away. Shortly after Grandmother was laid to rest in her simple wooden coffin, Grandfather began having terrible nightmares, nightmares in which his beloved wife was desperately trying to claw her way out of the grave. After repeated nightmares, and after repeated entreaties, Grandfather finally convinced the local authorities to exhume the body. When the coffin was opened, everyone was horrified to see that Grandmother's nails had been bent back and the inside of the coffin was covered with scratches!
OK, so this story might not be entirely true; in fact, the more we think about it, the more we realize that it isn't the least bit true. Nevertheless, the story has an important lesson to teach us. We have no idea what it is, but it's in there somewhere.
Wait a second, now we remember! Coffins were originally designed to protect the deceased from the elements and to help prevent the body from decomposing. Unfortunately, coffins had an unintended consequence: they make it possible, in theory anyway, for you to bury a person alive and ensure that he can never get out. As the story of the Scripting Great-Great-Great Grandmother clearly shows, even the best-laid plans of mice and men can result in disaster, and in people being buried alive! (Cue the ominous music one more time.)
Note: Unless, of course, you opt for the Improved Burial Case, invented by Franz Vester in the 1860s. This coffin included a string connected to a bell that remained aboveground; in the case of premature burial, the "deceased" could simply ring the bell and summon help. The Improved Burial Case also included a folding ladder, although it's not entirely clear to us how a folding ladder will help you escape from a coffin buried six feet underground. If you happened to be buried on top of someone's garage, sure, a folding ladder would be useful. Otherwise ....
This very same thing (the thing about best-laid plans leading to disaster) is true of Internet firewalls. (Well, sort of.) Firewalls were originally designed to keep the bad guys out: they block incoming network traffic, which helps to keep hackers and intruders away from your computers. That's great, but—like the problem of being buried alive—there's also an unintended consequence here: firewalls can also keep the good guys out. This is especially true in the case of Windows® Management Instrumentation (WMI), which relies on DCOM to perform administrative tasks on remote computers. Firewalls tend to block all incoming DCOM traffic, something that makes it very difficult (if not downright impossible) to programmatically manage computers over the Internet. In fact, without opening up additional ports on the firewall and thus making you more vulnerable to hackers and crackers, this is downright impossible. Unless, of course, you opt for WinRM: Windows Remote Management (folding ladder not included).

What Is Windows Remote Management?
According to the WinRM SDK (msdn2.microsoft.com/aa384426), Windows Remote Management is "the Microsoft implementation of WS-Management Protocol, a standard SOAP-based, firewall-friendly protocol that allows hardware and operating systems from different vendors to interoperate." Impressive, huh? We're not going to discuss the details of the WS-Management Protocol in this month's column, so we recommend reading the WinRM SDK for the details. For now, all we care about is that WinRM is available on Windows Server® 2003 R2, Windows Vista®, and Windows Server 2008, and that it enables you to manage computers over the Internet. WinRM does this using port 80, a standard Internet services port that most firewalls leave open. (However, the port used by WinRM and the default transport mechanism, HTTP, can be changed as needed.)
We also won't spend any time in this month's column discussing how to install and configure WinRM. There's already plenty of information available to help you with that (msdn2.microsoft.com/aa384372). However, we will take a moment to emphasize one important point: if you want to use WinRM to retrieve information from a remote computer (which, of course, is the primary reason for using WinRM in the first place), then both your local machine and the remote computer must be running WinRM.
What does that mean? Well, it means that if you haven't upgraded your client computers to Windows Vista (say it isn't so!), or you haven't upgraded your servers to Windows Server 2003 R2 or Windows Server 2008, you won't find WinRM to be particularly useful, at least not today. Needless to say, however, that probably won't be the case tomorrow. (And, of course, assuming your firewall allows it, you can always use WMI and DCOM to manage remote computers.)

Returning All the Properties and Instances of a Class
But who cares about caveats and disclaimers, right? Instead of all that mumbo jumbo, let's see if we can figure out how to write a script that takes advantage of WinRM. Coincidentally enough, we just happened to have a simple little script that, using the HTTP protocol and port 80, connects to a computer named atl-fs-01.fabrikam.com and then returns complete information about all the services installed on that computer. See Figure 1 for the script in all its glory.
Figure 1 Silverlight project components (Close Figure 1)

File | Description
CreateSilverlight.js | This is a JScript script (in its initial release, Silverlight only supports JScript scripting) used to specify the initial Silverlight startup settings, including the XAML file used to configure user interface and graphic objects.
SampleProject.js | This is just a blank file where you can put JScript functions.
Silverlight.js | This file is used to initialize the Silverlight control.
SampleProject.html | This is where all the fun takes place. SampleProject.html is simply an HTML file that includes code for reading in the three .js files. It also includes code for instantiating the Silverlight control.
SampleProject.xaml | What's this for? To find out, you'll have to go back to the main text of this column.
Figure 1 Listing services on a remote machine (Close Figure 1)
strComputer = "atl-fs-01.fabrikam.com"

Set objWRM = CreateObject("WSMan.Automation")
Set objSession = objWRM.CreateSession("http://" & strComputer)

strResource = "http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Service"

Set objResponse = objSession.Enumerate(strResource)

Do Until objResponse.AtEndOfStream
    DisplayOutput(objResponse.ReadItem)
Loop

Sub DisplayOutput(strWinRMXml)
    Set xmlFile = CreateObject("MSXml2.DOMDocument.3.0")
    Set xslFile = CreateObject("MSXml2.DOMDocument.3.0")
    xmlFile.LoadXml(strWinRMXml)
    xslFile.Load("WsmTxt.xsl")
    Wscript.Echo xmlFile.TransformNode(xslFile)
End Sub
As you can see, we start out by assigning the DNS name of the computer (atl-fs-01.fabrikam.com) to a variable named strComputer. Alternatively, we could make the connection using the computer's IP address (or even its IPv6 address). For example:
strComputer = "192.168.1.1"
After assigning a value to strComputer we next create an instance of the WSMan.Automation object, then we call the CreateSession method to connect to the remote machine, in this case using the HTTP protocol (just like we said we were going to do):
Set objSession = objWRM.CreateSession _
    ("http://" & strComputer)
As we noted, we want to return information about the services installed on the remote computer. In addition, and at least for this first example, we want information about all the properties of all the services. What does all that mean? That means that we need to specify a URI Resource that binds us to the Win32_Service class on the remote computer:
strResource = _
  "http://schemas.microsoft.com" & _
  "/wbem/wsman/1/wmi/root/cimv2" & _
  "/Win32_Service"
Granted, that's not the prettiest URI we've ever seen. (Although, come to think of it, we're not sure we've ever seen a pretty URI.) Fortunately, though, most of the URI is boilerplate; all you need to worry about is the WMI path at the very end:
http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Service
That should be pretty straightforward. What if you wanted to connect to the root/cimv2/Win32_Process class? Well, then you just modify the URI path accordingly:
http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Process
Interested in the root/default/SystemRestore class? Well, once again, just modify the URI class, taking care to specify the default namespace (rather than the cimv2 namespace):
http://schemas.microsoft.com/wbem/wsman/1/wmi/root/default/SystemRestore
And so on.... It's a bit of a shame that you need to include the http://schemas.microsoft.com/wbem/wsman/1/wmi portion of the URI as well, but....
At this point we're ready to get back some data. To do that, we simply call the Enumerate method, passing the variable strResource as the sole method parameter:
Set objResponse = _
  objSession.Enumerate(strResource)
Will that line of code truly populate objResponse with information about the services installed on the computer atl-fs-01? You bet it will. However, unlike standard WMI scripts, you don't get back a series of objects, each with its own properties and property methods. Instead, you'll get back a big old XML blob that looks a little like what you see in Figure 2.
Figure 2 Pretty colors (Close Figure 2)
<Canvas
 xmlns="http://schemas.microsoft.com/client/2007"
 xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
 Width="800"
 Height="300">

 <Canvas.Background>
 <LinearGradientBrush>
 <GradientStop Color="Blue" Offset="0.0" />
 <GradientStop Color="Black" Offset="1.0" />
 </LinearGradientBrush>
 </Canvas.Background>

</Canvas>
Figure 2 Big old XML blob (Close Figure 2)
<p:Win32_Service xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:p="
http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/Win32_Service" xmlns:ci
m="http://schemas.dmtf.org/wbem/wscim/1/common" xsi:type="p:Win32_Service_Type"
xml:lang="en-US"><p:AcceptPause>false</p:AcceptPause><p:AcceptStop>false</p:Acce
ptStop><p:Caption>Windows Media Center Service Launcher</p:Caption><p:CheckPoint
>0</p:CheckPoint><p:CreationClassName>Win32_Service</p:CreationClassName><p:Desc
ription>Starts Windows Media Center Scheduler and Windows Media Center Receiver
services at startup if TV is enabled within Windows Media Center.</p:Description
><p:DesktopInteract>false</p:DesktopInteract><p:DisplayName>Windows Media Center
If you're an XML whiz, that's no big deal; anyone familiar with XML should be able to parse and output this information without too much trouble (even though, in the words of the WinRM SDK, this information is not in "human-readable format"). But what if you're not an XML whiz? In that case, you have two choices. One, you can wait until next month, when we'll show you a few tricks for working with WinRM's XML. Or two, you can do what we did in our sample script: employ the XSL transform that gets installed alongside WinRM.

The XSL What-form?
An XSL transform is nothing more than a template that describes how an XML file should be displayed. A complete discussion of XSL files goes way beyond what we can do in this month's column—for that matter, even a cursory discussion of XSL files goes way beyond what we have the capacity to do in this month's column. Therefore, we won't try to explain how WsmTxt.xsl (the name of the built-in transform) actually works. Instead, we'll simply show you how you can use that transform in your script.
When you call the Enumerate method, WinRM sends back a stream of XML data. The easiest way to work with this data is to set up a Do Until loop that continues to run until you reach the end of the data stream. That's what we do here:
Do Until objResponse.AtEndOfStream
    DisplayOutput(objResponse.ReadItem)
Loop
As you can see, inside our loop we call a subroutine named DisplayOutput. When we call that subroutine, we pass along the value of the stream's ReadItem method as the subroutine parameter. (As this whole approach implies, the XML stream is sent back in separate pieces rather than as one large blob of data. Our script, in turn, reads the XML data one piece, or one item, at a time.)
Meanwhile, the DisplayOutput subroutine looks like this:
Sub DisplayOutput(strWinRMXml)
  Set xmlFile = _
    CreateObject("MSXml2.DOMDocument.3.0")
  Set xslFile = _
    CreateObject("MSXml2.DOMDocument.3.0")
  xmlFile.LoadXml(strWinRMXml)
  xslFile.Load("WsmTxt.xsl")
  Wscript.Echo xmlFile.TransformNode(xslFile)
End Sub
In brief, we start out by creating two instances of the MSXml2.DOMDocument.3.0 object. We load the XML data stream (strWinRMXML) into one object, then we load the XSL file (WsmTxt.xsl) into the other object. At that point we call the TransformNode method to use the information in the XSL file to format and display the data grabbed from the XML stream.
Yes, it's a little confusing. But at least the output (while far from perfect) is a bit easier to read (see Figure 3).
Figure 3 Painting text (Close Figure 3)
<TextBlock
 Name="Test"
 FontSize="40"
 FontFamily="Georgia"
 FontWeight="Bold"
 Canvas.Top="20"
 Canvas.Left="20"
 Text="The TechNet Script Center">

 <TextBlock.Foreground>
 <SolidColorBrush Name="test_brush" Color="red"/>
 </TextBlock.Foreground>

</TextBlock>
Figure 3 A tidier version of the XML (Close Figure 3)
Win32_Service
    AcceptPause = false
    AcceptStop = true
    Caption = User Profile Service
    CheckPoint = 0
    CreationClassName = Win32_Service
    Description = This service is responsible for loading and unloading user profiles. If this service is stopped or disabled, users will no longer be able to successfully logon or logoff, applications may have problems getting to users' data, and components registered to receive profile event notifications will not receive them.
Like we said, this is good, but it's not necessarily great, which is all the more reason to tune in next month, when we'll show you a few ways to manipulate the XML output by yourself.

Returning Selected Instances and Properties of a Class
Needless to say, this is all very cool, except for one thing: it might not be fully reflective of the way you typically do your job. Yes, there will be times when you want to return all the properties of all the instances of a class; however, there will also be times (perhaps many more times) when you want to return only selected properties or instances of a class. For example, you might want to return only information about services that are running, something you do in a regular WMI script by using code similar to this:
Set colItems = objWMIService.ExecQuery _
  ("Select * From Win32_Service " & _
   "Where State = 'Running'")
That's nice. But how in the world are you going to modify your Resource string to make it equivalent to that?
Well, to be perfectly honest, you aren't going to modify your Resource string to make it equivalent to the ExecQuery statement. You will definitely need to modify the Resource string, but you'll need to do a few other things as well.
With that in mind, let's take a peek at Figure 4. This is a WinRM script that returns information about the services that are running on a computer (as opposed to all the services installed on that computer).
Figure 4 TextBlock tricks (Close Figure 4)
<TextBlock.Triggers>
 <EventTrigger RoutedEvent=
     "TextBlock.Loaded">
 <BeginStoryboard>
 <Storyboard>
 <DoubleAnimation
 Storyboard.TargetName="Test"
 Storyboard.TargetProperty="Opacity"
 From="0.0" To="1.0"
 Duration="0:0:5" />
 </Storyboard>
 </BeginStoryboard>
 </EventTrigger>
</TextBlock.Triggers>
Figure 4 Finding running services (Close Figure 4)
strComputer = "atl-fs-01.fabrikam.com"

Set objWRM = CreateObject("WSMan.Automation")
Set objSession = objWRM.CreateSession("http://" & strComputer)

strResource = "http://schemas.microsoft.com/wbem/wsman/1/wmi/root/cimv2/*"
strFilter = "Select * From Win32_Service Where State = 'Running'"
strDialect = "http://schemas.microsoft.com/wbem/wsman/1/WQL"

Set objResponse = objSession.Enumerate(strResource, strFilter, strDialect)

Do Until objResponse.AtEndOfStream
    DisplayOutput(objResponse.ReadItem)
Loop

Sub DisplayOutput(strWinRMXml)
    Set xmlFile = CreateObject("MSXml2.DOMDocument.3.0")
    Set xslFile = CreateObject("MSXml2.DOMDocument.3.0")
    xmlFile.LoadXml(strWinRMXml)
    xslFile.Load("WsmTxt.xsl")
    Wscript.Echo xmlFile.TransformNode(xslFile)
End Sub
At first glance this might appear all but identical to the first WinRM script we showed you; however, there are some very important differences. For one thing, take a look at the value we assign to the Resource string:
strResource = _
  "http://schemas.microsoft.com" & _
  "/wbem/wsman/1/wmi/root/cimv2/*"
Notice that, when writing a filtered query, we don't specify the actual name of the class (Win32_Service) that we want to work with; instead, we simply connect to the namespace (root/cimv2) where that class resides. Just remember to tack the asterisk (*) onto the end. If you don't, the script will fail with the resulting message "... the class name must be '*' (star)," which simply means that you need to make the class name *.
In addition, we also need to define a Filter and a Dialect:
strFilter = _
  "Select * From Win32_Service " & _
  "Where State = 'Running'"
strDialect = _
  "http://schemas.microsoft.com" & _
  "/wbem/wsman/1/WQL"
The Filter should be easy enough to figure out; that's where we put our Windows Management Instrumentation Query Language (WQL) query ("Select * From Win32_Service Where State = 'Running'"). The Dialect, meanwhile, is the query language used when creating the Filter. At the moment there's only one query language allowed: WQL. Nevertheless, the Dialect must be specified or the script will fail with the note that the "filter dialect ... is not supported."
Note: interestingly enough, the error message suggests that you remove the Dialect when you call the Enumerate method. That's a recommendation that you shouldn't follow. When doing a filtered query, the Dialect must be specified and must be WQL. Period.
The only other change we need to make occurs when we call the Enumerate method. At that point we need to pass along the variables representing the Filter (strFilter) and the Dialect (strDialect) as well as the variable representing the Resource (strResource):
Set objResponse = _
  objSession.Enumerate _
  (strResource, strFilter, strDialect)
Give that a try and see what happens.
Now, what about returning only selected properties of a class? For example, suppose you are interested in returning just the Name and DisplayName of all the services running on a computer. What then?
Well, in a case like that you can try to manipulate the XML so that only the Name and DisplayName are displayed. That's possible, but definitely a bit tricky. An easier way to do this is to specify just those properties when assigning the Filter:
strFilter = _
  "Select Name, DisplayName " & _
  "From Win32_Service " & _
  "Where State = 'Running'"
Do that and all you'll get back are the Name and DisplayName of each service, like so:
XmlFragment
    DisplayName = Windows Event Log
    Name = EventLog

XmlFragment
    DisplayName = COM+ Event System
    Name = EventSystem
Granted, the formatting is a bit goofy. (What is the deal with that XmlFragment stuff?) All the more reason to tune in next month.

Wait for It
With any luck, that should be enough to get you started with the wild and wonderful world of WinRM. Of course, we couldn't end a column on WinRM without mentioning the "waiting mortuaries" that were once prevalent throughout Germany. In cities with waiting mortuaries, corpses weren't immediately buried. Instead, they were placed in warm rooms with a number of strings and wires attached to their fingers and toes. The idea, of course, was that the slightest movement would trigger an alarm and summon help. The bodies were kept in these waiting mortuaries until it became evident that these people were truly beyond hope and would never do anything ever again.
Now that you mention it, a waiting mortuary is very similar to being assigned to the Scripting Guys team, isn't it? Of course, no one ever came back to life in any of the waiting mortuaries, whereas the people assigned to the Scripting Guys team ...

Dr. Scripto's Scripting Perplexer
In June 2007, the Scripting Guys attended the Tech•Ed 2007 conference in Orlando, Florida. Not content to just attend the conference, we decided to have some fun. Not only that, but we thought everyone else could use some fun, too. With that in mind we came up with Dr. Scripto's Fun Book, a booklet filled with scripting-related puzzles and various other bits of information. We teamed up with TechNet Magazine—meaning we talked them into giving up a tiny corner of their booth in the Expo Hall—and handed out Fun Books to anyone who wandered by.
As it turned out, the Fun Book was a pretty popular item (maybe not as popular as the Dr. Scripto bobblehead dolls, but close). The fine and opportunistic people at TechNet Magazine, seeing one more way they could capitalize on the Scripting Guys's success (since the Scripting Guys never seem to be able to capitalize on their own success), asked us to create some puzzles for them. When Scripting Guy Jean Ross turned her back for just a moment, Scripting Guy Greg Stemp said, "Sure, we'll do that!" And here we are: Dr. Scripto's Scripting Perplexer. Enjoy.

Drop-In Scripting
In this puzzle, all the letters in the top section unscramble to create a script (in VBScript). But don't worry, you don't have to unscramble the whole thing; instead, you simply need to unscramble one column at a time. The letters in each column of the top section fill in the blank spaces in the same column of the bottom section. Here's an example:
As you can see, in column 1 we have the letters S, C, and T. Those three letters belong in the grid below in some unknown order. But when all the letters are dropped down in the proper order, the bottom grid—read left-to-right—becomes something logical. Take a look at the solution:
You can see that the letters S, C, and T in column 1 move down in the order T, S, and C. These turned out to be the first letter in each word of "The Script Center." The actual puzzle is a little more difficult because, well, it's longer and because the final result is a full script.
Hint: the final script starts out with a full path to a file then parses out and displays just the file name.
Good luck!

Dr. Scripto's Scripting Perplexer
Answer: Drop-In Scripting, November 2007
In this puzzle, you needed to drop characters within a column into the correct boxes below so that the bottom rows created a script. Here's the standalone script:
name = "C:\Scripts\Test.txt"
arr = Split(name, "\")
index = Ubound(arr)
Wscript.Echo "Filename: " _
& arr(index)
Here's what it looks like in the puzzle grid:

The Microsoft Scripting Guys work for—well, are employed by—Microsoft. When not playing/coaching/watching baseball (and various other activities) they run the TechNet Script Center. Check it out at www.scriptingguys.com.


 Security Watch: Secure Access Anywhere
Security Watch
Secure Access Anywhere
John Morello

One of the undeniable trends in technology today is the increasingly connected and mobile workforce. Many organizations have staff that operate from far-flung offices, telecommute, or frequently travel to visit customers. Empowering these users to easily access applications and data, regardless of location, makes them more
productive. However, until recently, enabling secure remote access often meant installing client software, typing in arcane commands, and long connection times.
Over the past several years, a number of approaches have come along to make remote access a simpler and more accessible technology. For example, Outlook® Web Access (OWA) provides a simple, browser-based way for users to access their e-mail, calendar, and contacts without the complexities of a full Layer 3 Virtual Private Network (VPN). While technologies like OWA can supply an important piece of an "anywhere access" solution, most organizations have many key applications that don't permit the same kind of integrated browser experience. In these cases, solutions like Terminal Services provide an effective method for giving users access to their applications from wherever they may be.
In Windows Server® 2008, Microsoft greatly enhanced the "in the box" feature set of Terminal Services. Terminal Services now includes a seamless window, per-application publishing feature called RemoteApp, a universal print driver capability in EasyPrint, and a browser-based portal called TS Web Access. Additionally—and key to the anywhere-access scenario—Windows Server 2008 also includes the TS Gateway component that provides SSL encapsulation for the Remote Desktop Protocol (RDP), allowing it to easily and securely traverse firewalls and Network Address Translation (NAT) devices. The TS Gateway feature also integrates with another new technology in Windows Server 2008, Network Access Protection (NAP), to provide endpoint client health inspection. When all of these components are combined, organizations have the ability to build solutions that let users easily and securely access their applications and data from anywhere.
This column focuses on the network and security design aspects of an anywhere-access solution, rather than providing details on managing the Terminal Service components. I'll describe the methods and best practices for creating such a solution based on technology included with Windows Server 2008.

What's the Problem with Layer 3 Virtual Private Networks?
When thinking about any new technology, it's important to understand its advantages over current approaches so you can accurately assess the value of the new model. In the case of classical Layer 3 VPN, there are typically two major problems that need to be addressed: security and ease of use.
It may seem counterintuitive to list security as a problem with many of today's Layer 3-based VPNs. Isn't the whole point of a VPN to provide a secure tunnel through the Internet? To understand, let's take a higher-level view of what you might typically consider to be VPN threats. I'm not saying that data passing over these VPNs is at risk of interception or tampering; most modern Layer 3 VPNs provide excellent encryption of the data stream. Instead, think of the threats posed by remote machines that are given full "all ports, all protocols" access to your organization's network. The problem is not the data streams the Layer 3 VPNs encrypts and carries over the network, but rather the remote clients connecting through these tunnels that pose increased risk to your internal network. Recall that most organizations impacted by malware like Slammer or Blaster were not infected by machines on their internal network or hackers traversing their firewalls. Instead, the infestation came via remote workers connecting to their networks over VPNs from infected machines. When these VPNs create fully routed, Layer 3-based connections, the remote machine often has as much access (both good and bad) to internal resources as a machine sitting on the datacenter floor.
Moreover, Layer 3 VPNs are often expensive to operate because they require software installation and configuration on machines that are not managed by an organization's IT group. For example, installing a VPN client on a user's home machine may involve creating custom installation packages, detailed installation directions for the user to follow, and troubleshooting conflicts with apps on the user's machine. Moreover, management costs can be high if new versions of the client need to be deployed or configuration data changes (such as adding a new VPN endpoint). For the user, working via this VPN is often a non-intuitive experience. Since it only provides the Layer 3 connection, the user's business apps and data are not made easily accessible and visible to him.

Solving the Problems with Terminal Services
Terminal Services, and other so-called Layer 7 or application layer VPN technologies, solve these two issues by providing much finer control over which resources and protocols users are able to access and by making the end-user experience much simpler and more intuitive than before.
From a security standpoint, the most important differentiating feature between a Layer 3 VPN approach and one that utilizes Terminal Services and TS Gateway is that the connection to the internal network is not a completely open pipeline. Specifically, whereas a Layer 3 approach will create a virtual interface on the local machine with full routing capability into the internal network, the TS Gateway approach only allows RDP-based packets to reach the internal network. Thus, the overall attack surface is greatly reduced and more granular controls can be applied within RDP. For example, while RDP provides native support for drive redirection, organizations could configure their TS Gateways to integrate with NAP and to not allow this capability unless the remote machine could prove that its antivirus software was up-to-date.
For end users, the most obvious difference between a Layer 3 VPN and a Terminal Services-based approach is a vastly simpler setup (often, there is no setup at all) and much easier and more intuitive access to their applications and data. Because the Remote Desktop Connection client is built into Windows (and kept up-to-date through normal servicing technologies like Windows® Update), there is typically no client software to install. Also, by leveraging TS Web Access, users can visit a single URL to find all their applications and data. Simply clicking on the appropriate application starts TS Gateway securely tunneling the connection over the Internet, and the application is seamlessly delivered to the user's desktop. In other words, the application will look and behave as if it is installed locally, including having the ability to copy and paste and minimize to the taskbar. By making applications and data more discoverable and setup instantaneous, a Terminal Server-based approach to remote access successfully increases user satisfaction and reduces support costs.

Network Architecture Choices
There are two basic network design approaches that can be used to make a TS Gateway server available over the Internet. The first sandwiches the TS Gateway into the perimeter network between two Layer 3/4 firewalls. The second keeps the TS Gateway on the internal network and uses an application-layer firewall (such as Microsoft® ISA Server, Microsoft Intelligent Application Gateway, or a variety of third-party solutions) to sit on the perimeter network and inspect inbound HTTPS frames. Only after these inbound sessions have been analyzed are the packets forwarded on to the internal TS Gateway server.
If an organization has only Layer 3/4 firewalls with basic stateful packet inspection, the TS Gateway device can be placed directly in the perimeter network as shown in Figure 1. In this design, the Internet-facing firewall restricts connectivity to the TS Gateway to only allow HTTPS traffic from the Internet to reach it. However, the firewall does no inspection at the application layer of this traffic; it simply forwards the traffic on to the TS Gateway. The TS Gateway server then extracts the RDP frames from the HTTPS packets and forwards them to the appropriate back-end server. These back-end servers are often separated from the perimeter network by a second firewall configured to allow RDP frames from the TS Gateway to reach the appropriate internal servers.
Figure 1 With a Layer 3/4 firewall, the TS Gateway is placed in the perimeter network
Figure 1 With a Layer 3/4 firewall, the TS Gateway is placed in the perimeter network
While this is a fully supported scenario and one that may be useful to many organizations, it has two key disadvantages. First, because the TS Gateway receives traffic directly from the Internet, it is exposed to a relatively higher level of risk from external malicious parties. These malicious parties could attempt to attack the gateway through the SSL session and, because the front-end firewall only checks the headers and not the payloads, these sessions would reach the gateway. This is not to say that the TS Gateway is a vulnerable component; it has gone through the same rigorous security design and testing as the rest of the Windows Server 2008 product. However, its relative risk is higher in this scenario simply because it would be handling unfiltered traffic directly from the Internet. The second major disadvantage is the increased amount of traffic that must be allowed between the gateway and the back-end firewall. To authenticate users, the TS Gateway needs to communicate with Active Directory®. To allow this communication, the back-end firewall must then have exceptions for a much wider range of ports and protocols than simply HTTPS. This increased level of allowed communication again represents an increase in the relative risk of the design.
A better approach for exposing a TS Gateway to the Internet is to use an application-layer (Layer 7) firewall to handle inbound HTTPS sessions before they reach the gateway (see Figure 2). In this design, there may still be the traditional layer 3/4 firewalls forming a perimeter network. But rather than the TS Gateway sitting in the perimeter, the Layer 7 firewall is placed there. As traffic reaches the outward-facing firewall, that firewall filters out everything except HTTPS frames, then forwards those frames on to the layer 7 firewall. The Layer 7 firewall will terminate the SSL session, inspect the unencrypted contents of the stream, block malicious traffic, then send the RDP frames through the back-end firewall and on to the TS Gateway. Note that, if desired, the Layer 7 firewall could also re-encrypt the frames before sending them on to the TS Gateway. This approach may not be necessary on an organization's private network, but it can be very useful in hosted datacenters or shared networks.
Figure 2 Using an application-layer firewall with a TS Gateway device
Figure 2 Using an application-layer firewall with a TS Gateway device
This design avoids both of the disadvantages of the previous solution. Because the TS Gateway server only receives traffic that's been inspected by the Layer 7 firewall, there is less risk of Internet-borne attacks being passed through. The Layer 7 firewall should filter these attacks and only send clean, inspected traffic back to the gateway.
The second major advantage of this solution is in the placement of the gateway relative to the internal network. Because traffic coming from the Internet is inspected by the Layer 7 firewall before reaching the gateway, it can stay on the internal network and have direct access to domain controllers for authentication and RDP hosts for user sessions. Furthermore, the back-end firewall can have a much more restrictive policy. Instead of allowing both RDP and directory authentication traffic, it simply needs to allow RDP sessions from the Layer 7 firewall to the TS Gateway. Using a Layer 7 firewall to front end your TS Gateway deployment provides a more secure solution that's easier to manage without requiring a redesign of your existing perimeter network.

NAP Integration
Terminal Services Resources
  • Terminal Services in Windows Server 2008
  • Windows Server 2008 Terminal Services TechNet Forum
  • Terminal Services Team Blog
  • Microsoft Intelligent Application Gateway 2007
  • Network Access Protection (NAP) for Windows Server 2008
  • Network Access Protection (NAP) Blog
  • Windows Server 2008 Beta 3 Step-by-Step Guides

While a good perimeter design is a key element of your anywhere-access solution, it's equally important to ensure the compliance and security of your endpoint devices. Because RDP is a rich protocol that allows for many types of device redirection, such as RDP sessions and printers, it's critical that the clients connecting to your solution meet your organization's security policies. For example, even with a secure network topology based on best practices like those discussed earlier, an end user connecting to a terminal server from an unsecure machine can still end up losing confidential data or attempting to put malicious files on to the terminal server. Even though the amount of connectivity and the potential for damage is far less than if that user were connecting over a Layer 3 VPN, it's still important to manage the risk of data loss and ensure compliance with your IT policies.
Network Access Protection (NAP) is a new technology in Windows Server 2008 that allows you to control not just who can connect to your network, but what kinds of systems they can connect from. For example, you can use NAP to ensure that only machines with their firewalls running and antivirus up-to-date can connect to the network. NAP is an extensible solution that covers not just the organization's internal network but also remote users, including both those connecting over Layer 3 VPNs and those connecting through TS Gateway. By integrating NAP with your TS Gateway design, you can ensure that only those systems that meet your security policies can connect to it. For more details on NAP, see my Security Watch column in the May 2007 issue of TechNet Magazine, available online at technetmagazine.com/issues/2007/05/SecurityWatch.
Integrating NAP into a TS Gateway deployment is a straightforward process that involves a single additional service in the design. This service, Network Policy Server (NPS), can be installed either on the TS Gateway server itself or on a separate operating system instance. The TS Gateway MMC is then used to create Connection Authorization Policies (CAP) that define what RDP capabilities are allowed for a given health state. The TS Gateway server is configured to check with NPS each time a new connection is attempted and to forward the Statement of Health (SoH) for that connection to NPS. Network Policy Server then compares the SoH against its policies and informs the TS Gateway whether or not the connection should be allowed based on the health of the client.
As Figure 3 illustrates, if a noncompliant system is not configured to use Windows Update, but the organization's security policy requires that automatic updates be enabled, when a user attempts to connect to the TS Gateway his machine will generate and forward an SoH. This SoH would say something like, "My firewall is on and my antivirus is up-to-date, but automatic updates are disabled." The TS Gateway would simply forward this SoH to the NPS (the TS Gateway has no logic to decode the SoH on its own) and the NPS would compare the SoH against the policies its administrator had defined. Because automatic updates were disabled, NPS would determine that the connection matched the "noncompliant" policy and would inform the TS Gateway to not allow the connection. The TS Gateway would drop the connection and notify the user that the machine did not meet the organization's security policies. The user could then take whatever corrective action was required (in this case, simply enabling automatic updates) and reattempt the connection. As a result, a new SoH would be generated, NPS would determine compliance, and the TS Gateway would allow the connection to proceed.
Figure 3 Only compliant systems can connect
Figure 3 Only compliant systems can connect

Conclusion
Windows Server 2008 includes the key building blocks for a secure anywhere-access solution. TS Gateway provides a secure method for tunneling remote desktop sessions over the Internet and gives organizations a great deal of choice in how it gets integrated into their existing networks. Options include either placing the TS Gateway directly on the perimeter network or using a Layer 7 firewall, like ISA Server or Intelligent Application Gateway, in front of it. NAP can be combined with TS Gateway to add endpoint health inspection capabilities into the solution. The endpoint checks allow organizations to ensure that not only are remote connections coming from valid users, but also that they are coming from machines that are compliant with IT security policies. When combined, these technologies provide organizations with remote access capabilities that are more secure, more efficient to operate, and that offer better experiences for end users. You can find much more information by exploring the sites listed in the "Terminal Services Resources" sidebar.

John Morello has been with Microsoft since 2000. As a Senior Consultant, he has designed security solutions for Fortune 100 enterprise and Federal civilian and defense clients. He's currently a Senior Program Manager in the Windows Server group working on anywhere access technologies. You can read his team's blog at blogs.technet.com/WinCAT.


 The Desktop Files: The Truth about Defragmentation
The Desktop Files
The Truth about Defragmentation
Wes Miller

As long as there have been hard disks, there has been disk fragmentation. If you're concerned about your systems at all, you should be defragging them regularly. This probably reminds you of that flossing conversation your dentist has with you every year—you know the one. Well, defragmentation is just as important.
Unless your computer is sitting in the corner always turned off, it's getting more and more fragmented. In a moment, I'll show you how this happens, but first let's go over a little defragmentation history.

Defragmentation and Windows NT
Windows NT® didn't have a built-in defragmentation utility, and conventional wisdom at the time was that you didn't need one. After all, NTFS was built with optimization in mind, and it did not suffer from fragmentation as dramatically as a comparable FAT volume might have. But over time, even NTFS performance can suffer due to less-than-optimal file placement. This gave birth to a thriving market of third-party defragmentation tools. One of the most popular products among IT pros was the Executive Software (now Diskeeper Corporation) Diskeeper product. When Windows 2000 incorporated defragmentation functionality, Microsoft licensed the technology from Executive Software. The Windows® 2000 defragmentation functionality was available in its own Microsoft® Management Console (MMC) snap-in (dfrg.msc), which could be invoked by clicking on a volume or by manually running defrag.exe from the command line. Though no scheduling and only a constrained command line were available, many IT pros still created functional scripts to automate defragmentation using this built-in version. Defrag.exe in Windows XP and Windows Server® 2003 improved a bit upon the earlier version, but only provided limited automation—you still had to script it.
A number of vendors, including Winternals Software (my previous employer), produced software for the enterprise defragmentation space. Most of these products work the same way, with a centralized console, an agent of some type that lives on client systems (sometimes all the time, sometimes only on demand), and a structure for dictating when and how a defragmentation task runs.
Windows Vista® largely continues the trend of single-system defragmentation, while optimizing the defragmenter and including options such as defragmenting small file fragments (64MB or smaller), which speeds up defragmentation but leaves many larger file fragments across the disk. The defragmentation engine itself is more powerful in Windows Vista and Windows Server 2008 than in earlier versions of Windows. It also runs automatically and does not require any manual scheduling. But if you run a large organization or want to optimize your systems in ways that defrag may not provide by default, you may still need to license third-party software. And as in earlier versions, you may have difficulty defragmenting a volume where NTFS compression is enabled and used extensively. A third-party tool may be helpful in such a scenario as well.

How Does Fragmentation Occur?
Fragmentation occurs because files change over time. Ideally, Windows files themselves don't change. Or at least they don't change very often—really only with service pack and software update installations. Otherwise, the system files remain relatively constant. After performing a large update such as the installation of a service pack, fragmentation will naturally occur since the files being updated usually cannot be overwritten where they are on disk—and may require a reboot to be updated—resulting in further fragmentation.
User files and data (and the Windows registry), on the other hand, are subject to constant change. Reading, writing, editing, copying, and deleting files all the time causes a great deal of fragmentation, especially as the drive capacity is filled.
To visualize this, imagine a perfectly arranged disk, in which files occupy space contiguously with no space between them. Suppose you open a file and edit it, and then you try to save it. If the file has grown, Windows either has to save the file in its entirety somewhere else on the disk (imagine after file N) or save the newest data after file N. This means file 2 is now fragmented. If you now edit file 1, you'll have two fragmented files. Keep repeating this over time and you will end up with a considerably fragmented system where the drive has to look in multiple locations to access an individual file. With large database files, expansive hard disk files for virtual computing technologies, and copious amounts of video and audio stored on the average disk, fragmentation is common. Of course, the ever-larger size of current hard disks makes fragmentation less of a problem, but it doesn't make it go away. And older laptops with smaller hard disks will become less and less responsive, slower to boot, and take longer to open and save files.
The key concern isn't necessarily how fragmented the system is, but how fragmented the individual files are. Ever since the first disk defragmenters arrived on the scene with multicolored visuals, users have looked at them and thought, "I want a solid bar of color." But actually that is the last thing that you want. Some products try to make their disks appear as two blocks—one colored (file data) and one generally white (free space). Unfortunately, this is one of the worst things you can do to a disk if your intention is to minimize the frequency and cost of defragmentation. If you've aggressively compressed all file data to the front of the disk and removed much of the free space when you make additional edits to a file, all those edits will have to be written to the end of the file data, so by defragmenting in that way, you've actually caused fragmentation, beginning with the next file edit. Instead of two blocks of neatly organized file data, a good defragmentation results in an image that is not necessarily perfect. For an example, see Figure 1 for my test system before defragmentation. Then look at Figure 2 to see the same system after Windows had completed defragmentation.
Figure 1 My drive before defragging
Figure 1 My drive before defragging
Figure 2 My drive after defragging
Figure 2 My drive after defragging
Note that Windows focuses not on the fragmentation of the files across the disk, but instead focuses on the individual file fragmentation itself (see Figure 3 for the section of the defrag report on the Most Fragmented Files before I defragmented this system).
Figure 3 Defragmentation report
Figure 3 Defragmentation report
As a result, even after defragmentation, Windows will not show you a completely compacted set of file data at the front of the disk. Also note the large section of green data in both Figures 1 and 2. That is the Windows pagefile, which I'll discuss shortly. If you have hibernation enabled on your Windows system, it will show up similarly in a defragmentation.

Regular Defragmentation Is Essential
Just as with flossing, the main thing to remember is to defragment on a regular basis. It becomes even more important to defragment frequently on high-use systems such as servers, because if you wait, the process may take too long and leave your server unavailable or unresponsive (because defragmentation can take a considerable amount of time and certain phases of defragmentation can be CPU-intensive as well).
Even the best online defragmenters can't defragment everything. For example, they can't always defragment open or locked files (such as the pagefile or registry files) and must work around the hibernation file if it exists (which is as large as the RAM in use on your system). To optimize locked files, see the upcoming section on defragmentation tricks.
I think it's important to understand the myths and facts associated with defragmentation. Figure 4 and Figure 5 list some common topics I've heard brought up when discussing defragmentation and explain the real chances of these problems occurring.
Figure 5 Myths about defragmenting (Close Figure 5)

Myth | Likelihood
Systems will crash. | Not likely—Crashes are usually due to driver or hardware issues (memory corruption, thermal issues causing reboots, or disk failure due to disk thermal abuse).
Applications will crash. | Not likely—As with an entire system, a properly engineered application should not crash due to fragmentation, so unless it can be constrained to be the cause, you shouldn't assume that fragmentation is causing it.
Registry corruption will occur. | Not likely (see above)—Fragmentation does not cause or lead to registry corruption.
Defragmentation should result in a solid bar in any visual representation of the disk. | Not likely—You should not generally squeeze all files on the system together unless you are specifically planning to image the system with a sector-based imaging tool that doesn't exclude free space or shrink or split the partitions. You need to get the files to the front of the disk; otherwise, aggressive file consolidation leads only to further fragmentation.
Figure 4 Common concerns about defragmenting (Close Figure 4)

Concern | Likelihood
System performance will suffer. | High—Specifically depending on the usage of the system.
Recovering accidentally deleted files may become difficult or impossible. | High—Most file recovery tools depend on heuristics (to determine the data type) and file cluster proximity. Scattered files result in a more complex or impossible recovery operation.
It will be costly to defragment the system. | Medium—There is a direct correlation between how much the system is used and how much files change on the system over time. If users are aggressively editing files but saving them only to server shares, for example, you will not see much degradation over time.
Disk damage will occur. | Low—Not really likely. Again, the more you abuse a system (leave it on with disks powered up all of the time, editing, modifying, or deleting files) the more likely this is. But with today's hardware, you should not see this occur simply due to file editing.

Defragmentation as a Part of Your Infrastructure
You should seriously consider running a defragmenter on a regular basis on all of your systems. Your options are to use the defragmenter built into Windows or to look into third-party options. Basically, it comes down to how much engineering you want to do. Windows XP and later versions let you add defrag.exe as a Scheduled Task—the Knowledge Base article at support.microsoft.com/kb/555098 shows you how. Note that laptop defragmentation comes with some special considerations. You want to defragment a user's system when it is powered on—ideally via AC power—but you also don't want to take CPU or disk cycles when a mobile user is attempting to get work done. Unfortunately, however, when a mobile user isn't working on his system, odds are it's disconnected from AC power and either in standby or hibernate (which is powered off from the perspective of power management and Scheduled Tasks). At Winternals (as likely everywhere), we spent quite a bit of time trying to design an ideal mobile user scenario. If you're using Windows Vista, a solution using the built-in defragmenter and Scheduled Tasks, which has much better power awareness than earlier versions, may well provide enough of a solution. If not, a third-party solution for mobile systems may be necessary. Some of these actually kick off a defragmentation task, let you know that it ran, when it ran (and how often), and how long it took.
In addition to mobile scenarios, you will want to think about fragmentation above the file system. This isn't something you'll generally need to be concerned about with desktop or mobile systems, but servers, Microsoft Exchange, SQL ServerTM, or other data stores may require their own defragmentation. In such cases you'll need to decide whether that should be done before or after you've performed a file system defragmentation.
If you are using a third-party defragmenter, you'll want to ensure that it uses the Windows defragmentation API. Since Windows NT 4.0, Windows has had its own defragmentation API to prevent unexpected data loss. For example, if you're in the middle of moving a file and your system loses power, the last thing you want is an incomplete fragment move, which could result in data loss and even an unbootable system.
Finally, if you are using any virtual computing technology, you may want to consider how you manage the defragmentation of your virtual disks as well. They are just as likely to have fragmentation as a physical spindle, and in many senses may be worse due to size constraints. Plus, if your virtualization solution allows you to shrink disks or partitions, you may need to defragment first in order to free up space (or at a minimum, reduce the amount of time that a shrink operation will take).

Additional Tools
By now, you're surely convinced that defragmentation is an important and valuable function. Even if you do not use a third-party tool for defragmentation itself, you may want to consider a tool by Mark Russinovich called PageDefrag, which you can get from the Microsoft Sysinternals tool site (see microsoft.com/technet/sysinternals/FileAndDisk/PageDefrag.mspx). As I mentioned earlier, the Windows page file cannot be defragmented while Windows is online. In fact, the same is true of the Windows registry files and the event logs. PageDefrag allows you to defragment these locked files before Windows has finished booting and locked these files, using a small driver. To use it, simply run PageDefrag and set it to execute either on the next reboot or on every reboot. When the system reboots (at the point in the boot process where you may have seen a disk check run before), it will defragment the files specified in the list (see Figure 6).
Figure 6 Files to defragment in PageDefrag
Figure 6 Files to defragment in PageDefrag
Once PageDefrag has completed, the system will finish booting normally. If you're looking for more on PageDefrag, Lance Whitney wrote about it in the September installation of Utility Spotlight (see technetmagazine.com/issues/2007/09/UtilitySpotlight). Note that PageDefrag is only supported on Windows NT 4.0, Windows 2000, Windows XP, and Windows Server 2003—it is not currently supported in Windows Vista.
In addition to PageDefrag, there is another tool available from Sysinternals called Contig, which allows you to defragment a specific file manually (see microsoft.com/technet/sysinternals/FileAndDisk/Contig.mspx). This can be useful if a specific file was not able to be defragmented via a normal defragmentation task or if it has become fragmented since (but the remainder of the volume has not become as fragmented).
Contig can also defragment an entire directory or subdirectories by using wildcards. The following will defragment all files under the Windows directory, regardless of where they are located.
Contig –s c:\Windows\*.*
Applying the –v switch will make the operation verbose, and –q will make it run quietly.

Wes Miller is a Development Manager at Pluck (www.pluck.com) in Austin, Texas. Previously, he worked at Winternals Software in Austin and at Microsoft as a Program Manager and Product Manager for Windows. Wes can be reached at technet@getwired.com.


 Field Notes: The Cod of Conduct
Field Notes
The Cod of Conduct
Betsy Aoki

Sometimes what you need may not be a bigger server, or a better-educated user base who refrains from beeping you just as you are about to leave early for the weekend. Sometimes it's not managing the code that's the problem; what you need is a great big fish.
About a year ago, I was working on abuse reporting for the Live QnA site (qna.live.com). We manned the Delete buttons and monitored the site for idiocy attacks and poor behavior. We had internal tools. We had external policies. We confronted a couple of real wackos who attempt to sabotage our questions with nasty content.
Meanwhile, I was burning out. After all, the community health of the site was my charter. I was the one who helped work on the code of conduct that set the tone for the site. I was the Gatekeeper, and everyone was just going to darn well BEHAVE.
Except they didn't. People are people—that is to say, capable of the greatest perfidy and the wildest saintliness—and what I wanted didn't really factor into whether someone would call another user a butthead or not.
So one Saturday I was sitting in my kitchen, thinking about the code of conduct I was defending. As I was doodling around with my pen, I had one of those "brain-moves-sideways" motions in my head. What if there was no code of conduct, but a cod of conduct? And if so, what would this cod look and act like? Well, I grew up in New England where we ate cod. White, flaky, boring cod. I wanted to be able to chide and cajole my QnA membership with the snide, matter-of-fact tone of a cod. Not just the smiting hammer of the Delete button, which I was using with too much frequency of late, but with the scathing wit that would make someone laugh even as they vowed not to cross the line again. Thus, Norbert the Cod of Conduct was born. I pondered giving him a pipe, but I figured the bowtie was enough.
The Cod of Conduct
The Cod of Conduct
I uploaded my drawing of the fish to a new LiveID, gave it a QnA user name, and Norbert, the QnA Cod of Conduct, was ready for business. He wasn't always me, mind you. Sometimes another team member would take on the Cod persona and chide the miscreants. Who is the Cod of Conduct? See go.microsoft.com/fwlink/?LinkId=99044. Does the Cod of Conduct hurt people's feelings? See go.microsoft.com/fwlink/?LinkId=99045.
QnA community members took to him like ... er ... a fish takes to water. They bantered with him, summoned him, got cranky about him. He educated the newcomers, chided the folks we thought could be turned from darkness to light, and provided a lighthearted way of looking at the rules beyond just reading a dense legalese-ridden Web page.
Rules—whether they are company policy, tribal law, or playground pecking order—all depend on the goodwill of the folks who feel the effects of the enforcement. As an IT person or a community Web site developer, you can create tools that assist in the enforcement of the rules, but sometimes the best tool for the job will be something that arises out of your own personality or the collective tribe's sense of humanity. Rather than technical engineering, a bit of social engineering spit and polish can be more effective. Or perhaps, in our case, spit, polish, and a fish.

Betsy Aoki is a Community Program Manager for Microsoft on the Xbox team. Her gentle neuroses are avidly expressed in her e-mails, blog posts, and, hopefully, on the XNA Framework.


 Windows Confidential: The File System Paradox
Windows Confidential
The File System Paradox
Raymond Chen

If you take a look around your hard drive, you probably have noticed that a good number of operating system files hang out in the root directory of the boot drive. If you take a closer look at those files, you may notice a theme: boot.ini, NTLDR, and NTDETECT.COM are critical boot files, and hiberfil.sys is the hibernation file. These are all files that are involved in the crucial first stages of booting the operating system or, in the case of hiberfil.sys, resuming from hibernation, and they have to go into the root of the boot drive. Why can't you move them somewhere else?
My colleague Adrian Oney explained: in order to read the boot files off the disk, you need the file system driver, but the file system driver is on the disk, which you can't read until you've loaded the file system driver. Oh no, Catch-22!
The vicious cycle is broken by having a miniature file system driver built into the critical boot files. This miniature driver knows just barely enough to locate files in the root directory and load them into memory. Those files can, in turn, get the operating system off the ground, at which point the real file system driver can take over and look for files in much fancier places like subdirectories.
Hibernation follows a similar pattern. Hibernating the operating system means dumping the entire contents of memory into the hibernation file; restoring from hibernation entails sucking that file back into memory and pretending nothing happened. Again, it's another chicken-and-egg problem: to load the hibernation file, you need the file system driver, but the file system driver is in the hibernation file. If you keep the hibernation file in the root directory of the boot drive, the miniature file system driver can be used instead.
Those who have used early versions of MS-DOS® are well aware of this constraint on the location of the operating system boot files. The old SYS command made a boring old floppy disk into a magic bootable floppy disk. However, if you ran it on a floppy disk that already had data on it, there was a good chance the conversion would fail because early versions of MS-DOS required that the boot files occupy specific locations on the floppy disk in order for the boot sector to find them. Later versions of MS-DOS relaxed these requirements by making improvements to the boot loader, but even with the improved version, the system files still must reside in the root directory.
One customer asked, "Can I move the hibernation file into a subdirectory?" After all, why does it matter where the file is? The ACLs on the file would be the same regardless of where you put it. But these folks wanted to put it on a drive different from the boot drive and had been told that can't be done. They didn't really understand the reason for this restriction, so they tried an end run: well, if I can at least move it into a subdirectory, then I can create a directory and make that a junction that points to where I really want the hiberation file. Woo hoo—I relocated the hibernation file!
Of course, the miniature file system driver doesn't understand mount points, because mount points mean talking to the disk management service, and at the time the system is booting, the computer hasn't even loaded the operating system yet, much less some fancy advanced service that knows how to map drive letters back to ARC paths, and even if you somehow got that service running, you still have to find the device drivers for those other hard drives and load the corresponding file system drivers.
It's only a matter of time before somebody asks for a way to put the hibernation file on a DFS junction.

Raymond Chen's Web site, The Old New Thing, and identically titled book (Addison-Wesley, 2007) deal with Windows history and Win32 programming. His Web 2.0 name is Raymond, or maybe Chester.