 TechNet Magazine August 2006

TechNet Magazine August 2006

Disaster Recovery: A Guide to Exchange Disaster Recovery Planning

Michael Lee Disaster Recovery: Simplify File Recovery with Data Protection Manager

Microsoft IT Showcase and Laura Euler Disaster Recovery: Plan Ahead to Save Your Web

Chris Adams Performance: Playing with Exchange in a Sandbox

Greg Taylor and Simon Shepherd Windows Administration: Problems of Privilege: Find and Fix LUA Bugs

Aaron Margosis Windows Administration: Find Anything with Windows Desktop Search

Matt Hester Windows Administration: Manage Printers with New Group Policy Features in Windows Server 2003 R2

Jeremy Moskowitz From the Editor:

Joshua Trupin Toolbox: New Products for IT Professionals

Greg Steen Utility Spotlight: Scripting Trouble-Free Downloads with BITS

Michael Murgolo SQL Q&A: Clusters, Snapshots, Log Shipping, and More

Edited by Nancy Michell How IT Works: Certificate Services

Randy Muller Hey, Scripting Guy!: How Can I Delete a Key and its Subkeys from the Registry?

The Microsoft Scripting Guys Inside Microsoft.com: Analyze Web Stats with Log Parser

Jeffrey A. Johnson Security Watch: The Most Misunderstood Windows Security Setting of All Time

Jesper Johansson How IT Works: Roaming in SMS 2003

Steve Rachui Field Notes: TLC for Your Server

Jay Shaw What is TechNet?: Resources for the IT Professional

Windows Confidential: A Directory by Any Other Name

Raymond Chen


 Disaster Recovery: A Guide to Exchange Disaster Recovery Planning
Disaster Recovery Michael Lee

A Guide to Exchange Disaster Recovery Planning
At a Glance:
  • Basic Exchange Server disaster
  • recovery strategies
  • Using Exchange on a Windows cluster
  • Recovering Exchange databases
  • New recovery features in Exchange Server 2007

In all but the smallest organizations, Exchange Server 2003 runs as a distributed system. This architecture provides Exchange with a great degree of resiliency. In fact, the great majority of an Exchange
server's configuration information is stored away from the server, in Active Directory. This distributed design makes rebuilding a damaged Exchange server nearly automatic. In addition, Exchange works well with the Microsoft® Windows® Clustering Service. Running Exchange in a Windows cluster (see Figure 1) can provide you with even faster recovery and greatly reduce maintenance downtime for tasks like installing service packs and security upgrades.
Figure 1 Running Exchange Servers as Clusters
Figure 1 Running Exchange Servers as Clusters
For Exchange to work, the network, DNS, and Active Directory® must all be working properly. Increasing the reliability of these vital infrastructure components is often the best thing you can do to increase Exchange reliability and availability. A detailed discussion of architecting and recovering your network and Active Directory is beyond the scope of this article, which, for the most part, deals with how Exchange servers can be affected by a disaster.
The most fundamental aspect of preparing for and recovering from an Exchange disaster is making sure you have good backups of your databases, and I'll spend a good bit of time here on Exchange database recovery. I'll also discuss recovery of server installations and configuration information, and re-assignment of special Exchange server roles after a disaster (see the sidebar "Special Server Roles").

Basic Strategy
Redundancy is the first principle of disaster recovery. Redundant network paths, DNS and Active Directory servers, and Exchange front-end (client access) and clustered servers turn failures into inconveniences instead of disasters.
Some failures can be automatically handled with little apparent effect on the system. For example, an emergency generator can monitor electrical power. In case normal power fails, the system automatically fails over to the generator with a negligible interruption in service. Other failures can be mitigated by having hardware backups or spares. If a single light bulb burns out, for example, you simply replace it with a new one.
Some components of an Exchange system can be designed for recovery by failover: Active Directory, DNS, front-end servers, public folder databases, message routing and connectors, clustered Exchange virtual servers, and general network infrastructure. Other components typically rely on backups, manual reconfiguration, or failover to an alternate server or application after a disaster: mailbox databases, special server roles, standalone Exchange server configuration information, in-transit messages, and client applications.
Several third-party vendors offer replication capabilities for Exchange Server 2003 mailbox data, so you can devise a failover-based recovery strategy for mailbox data, but it requires purchasing add-on products and services. Exchange does not currently provide this capability out of the box. (Later in this article, I'll preview new mailbox database replication features coming in the next major version, Exchange Server 2007.)
For reduced downtime and system impact, it is always better to design for failover instead of restoration or reconfiguration. But failover strategies typically cost more, not only in monetary terms but also in complexity and management. What's most important is to have a strategy for protecting and recovering each of the components noted previously. Regardless of the details of your strategy, you should take into account the following:
Recognize the scope and limits of your strategy What disasters won't your strategy handle? What if there is no electric power for more than a day? What resources are you assuming will be available after a disaster? What if an entire building is destroyed or the entire Internet is down? Understanding what your strategy can't handle is critical for designing Service Level Agreements (SLAs) and for assessing risks. The bigger the scope of a disaster, the less effective are detailed plans for dealing with it. That doesn't mean you can't plan effectively, but you need to build in flexibility and ensure that you have the necessary resources to solve all unexpected problems. Step-by-step procedures save time and reduce mistakes, but they are no substitute for people who can creatively handle unexpected situations.
Test your strategy regularly When you devise a plan on paper, you'll inevitably miss certain elements that you'll discover only during a fire drill. For example, if your strategy for replacing a burned out light bulb is to replace it with another one, it's a good idea to find out before the light goes out that you'll need a ladder to reach the fixture as well as a flashlight to find the replacement bulbs and to help make sure you don't fall off the ladder in the dark.
Update the strategy periodically As conditions change, parts of your strategy will become obsolete. Maybe an updated version of your backup program requires different restoration procedures. People change jobs or phone numbers. Other groups you depend on may reorganize or disappear entirely. You should schedule periodic reviews to evaluate and update your plan. Quarterly reviews are common. Your first quarterly review may reveal whether that is often enough.
Have a backup plan for your backup plan Do you know how to quickly find a different ISP or hardware provider? If key personnel are unavailable, do you have the budget and contacts to get substitute expertise quickly from a vendor or system integrator? Redundant strategies are as important as redundant hardware.
Train more than one person It's not just technical knowledge that matters. What about passwords? Account numbers? Telephone numbers? Don't let the absence of a key person or a changed cell phone number become your most important single point of failure.

Recovering Configuration
Nearly all the configuration information for an Exchange 2000 or 2003 server is stored in the Active Directory database. If Active Directory replicas are kept on servers separate from Exchange (which is the case in all but the smallest Exchange systems), then the configuration for an Exchange server can survive the loss of the entire server.
Of course, this means that it is very important to take good care of Active Directory. You should have multiple replicas of the Active Directory database for each domain in your organization or do frequent backups of the database. If Active Directory is entirely destroyed but an Exchange server survives, you can still recover but it is not a simple process.
The Exchange Setup.exe program can be run in three different modes: initial setup, maintenance, and disaster recovery. Initial server setup runs automatically if the setup detects that Exchange has never been installed on the server and there's no configuration information already in Active Directory. If you are installing Exchange to a standalone server, setup will install all necessary Exchange program files, configure default registry and Internet Information Service (IIS) metabase entries, and then store the server's configuration information in Active Directory. If you are installing Exchange into a Windows cluster, setup will install all necessary Exchange program files, configure default registry and IIS metabase entries, and stop. Configuration of Exchange virtual servers inside a cluster is done by the administrator after setup has completed. I'll explain how this works and other differences between Exchange on a standalone server and in a cluster shortly.
Maintenance mode setup runs automatically if setup detects that Exchange has already been installed on the server. This mode is useful for adding new Exchange components (such as foreign connectors) or for reinstalling Exchange program files and default settings to try to repair a damaged Exchange installation.
Disaster recovery mode setup (see Figure 2) must be run manually by starting Setup with the /Disasterrecovery command-line option. For this mode to work, Setup must find intact configuration information for this server in Active Directory. Use this mode when an Exchange server installation has been damaged badly enough that maintenance mode setup does not detect that Exchange is already installed on the server. Setup /Disaster recovery will install Exchange, configuring the server automatically with the server information already in Active Directory. This mode can also be used to transfer ("forklift") an Exchange server to different or upgraded hardware.
Figure 2 Disaster Recovery Setup
Figure 2 Disaster Recovery Setup
The /Disasterrecovery option for Setup.exe is not available for an Exchange cluster, but there is equivalent functionality in a clustered environment. To understand how this works, you must first know something about how Exchange running in a cluster differs from Exchange running on a standalone server. And for that, you need to know the basics of Windows clustering.

Windows Clustering Basics
A Windows Server™ 2003-based cluster is formed by defining a cluster name in the Cluster Administrator console and then configuring two or more computers (or nodes) as members of the cluster. Note that it is possible to create a single-node cluster on any computer running Windows Server 2003 for testing and learning purposes. You can even install Exchange on a single-node cluster to see how it works. But a single-node cluster isn't much good for anything except testing and learning.
Cluster nodes share a quorum disk and a cluster database. (Quorum types other than shared disk are now supported by Windows Server 2003. They won't be discussed here because they don't affect basic clustering principles.) These resources store information about the cluster state and about the configuration of applications running in the cluster. At any given time, one node in a cluster is responsible for running a particular application that's under cluster control. If this node fails, or at administrator discretion, another node can take over running the application. Switching an application to run on a different node is called failing over. Other nodes can also run the application because the application has been installed in a cluster-aware configuration on each node, and all information needed to run the application is replicated continuously to every node.
In general, clustered applications run as virtual servers. This means that the application appears in the Cluster Administrator as a set of resources. For example, an Exchange virtual server typically has four resource types in it: a network name, an IP address, physical disks, and services.
The network name is the computer name that clients on the network see when connecting to the Exchange virtual server. This name is different from the cluster name and from the computer names for each node in the cluster. Typically, only cluster administrators connect directly to the cluster or its nodes. All other clients connect to virtual servers running in the cluster which appear to them to be full-fledged standalone computers.
The IP address is the address associated with the network name. It is the IP address clients can ping for the virtual server. In DNS, there will be an entry for the network name associating it with this IP address.
Typically, there will be multiple physical disks in an Exchange virtual server to support multiple databases and transaction log file streams. These disks share a physical bus of some kind among all the computers that belong to the cluster. When the virtual server is failed over to a node, the node can acquire exclusive access to the disks that belong to the virtual server.
All of the services you typically see on a standalone Exchange server are configured as resources in an Exchange virtual server. Instead of starting and stopping Exchange services from the Services console, you do it in the Cluster Administrator console. (Stopping services incorrectly in a cluster will probably make the cluster think the service has failed and cause the entire virtual server to fail over to another node.)

Setting up an Exchange Cluster
Recall that when you set up Exchange on a cluster, only the Exchange program files, default system registry, and the IIS metabase entries are configured. No configuration information is put in Active Directory by Exchange cluster setup. The reason for this is that while you must install Exchange program files on a specific cluster node, you will run Exchange in a virtual server that floats among multiple nodes in the cluster. You set up Exchange virtual servers through the Cluster Administrator program, not Exchange setup. In a cluster, all Exchange setup does is prepare the node to be a possible owner of an Exchange virtual server. It gets the program files needed to run Exchange installed on each cluster node.
To create an Exchange virtual server, you must create a new virtual server in Cluster Administrator and assign it an IP address, a network name, and one or more physical disk resources (see Figure 3). These basic resources are available and configurable from Cluster Administrator's menus. After configuring them, you're ready to add a Microsoft Exchange System Attendant resource to the virtual server.
Figure 3 Configuring Exchange in a Window Cluster
Figure 3 Configuring Exchange in a Window Cluster
When you add this resource to a virtual server, Exchange checks Active Directory for an Exchange virtual server with the same network name. If a virtual server with the same name already exists, the rest of the setup proceeds in much the same way as it would during a disaster recovery setup on a standalone Exchange server. Information already in Active Directory is used to complete the configuration of the virtual server.
If there is no matching virtual server in Active Directory, initial setup is completed and configuration information for the virtual server is added to Active Directory. In all cases, the other services required by an Exchange virtual server are automatically created when you create the System Attendant resource.

Advantages of an Exchange Cluster
In a cluster, Exchange virtual server configuration information is shared and replicated among all cluster nodes. If a particular node is destroyed, the impact on Exchange is minimal; another node simply takes ownership of the virtual server and keeps running the app. You can add or remove nodes from the cluster without disturbing other nodes or apps already running in the cluster. If a cluster node is destroyed, it can be replaced simply by adding another Windows server to the cluster and installing Exchange program files on it. You don't have to run a disaster recovery setup or incur any user downtime.
If you need to install service packs, firmware upgrades, or security updates, you can move Exchange to a different node while you complete maintenance at your leisure—even during peak usage hours. Typically, moving Exchange to a different node takes only a minute or so. If your end users run Outlook® 2003 in cached mode, they are unlikely to even notice the failover. In a standalone configuration, maintenance or upgrades that require a server reboot usually mean you have to stay late or work weekends so you can take a prolonged outage during off peak hours.
So why doesn't everyone run Exchange in a cluster? There are two important issues to consider when deciding whether to cluster Exchange. The first is hardware. To be fully supported by Microsoft, you must run Exchange on Windows cluster-certified hardware and configurations. Consult the Windows Catalog for a list of systems that are cluster certified. Cluster certification goes beyond ordinary Windows Hardware Certification List (HCL) approval. Because cluster machines must work together as a single system, cluster hardware is certified as a full working system, not just component by component. This means that you may have to replace or upgrade hardware to run Exchange in a cluster.
The second factor is training. As you've just seen, Exchange behaves differently in a cluster than on a standalone machine. Exchange administrators must be trained to understand not only how Exchange behaves in a cluster, but also how to administer clusters in general. Well-managed standalone Exchange servers in a stable environment can easily achieve 99 percent or better availability. If you are running Exchange on standalone machines with less than 99 percent uptime, switching to clusters is unlikely to improve things. In fact, it may make them worse because you're adding extra complexity to an already unstable environment.
If you are currently running below 99 percent Exchange uptime, improvements other than clustering are likely to be more effective. Very often, these improvements do not even cost anything, but are achieved through more effective management and operational procedures.
If you are considering clustering for the first time, choose Windows Server 2003 instead of Windows 2000 Server. Clustering has been made more simple and robust in Windows Server 2003 and the learning curve for new administrators is greatly reduced. Clustering is installed by default in Windows Server 2003, and initial cluster setup has been streamlined and improved.
Finally, there is one important respect in which running Exchange in a cluster offers no additional protection over running Exchange on a standalone server: protecting databases. If something damages the databases, failing over to a different node will not help you. The damaged database disk is transferred to the new node. It will fail there in just the same way as on the previous node.
The next version of Exchange is expected to provide Cluster Continuous Replication (CCR) of databases, which will remove this single point of failure. You will be able to fail over to a completely independent copy of the databases. There will also be Local Continuous Replication (LCR) available for standalone servers.

Special Server Roles
There are several functions all Exchange servers need that can be delegated to a single server. These roles are easily transferable to different servers in case of a disaster, but it is important to document which server carries each role in case you need to reassign it after a disaster. Many an Exchange administrator has gone through hours of unnecessary troubleshooting after forgetting something like which server is the Routing Group Master—and that this server is down. Here are the main roles to document:
Domain Recipient Update Service (RUS) There must be at least one Exchange server designated for this role per Windows domain with Exchange recipients. This server is responsible for managing and updating mailbox-enabling attributes for all Exchange users in the domain. Without it, new mailboxes can’t be enabled. To change the Domain RUS Server, change the designated server on each Domain RUS object in Exchange System Manager.
Enterprise RUS There is one Exchange server designated for this role per Windows forest. This server is responsible for managing and updating mailbox-enabling attributes for all Exchange system mailboxes in the Active Directory Configuration container. Without it, System Attendant functions on new servers will fail and critical system messages may not be delivered. To change the Enterprise RUS Server, change the designated server on the Enterprise RUS object in Exchange System Manager.
Routing Group Master There is one Exchange server designated for this role per Exchange Routing Group. This server is responsible for coordinating and updating routing information between all servers in the routing group and between routing groups. Without it, mail may get stuck in queues or be returned as undeliverable when routing topologies change. To change the Routing Group Master, open the Routing Group in Exchange System Manager and select a different server by right-clicking its object and setting it as the new Master.
Offline Address Book Generation Server There is one Exchange server designated for this role per Offline Address Book. This server is responsible for creating the Offline Address Book that offline Outlook clients use to address messages. Without it, updates and changes to the address book will not be available to clients. To change The Offline Address Book Generation Server, open the Offline Address Lists container in Exchange System Manager and select a different server in the properties of each list.
Free/Busy Folder There can be multiple Exchange servers that have replicas of this folder, but typically there’s only one. Clients connect to servers with replicas of this folder to upload Calendar information that allows others to check when they are free or busy. If a replica of this folder is unavailable, scheduling efficiency is impacted and this problem is usually very visible to end users and executives. If the original folder is unavailable and you configure a new replica on a different server, clients must all make changes to their calendars or run Outlook for a period of time before their information will be up to date.
Front-End Servers Front-end or Client Access servers (CAS) do not host mailboxes but are used for load balancing and convenience in client access and firewall configuration. Typically, a downed front-end server can be simply replaced with another one that is similarly available to the same clients.

In-Transit Messages
When disaster strikes, there may be messages "in flight" within the system. A disaster may prevent the server from functioning, but it may leave messages intact that have been queued for delivery. Depending on the scope of the disaster, these messages may be stranded somewhere, but still be recoverable. In such cases, many administrators simply instruct users to resend recent messages that may not have gotten through before the disaster. But that is not the only option. You may be able to locate and save the messages for later delivery after the system has been recovered.
Exchange supports several different message transport mechanisms. Each transport protocol queues messages in a different way. The default transport for messages in Exchange 2000 and Exchange 2003 is SMTP. If this is the only message transfer method you use, then saving and replaying messages for later delivery is simple. If you copy a properly formatted SMTP message into the Pickup directory for the SMTP service, SMTP will grab the message and try to deliver it. This works even if you copy a message that was stranded on one server to the Pickup folder on another SMTP server. See the Microsoft Knowledge Base article "Supported Methods to Replay Outgoing SMTP Messages in Exchange 2003 or in Exchange 2000" for more details.
Third-party connectors and other message transfer protocols may be more difficult to preserve and replay for delivery. If in-transit messages are of concern, refer to Microsoft or third-party documentation for descriptions of how messages are queued and transmitted in each connector.

Recovering Exchange Databases
The information presented so far should give you a good foundation for understanding the various elements of an Exchange system you need to protect and get running again after a disaster. It may seem like there are a lot of moving parts to keep track of, but if you know how to install Windows and Exchange and are reasonably familiar with the Exchange System Manager program, getting the system back up and running depends more on having spare parts than anything else. The actual tasks are relatively simple.
After an Exchange server is back up and configured correctly, you come to what is often the most difficult—or at least time-consuming—part of disaster recovery: restoring user data. More and more information is being kept and sent through e-mail. Exchange databases are frequently 50GB or more in size and there are often 20 of these databases on a single server. Without specialized (and still very expensive) disk hardware, backing up and restoring that amount of data just takes time.
Exchange includes many features to help you back up, restore, manage, and repair Exchange data efficiently and flexibly. In most cases, recovering Exchange data after a disaster is simple: restore your last backup and let automatic transaction log replay roll the database forward. Things get complicated, though, if you discover that your backups haven't run for weeks—and you forgot to check whether they were working. Loss of transaction logs also makes recovery more difficult, but you're not out of luck even then. Exchange has a robust suite of tools and utilities for repairing databases and salvaging and merging mailboxes between databases. Still, your best recovery strategy is to do the things you should to avoid needing to use such tools.
Many companies don't think of Exchange as a Tier 1 or database application. They don't provide the same quality of hardware or protection they do for other mission critical applications. Sometimes it takes a disaster before companies realize that e-mail is mission critical and they should be more serious about backing it up and protecting it. Ask yourself (and perhaps your boss) these questions: How would people react if the mail system was shut off for a day? What would the impact on your business be if all the Exchange databases were suddenly deleted? If you don't use a reliable method to back up your Exchange data, you're just one disaster away from finding out the answers to those questions.
The first line of defense in recovering an Exchange database is its transaction log files. Not only are the transaction log files a good crash-recovery mechanism, they also play an important role in recovery using backups. Because every change to the database is recorded in a transaction log file, you can use the logs to bring a restored copy of the database completely up to date. Essentially, the transaction logs allow you to restore from backup and then fast-forward through everything that happened since the backup. If you have all the transaction logs generated after the backup, the result of restoration will be a database that is logically identical to the one you just lost, right up to the moment that disaster struck.
A critical best practice is to store the transaction log files away from the database files on a separate physical disk or array. This decreases the chance that both the database files and the log files will be damaged or lost in the same failure. If you keep the database and log files on the same drive, you are storing all your eggs in one basket.
There is no need to take a database down to back it up. Microsoft provides two application programming interfaces (APIs) for backing up Exchange databases while they are running and available to clients. Depending on hardware capabilities, the Windows Volume Shadow Copy Service (VSS) API in Windows Server 2003 is capable of making a backup copy of a database of any size in a backup window of only a few seconds. This API is implemented by several third-party hardware providers and backup applications. The Windows Backup streaming backup API has been available since the first release of Exchange. It is also implemented by many third-party applications. To make Exchange streaming backups available in Windows Backup, all you have to do is install Exchange administration tools on the backup server.

Database Recovery Strategies
If a disaster leaves Exchange database and transaction log files intact on disk, database recovery will be automatic. Exchange will replay necessary transaction logs, bring the databases to a logically consistent state, and then allow you to start them. This works even if you have to move disks to different hardware.
The success of transaction logging as a crash recovery mechanism depends on the database files and the transaction log files surviving a crash with no damage. If a crash causes file corruption or loss of an entire disk, the automatic recovery mechanism can't work. What you should do to recover after that depends on whether you have lost a database drive or a transaction log drive.
Strangely enough, if you have to choose between losing the Exchange transaction logs and losing the database files, lose the databases—if you have a good backup. After the loss of database files, you can restore from backup and roll the database forward with the transaction log files for a zero-loss restoration. But losing transaction log files that haven't been backed up means you lose all changes recorded in them.
In short, if no transaction logs are missing after a disaster and you have a good backup, full recovery is simple. If you are missing transaction logs, you have four recovery choices. First, if the database files are logically inconsistent but otherwise intact, the database can be examined with the Eseutil and Isinteg utilities to detect and correct any transactional inconsistencies. Because all data in the files must be carefully examined, this repair can take several hours for a large database. Repair is typically very successful in this circumstance and may result in zero or very minimal data loss.
However, many Exchange administrators follow a policy of not leaving a repaired database indefinitely in production. After repair, they move all mailboxes or folders to a different database.
Second, you can restore from backup. If none of the transaction log files generated since the backup are available, you'll lose all changes to the database since the backup. Note that it is possible to do partial replay of transaction log files. Suppose only the last several files were lost in the crash. You can still replay changes into the database up to the point of the first damaged or lost transaction log file.
The third strategy is to restore, repair, and merge. This is a hybrid of the first two strategies with an additional twist—merging the contents of two databases together. You restore a previous database and allow clients access to it. They don't have immediate access to recent data, but do have access to everything from before the last backup. At the same time, you repair the inconsistent copy of the database in a Recovery Storage Group (RSG) or on a separate lab server. Prior to Exchange 2003, it was necessary to build a separate Exchange server to perform parallel repairs. The Recovery Storage Group feature removes this requirement.
There are a number of ways you can merge the contents of two databases or mailboxes together. The most basic method is to open the two mailboxes in Outlook, then drag items from one mailbox and drop them into the other. This is impractical, though, if you need to recover a large number of mailboxes. You can do bulk mailbox merges using either the Exchange Mailbox Merge Wizard (ExMerge.exe) or the Recover Mailbox Data wizard, which was first introduced in Exchange 2003 Service Pack 1 (SP1).
The ExMerge utility, shown in Figure 4, works for any version of Exchange. ExMerge is very powerful, with multiple filtering and selection options. It is useful in many scenarios other than disaster recovery and has been an Exchange administrator favorite for years. You can download ExMerge from go.microsoft.com/fwlink/?linkid=68533.
Figure 4 Using ExMerge to Recover Mailbox Data
Figure 4 Using ExMerge to Recover Mailbox Data
The Recover Mailbox Data wizard lets you do bulk mailbox merges right from within Exchange System Manager (see Figure 5). It does not have the advanced filtering and selection capabilities of ExMerge, but administrators typically don't use those features in a disaster recovery situation. There is no setup needed to use this tool, and merging mailboxes is as simple as selecting multiple mailboxes running in an RSG and right-clicking to merge their contents back to the production databases.
Figure 5 Recover Mailbox Data Wizard
Figure 5 Recover Mailbox Data Wizard
There is no downtime required with either tool. Both work while databases are online and both have excellent duplicate-item detection and suppression. The client experience is that lost data suddenly reappears as they are working.
The fourth strategy is to provide send/receive functionality immediately and restore the data later (also known as dialtone recovery). This strategy is essentially the same as the previous hybrid strategy, except that instead of restoring a backup for clients to use, you start over with a new database as soon as possible after a disaster. Clients see an empty mailbox, but they have send and receive capability more quickly than if they had to wait for restoration or repair to complete.
From the administrator point of view, this strategy is the most complex. At some point, it may be necessary to merge the contents of three separate copies of the database: the "dialtone" database, a restored backup, and a repaired copy of the database. This is the best strategy for use when send and receive capability is mission critical, such as for users who handle very time-sensitive customer orders. (Though even for these users, it may be more efficient to set up temporary mailboxes and reroute mail to them.) You might also use this strategy in situations where it will take a very long time to restore or repair previous data, or in single storage group outages. There can be only one RSG on a server, and it can be used for only one production storage group at a time. This means that repair and merge operations will have to be done serially after a full server outage or that separate recovery servers will be needed to handle multiple storage group recoveries in parallel.

The Notorious Error -1018
Experienced Exchange administrators know and dread error -1018, an example of which is shown in Figure 6. In most cases, this error will not cause an immediate outage or other obvious problems, but it is still critical because it warns of the possibility of imminent hardware failure. Error -1018 means that a page in an Exchange database has been damaged at the file system level. An Exchange database is divided into pages, and each page has a checksum on it that validates the integrity of the page. When Exchange reads a page and checksum verification fails, a -1018 error is the result. Error -1018 is recorded in the server's Application log each time Exchange tries to touch the bad page.
Figure 6 The Dreaded Error - 1018
Figure 6 The Dreaded Error - 1018
A single appearance of the error may happen on perfectly good hardware—even the best hardware will have a glitch now and then. But if you see it more than once on the same server, it is likely that the hardware is defective or failing.
A great deal of the information in a database is client data that is seldom accessed (such as items in the Deleted or Sent folders). Because checksum errors are only detected when a page needs to be read, a -1018 problem could lurk undetected in a database for a very long time on a page that is seldom retrieved. Online backup is therefore used to ensure timely detection of -1018 problems. During backup, the entire database must be read or copied and so this is a good time to check the whole database.
Exchange will not let you complete an online backup of a database with a bad page in it. This ensures that your last backup is good and allows you to expunge a -1018 error by restoring and rolling forward. You can also repair a database to remove a -1018 page, but this will result some data loss—at least the loss of the data that was on the bad page.
Because you can't do a new online backup of a database with a -1018 error in it, there is some urgency in resolving such errors. To back up the database before correcting the problem, you'll have to take it offline and back it up as a set of raw files. No transaction log files will be removed from disk until you do another successful backup. This ensures you will be able to roll forward all the way from the last backup, but it also means that if you neglect dealing with the problem you will eventually run out of space on the transaction log drive.
Moving all mailboxes from a database with a -1018 page error is also an effective strategy and causes minimal downtime. Moving mailboxes will result in the loss of whatever was on the bad page. Restoring and rolling forward is the only strategy that works to recover what was on the page before it was damaged.
As of Exchange Server 2003 SP1, there is a new defense against -1018 errors. A new Error Correcting Code (ECC) checksum has been added to each database page. The previous page checksum reliably detects bad pages but can't correct them. The new ECC checksum lets Exchange fix the page when the problem is a single-bit error (also called a "bit flip"). Bit flips are a common hardware glitch, and historical analysis of -1018 database errors revealed that approximately 40 percent are caused by bit flips.
The ECC checksum is a good thing for administrators because it means that 40 percent of the time it's not necessary to restore, repair, or abandon a damaged database. But there is a new danger too: it is now easier to ignore a -1018 error. Just because Exchange can fix up the damage doesn't mean it is less serious. The implications of a -1018 error remain the same, whether or not the error can be fixed. A -1018 indicates the possibility of progressive hardware failure. Ignoring these errors is just as dangerous as before, even if Exchange helps you recover more easily.

Coming Up: Easier Recovery
Many of the data recovery features that are found in Exchange Server 2003 will be carried over and incrementally improved in the next major revision—Exchange Server 2007, and exciting new features are being developed as well. The most significant new features are Cluster Continuous Replication (CCR) and Local Continuous Replication (LCR). CCR and LCR provide application-level replication of mailbox databases in near real time.
In Exchange Server 2003, it's possible to replicate Exchange mailbox databases using third-party solutions. Many such solutions work very well but they can be expensive and they are all resource and bandwidth intensive. This isn't the fault of the solution vendors—there is a lot going on in an Exchange database, with hundreds or thousands of users constantly updating their mailboxes and thousands of new messages coming in and going out. Vendors who implement file or disk-based replication for Exchange have to replicate every single change, in correct order, made to both the database and transaction log files. This imposes severe limits on how far away the replicated databases can be located and it means that you need a very high-speed network between the locations. Replicating an Exchange mailbox database can reduce the number of mailboxes you can put on it by half or more because you often must scale down to allow the replication engine to keep up. There is currently no way for disk- or file-based replication solutions to be efficient about which changes are replicated because Exchange Server 2003 has no application-aware replication mechanism for mailbox databases. (Exchange has always had application-aware replication for public folders.)
In Exchange Server 2007, transaction logging will again be taken advantage of to provide additional functionality, this time for replication. You will be able to copy a full mailbox database once to a different disk or remote location and then update it by subsequently copying only the transaction logs. The transaction logs will be applied to the replica database soon after they've been copied, and you'll thus be able to fail over to the copy with minimal startup delay. This kind of incremental log file replay can't be done in Exchange Server 2003. A number of very difficult technical problems had to be solved to make it possible in Exchange Server 2007.
CCR will be fully integrated with Windows clustering. You'll have two copies of the mailbox database that are closely synchronized. During a controlled failover, you'll be able to completely synchronize the DataSets and fail back and forth between each as you wish. In case of a disaster that requires an uncontrolled failover, data loss will be minimal to zero (administrators can define individual tolerances for data loss and failover triggers). CCR will also be able to query SMTP bridgehead servers for messages delivered during the data loss window and can ask to have them redelivered. This message redelivery capability will not be available for LCR.
LCR will be available to every Exchange administrator, from Small Business Server installations with half a dozen users to the largest standalone Exchange server with thousands of mailboxes. LCR failovers must be manually initiated and you must set up replication again after a failover. Still, LCR can get you back in business in seconds or minutes, even after catastrophic loss of both databases and transaction log files on your primary disks.
There are several advantages to using LCR or CCR compared to file and disk-based replication methods. First, there is a dramatic reduction in the amount of data that must be transmitted across the network. Second, replication is simplified because rapid changes to the open database files no longer need to be replicated, and it is no longer necessary for transaction log and database changes to be perfectly synchronized and write-ordered as they are replicated. Third, LCR and CCR are more tolerant of network glitches and interruptions. Resynchronization even after prolonged interruptions in replication is automatic.
CCR and LCR replicate transaction log files that have been filled and closed. They will not replicate the current transaction log file. Thus, the data in that last log file is at risk in case of a disaster. To reduce the amount of data at risk, Exchange Server 2007 log files will be reduced from the current 5MB to 1MB.
Neither LCR nor CCR replace backups. You still need backups for point-in-time snapshots of data and for legal and archival purposes—and as a fallback strategy in case a large-scope disaster destroys the LCR or CCR database copies. But LCR and CCR make it much less likely that you'll ever have to restore a backup to recover from a disaster. They enable dramatically quicker data recovery in the most common disaster scenarios and will let you select backup cycles and strategies that focus on archival features instead of being driven by the need for fast restoration.
A nice side-effect of using LCR or CCR is that you can do your backups from the replica instead of from the primary copy of the database. While you have always been able to back up Exchange while it is online and serving users, Exchange administrators know that performing backups during peak hours has significant impact on performance. By backing up from replica, you remove nearly all of the I/O impact. As another bonus, you no longer have to balance your online database maintenance window with your backup window. Backup from replica will require using the VSS API, not the legacy streaming backup API. You won't be able to do it with NTBackup, but will need a third-party VSS solution.

Conclusion
This article has been a whirlwind tour through multiple strategies for making your Exchange deployment more reliable and available. The goal has been to give you an overview of how the system works and what to think about before things go horribly wrong.
Hopefully, you have never experienced an Exchange-related disaster. Things have been humming along for years, and so disaster planning for Exchange has always been a low priority. That's fine: doing good backups and calling Microsoft when things go wrong is a respectable default strategy for many organizations.
But if e-mail is mission critical, you have work to do. Because Exchange offers such a wide variety of recovery mechanisms, you need to understand in advance how to choose the right one and how to make all the parts work together smoothly.
Database recovery is the most critical "last mile" of disaster recovery. If you have preserved your databases, you can lose everything else and still recover completely. An entire Exchange organization can be rebuilt from scratch, if necessary. But end users will be after you with pitchforks if you lose their irreplaceable mailbox contents. So, if you remember only one thing, it should be: back up and protect your Exchange databases. If you do that right, you can recover from almost any other kind of disaster.

Michael Lee has worked on Exchange Server at Microsoft for eight years, both in technical support and, for the last few years, in the Exchange product group. He is currently a Program Manager with the Customer Experience Team.
© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


 Disaster Recovery: Simplify File Recovery with Data Protection Manager
Disaster Recovery Microsoft IT Showcase and Laura Euler

Simplify File Recovery with Data Protection Manager
At a Glance:
  • Using DPM for backup at Microsoft
  • The DPM architecture
  • End-user data recovery
  • Configuring backup and recovery

Most large companies use magnetic tape for backing up data. Tape is cheap and useful for long-term retention and off-site storage, but it can be inefficient and unreliable. It’s physically fragile and
vulnerable to misalignments. It also takes a long time to back up data to a tape. Even when data has been backed up successfully, tapes are subject to loss, damage, or prolonged delivery times due to off-site storage.
Some industry analysts estimate that more than 40 percent of companies have had restorations fail because the data was not correctly written to tape, was corrupted, or was in other ways unusable. Verification procedures are often necessary to ensure that data has been properly backed up, but that adds more time and overhead.
Is there a better way? Hard disk space gets cheaper all the time, so it’s now more cost-effective to back up on disk and avoid tape backups altogether. Disks aren’t subject to the same pitfalls as tapes. They aren’t as fragile, they can be overwritten more easily, and they’re faster for accessing data. And management tools can make the task of keeping track of your data easier.
Microsoft® Data Protection Manager (DPM) 2006 is a server software application that optimizes disk-based backup and restoration. The DPM server routinely synchronizes with your production servers, capturing only the changes. This replication occurs at the byte level; instead of replicating an entire file when a change occurs, DPM replicates only the bytes that actually change within each file.
A test by the Microsoft Data Protection Services Group using a beta release demonstrated that DPM could protect a branch office with 300GB of data to back up every day through a single nightly synchronization session that lasted only 10 minutes. This session replaced the eight-hour tape backup session formerly required.
For users, DPM offers instant recoverability. If a user’s data becomes overwritten or lost, a backup copy is available without the need to request a tape. An employee can use Windows® Explorer or any Microsoft Office 2003 (or higher) application to right-click the file and initiate the restoration. If a catastrophic server failure occurs, the DPM image of the data can be set to read-only mode and shared over the network to provide users access to their data while the server is rebuilt.

Microsoft Case Study
With about 150 sites, including 115 branch offices around the world, Microsoft needed a backup and restoration solution that would enable it to centrally manage all remote locations. A big part of centrally managing support for remote locations is having the ability to restore data without requiring local personnel to pull a tape from a library and mount the media for restoration. Loading tape cannot be done from a central location. And a WAN-based solution didn’t work, either. It combined the unreliability of a tape backup with the slow transmission rates of the WAN.
Microsoft also wanted to reduce the cost of maintenance and repair support, as well as minimize backup errors. The monitoring tools used at Microsoft to verify tape-based backups identified about 16,000 errors each month for more than 5,000 servers. About 14,000 could be resolved via automation, but another 2,000 errors had to be addressed. But with at least 150 different error codes, resolving each was time consuming.
Yet another consideration was the size of the Microsoft datacenter servers. Some of the these servers contain so much information that the Microsoft Data Protection Services Group found that tape was simply unable to protect all the information.
The solution was DPM. Microsoft IT configured its DPM servers to store between 14 and 21 days of backup data in the form of easily accessible snapshots—dedicating 180 terabytes of storage connected to the DPM servers to cover all 115 branch offices. Tape backups for long-term retention and off-site backup are made from the DPM servers, completely freeing the branch offices from the expense of managing tapes locally. The 12 DPM servers replaced 115 tape libraries, media servers, and associated infrastructure at these sites.

DPM Architecture
An architectural overview of a typical DPM installation can be found in Figure 1. The DPM server software is installed on a dedicated server running Windows Server™ 2003 Service Pack 1 (SP1) or Microsoft Windows Storage Server 2003 SP1. Windows Server 2003 R2 also supports DPM. The DPM setup process also installs components of SQL Server 2000 (the DPM product includes a restricted license of SQL Server for use only with DPM).
Figure 1 Typical DPM Installation
Figure 1 Typical DPM Installation
An Active Directory® service domain is required for the discovery of servers and to maintain the security settings of files and folders through access control lists. The Active Directory schema also holds the configuration settings required for the user recovery client to retrieve shadow copies from the DPM server. The DPM server must be in the same domain as the servers being protected.
The DPM agent software is installed on each protected server; it captures and logs all the changes made to the file system. These servers may run Windows 2000 (SP4 with the update rollup), Windows Server 2003, Windows Storage Server 2003, Windows Server 2003 R2, or Windows Storage Server 2003 R2. The agent is installed from within the DPM administrator console.
The clients run Windows XP and Windows Server 2003. The Previous Versions Client and Volume Shadow Copy Service (VSS) on these systems enable users to access and recover previous versions of their files. (Note that enabling end-user recoveries via DPM requires an update for computers running Windows XP SP2 and Windows Server 2003 SP1. For more information, see "An update is available to optimize the way that the Shadow Copy Client accesses shadow copies in Windows Server 2003 and in Windows XP".)
You might have noticed the final tape backup solution in Figure 1. This last step is optional, but for off-site archiving or long-term retention (especially for legal and financial purposes), tape is still the most cost-effective medium. However, for fast backups and restorations of recent data, it’s more efficient to restore directly from the DPM disk.
Synchronization is the process by which DPM transfers only the changes made on file servers to the DPM server and applies those changes to the replica of protected data. DPM updates data at the byte level within protected files. File operations such as renaming, deleting, and creating also are replicated for protected files.
DPM synchronization is asynchronous, meaning that synchronization occurs without blocking disk I/O on the protected objects. Changes are stored in the agent synchronization log and then replicated across the network to the DPM server, where they are stored in the transfer log. Data in the transfer log is later used to construct complete replicas that can be used to create a shadow copy of the protected data.
The Windows Server 2003 VSS creates consistent point-in-time copies of data known as shadow copies. After the DPM agent has synchronized the changed data with the DPM server, VSS creates replicas within the DPM server. Users can then browse and recover copies of deleted or corrupted files from various points in time. Eventually old shadow copies are deleted—as soon as the size of all shadow copies reaches either a configurable maximum or 64 shadow copies per volume (whichever occurs first).
Prior to VSS, there was no standard way to produce uncorrupted snapshots of a volume. You have to repair corruptions due to interrupted writes using tools such as Chkdsk.exe. VSS prevents incomplete writes by enabling applications to flush partially committed data from memory.

End-User Recovery
Allowing your users to selectively restore their own data improves productivity and satisfaction. Instead of being dependent upon someone from IT to locate and mount the appropriate backup tape, the user just navigates to the DPM file share and downloads the lost information. End-user recovery also lowers operation costs since helpdesk and administrators aren’t involved. Various industry studies have found that more than 90 percent of all tape restores are for single files, and nearly the same percentage of cases are for files that are less than 14 days old. So it makes sense to let your users recover their own data.
End users can recover files by browsing through shadow copies on the DPM server, either by using Windows Explorer or by using the Recover Previous Version command on the Microsoft Office 2003 Tools menu (see Figure 2).
Figure 2 Folder Recovery
Figure 2 Folder Recovery

Backup Verification
Because DPM is disk based, it can take advantage of a redundant array of independent disks (RAID) for protection—unlike tape, with the potential to be a single point of failure. IT administrators can easily browse through the server running DPM to confirm that the shadow copy has been made, and they can check and verify the backup online. Being able to confirm the redundancy of data without continually restoring from tape may also benefit auditors doing compliance checks in regard to business continuity or disaster recovery planning.
DPM monitors its backup jobs to ensure that they complete without error. If an error is detected, DPM has a two-stage error-correction process. First, DPM automatically validates the replica against the production server to ensure that the replication is consistent and has occurred as planned. If inconsistencies between a data source and its replica are found, the fix-up activity resends the object or objects from the data source to the replica.
With tape-based systems, about 40 percent of the IT staff’s time is spent monitoring and correcting backup operations. DPM eliminates this chore because it automatically validates and reworks the replica to help ensure consistency with the production server.

Scheduling Backups
You can deploy DPM to perform a nightly synchronization and shadow copy; you can also schedule nearly continuous protection. The scheduling engine supports a wide range of protection options and frequencies.
In its default settings, DPM provides hourly synchronizations so that even complete server failure results in less than an hour of data loss. With shadow copy scheduled at regular points throughout the day, users can roll back to a previous copy of data that is less than two hours old—instead of being limited to the previous night’s tape. Other organizations might configure for less frequent synchronizations and perhaps only daily shadow copies. With 64 instances per volume, a once-a-day schedule would enable more than two months of recovery capability from disk.
DPM essentially eliminates having to schedule backup windows because it logs and replicates byte-level changes to the files on the production servers. This also makes backups more efficient. Because DPM captures changes as they happen instead of copying entire files, DPM backups place less load on production servers than conventional backup tools that have to copy entire files if even a single byte changes. (Note: the DPM administrator console provides a throttling feature that enables you to define the maximum amount of total bandwidth that the backup procedure can use.) Collapsing the backup window from hours to minutes also lets you use fewer servers than you’d need using slower tape backups.
DPM also provides an alternative to tape backup for servers that hold so many millions of files or terabytes of data that attempting to use a tape-based backup solution would not be practical. Full tape backups take more than 24 hours on such systems, and they are often unsuccessful due to the millions of files and folders that need to be cataloged. On these systems, even incremental backups are problematic, due simply to the sheer volume of files.
The IT department at Microsoft found that the tape-based solutions it tried with a 10-million-file server failed consistently because crawling through that directory structure took so long. They also found that as the backup length increases, so do the odds of encountering a drive failure or some other issue that causes the backup to fail. Because DPM captures the data changes as they occur, no crawling of the directory structure is needed. Using this approach, Microsoft was able to successfully protect these very large servers within the already existing backup window.

Selecting the Data
After you select the data you want to protect, you can plan how to collect the data in protection groups. A protection group is a set of volumes, folders, or shares under the same protection policy. Items included are called members. The protection policy contains both the synchronization schedule and the shadow-copy schedule.
The key consideration in creating protection groups is tolerance for data loss. Typically, data with a relatively low loss tolerance is in one protection group, data with a relatively high loss tolerance in another, and medium loss-tolerance data in another. A given protection group may contain members from different volumes and servers, although each protected object can be associated with only one protection group, and only one protection group can protect data sources on any single volume.
Although the complete data of a server can be easily restored through DPM, special steps are required to enable DPM to support recovery that includes restoring the operating system and system-state data. To accomplish this, DPM recommends that you back up the protected server using the backup feature of that server’s OS (meaning the backup utility included with Windows 2000 Server or Windows Server 2003).
You can use these backups to restore the server to a bootable state; store these backups on a volume that is added to a protection group on the DPM server. When you require a recovery, retrieve the system state data from the DPM server and write the data to restore media. You can then use the media with the appropriate backup tool to restore the system to a bootable state. Then you can restore protected data to the server using the normal DPM workflow.

Allocating Disk Space
Obviously, the amount of disk space that an organization allocates on the DPM server will directly affect how extensive its backup history will be. The set of available storage for replicas and shadow copies is known as the storage pool. DPM enables an organization to distribute a single server’s storage pool across multiple disks, adding more space to the pool when necessary.
The default DPM cumulative space allocation for a given protected volume includes the following:
  • A replica allocation of the smaller of 1.5 times the space used on the protected volume or the total volume capacity (with a minimum of 1.5GB). For a 10GB volume that is 40 percent full, this allocation equals 6GB.
  • A shadow-copy allocation of 20 percent of the size of the replica allocation (with a minimum of 550MB).
  • A transfer log allocation of 1.4 times the synchronization log space allocated on the protected server (with a minimum of 700MB).
In addition, DPM includes a calculator that can more precisely estimate the amount of space required to protect a given volume of data, and you can always manually specify space allocations either when creating the protection group or afterward.
Figure 3 shows actual DPM server utilization on the Microsoft corporate network. As you can see, the DPM server that protects the most branch offices is DPM #2, which is located in the Redmond datacenter and uses 1.55 terabytes of its allocated 3.69 terabytes of storage. The 12 datacenter DPM servers protect a total of 149 servers and hold 20.79 terabytes of data; the smallest DPM server has a 3-terabyte capacity, and the largest has a 10-terabyte capacity.
Figure 3 Group Utilization by DPM Server (Close Figure 3)

Data Center DPM Server | Branch Servers Protected | Disk Capacity (in terabytes) | Disk Allocated (in terabytes) | Disk Used (in terabytes)
Dublin DPM #1 | 9 | 3.07 | 1.90 | 0.97
Dublin DPM #2 | 11 | 6.14 | 4.52 | 2.11
Dublin DPM #3 | 13 | 7.16 | 6.47 | 3.37
Dublin DPM #4 | 6 | 5.11 | 4.90 | 2.54
Dublin DPM #5 | 8 | 6.14 | 2.89 | 2.03
Redmond DPM #1 | 16 | 4.09 | 3.16 | 1.55
Redmond DPM #2 | 22 | 8.18 | 3.69 | 1.55
Redmond DPM #3 | 21 | 10.23 | 6.19 | 2.90
Redmond DPM #4 | 20 | 5.11 | 3.57 | 1.56
Singapore DPM #1 | 6 | 3.99 | 2.22 | 0.92
Singapore DPM #2 | 6 | 2.99 | 1.78 | 0.90
Singapore DPM #3 | 11 | 2.99 | 1.62 | 0.86
Totals: | 149 | 63.76 | 41.98 | 20.79
Figure 4 shows the average number of files and the amount of total data stored on branch servers. Some protected servers hold more than 6 million files and have an average disk capacity of 218GB.
Figure 4 Average Branch Office Server Workload (Close Figure 4)

Metric | Per Volume | Per Server | Per Domain
Count | 660 | 213 | 7
Average number of files | 110,093 | 341,134 | 10,380,222
Maximum number of files | 1,672,039 | 6,072,431 | 36,244,679
Average capacity in GB | 70 | 218 | 6,629
Maximum capacity in GB | 417 | 1,452 | 19,033
Analyzing the change workload—how much data on a protected volume or server actually changes between backups—underscores the value of DPM, which is based upon copying only data blocks that have changed. Whereas Figure 4 showed that the average protected server holds 218GB of data, Figure 5 shows that the average branch office server has a data change of only about 1.8GB per backup session.
Figure 5 Average Branch Office Change Workload per Server (Close Figure 5)

Changes | Per Volume | Per Server | Per Domain
Average change in GB | 0.6 | 1.8 | 54
Longest changes in GB | 31 | 32 | 201
Heavy day changes in GB | 3.4 | 11 | 323
Maximum changes in GB | 189 | 194 | 1,204

Conclusion
DPM is a handy solution for organizations that need advanced data protection. DPM enables users to recover their own documents quickly and easily. Recoveries that once took hours, while IT staff located and mounted tapes from a library, can now be achieved in seconds. Because tape backups can easily be made from the DPM server, organizations can still use tapes for long-term and offsite storage, while eliminating the long backup windows that tape requires.
DPM is especially useful if you need to manage backup and restoration services for branch offices and other remote locations from a high-security centralized datacenter, while at the same time reducing the cost of local administration.
Best Practices for DPM
In addition to the suggestions made in this article, if you plan to deploy DPM, your organization can benefit from a number of specific best practices, including:
  • Checking deployment requirements.
  • Scheduling strategies for DPM backups.
  • Integrating with Microsoft Operations Manager (MOM).

Deployment Requirements
For an organization to deploy a DPM server, the server must be running the Standard or Enterprise Edition version of either Microsoft Windows Server 2003 or Windows Storage Server 2003 SP1 or later.
In addition, your server must be a member of the same Active Directory domain as the file servers it protects and it should be an ordinary single-purpose server. The DPM server cannot be an Active Directory domain controller. Your server should be equipped with at least one logical volume (which may be made of multiple physical disks) and at least one additional unused disk.
Every file server that DPM will protect must have the DPM agent installed on it and must meet the following requirements.
First, the server must be running only one of the following operating systems: Windows Server 2003 SP1 or later; Windows Storage Server 2003 SP1 or later; or Windows 2000 Server with both SP4 and the Windows 2000 update rollup installed. Second, the server must be a member of the same Active Directory domain as the DPM server that protects it.
DPM can be used to protect standalone file servers today. Protecting clustered file servers will be supported with SP1 for DPM, which is scheduled to release in 2006.

Scheduling Strategies
Scheduling DPM backups involves a tradeoff between how often you want to back up with how many days of backed-up data to retain, because there is a 64-image limit for data retention. Scheduling two jobs per day, for example, will allow storing about 32 days’ worth of protection. More days can be saved if data has not changed: DPM does not create a shadow copy if there have been no changes.
With 64 images available, the most common rotations for a five-day business week would provide:
  • One per business day for over three months of daily restores available from disk
  • Two per business day would give six weeks with recovery points at the beginning of each business day and noon—resulting in a four-hour RPO (recovery point)
  • Three per business day—perhaps at 9:00 A.M., 12:00 P.M., and 3:00 P.M. would provide a month of backups
And when roll-back becomes the primary goal, perhaps the most common method is to synchronize hourly and initiate shadow copies at every even business hour (8:00 A.M., 10:00 A.M., 12:00 P.M., 2:00 P.M., and 4:00 P.M.). This gives two outcomes: if the primary server were to have a catastrophic failure, the DPM server would have data that is no more than one hour old, available for restoration. If data was deleted or overwritten, the DPM server would have a version no more than two hours old, available for easy and predictable restoration back to the last even hour.

Integrating with MOM
DPM already provides reporting through SQL Reporting Services, which comes included with DPM, as well as notification via e-mail, alerts through the DPM interface, Event Log and Performance Monitor statistics. But for more of an enterprise perspective, the DPM management pack for MOM 2005 allows you to centrally monitor data protection. You can check state, health, and performance of multiple DPM servers from a MOM management server. Health and status information can be viewed and analyzed in the context of the health and status of the network and other service.
DPM already provides reporting through SQL Reporting Services, which comes included with DPM, as well as notification via e-mail, alerts through the DPM interface, Event Log and Performance Monitor statistics. But for more of an enterprise perspective, the DPM management pack for MOM 2005 allows you to centrally monitor data protection. You can check state, health, and performance of multiple DPM servers from a MOM management server. Health and status information can be viewed and analyzed in the context of the health and status of the network and other service.
For more information about preparing to deploy DPM, see "Data Protection Manager Technical Overview" at "Data Protection Manager Technical Overview".

Microsoft IT Showcase presents an inside view of the Microsoft IT process for developing, deploying, and managing Microsoft solutions—from Microsoft IT professionals to IT professionals—peer to peer. The resources they provide reveal how Microsoft uses technology to solve specific business problems. Find out more at Microsoft IT: Showcase.
Laura Euler runs the universe from an underground fortress in an undisclosed location. You can contact her at lauraeu@speakeasy.net.


 Disaster Recovery: Plan Ahead to Save Your Web
Disaster Recovery Chris Adams

Plan Ahead to Save Your Web
At a Glance:
  • Back up your forgotten data
  • Back up the metabase
  • Don’t forget your content
  • Disaster Recovery Automation 101

I am asked at times about strategies for recovering from disasters. My response is always the same: if software or hardware problems put your system in an unusable state, knowing is half the battle.
I started at Microsoft as a systems engineer for Product Support Services (PSS) and in that role I helped our customers many times in disaster recovery situations. It was always intense and pressure packed as the engineer struggled through this backup tape or that backup tape while all along not knowing whether the lost data was even recoverable. Preparation is the key to handling these tough situations. I’ve succeeded sometimes by helping the customers get their systems restored to a suitable state, but I’ve also failed a few times. This article will help you prepare for these tough situations in hopes that if that day comes, you will be ready to succeed. As Vince Lombardi once said, "The only place Success comes before Work is in the dictionary." This is never truer than in disaster situations.

Back Up the Forgotten Data
When restoring from a disaster, the most obvious items are usually the most costly. IIS 6.0 can rely heavily on Windows® system-level features that many Web administrators fail to recognize, and thus often miss in their backup strategy, so let’s discuss them. IIS 6.0 doesn’t require that you use all of these features, so do your homework to determine which ones you are using and need to back up.
HTTP.sys was introduced in Windows Server™ 2003 and is known in the Web world as being the kernel driver that IIS 6.0 relies on for connections, caching, and HTTP compliance. HTTP.sys stores key data in the registry as shown in Figure 1.
Figure 1 HTTP.sys Configuration in the Registry
Figure 1 HTTP.sys Configuration in the Registry
Make sure that you have a current version of the registry backed up in case of a disaster. This is important when you have built a custom configuration for HTTP.sys and you need to ensure you have this key backed up. You should be familiar with the default HTTP.sys configuration so you can evaluate whether your servers are using anything custom.
The most common use for a custom configuration is when you are running multiple Web servers on your system. A lot of server software takes advantage of the HTTP protocol, though they all must honor one rule: a service can only own a single IP:Port:HostHeader endpoint per server. When you want to run more than one Web site, you will have to prevent HTTP.sys from listening on each IP address configured on the server, and you do this through a custom configuration setting. The supported method of modifying this behavior is to use Httpcfg.exe on the server to add a particular IP:Port binding to the IP Listen List.
How you modify this behavior isn’t as important as ensuring you have the resulting configuration information available when you restore after a disaster. The easiest way to back up this key is to use Windows Backup (NTBackup), performing a System State backup that captures a copy of the Windows registry, as shown in Figure 2. The following steps will help you do this manually, but it’s a good idea to schedule this as a task to be run as often as your environment calls for.
Figure 2 Backing Up System State
Figure 2 Backing Up System State
  1. Choose Start | Programs | Accessories | System Tools | Backup.
  2. Click the Backup tab.
  3. Click to select the System State checkbox. (All of the components to be backed up are listed in the right pane. You cannot individually select each item.)
During the system state backup, you must choose to back up the \Winnt\Sysvol folder. You must also select this option during the restore operation to have a working \Sysvol folder after the recovery. For more information, please review the Microsoft® Knowledge Base article "How to Use the Backup Program to Back Up and Restore the System State in Windows 2000".
Another item that’s often forgotten is SSL certificates. You might forget about the certificates because it’s commonly believed that backing up the IIS metabase captures these for you. I am sorry to say this isn’t the case. But, unlike Windows 2000 Server, Windows Server 2003 and IIS 6.0 offer a handy COM+ interface that allows you to manage certificates on your server.
A script that is part of the IIS 6.0 Resource Kit Tools called IIsCertDeploy.vbs can grab all your SSL certificates and export them to a .pfx file. Learn more about the IIS 6.0 Resource Kit Tools and download the resource kit at IIS 6.0 Resource Kit Tools (IIS 6.0).
Assuming you choose the default installation settings, IIsCertDeploy.vbs will be located in %programfiles%\IIS Resources\IIsCertDeploy. You will need to either add this to your path or copy the script to a more convenient location. A certificate might exist at a top level for all sites that are defined in the IIS configuration file Metabase.xml (I will talk about backup of this file a bit later). You will need to call IIsCertDeploy.vbs for each site where a certificate is installed.
iiscertdeploy.vbs -e d:\webbackups\certs\site1.pfx
 -i w3svc/1 -p site1pwd -q On
This will produce a file with the extension .pfx that you can store on disk and safely restore in the case of a disaster. Remember, certificates are not volatile data; usually you don’t have to back these up each time. Instead, you can capture them only when you have recently added SSL to a site or renewed your certificate.
Certificate backups will require you to fully export the certificate, including the private key. If you fail to export the private key, your certificate will be unusable in the case of disaster. I mention this because you might choose to back up using the Certificate Manager MMC snap-in, as described in the Microsoft Knowledge Base article "How to Back Up a Server Certificate in Internet Information Services 5.0", and fail to export the private key. And be careful when exporting your certificates with your private key: if lost or stolen, these certificates are fully usable.

Back Up the Metabase
You were probably worried when I didn’t start with backing up the IIS configuration file, more commonly known as the metabase. The metabase is composed of the raw IIS configuration file Metabase.xml as well as the IIS schema stored in Mbschema.xml. These files are located in the %windir%\System32\Inetsrv directory and are protected by administrator-only privileges.
From my PSS days, I remember how to restore IIS 4.0 after a disaster occurred.
  1. Install Windows NT® 4.0 SP 6a.
  2. Install IIS 4.0.
  3. Reinstall Windows NT 4.0 SP 6a.
  4. Rebuild the metabase.
  5. Install the latest FrontPage® Server Extensions (if used).
  6. Install the latest Microsoft Data Access Components (MDAC).
Rinse and repeat if running in a Web farm environment. All kidding aside, we have come a long way. During the old days, if you failed to do this in the correct order, you would often get mismatched DLLs leading to failures when you attempted to get your IIS services to start. The reason this was so painful is that the installer failed to include a copy of the default metabase upon completing the install. It was only with blood, sweat, and tears that this core problem was resolved. IIS 5.0 offered the first opportunity to back up the metabase using the Internet Information Services Manager. Many admins didn’t realize this feature had been added and thus did not take advantage of it. If you did, pat yourself on the back.
For IIS 6.0, one key user request was to make this functionality support automation without you having to write a single line of code (or script). Now you have a mechanism to back up your server easily and effectively whether you have one IIS Web server or many. IIS 6.0 shipped a command-line script called iisback.vbs, aimed at making the backup and restore process of the IIS 6.0 metabase much easier. This tool, located in the system path by default (%windir%\System32), will back up the IIS configuration locally or remotely, making it very useful in large Web farms or on single-server installations. The following sample provides you with a quick and easy command to back up your IIS 6.0 server’s metabase configuration in a single server scenario:
iisback /backup /b MyMetaBackup /e BackupPass /overwrite
Why is all this necessary? Why can’t you just copy the file itself? Or, better yet, why not use the backup your backup program already has backed up? You can’t do this because the IIS configuration includes machine-specific settings like IP:Port bindings, paths to custom ISAPI filters and extensions, log file and custom error paths, and even credentials such as the anonymous user account. Upon installation of Windows Server 2003, you make some fundamental decisions such as the IP address of the server, the system directory (%windir%), and even down to the Web root path if you are automating the installation.
For example, the installation of IIS automatically creates two accounts on your local server—IWAM_MachineName and IUSR_MachineName—where MachineName is replaced with the server’s name. This fundamentally breaks your ability in IIS 6.0 to copy the file from server to server with no loss of service.
This machine-specific information isn’t detrimental to the backup process but creates more work for you. IIS 6.0 offers you the ability to back up with or without a password, just to ensure you have a valid metabase configuration in case of a disaster. If you choose to not use a password, IIS does not remove the machine-specific data that is stored in the metabase encrypted. What’s used to encrypt it? The data is stored using the Windows APIs for encrypting data and is based on the machine key of the operating system. In case you didn’t know, each installation of Windows has a unique machine key that isn’t reproducible unless you have a full backup of the system. Thus, the IIS 6.0 configuration can’t be decrypted and will cause services to fail if the machine key isn’t correct or is unavailable. If you want a backup just for the sake of having a backup (you’re not as concerned, for whatever reason, with disaster recovery) then select the Do not use Password option.
If, on the other hand, you do plan to be prepared in case of loss of operating system, select the password option. The primary difference here is what is used to protect access to the sensitive or otherwise encrypted data. Instead of using the operating system’s machine key, the system uses the password you provided.
Is this helpful? You bet it is. Now you can have a generic backup of your IIS 6.0 server’s configuration, supply key data into that file when you need to, and off you go with the restore. It isn’t the most straightforward backup-and-restore approach but it works. Future releases of IIS should improve this process and hopefully make your life that much easier.

Don’t Forget Your Content
The most obvious aspect of all backups is the backup of files and directories. Nothing changes here for IIS 6.0. Without your content, your users will probably be upset. I often find, though, that many people don’t really understand the full range of items that are considered content. It goes way beyond the images or the .css files, and also into your business logic. Developers today build very robust and powerful applications and they each do it differently.
Prior to Microsoft releasing ASP.NET, many developers built applications using the Microsoft Windows DNA architecture. This architecture advised developers to separate the presentation layer (browser), business logic (COM+), and data (SQL Server™). These designs are still prevalent in the community, but what is most important to you now is identifying the business-logic layer. Do you have any COM+ applications stored on the server or servers?
Poke around and you might succeed in locating these applications. Start by using the Component Services MMC snap-in and navigating into the applications for this particular system (see Figure 3). Unfortunately, in many cases you will have to depend on your developers to help you find where they installed these bits. But be prepared, because without this business logic your site might be restored, your services running, but your clients and management will still be unhappy. The nice thing is that you have the ability to back these up as well if you do locate them.
Figure 3 Apps in the Component Services Snap-In
Figure 3 Apps in the Component Services Snap-In
Using the same Component Services MMC snap-in, you can export the COM+ application to an .msi installer that can be installed on other servers if necessary. Double-click on Component Services, then on Computers. Click to expand the My Computer node and then the COM+ Applications node. Now locate the app that represents your business logic, right-click and choose Export. In the Welcome to COM+ Application Export Wizard, click Next, then enter a full path for the export file for the app, along with any other pertinent options (discuss with your developer) and click Next. Click Finish to close the COM+ Application Export Wizard.
This is very similar to SSL Certificates in not needing massive automation, as the data isn’t often volatile enough to warrant it. In some environments, business logic is added, modified, or changed often; in this case you would be better off using the APIs provided to do this automatically. For most Web administrators, though, the procedure I described should suffice.
At the end of the day, you now have your files, directories, and your business logic all packaged up and ready for deployment when the time comes (if it does).

Disaster Recovery Automation 101
I have hopefully cleared up the "what." Now it is time to close down on the final key—how. How can you be efficient and protect yourself with the least amount of effort? Figure 4 shows the simplest of batch files I created to help you. The vast majority of the work is up to you: find the data, understand how to grab the data, and centralize it in a safe place. Centralize as much as possible and always know where this central location is. When I backed up my system while writing this article, I created a single directory on the server called WebBackup where I can quickly and easily tell my backup program to put the files during the backup. This one directory will get my operating system components, IIS configuration, and also my content.
Figure 4 WebBackup Script (Close Figure 4)
REM Backup to Remote Share (WebBackup.bat)
REM =============================================
REM

REM Connect to Remote Backup Server
net use s: \\[server ip address\name]\WebBackups

REM Get SSL Certificate
cd \
cd %programfiles%\"iis resources"\iiscertdeploy
iiscertdeploy -e [pathtostorefile]\mySSLCert.pfx -p 1 -i w3svc/[SitePath] -q on
cd \

REM Backup IIS Metadata
iisback /backup /b MyMetaBackup /e BackupPass /overwrite
xcopy %windir%\system32\inetsrv\metaback\MyMeta*.* c:\WebBackups\metadata /y

REM Copy Content & Export App...
xcopy /o /x /e /h /y /c c:\webbackups\metadata\*.* s:\Metabase\
xcopy /o /x /e /h /y /c c:\webbackups\ContentAppDir\*.* s:\Content\
xcopy /o /x /e /h /y /c c:\webbackups\Certs\*.* s:\Certs\

REM Delete connection to Remote Backup Server
net use s: /delete /y
When you put all the data in one place, the restoration process becomes almost silly. All you have to do at this point is to use a few easy commands and everything is snapped right back into place. In Figure 5, I use a simple batch file that will restore of all my key data.
Figure 5 Restoring Your Data after a Disaster (Close Figure 5)
REM Restore from Remote Share (Recovery.bat)
REM =============================================
REM

REM Connect to Remote Deployment Server
net use s: \\[server ip address\name]\WebBackups

REM Restore Web site
xcopy /o /x /e /h /y /c s:\metabase\*.* %windir%\system32\inetsrv\metaback
iisback /restore /b [BackupName] /e [BackupPass]

REM Restore Content
xcopy /o /x /e /h /y /c s:\Content\*.* [SiteWebRoot] (i.e. c:\inetpub\wwwroot)

REM Restore SSL Certificates
REM Either add to server path or go directly to path for IIsCertDeploy.vbs
cd \
cd %programfiles%\"iis resources"\iiscertdeploy
xcopy /o /x /e /h /y /c s:\certs\ c:\webbackups\certs\
iiscertdeploy.vbs -c c:\webbackups\[pfxfilenamecreatedduringexport].pfx
    -i w3svc/[SitePath] -p 1 -q on

REM Delete Connection to Remote Backup Server
net use s: /delete /y

Wrap-Up
The result of this process is you sleeping more soundly while avoiding downtime for your customers. Web farms can be your saving grace, protecting you from having to work under the gun while customers can’t use your system. But executing a broad, inclusive backup and recovery strategy for IIS means you are prepared to succeed whether one system goes down or your entire server room is in an unusable state. All the essential data needed to get your Web sites back up will be ready at your fingertips.

Chris Adams is the Web Platform Supportability Lead at Microsoft. Chris is currently certified as a MCP, MCSA, and MCSE on Windows NT 4.0, Windows 2000, and Windows Server 2003. Reach him at chrisad@microsoft.com.


 Performance: Playing with Exchange in a Sandbox
Performance Greg Taylor and Simon Shepherd

Playing with Exchange in a Sandbox
At a Glance:
  • Calculating IOPS
  • Configuring JetStress
  • Gathering data with Exchange Profile Analyzer
  • Exchange Server 2003 Load Simulator
  • Exchange Server Stress and Performance tool

"Looks great! Now prove it works before we deploy it." If you’ve ever planned and pitched an Exchange Server solution, you’ve probably heard something like this. Many a messaging architect has quickly
responded that they can and will prove the design—throwing around names like LoadSim, JetStress, IOPS, Heavy Profile and Medusa for good measure. And then they proceed to either not run the tests properly or, worse, not run them at all. There’s a reason why that happens: these tools are not always easy to use!
Designing servers for use in anything but the smallest of Exchange deployments is complicated. Availability and reliability are key in Exchange design. So how can you be sure that the servers you have specified can cope with the load that will be placed on them if you haven’t been able to realistically test them over any period of time? Deploying into production should never be the time at which testing takes place!
This article focuses on validating and testing mailbox servers. These tend to rely on fast and reliable storage and also consume more memory the more users are serviced. We’ll discuss some important considerations when planning your tests and a number of handy tools to help you perform them. For more on the performance characteristics of Exchange Server, be sure to read "The Exchange Server 2003 Performance and Scalability Guide". And for some advice on determining the number and speed of processors you’ll need for your deployment, read the sidebar "Calculating Processor Power."

Simulating Real World Users
Sure, you can set up a test environment. But you need "users" to put the environment through its paces. How do you test and validate your designs under realistic conditions without recruiting a few thousand Outlook® users to log on to your test environment, send a few e-mails, and make a few appointments? Simple: you simulate the work. The problem is configuring these tools to represent your users. There are defaults, but how many default users do you have? If you don’t know what load your users place on the server or how many mails they typically send and receive, you can’t accurately simulate their activity.
This article will help you gather and calculate data for configuring the test tools beyond the defaults, so they better simulate your real users. We’ll look at how you can measure the impact users have on the disks of a server, which ultimately relies on how many messages each user sends and receives. We’ll discuss how you can calculate the number of inbound SMTP mails received each day. And we’ll show you how to use the new Exchange Profile Analyzer to look inside existing mailboxes to accurately see how users actually use the messaging system. Then we’ll move on to using these numbers for testing.
None of these calculations are an exact science. But by taking the right approach and properly interpreting the data, you should be able to profile and simulate your users’ impact with confidence, making your tests more accurate and reliable.

Measuring the Impact of Users
There are two ways to profile your users: one relies on actual measurement of existing data, the other relies on educated guesswork. If you already have Exchange, you can use the real data from that implementation to help you plan the new deployment. If you currently use a different messaging system, you’ll have to use a less accurate approach.
Real Data Approach There are several ways to measure performance and work load. Probably the best way to calculate load is to measure disk I/O operations per second, per mailbox (otherwise known as IOPS). This is a measure of the raw database usage. It’s also the key input parameter to JetStress (which we’ll discuss), so working this number out is essential to configuring the tool accurately.
The time of day at which you observe your data will significantly affect the results. For a fair representation of day-to-day usage, we recommend that you gather data during peak usage—Monday morning between 9 A.M. and 11 A.M., for example.
To measure IOPS per mailbox (IOPS/mailbox):
  • Select a production server with a typical user load.
  • Use the Performance Monitor tool (perfmon.msc) in order to monitor the Logical Disk\Disk Transfers/sec counter over the peak two hours of server activity on the database drives.
  • Calculate your current IOPS/mailbox using the following formula:
    IOPS/mailbox = (average disk transfer/sec)
/ (number of mailboxes)
For example, if your server averages 2,100 disk transfers per second over the two hour period, and it hosts 4,000 mailboxes, the calculation would be:
2100 / 4000 = 0.525 IOPS per Mailbox
In this case, users are causing Exchange to perform 2100 disk read/write operations per second. That sounds like a lot and it is. This is why Exchange solutions generally require a lot of disks—not for capacity, but for performance.
Knowing the number of mailboxes to divide by is important. Counting the number of mailboxes on a server should be simple, but often isn’t. Many companies have a large number of dormant mailboxes sitting on servers—for former employees, for example. Should you count these? It depends—if those mailboxes still receive mail, then yes. If not, you can rule them out.
We use the logical and not the physical disk counters—if the server is clustered and we use the physical disk counters, we would get the data for all the disk resources in all the resource groups, whether they are on an active node or not. There won’t be any data in them, but it unnecessarily adds to the amount of data being collected.
You can’t just get a number from your disk vendor for how many IOPS their disks can provide and divide that by the average per user to come up with the number of users you can support on that disk. It’s important to factor in disk latency, RAID penalty, the read/write ratio, and throughput when calculating storage requirements. These factors are outside the scope of this article, as they relate more to the design of the storage. For now we just need to see how many IOPS each user generates so we can use that as input for our testing tools. For more information, see How to Calculate Your Disk I/O Requirements.
Finger in the Air Approach Sometimes, measuring real data just isn’t an option. For instance, there may be so many other applications running on the Exchange Server that you can’t isolate the work Exchange is doing. In situations like this, you have to guess. But we can give you some guidance to help you make more accurate estimates.
There are four fairly tried and tested Exchange Server 2003 profiles that have equivalent IOPS numbers, based on Outlook/MAPI users. Figure 1 provides typical numbers for each of these profiles. It is important to note that IOPS will increase as mailbox size increases—we’ll come back to that later.
Figure 1 User Profiles and Usage Patterns (Close Figure 1)

User Type | Database Volume IOPS | Sent/Receive Per Day | Mailbox Size
Light | .5 | 20 sent/50 received | 50MB
Average | .75 | 30 sent/75 received | 100MB
Heavy | 1.0 | 40 sent/100 received | 200MB
Large | 1.5 | 60 sent/150 received | 500MB
By choosing one of these profiles and using the numbers shown, you may be able to estimate the IOPS for your environment. This number will be useful when configuring LoadSim and JetStress. When using an estimate, however, it is particularly important to test your solution thoroughly before going anywhere near production.

Measuring Mail Flow
You need to measure mail flow so you can configure the Exchange Server Stress and Performance tool (ESP) to send the right number of SMTP mails inbound to the test environment. Measure on the SMTP Gateway server that accepts mail coming in from the Internet. You don’t need to measure outbound mail, since LoadSim cannot be configured to send mail to recipients that are not other LoadSim users.
To measure mail flow:
  • Select an SMTP gateway server with a typical load.
  • Use the Performance Monitor tool to monitor the SMTP Server\Messages Received Total counter over the peak 2 hours of server activity.
  • If you are able to restart the SMTP service, do so; this will zero all the counters and make calculations easier.
  • Calculate messages per second as described in the following formula:
    messages per second =
(total number of messages) / (number of seconds measured)
For example, if your server receives 15,654 messages over the two hour period, the calculation would be:
15,654 / 7200 = 2.17 messages per second
The number you come up with will be used to configure the instances field in ESP.

More Factors to Consider
There are a few more factors to consider before you use these numbers to configure the testing tools.
Concurrency You don’t necessarily need to design a system for 100 percent concurrency. If you know for sure that only 10 percent of the users will ever be logged in at any one time, adjust your figures accordingly. Be sure to allow for fluctuations.
Increase in Usage The number of employees may go up, e-mail usage will likely increase, and the average size of e-mail attachments will probably increase over time. When designing your system, account for these changes.
Large Mailboxes Large mailboxes require more IOPS. The ratio is pretty constant. You can assume that when the mailbox size doubles the IOPS also double.
Remote Device Usage When devices like Smartphones and Blackberries are used to access mailboxes, they affect the IOPS (to varying degrees). You need to assess the effect that a growing mobile workforce will have on your deployment.

The Tools of the Trade
Now let’s take a look at the tools you use to simulate the workload. We’ll look at four, each of which performs a different task. These are outlined in Figure 2.
Figure 2 Simulation Tools (Close Figure 2)

Tool Name | What It Does | When You Would Use It
JetStress | Used to test the performance and stability of the disk subsystem. | Storage performance validation; storage reliability testing; end-to-end testing of storage components.
Exchange Profile Analyzer | Used to measure actual user profiles, which can be used as input to LoadSim. | LoadSim fine tuning; real-world data analysis.
Load Simulator 2003 (LoadSim) | A benchmarking tool used to test the response of servers to mail loads. | Comparison tests; scalability tests; benchmarking; MAPI load generation; end-to-end testing with third-party solutions.
Exchange Server Stress and Performance (ESP) | Used to test stress and performance of Internet protocols (SMTP, IMAP4, POP3, OWA, and so on). | Comparison tests; scalability tests; benchmarking; POP3, IMAP4, SMTP, DAV (OWA) load generation; end-to-end testing with third-party solutions.
The Exchange Server 2003 Load Simulator (or LoadSim) is probably the most well known and commonly used of these tools. You can use it to simulate MAPI clients. JetStress, which is also well known and has been updated recently to add a GUI, lets you test storage components. The Exchange Server Stress and Performance tool is used to send SMTP e-mails. It’s less commonly employed, probably because it’s very difficult to configure. The Exchange Profile Analyzer (EPA) is not used for simulating workload. It gathers data about your real environment, which you can then use to customize LoadSim for more accurate testing.
This article will not give you step-by-step instructions on utilizing these tools (you can get that from the documentation). Instead, we’re going to help you configure the tools for better results and introduce some important points you might not have considered.

Configuring JetStress
JetStress is used solely to test the storage subsystem of your Exchange servers. Typically, this is the first test you will run. Before getting started, you need to hear some warnings. Do not use JetStress on a production server, do not run it on a machine that already has Exchange running on it, and do not use the other testing tools at the same time. JetStress isn’t interested in users, mail flow, or any of those things—it just wants to beat up on the disks in the same way Exchange does, so test your system using JetStress, remove it, and then move on to the other tools.
Now let’s look at what you should do. Here are 15 of the most important points you need to know for accurately configuring JetStress. If you bear all these points in mind and run the tests several times you should get consistent, reliable results.
  • Download the latest version. There have been several important updates to JetStress over the last year or so. Most noticeably a UI was added!
  • Read the documentation that comes with the tool—it is packed full of valuable information.
  • Disable Storage Area Network (SAN) data replication while tuning JetStress. Once you have reached a steady state you can reenable SAN replication if required.
  • If other applications share the same disks (not recommended, by the way) or SAN, then use a tool like Iometer (www.iometer.org) to simulate the additional load.
  • Make sure the versions of the ese* files you use are the same Service Pack version as those you will use in production. Some significant advances were made in SP2 for Exchange 2003.
  • Create databases equivalent in size to those you will use in production. Ten percent of the size of those planned for production is no longer recommended because disk latency increases as the disk fills up.
  • The number of disks field refers to the number of logical disks (mount points included) used in the storage group. One logical drive with five databases on it is one disk. If each database gets its own drive letter, the number of disks would be five.
  • Databases and logs must go on separate disks. If you have to run them on the same disk, check the NAS storage checkbox to bypass the drive logic check or use the command-line version of the tool.
  • The automatic tuning feature works well and should always be used, unless you really need to fine-tune the process or if JetStress reports that auto tuning won’t work in your configuration.
  • Don’t use the automatic tuning on several machines using shared storage at the same time.
  • If you have to manually configure the tool, start with the back of an envelope calculation of 1 thread being able to deliver 100 IOPS across 4 storage groups each with 1 database. Then you can build from there.
  • When manually configuring parameters like the number of threads, run several iterations of the tests in order to get those numbers right.
  • If a stress test will last 24 hours, you need to account for database growth and initialize your databases at approximately 50 percent of the actual limit you would use in production.
  • The report at the end of a test run will provide a summary. You can validate the conclusions yourself by using the Performance console to look at the .blg file produced during the test run.
  • Finally, use the backup facility (built into the latest versions). You will run the test a few times to get your results and using backups will save you a significant amount of time.
Since JetStress is all about disks, the counters you need to analyze are all disk related. Figure 3 outlines the performance counters you should be most concerned with and the values you should anticipate. Remember that averages can be deceptive. If the Performance Monitor tool gives you an average number, don’t just use it. Look at the chart and take a visual average, asking whether it agrees with the calculated average. Eliminate peaks and valleys from your calculations, instead looking at trends. Adjust the timeline to cut out the early test peaks to see how that affects the system-generated numbers.
Figure 3 Key Performance Counters (Close Figure 3)

Object | Counter | Instance | Definition | Expected Value
Logical Disk | Disk Transfers/sec | All | The amount of disk I/O per second. | Total of Reads/Writes—this is the actual IOPS figure.
 | Disk Reads/sec | All | The rate of read operations on the disk. | No expected value since this varies widely based on environmental factors.
 | Disk Writes/sec | All | The rate of write operations on the disk. | No expected value since this varies widely based on environmental factors.
 | Avg Disk sec/read | All | The latency associated with each read from disk. | At all times, this value should remain below 20ms for the database drives and 10ms for log file drives.
 | Avg Disk sec/write | All | The latency associated with each write to disk. | This value should be below 10ms for the database and log file drives at all times.

Using Exchange Profile Analyzer
LoadSim generates workload against Exchange to simulate Outlook client activities. The difficulty is in accurately specifying that workload. LoadSim provides several predefined profiles—Medium and Heavy being the recommended profiles for pre-deployment scalability testing. The profiles represent collections of Outlook tasks, such as reading and sending e-mail, creating meetings, and browsing public folders. To get the most out of LoadSim, you need to customize the workload profiles. This is where the EPA comes in.
The EPA generates statistical usage data from Exchange by analyzing existing mailboxes. The results can then be interpreted and used to create a custom LoadSim workload profile.
We recommend that you run EPA remotely and use it to profile Exchange 2000 SP3 or Exchange 2003. The tool is intended for use in production environments, which introduces some potential concerns regarding security and system performance.
EPA accesses mailboxes directly using WebDAV. EPA requires access to every mailbox it analyzes, and this is explicitly denied by default to members of the Enterprise and Domain Administrators groups. Therefore, you must create an account for EPA that is not a member of either of these groups and grant the account Exchange View Only permissions at the Administrative Group or Organization level using Exchange System Manager (ESM). You also need to grant Receive As and Send As permissions to the account on each server with mailboxes being profiled. No changes are made to the mailboxes being profiled. Unread items, for example, remain unread. The only change is the last logon time.
EPA profiles mailboxes at a rate of about 0.5 to 1.0MB per second. You may want to target only a representative subset of Exchange users. The impact of EPA is similar to a cached mode full sync. Due to the impact EPA can have on a mailbox being profiled, we suggest you run the tool outside of normal hours. (A profiling job can be scheduled to run using a configuration file and the command-line version of the tool. But in this article, we’re concentrating on the GUI version of EPA.)
An EPA job can be targeted at the store level, and even at only those mailbox items created within a particular timeframe. You could tell EPA to look at all the mailbox stores in Storage Group One, but only items from the last 72 hours. This flexibility can help you limit the impact on the production environment.
For your first look at EPA, consider using LoadSim to initialize some test mailboxes using the built-in LoadSim profiles. Create two storage groups and initialize the first with some Medium users and the second with the Heavy profile. Then install EPA, fire it up, and select Connect to Active Directory. The tool offers a number of configuration options, some of which are detailed in Figure 4.
Figure 4 Configuration Options in EPA (Close Figure 4)

Option | Description
Active Directory | You can specify a Global Catalog (GC) server in the forest that is hosting Exchange or you can leave this blank to use a GC in the current forest.
Open Configuration | Use this to load a previous configuration. The GC and login account can be saved to an XML file for reuse.
Logging\Stats Option | This lets you specify logging level. You can specify whether to collect statistics on individual mailboxes, which will increase the size of the output log file. By default, data is reported anonymously with each mailbox represented by an ID number.
Specify a time frame | This, obviously, is used for specifying a time frame to look at. By default, it looks at yesterday and will not analyze your LoadSim data if you created it today.
Select the mailbox stores to be profiled | EPA will display the Admin Groups, servers, and mailbox stores that can be profiled. This lets you select a representative set of information stores. Bear in mind the time it can take to profile mailboxes. And note that the System Attendant mailbox will generate an exception as this URL can’t be crawled.
Once you’ve specified all your configuration options, you can click Start Collect. If you have multiple stores with the same name on the same server, the respective report can’t be viewed at the individual store level. Be sure to rename any stores with duplicate names. For example, if you have multiple storage groups, each with a Mailbox Store One, you will need to rename them to something like Mailbox Store One (First Storage Group) and Mailbox Store One (Second Storage Group).
An EPA run produces three files in the folder below C:\Documents and Settings\<User Name>\Application Data\Microsoft\EPA. The log file records details of mailbox access during profiling. The logging selection found on the Configuration tab lets you control the level of detail. An XML file contains the raw data collected during the profiling process, and an HTML file contains the actual report. This includes details like the average mailbox size, the average number of messages and folders, and the like—all of which can be used to customize LoadSim.
You can analyze the report on many levels. For example, the Overall Statistics option gives you a summary for all servers and stores profiled. Information Store summarizes just the selected information store.

Customizing LoadSim
Now that you know how to use EPA to gather some profile data, let’s customize LoadSim. The workload can be customized at three levels: Topology, Initialization, and Profile. Let’s look at each level, discussing how some of the data you produced using EPA can be mapped into LoadSim to increase workload accuracy.
Topology The LoadSim topology defines the high-level structure of the simulated Exchange organization. The mailboxes required for the test are allocated to each store, and Distribution Lists (DLs) and Public Folder requirements are configured. Active Directory® is populated with the required mailbox-enabled accounts and DLs when the topology is generated.
It is only necessary to include the servers, storage groups, stores, and mailboxes required for the LoadSim test. Even if the production environment contains more servers, adding them to LoadSim will not make the results any more valid.
The DL values are critical, though often overlooked and derogated. The default values for DLs are: 1000 per site, Membership, Minimum 2, Average 10, and Maximum 20. For many organizations, the LoadSim defaults are not representative of the production system DLs. (One thousand DLs is quite a lot for a small or medium sized organization.) EPA can’t help you examine the production DLs in your organization, so you will need to perform a Directory export to get at that data. To export this information from Active Directory, use the Windows® Support tool csvde.exe and the following command line:
csvde.exe – d "ou=loadsim users, dc=forest,
    dc=net" –r  "(objectclass=group)" –f
    c:\export.csv
The resulting export file can be manipulated in Microsoft Excel® to identify the values you should use to replace the defaults.
Changing the LoadSim default DL values can have quite a dramatic impact on the generated load. For any given user profile, the number of messages delivered will vary according to the Distribution List settings. For example, by using the default average membership of 10, a Medium user profile will result in 141 messages being received per user. If the average and maximum membership values are increased from 10 to 100 and from 20 to 200 respectively, then the number of messages received increases to 5,621. So while the LoadSim test will be submitting the same number of messages per user, the number of messages delivered will have rocketed due to DL expansion.
LoadSim is typically run for 8 to 24 hours and has the potential of using all of the available DLs during the test cycle. Organizations typically have a small number of large DLs that are used infrequently (perhaps composed of all full-time employees or all users in Europe, for example). This, unfortunately, is difficult to model in a short test cycle. The important thing is that the servers can cope with the normal required throughput. If this throughput can be generated with the slightly artificial DL values provided, the test results are valid. The test does not need to be run with DL values that match the production values, because LoadSim will not be able to generate a representative load. Of course production DLs are also typically out of date and may include unused DLs. LoadSim assumes that housekeeping is fully up to date and that all DLs are active.
Once the topology has been created, any changes to it will require that the initialization steps described in the next section be repeated. And you must ensure that the .sim file used to save the topology creation parameters is the same file you use for initialization. Small changes and mistakes in these areas can seriously affect the success of your test.
Initialization By default, the LoadSim profiles create mailboxes of approximately 60 to 70MB for a Medium profile and 100 to 120MB for a Heavy profile. This might not be appropriate for your testing. When LoadSim initializes the mailboxes, it uses a probability algorithm; so although all the mailboxes will have the same number of items in them, sizes will vary. By tuning the number of messages and folders, you can create mailboxes of any size.
Open the EPA report and look for the values in the left columns from the Overall Statistics, Server Statistics, or Mailbox Store Statistics. These values can be used to perform this tuning. The EPA report values are totals for the servers and stores profiled. The LoadSim values are totals per mailbox. Use the Mailbox: Total Count value from the EPA report to calculate the per mailbox values, remembering that the mailbox size for the stores profiled is shown as Mailbox: Aggregate of mailbox size.
Figure 5 shows how the EPA values map to LoadSim parameters. This allows you to fine-tune the size of the mailboxes that LoadSim creates.
Figure 5 EPA and LoadSim Initialization Mapping (Close Figure 5)

EPA Report Value | LoadSim Parameter
Folder Hierarchy: Total number of user-created folders | Initialization: Number of new folders
Message Counts: Total number of messages | Initialization: Messages per new folder
Message Counts: Total number of messages in Inbox | Initialization: Number of messages in Inbox
Message Counts: Total number of messages in Deleted Items | Initialization: Number of messages in Deleted Items
Calendar: Aggregates of appointments per weekday | Initialization: Number of Appointments
Contacts: Aggregates of contacts count | Initialization: Number of Contacts
Rules: Aggregates of rules count | Initialization: Number of rules in Inbox
Folder Hierarchy: Total number of Search Folders | Initialization: Number of Smart Folders
Profile A LoadSim profile has many settings to define the actions a "user" will perform during a test run. Figure 6 shows some of the settings and parameters the profile takes. These parameters generate various tasks over the duration of the test. Figure 7 lists the fields that can be mapped from your EPA reports into LoadSim parameters to create profile users that are more like your real users. At the beginning of the test, LoadSim will look at the profile and test duration and create an appropriate workload. Decreasing the duration of the test day length will increase the workload by increasing the task intensity. (Hint: If you want to increase IOPS, simply shorten the day length.) You can view a projection of your customized workload in LoadSim by looking at the Test\Logon tab of the Customize Tasks dialog.
Figure 7 EPA to LoadSim Workload Profile Mapping (Close Figure 7)

EPA Report Value | LoadSim Tasks\Parameter
Message Frequency: Aggregates of messages sent per weekday | Send Mail: Frequency per day
Recipients: Aggregates of number of recipients across all messages sent out | Send Mail: Recipients per message
Recipients: Aggregates of DLs across all messages sent out | Send Mail: Add a single DL to X% of messages
Message Frequency: Aggregates of replying messages per weekday | Process Inbox: Reply
Message Frequency: Aggregates of forwarding messages per weekday | Process Inbox: Forward
Message Frequency: Aggregates of messages deleted per weekday | Process Inbox: Delete
Calendar: Aggregates of meetings per weekday | Request Meetings: Frequency
Calendar: Aggregates of meeting request received per weekday
Calendar: Aggregates of appointments per weekday | Make Appointment: Frequency
Folder Hierarchy: Total number of Search Folders | Smart Folders: Number of Smart Folders
Rules: Aggregates of rules count | Rules: Number of rules
Contacts: Aggregates of contacts created per weekday | Create Contact: Frequency
Figure 6 LoadSim Workload Profile
Figure 6 LoadSim Workload Profile
The EPA tool does not provide data for every possible task customization in LoadSim, but it does provide sufficient data to improve on the per-defined profiles and generate more accurate workloads. Before starting the LoadSim test, let’s look at how you can add ESP into the environment to add yet another level of realism to your test environment.

Exchange Stress and Performance
Now let’s configure ESP to send inbound SMTP mails to your system (that is, to your LoadSim users). You do this configuration at the same time LoadSim is running, throwing some regular old Internet mail into the system. This creates an even more realistic setup for your tests.
To configure ESP you need three things: a list of users to send messages to, a script that actually sends the mail, and some sample e-mail messages. The easiest way to get the list of users is by exporting the SMTP addresses of the users created by LoadSim. Export these to a CSV file using Active Directory Users and Computers, edit the list in Excel so that just the actual addresses are left one per line, name the file SMTP_RECIPIENTS.TXT, and save it to the C:\ESP folder.
To create the script file, simply copy the following script into Notepad:
CONNECT
EHLO SMTP.DOMAIN.COM
MAIL FROM: <testuserRANDNUMBER(1,10000)@domainRANDNUMBER(1,100000)>
RCPT TO: <RANDLIST(SMTP_RECIPIENTS.txt)>
DATA RANDNUMBER(1,10).eml
QUIT
SLEEP RANDNUMBER(500,900)
Save this file as SMTPSCP.TXT and place it in the C:\ESP folder. The script randomly selects an SMTP recipient from SMTP_RECIPIENTS.TXT and attaches an e-mail file (.eml) to the message.
You can use Outlook Express to create a message. Don’t put any recipients into the message—just save it to the C:\ESP directory as an .eml file. Do this 10 times (if you are using the script we have provided) putting differing amounts of text into each of the files. If you want, you can also place various-sized attachments in some of the messages.
Now fire up ESP and add a host (see Figure 8). You can just use LOCALHOST to target the machine you installed ESP onto. Then click Connect Host and set a time duration (leaving that field blank will result in a test that runs infinitely).
Figure 8 Basic ESP Connection
Figure 8 Basic ESP Connection
Right-click the machine you just added in the ESP console, select the Add Module option, and select SMTP. Go into the properties of the SMTP module (see Figure 9) and enter the path to the script file C:\esp\smtpscp.txt. (You may notice the browse button doesn’t seem to work—that’s why we’re explaining this step by step.) Enter the server name or IP address running Exchange, and finally enter the number of script instances. This indicates how many instances of the script will run concurrently—entering 10 means 10 e-mails will be sent in at any one time.
Figure 9 SMTP Script Parameters
Figure 9 SMTP Script Parameters
ESP can produce an amazing load using very limited resources. A 5-minute duration with 10 instances using the test script we have provided will send nearly 2,500 e-mails. So use the instances parameter and the sleep parameter within the script to control the load being applied.
Each instance of our supplied script takes approximately 1 second to run; therefore, in the example mail flow calculation we showed you earlier, this value would be 2. As we calculated, there are approximately 2 messages per second inbound to our production system.

Testing Methodology
Once you’ve gathered all your data, done your calculations, and configured your tools, you’re ready to start testing. Remember the three tools weren’t designed to all work together. First use JetStress to performance- and stress-test the storage subsystem. This will help you confirm whether the disk configuration you have designed is working as it should. Once you’ve done that, move on to LoadSim and ESP.
LoadSim and ESP provide more of an end-to-end test of your Exchange system, helping you identify bottlenecks and misconfigurations. Be sure you approach your testing in a logical manner—if you plan on using multiple workstations to deliver the load, first configure one and test it and then scale up. Don’t just configure 10 clients with 1,000 users each and then launch them all at once.
Our final piece of advice is this: measure the counters at the right time. Exchange does a lot of optimization as it runs, and the first few hours of your test run will not be representative of how Exchange will run after several days or weeks. Let it settle, give the servers a few hours to adjust themselves, and then when the system is in a steady state start measuring numbers.
We hope this article has shown you why testing is more than just running LoadSim or JetStress in isolation. But we also hope you are now better equipped to use these tools to simulate your environment. Once you do that, you can really prove that your design works exactly as expected.
Calculating Processor Power
A common problem when designing an Exchange solution is determining the speed and number of processors you need in each server. A useful measure for calculating this is megacycles per mailbox, the amount of processor usage consumed by each mailbox. This calculation isn’t used as an input to any of the load simulation tools, but it helps ensure that the processors in the new servers will satisfy the load requirement.
To calculate megacycles per mailbox:
  1. Select a production server with a typical user load.
  2. Use the System Monitor tool to monitor the Processor\% Processor Time counter over the peak two hours of server activity.
  3. Calculate your current megacycles per mailbox as described in the following formula:
    Megacycles per mailbox =
((average CPU usage) ×
(speed of processors in megacycles) ×
(number of processors)) / (number of mailboxes)
For example, if your server has two 1.8GHz processors, it hosts 1,500 users, and the processor time counter averaged 70 percent, the formula would be:
((1800 x 2) x 70%) /
1500 = 1.68 megacycles
per mailbox (per second)
Based on this information, if you are planning to buy a new server with four processors, each running at 3GHz, you can calculate the approximate processor utilization these same users will place on the new server by plugging in the appropriate numbers. And, in turn, you can calculate how many users that server can support, at least in terms of processor. (As a guide, we always aim to load servers to 80 percent as this allows for spikes in usage.) So, to calculate CPU usage, use the following formula:
CPU Usage = ((number of users) ×
(current megacycles per mailbox)) /
((number of processors) x (speed of processors in megacycles))
The 4 x 3GHz server gives a processor utilization of just 21 percent:
(1500 x 1.68) / (4 x 3000) = 0.21 (21%)

Nmax users = 0.80 × (number of processors)
 × (speed of processors in megacycles)
 / (current megacycles per mailbox)

0.80 x (4 x 3000) / 1.68 = 5714 users
Wow! So the new server should be able to support over 5,000 users with the same workload! Sounds great, but Exchange is about more than processors, as you know. Exchange performance is holistic by nature and the actual performance will depend on the combined performance of all the components. So don’t go loading that server up just yet. First you’ll need to do some thorough testing.

Resource
  • Download dozens of tools that can help you test, manage, and add new features to your Exchange solution.
  • Get a detailed overview of tools and techniques for monitoring Exchange and isolating performance issues.
  • This guide will help you plan an Exchange solution, offering information on deployment, server tuning, client configuration, and more.
  • Check out this in-depth guide to configuring Exchange for high availability.

Greg Taylor is a Senior Consultant for Microsoft in the UK. He has been working for Exchange for seven years.
Simon Shepherd is a Principal Consultant for Microsoft based in the UK.


 Windows Administration: Problems of Privilege: Find and Fix LUA Bugs
Windows Administration Aaron Margosis

Problems of Privilege: Find and Fix LUA Bugs
At a Glance:
  • Least-Privilege User Accounts
  • Determining a true LUA bug
  • Techniques for fixing LUA bugs

You’re probably familiar with the Principle of Least Privilege, which states that a user should be given only the privileges necessary to accomplish his task. It’s an important rule to keep in mind when managing security, and TechNet Magazine has discussed it a number of times. LUA can also introduce some unexpected challenges.
The acronym LUA generally refers to Least-Privilege User Account, but is sometimes defined as Limited User Account, Least User Access, and several other variations. But whatever the letters stand for, the concept is the same. LUA is a computer user account that cannot make changes that affect other users of the system or the operating system itself. In Windows®, these are typically members of the built-in Users group. Members of this group are explicitly not members of powerful groups (such as Administrators, Power Users, and Backup Operators) and they do not hold elevated privileges (like Load and unload device drivers, and Act as part of the operating system). Unfortunately, LUA can surface a number of issues.
A LUA bug occurs when an application—or a feature of an application—works correctly when run with elevated privileges but fails to work for a LUA user when there is no technical or business reason for requiring elevated privileges. A common example is when an application saves its settings to a registry key under HKEY_LOCAL_MACHINE (which is read-only to LUA users), instead of to HKEY_CURRENT_USER.
Another problem arises with system time. Windows XP doesn’t allow LUA users to change the system time. This is not a LUA bug, because changing the system time has security implications with respect to auditing and the Kerberos protocol. The fact that Windows XP doesn’t allow LUA users to change the time zone, however, is arguably a LUA bug. So is the fact that double-clicking the clock in the taskbar’s notification area gives you an error message instead of a read-only view of the Date and Time applet. (Note that Windows Vista™ is heavily focused on a more seamless LUA experience, including improvements to the Date and Time applet.)
By far, the majority of LUA bugs are due to registry and file system access permissions. A program might, for instance, try to save its settings into its installation folder under %ProgramFiles%, or it might try to open a key under HKLM for "All-Access" even if it only needs Read access. Other types of LUA bugs include attempts to start or stop a service, load a device driver, access hardware resources directly, create or manage file shares, or even explicitly check whether the current user is a member of the Administrators group.
There are always one or more low-level operations (API calls) that succeed when performed as admin but fail when performed as LUA. You can see some of these yourself using tools like Regmon and Filemon from SysInternals. Is every one of these really a LUA bug? It depends on how the application responds to the failure. The responses I have seen can be categorized into three groups:
Fire and forget: The application invokes the operation, doesn’t check the result, but doesn’t depend on the operation having succeeded in order to continue working correctly. Not a LUA bug.
Gracefully degrade: The application invokes the operation, checks whether it succeeded, and handles failure in an appropriate way. Not a LUA bug.
True LUA bug: The application invokes the operation, assumes it succeeded, and depends on the operation having succeeded in order to continue working correctly. A variation on this is that the app checks whether the operation succeeded, but handles the failure inappropriately, such as by displaying an error message and falling over dead.
If you’ve ever used Regmon to monitor a GUI app running as LUA, you’ve probably seen an example that can be categorized as fire-and-forget—a failed attempt to open HKLM\System\CurrentControlSet\Control\MediaProperties\PrivateProperties\Joystick\Winmm for All-Access. This occurs during initialization of the joystick subsystem for the process. The operation fails, but it does not affect the correct behavior of your application. However, I have seen advice on the Web (no doubt from people who have misinterpreted Regmon output) claiming that to fix some particular application you need to grant the user full access to this key. Absolutely not! This is not a true LUA bug. You should never need to change permissions on this key!
Before you make changes to security settings, verify that you’re fixing a true LUA bug and not just a phantom, making sure there aren’t better ways to correct the problem without increasing exposure.

How Not to Fix LUA Bugs
You have a standard application that you—or your users—need to run. The application isn’t designed to perform any system administration tasks on your computer, but for some unknown reason, it doesn’t work correctly unless it’s run from an account that has administrator-level access. You need to give your users access to this application, but you don’t want them running as administrators. What should you do?
The most common workaround is simply to add the user to the Administrators group. Sometimes this is done by a helpdesk technician seeing if that fixes the problem. The technician then forgets to remove the user from the Admins group, inevitably leading to another helpdesk call within a few weeks, when the user asks "Why is my computer running so slowly, and why are all these ads popping up whenever I log on?" Let’s just call this workaround a nonstarter and not give it any further consideration.
Two other common (and less than ideal) workarounds are to run just the one program as administrator, or to run the program as a regular user but after granting Everyone "Full Control" over the program’s installation folder, all of its registry keys under HKEY_LOCAL_MACHINE, and all of HKEY_CLASSES_ROOT.
Oh, and while you’re at it, you should remember to grant the user the Debug, "Take ownership", and "Act as part of the operating system" privileges (I’m kidding of course—these are seriously high-risk and should be avoided).
OK, so what should you do? In the rest of this article, I’ll lay out a systematic approach for working around LUA bugs while at the same time minimizing exposure. I’ll discuss approaches from most-preferred to least-preferred, listing some of the pros and cons of each. By the way, while the guidance in this article is primarily geared towards Windows XP, the techniques will also work on Windows Vista.

It’s a Bug—Treat it Like One and Make the Developers Fix It!
This is the best approach. If there is no legitimate reason (business or technical) for the app to require admin privileges, then failure of the app to work for a regular user account is a serious bug that compromises system security, stability, and manageability. Seriously consider having the developers fix the problem directly in the code.
If the development team responds to your request by saying something like "it’s mission-critical so it has to run as admin" or "the app has to run as admin because it writes to HKEY_LOCAL_MACHINE," you should respond by saying "you’re talking nonsense," and insist that they fix the bug.
Benefits: Once the development team has fixed the bug directly in the application code, you don’t need to carry forward any shims, tweaks, or workarounds. In addition, the developers may learn from the experience and avoid creating new LUA bugs. (Note that the number 1 cause of LUA bugs is developers running as admin when writing their code! I discuss this more in the "Loosen Access Control Lists" section.)
Drawbacks: The expense in both time and money can be prohibitive, especially if you have limited resources and a lot of apps that need to be fixed. The app may need to be rearchitected, and new bugs may be introduced in the process.
Another hurdle is that the developers or the source code may not be available. You could be dealing with third-party code from a company that no longer exists. The developers may be in rehab. Or in jail. Or, worse yet, working for your competitor. You get the idea.

Use the Application Compatibility Toolkit
The Application Compatibility Toolkit (ACT) offers useful LUA Mode shims. These shims detect attempts to write to systemwide locations in the file system and registry and silently redirect them to per-user locations. (Windows Vista includes a better equivalent called File and Registry Virtualization.) You can find the Application Compatibility Toolkit at "Application Compatibility: Overview".
Benefits: The really compelling reason for using the LUA Mode shims found in the Application Compatibility Toolkit is that it’s easy. This also doesn’t require you to elevate privileges.
Drawbacks: On Windows XP, the LUA Mode shims often do not work. (File and Registry Virtualization in Windows Vista is a complete rewrite and will have much higher compatibility marks than ACT LUA Modes in Windows XP.)
The added complexity of the underlying operations can make troubleshooting more complicated when things don’t work.
Working through a LUA Bug
When limited privilege causes a problem, there are a number of things you can try that may fix the issue. Some of these solutions are better than others, exposing less risk. If you encounter a LUA bug, follow these steps starting with the most preferred fix and trying the subsequent fixes only if necessary.

Copy Specific HKCR Keys to HKCU\Software\Classes
Prior to Windows 2000, HKCR was just a symbolic link to HKLM\Software\Classes that only administrators could write to. This meant that operations performed on HKCR\.txt actually occur in HKLM\Software\Classes\.txt. Windows 2000 introduced per-user registration data, so now HKCR is a merged view of HKLM\Software\Classes and HKCU\Software\Classes (which the user can write to). If a key exists in the latter, it takes precedence. So now an operation on HKCR\.txt occurs in HKCU\Software\Classes\.txt if that key already exists; if it doesn’t, the operation occurs in HKLM\Software\Classes\.txt as it had in the past.
The problem is that a number of applications write to HKCR at run time to reinforce their file associations, COM registration data, and so on. An error is raised if the write fails, even if the data to be written is already there. The same data is written every time the app runs. If that same registration data were stored in HKCU\Software\Classes, then the write operations would succeed, without changing program behavior.
To fix this, first you must identify the keys under HKCR that the application is trying to write to. Export those keys to one or more .reg files (in the Registry Editor, select File | Export and choose Selected branch). Then using a text editor, replace all instances of [HKEY_CLASSES_ROOT\ with [HKEY_CURRENT_USER\Software\Classes\ and save your changes. When you’re finished, import the edited .reg file into the registry of the user who needs to run the program.
Benefits: This technique fixes issues where applications perform operations in HKCR that should have been done only during installation. This is a better approach than loosening access control on systemwide resources under HKCR. Malware overwriting keys under HKCU will not affect operating system components or other users of the computer.
Drawbacks: Tools have only recently become available for identifying HKCR writes as the source of LUA bugs and isolating which keys are involved. LUA Buglight makes this task much easier.

Use IniFileMapping
Back in the days of Windows 3.x, before the advent of the registry we know and love, the OS and applications stored configuration and preference data to .ini files, such as win.ini. Windows did and still does offer API-level support for .ini files via the Profile APIs (such as WritePrivateProfileString). Many apps, including some Windows applets, still use these APIs to try to write to .ini-formatted files, often in folders where users are not supposed to write.
Windows NT® 3.1 encouraged the migration from .ini files to the more scalable and manageable registry, and provided a means for automatically redirecting .ini file reads and writes to registry keys. The internal implementation of the Profile APIs was augmented to use mappings found under HKLM\SOFTWARE\Microsoft\Windows NT\CurrentVersion\IniFileMapping\. If a mapping for an .ini file is not found under that key, then the operation is performed in the file system as before.
If access to an .ini-formatted file—via the Profile APIs—is the cause of a LUA bug, it can be corrected by adding a key under the IniFileMapping key to redirect access to HKCU. Note that IniFileMapping is under HKLM, and to configure this you’ll need administrative privileges. The configuration specifics are described in the documentation for the Profile APIs.
Benefits: This approach avoids loosening access control on systemwide resources in the file system. And again, malware overwriting keys under HKCU will not affect operating system components or other users of the computer.
Drawbacks: IniFileMapping entries specify the file name only and will affect all Profile API access to any file with that name, in any path. This can have side effects on other apps that have their own ini files with the same name.

Update the SafeDisc Driver
A number of games depend on the secdrv device driver—also known as SafeDisc—that comes from Macrovision. The SafeDisc driver that ships with Windows XP is a demand-start driver; users are not allowed to stop and start it, and this results in errors when programs attempt to access it. There is an update (available from Microsoft and from the Macrovision site) that configures the driver to load when the system starts so the user does not need permission to start it. This change allows some games to work correctly for a non-admin user.
(Note that at the time of writing, the Microsoft download page for this update said that this software "will not alter or patch any component on your system; it will only change the startup state of the system component." This is not true—it installs an updated driver.)
Benefits: This update is easy to implement, and there’s no need to make Access Control List (ACL) changes to systemwide resources.
Drawbacks: There aren’t really any drawbacks, except that it is primarily used for correcting problems with games (not productivity apps).

Loosen Access Control Lists
The most common underlying cause of LUA bugs is that developers (and often testers) typically run as admin. A developer may not intentionally require that the end user run as admin, but things that depend on admin access can creep into the code (for example, writing to files in the root folder of the C: drive, in the app’s installation folder under %ProgramFiles%, or in %windir%). The app works fine for the developers and testers, but then you try to run it as a regular user and it crashes.
One option is to change the access control list (ACL) on objects to grant your user the access that the program requires. Typically, the objects that need tweaking are in the registry or in the file system (if using NTFS). This approach requires serious consideration and you should note some caveats.
First, this approach should only be chosen after you’ve tried all of the more preferred options and confirmed that they don’t fix the LUA bug you’re dealing with. If you proceed, be very careful.
ACL changes should only be considered on application-specific resources, not on OS-wide resources. While it might be OK to change the ACL on %ProgramFiles%\VendorX\AppX\DataFolder, you should never change the ACL on %SystemRoot%\System32—to loosen or to tighten access. (For more about this, see the Knowledge Base article "Security configuration guidance support".) In addition, avoid changing ACLs on program code if at all possible. This is to prevent malware from infecting or replacing application files.
Also avoid changing ACLs on resources (particularly .exe and .dll files) that are used by administrators or services. Doing so increases the risk of elevation of privilege, which can lead to compromising the entire system. (Even so, the attack surface remains far smaller than in a situation where everything always runs as admin.)
Ideally, the resource should only ever be accessed by a single user. If the resource is accessed by multiple non-admin users, there is increased risk of one user causing another user’s account to be compromised.
Finally, you should grant the least amount of additional access that is required to the smallest possible number of resources and to the smallest possible number of users in order to allow the application to work. Granting full control to Everyone on a big chunk of the file system or registry should never be necessary.
Granting additional access only to the computer’s primary user is optimal, but that may be difficult to manage across a large number of systems (for instance, granting MARY permissions on one system, STEVE on another, and so on). If you can define a set of users who need to use the program, you should add them to a group and then grant access to that group.
Alternatively, you can grant access to the built-in INTERACTIVE pseudo group. This will grant the additional access only to whoever is interactively logged on at the time, without granting additional remote access to the resource. Note that in a terminal server or Fast User Switching scenario, there can be multiple simultaneous users on the computer with INTERACTIVE in their tokens.
Benefits: This technique offers a big return on the investment of your time. Most of the LUA bugs that my colleagues and I have seen revolve around file and registry permissions. Tweaking Access Control Lists will probably fix a larger share of your LUA bugs than any other approach.
Drawbacks: This approach is less-preferred, because ACLs are set this way for a reason. Changing them allows otherwise constrained users to change shared resources—for good or for bad—and it makes it easier for one user (or malware unintentionally run by that user) to affect others. If an admin becomes affected, the entire system can be compromised.
In addition, it is not easy to identify precisely which resources should be opened up and by how much. And it’s difficult to know for certain whether opening access to a resource will inadvertently expose an avenue for elevation of privilege, allowing system takeover.

Run Just the One Problematic App with Elevated Privileges
Some apps address LUA bugs by explicitly checking for admin group membership on startup and displaying an error message insisting that you simply have to be an admin to use the program. This may be due to developer laziness, incompetence, or arrogance (or all three), but these apps will be resistant to any other workarounds available to you. As a last resort, after you’ve confirmed that all else fails, you can consider running the one problematic app with elevated groups or privileges.
Instead of running the app as admin, try running it elevated but as less than full admin. You could run the app as a member of Power Users or with a specific privilege, such as SeLoadDriverPrivilege. But with just a little more work, many of these other groups and privileges can be used to take over an entire system.
So how do you do this? If you trust the user with the admin password (or you trust the user to make security decisions) you have four options:
  1. You can use RunAs. (For more details on this, see my postings ""RunAs" basic (and intermediate) topics" and "RunAs with Explorer".)
  2. You can use MakeMeAdmin. This is a batch file, which you can easily customize to run something other than a command shell. You can also tweak it to make the elevated context less than full-admin. (For more information on this, see my blog posting.)
  3. SysInternals offers PsExec and Process Explorer. These apps provide various RunAs-like options.
  4. Finally, you can use RunAsAdmin. This is an interesting and useful open source utility by Valery Pryamikov. RunAsAdmin takes an approach that is a little bit like the Windows Vista User Account Control feature (UAC), elevating the current user in place without requiring a password.
On the other hand, if you don’t trust the user with the admin password, there are a couple of third-party options that are worth considering.
  1. PolicyMaker Application Security by DesktopStandard uses a Group Policy extension to configure rules for modifying process tokens. It mitigates some of the drawbacks mentioned below. It can be configured so that child processes launched by a targeted app do not inherit its modified token. And it can perform granular token modification, to raise (or lower) permissions or add (or remove) privileges.
  2. Winternals (the commercial side of SysInternals) offers Protection Manager, a tool that uses a lightweight client-server application and a whitelist technique to block all untrusted applications. Protection Manager allows applications to have their process tokens and privileges elevated to that of an administrator or reduced to that of a user (in cases where end users are non-administrators or administrators, respectively). Protection Manager doesn’t allow a child process of an elevated app to run elevated unless it is also explicitly configured as an elevated app. All children of reduced privilege processes are reduced automatically. Applications can be allowed, blocked, elevated, or reduced as specified by an administrator via digital signatures, hashes, NTFS file ownership, or path.
PolicyMaker Application Security and Protection Manager both determine in kernel-mode code whether, when, and how to modify a process token. Since passwords are not used, they’re not at risk, and the modification decision cannot be interfered with by non-admins.
Other tools are available that perform RunAs-like operations with the admin account credentials encrypted—or sometimes just obfuscated. Even though this technique raises the bar and will stop some users from getting the admin credentials, those passwords still have to be decrypted within the user’s security context and therefore may be exposed to an attacker with the right tools.
A question I’ve heard asked frequently is whether the RunAs.exe /savecred option can be used as a shortcut to let a user run a single app as admin using a saved password (without requiring further password entry). But this can cause unanticipated problems and there are a number of issues you should be aware of. The credentials are not tied to any one shortcut; once the credentials have been saved they can be used to start any app. While the password is securely encrypted with a user-specific key, it will still be decrypted in the user’s security context and briefly exposed. And the /savecred option is not available on Windows XP Home Edition.
Benefits: With this approach, you are able to avoid always running everything as admin. Unfortunately, that’s about the only benefit for this one.
Drawbacks: Running an application with elevated privileges is much riskier than any of the other options I’ve already described. It’s very difficult to defend a system against a malicious user or a malicious app when there’s an application running as admin. Here’s a simple example. If you run Notepad as admin, and then choose File | Open, you are presented with a little Explorer-like window. You’re still in Notepad, which means you have full admin-level access to the entire file system—you even have the ability to launch programs as admin from here. This simple technique can be exploited by a malicious user or by malware pumping keystrokes or window messages into the elevated program.

Conclusion
The Least-Privilege User Account can be a handy option in your security strategy, but it can present some interesting challenges. Workarounds for these bugs vary greatly in effectiveness and security. Though it can be difficult, it’s important to determine the exact cause of the bug and then address it with the best solution. If you use one of the less preferred techniques, you should understand how to protect your systems as much as possible.
This article was adapted from posts that have been published on my blog. There you’ll find more information about the concept of running with least privilege and how to take advantage of the benefits as well as deal with the challenges.
LUA Buglight
LUA Buglight is a tool designed to help both developers and IT pros identify the specific causes of LUA bugs in applications running on Windows XP. Once the specific causes have been identified, the bugs can more easily be resolved by fixing the app’s source code or by making configuration changes, allowing the app to work correctly for non-admin users.
With LUA Buglight, the target app runs as non-admin, with approximately 200 Win32® APIs intercepted ("shimmed") using Detours 2.0 from Microsoft Research. When the target app calls a shimmed API, it actually calls into a LUA Buglight Detours-based DLL. That DLL can then perform various actions, including calling the actual underlying API and inspecting or modifying its results before returning control to the application.
LUA Buglight creates an access token representing the current user but as a member of the administrators group (referred to as the this-user-as-admin token) with full admin privileges. That token is available to the API shim code running within the process. LUA Buglight creates this token using an approach very similar to that of the MakeMeAdmin script. The user must supply credentials for a local administrative account to create this token—there is no magic trick for unauthorized elevation of privilege involved here.
The core of the LUA Buglight approach is that when the app calls a shimmed API, the shim first calls the underlying API. If that API fails with access denied (or in some cases, privilege not held), the shim takes the this-user-as-admin token, impersonates it, and calls the underlying API again. The result of the second call is what is returned to the application. If the call succeeds as admin, information about the call is written to a log file as a potential LUA bug. The net result is that the app will not fail due to LUA bugs, since it can use admin privileges when needed, but that each case where admin privileges are required is identified. For up-to-date information on the upcoming beta release of LUA Buglight, see blogs.msdn.com/aaron_margosis.

Aaron Margosis is a Senior Consultant with Microsoft Consulting Services. The author of the popular MakeMeAdmin and PrivBar tools, he is a passionate evangelist for the use of "least privilege" on Windows, and has been called "the God of non-admin" by Mark Russinovich. Aaron has been with Microsoft since 1999. Take a look at his "non-admin" blog.


 Windows Administration: Find Anything with Windows Desktop Search
Windows Administration Matt Hester

Find Anything with Windows Desktop Search
At a Glance:
  • Types of files indexed by WDS
  • How the WDS index is maintained
  • Deployment tips
  • Management and Group Policy

Have you ever wondered why it takes 3.2 seconds to search the Internet but 2.7 days to search your hard drive? Don’t you wish you could search your hard drive and e-mail as quickly as the Internet? You can with the newly improved Windows Desktop Search (WDS)
Think about how much information you have stored on your computer, and where that information resides. Now think about all the information your users have stored on their laptops, desktops, or on the network. I have all my rarely used information on my d:\ drive or a network share, my somewhat important information in My Documents, and my really vital information on the desktop and in the tons of e-mail that has built up over the years. (Not to mention the millions of other Outlook® items stored in contacts, calendar, and tasks.)
Finding anything in that morass is quite a chore. And not just for me. IDC conducted a survey that showed the average active computer user spends nine and a half hours a week looking for documents, which illustrates the serious need for a better solution. So how can WDS help?
Figure 1 Broad Range of Search Results
Figure 1 Broad Range of Search Results
Windows® Desktop Search is an extremely powerful search engine that lets you search through all of those pockets of information: your hard drives, your network drives (via UNC names), and even your e-mail. WDS can index over 200 unique file types (over 350 if you count different software versions). For a full list see "List of searchable file types". And, with an extensible API set, WDS can index anything, anywhere. WDS is available to all users in their taskbar and it runs automatically whenever the user logs into the machine. Figure 1 shows some typical search results. Built-in right-click capabilities allow you to copy, move, delete, print, open, reply, forward, and even preview your search results (see Figure 2).
Figure 2 Previewing Files
Figure 2 Previewing Files
Windows Desktop Search also provides great flexibility to multinational organizations. The optional Multilanguage User Interface (MUI) pack download offers support for 30 languages in addition to English (see Figure 3).
Figure 3 Supported Languages (Close Figure 3)

● Bulgarian | ● Korean
● Chinese (Simplified) | ● Latvian
● Chinese (Traditional) | ● Lithuanian
● Croatian | ● Norwegian
● Czech | ● Polish
● Danish | ● Portuguese (Brazilian)
● Dutch | ● Portuguese (Iberian)
● English | ● Romanian
● Estonian | ● Russian
● Finnish | ● Slovakian
● French | ● Slovenian
● German | ● Spanish
● Greek | ● Swedish
● Hungarian | ● Thai
● Italian | ● Turkish
● Japanese | 
If you tried the initial versions of WDS, you may have been deterred by the performance hit you took during indexing. The new version of WDS has significant improvements, though. WDS 2.6.5 reduces disk access by 50 percent compared with previous versions, and it eliminates the delay during system shutdown. WDS 2.6.5 offers three new policies (including policies for inclusions/exclusions of specific paths), considerable improvements in stability and results, and full integration in the Windows servicing model. There are also add-ins for Lotus Notes and for the MSG file format so WDS can index Outlook MSG files that have been saved outside of Outlook. Content within e-mail attachments are indexed and searched as well.
If you have multiple or dual-core processors, look for enhanced performance and even less noticeable system impact as the thread-optimized search engine code takes advantage of the additional processing threads.

Configuration Requirements
To take advantage of WDS you must be running Windows XP SP1 or higher, Windows Server™ 2003, or Windows 2000 SP4. As of now only the 32-bit versions are supported; support for 64-bit systems is expected in the near future. To index and search your e-mail, you need Outlook 2000 or Outlook Express 6.0 or greater. For full preview of Office documents you need Office XP or later.
From a hardware perspective, you need at least a Pentium 500MHz processor or better (1GHz recommended), 128MB of RAM (256MB recommended, 512MB is optimal), and 500MB of hard disk space is recommended for the index. The size of the index depends on how much content you have and how much you index. And for best viewing, a 1024 x 768 screen resolution is recommended.

Deployment
WDS is easy to deploy into any environment. It comes as a self-extracting executable file of only 4.5MB, so you can use Systems Management Server (SMS) to deploy it or you can turn to Group Policy. The size of the program makes it ideal for Group Policy package deployment. However, in order to use Group Policy for deployment, note that you will first need to wrap the WDS file in an .msi file.
WDS is distributed like other Windows components through the package installer (formerly update.exe), which means it can also be distributed very easily using a script. WDS supports both attended and unattended installation modes. Attended mode, which requires end user interaction, is more common.
For unattended installations you can create custom batch files using the command-line options that are shown in Figure 4, or by using SMS, Group Policy, or Windows Update Services.
Figure 4 Installation Command-Line Options (Close Figure 4)

Command-Line Option | Description
/quiet or /q | Provides no status dialog box during the extraction. Can be used in combination with /extract or /extract path, or this option will direct the installation to run in quiet mode.
/passive or /U | Provides a progress bar during the extraction, but does not prompt you for the destination folder name. Can be used in combination with /extract or /extract:path, or it will direct the installation to run in passive mode.
/extract or /X | Extracts package files without starting the installation. Prompts you for the path of the destination folder for extraction. When used in conjunction with the /q or /U switch, extracts the package file to a randomly named folder on the root folder.
/extract:path_name or /X:path_name | Extracts software update package files to the specified folder without starting the package installer or prompting you for a destination folder. When used in conjunction with the /q or /U switch, extracts the package file to the specified folder.
The command line offers many options for automated installation. When you use the /q option, for example, the WDS installation is completely silent, and if a restart is required, it occurs automatically. Alternatively you can use the /passive or /U option, which displays a progress bar and warns that a restart will occur if necessary. Passive mode also displays any errors that may be encountered. Figure 4 details the command-line options and their functions.
After extraction, the files are placed in the specified folder. If no folder is specified and the /extract option was used with /passive or /quiet, a randomly named folder (like 1ed6b742f546f) is generated and the setup files are placed there. When installation is finished, the folder and installation files are removed.
Figure 5 provides examples of commands you can use to extract the contents of a software update package.
Figure 5 Extraction Commands (Close Figure 5)

Example | Description
wds-x86-xx.exe /q /x:C:\WDS | Extracts and installs the contents of the package to a newly created folder called WDS on drive C.
wds-x86-xx.exe /extract /passive | Extracts the contents of the package to a randomly generated folder in the root folder of the current drive.
Once WDS is deployed, it will create and maintain an individual index of all the information on a user’s PC as well as any calendar appointments, contacts, tasks, and e-mails in Outlook. Each user maintains his own index and finds items only within his own scope on the system. The index also contains metadata for an item including the time a file was created, location, owner, file type, and so on. By default, WDS will index just the contents of the My Documents folder and the default e-mail location (stored either locally in a .pst or .ost file or on the Exchange Server). Indexing can be expanded to look into other drives and directories on your system including network drives via UNC paths.

WDS Components
WDS consists of four elements: WindowsSearch.exe, WindowsSafeFilter.exe, WindowsSearchIndexer.exe and WindowsSearchFilter.exe. WindowsSearch.exe is the primary component. It runs automatically at computer startup and is located in the Startup Program Folder. It manages the indexer process and provides the management interface via the desktop tray. Additionally, it defaults to replacing the standard Windows search GUI (Search Companion) with the enhanced search interface.
The WindowsSafeFilter.exe processes the information before it gets into the index and maintains the integrity of the index. WindowsSafeFilter.exe is the guardian of the WDS index, preventing attacks against it. It is the component responsible for a number of security tasks.
WindowsSearchFilter.exe is critical for the extensibility of WDS. It provides the host for IFilter add-ins, which enable the WDS indexer to open, read, and index the contents of new file types it would not otherwise be able to fully index. Many software programs on your PC already have IFilters installed. For example, if you use Microsoft Office Visio®, you automatically have the Visio IFilter plug-ins installed. Several free IFilters are available (StarOffice/OpenOffice, PDF, ZIP, and so on) and are located at Search Even More File Types and Content Sources. What’s great about IFilters is the extensibility they provide. WDS provides the architecture that lets you or your developers write customized IFilters. Windows Desktop Search is smart enough to recognize new IFilter plug-ins and will be able to index the contents of the associated file types the next time the index is rebuilt. Moreover, WindowsSearchFilter.exe was specifically designed to survive poorly written third-party IFilters, in order to maintain overall reliability.
The final component, WindowsSearchIndexer.exe, is the chief process for creating and querying the WDS index. This process runs per user, which ensures it indexes the documents in the user’s security context. In future releases the indexer is expected to become a system service where it will maintain the high level of security WDS currently offers. WindowsSearchIndexer.exe stores the index by default at: %UserProfile%\Local Settings\Application Data\Microsoft\Desktop Search.

The Index
The index is stored in the \Local Settings\Application Data\Microsoft\Desktop Search directory structure of the user’s profile (Figure 6), which looks rather complex. Two main components make up the WDS index: an inverted index and a properties cache database.
Figure 6 Desktop Search Directory
Figure 6 Desktop Search Directory
The inverted index is actually an index of words in your documents and it includes up to 120 additional properties, though the typical document (any file that’s indexed) will only have about 10 properties. The inverted index portion of WDS is made up of several files located in the desktop search directory. The .ci and .dir files are located in %UserProfile%\Local Settings\Application Data\Microsoft\Desktop Search\Applications\RSApp\Projects\MyIndex\Build\Indexer\CiFiles. The main index file will be the largest file in the directory; the other files are referred to as shadow indexes.

Keeping the Index in Sync
One of the challenges of maintaining a searchable index is making sure it produces accurate results even when documents change (moved, deleted, altered, and so forth). In this structure the database must be updated in order to reflect any changes, and this is where the multiple shadow indexes come into play.
The properties cache database (RSApp.edb) is crucial in processing the search results. RSApp.edb holds all the extra properties concerning the documents—typically path, owner, type and the like—and will work with the inverted index to return a full resultset. RSApp.edb provides the necessary properties to open the document, preview it, group the documents, sort the results, and display them.
Here is a simplified example that illustrates how the two databases work together to produce a resultset. Let’s say you’re looking for the word "phaser." If there are multiple words or wildcards in the search query, the inverted index will cross-reference the index to return accurate results. The inverted index might return a result something like Figure 7. Then the WDS engine would reference the properties database to display results like those in Figure 8.
Figure 8 Properties Cache (Close Figure 8)

Document | Location | Type
52 | C:/docs | .doc
Figure 7 Inverted Index (Close Figure 7)

Word | Document | Locations in Document
phaser | 52 | 4,8,15,16,23,42
 | 55 | 5,135,222
The shadow indexes along with the main inverted index keep the most current index. For example, if the word phaser gets changed in a document, the shadow index will keep the resultset up to date (see Figure 9). Over time the shadow indexes are merged into the main inverted index to reflect all changes.
Figure 9 Shadow Index (Close Figure 9)

Word | Document | Locations in Document
phaser | 52 | 4,8,15,16,23,42
 | 55 | 5,222
What happens under the covers when the initial indexing process begins and WDS needs to create the first index? During deployment, the process will determine when the documents were last modified and when (if ever) WDS last crawled them. If the process determines an index doesn’t exist, then the indexer begins to crawl the document directory structures (directory tree, e-mail folder tree, and so on).
After deployment, the behavior is somewhat different. When the documents on the system change, a file system notification will trigger and send the changed document straight into the process. These updates are prioritized and sent to the beginning of the indexing process, so if the WDS is doing other work, these changes are sent to the front of the line to be processed.
The way WDS indexes the documents is similar in both of these processes. The document to be indexed is sent to a protocol handler which turns it into a stream of data. The stream is then sent to the appropriate IFilter for parsing. When the document has been parsed, the appropriate information is then distributed to the inverted index or properties cache.
What all this updating means is that you should never have to rebuild the index. There is a specific policy you can implement to prevent your users from rebuilding the index on their own.
In addition to updating the indexes as you make changes on your PC, WDS preserves performance during busy periods using a smart back-off mechanism that forces WDS to wait until CPU load drops before resuming indexing. This back-off functionality is triggered automatically when WDS sees increased usage in any of the following: I/O, keyboard or mouse, or CPU. In other words, WDS works efficiently without consuming excess system resources while other higher priority tasks are running. For laptops, WDS can be configured to prevent indexing while the computer is running on battery power.

Security
WDS is designed with security in mind. It adheres to the Microsoft security and privacy model, so WDS does not index sensitive locations such as Web pages, temporary Internet files, the cache, and password protected Office files. It is also designed to index e-mail attachments in a sandbox and prevent the previewing of complex attachment formats (such as a ZIP or macro file) by default. This is accomplished with the WindowsSafeFilter.exe mentioned earlier. You can prevent the indexing of attachments altogether using a Group Policy setting called Prevent Indexing of E-Mail Attachments. WDS stores the index in a simplified encryption format. You can achieve even more security by placing the index on an NTFS volume and using the Encrypting File System (EFS).

Management
WDS can be managed via Group Policy, so it’s easily customized to match your organization’s business needs. There are over two dozen different settings that provide the flexibility to control your WDS installation. The three main areas you can manage with Group Policy are setup, index, and search. These settings not only allow you to manage WDS, but also enhance its functionality. In the search section, Add Primary Intranet Search Location and Add Secondary Intranet Search Location allow your users to push their search queries to your intranet search providers, such as SharePoint® Portal Server or SharePoint Services.
If you use either the /quiet or /passive switches for unattended install mode, you should enable the "prevent first run customization wizard" setting, which keeps users from seeing the first run wizard. A majority of the settings are located in the index section in GPO. These settings control what will be in the index and where WDS will look for the documents (see Figure 10).
Figure 10 Group Policy Settings
Figure 10 Group Policy Settings
WDS also offers a completely extensible API set that provides virtually limitless capability to customize the search experience to meet your needs. In addition to adding your own IFilters, you can extend WDS so it can index new data stores, such as the database of an e-mail application. And WDS doesn’t stop at your local environment. With one click, it can take your users’ search to the Internet directly from the interface, further integrating the silos of information your users and your organization need to access effectively.
The WDS interface can also be made more visible in your users’ environment. If you load the MSN® Search Toolbar suite in combination with WDS, the search interface will show up in Microsoft Internet Explorer® and Outlook, which makes it even easier to use.
Why not give Windows Desktop Search a try so you can finally unlock all those silos of information in your organization and truly integrate your data. To get started, grab a copy.
WDS Online Resources
For lots more on Windows Desktop Search, see the following sources:
  • Matt’s Blog
  • Searching Tips
  • Windows Desktop Search Administration Guide
  • Desktop Search Add-ins
  • MSN Toolbar
  • Windows Desktop Search SDK
  • Desktop IFilters on Channel 9

Matt Hester is a seasoned TechNet presenter, an Exchange Server insider who worked as an MCT for over eight years before joining Microsoft. Matt loves reaching out to users and customers in the local community and gets a thrill from installing a server that can send e-mail or provide other services.


 Windows Administration: Manage Printers with New Group Policy Features in Windows Server 2003 R2
Windows Administration Jeremy Moskowitz

Manage Printers with New Group Policy Features in Windows Server 2003 R2
At a Glance:
  • Windows Server 2003 R2 Print Management
  • Using Group Policy to control printer settings
  • Gaining control per user or per machine

Windows Server 2003 R2 has a truckload of neat new features, one of which, the Print Management Component, includes great tools for use with Group Policy. The Print Management Component can
centrally manage almost all aspects of the printers on your network. And even better, it lets you deploy printers to users or computers via Group Policy so you can ensure that whenever a user moves from machine A to machine B, she keeps her printer mappings. It can even ensure that every user on machine C will have the same printer settings.

Put Group Policy to Work
Before you can begin using the Print Management Component, you need to perform a number of steps. First you will have to update your Windows Server™ 2003 schema to match the Windows Server 2003 R2 schema. If you want to use Windows Server 2003 to perform printer management, you need to load the Print Management Component on Windows Server 2003. On the other hand, if you want to use Windows® XP for printer management, you need to load the Windows Server 2003 Administration Tools Pack onto your management station. In order to update the schema you’ll need approval from your Active Directory® Schema Administrator. Once you have the approval, this operation is best performed directly upon the Schema Master in your domain.
You need to do the schema upgrade so that printer connection objects get a new fast query lookup via LDAP in Active Directory. This way, the Print Management Console (which I’ll explore shortly) doesn’t have to inspect every Group Policy Object (GPO) in the domain to determine where printers are deployed.
When you pop in the R2 disks, you’ll be presented with the option to continue Windows Server 2003 R2 setup. If you do, however, you’ll get the message stating that setup can’t continue because the Active Directory schema version of the controller is not compatible with Windows Server 2003 R2 and that you must upgrade the schema. You are also instructed that you can upgrade the schema by running adprep.exe /forestprep on the schema operations master. Adprep.exe can be found in the Cmpnents\r2\adprep directory on the Windows Server 2003 installation disk 2. To update the schema, start out by locating Windows Server 2003 R2’s disk 2. In the \Cmpnents directory, run adprep/forestprep. Once this is done, you can continue the installation of R2 by clicking R2Auto.exe on the root of the R2 CD-ROM. The rest of the installation should be a piece of cake.
For the rest of the article, I’ll assume you’re using your Windows Server 2003 and have upgraded it to R2. I’ll also assume that you want to manage your printers from that server, as opposed to a Windows XP management station.

Install the Print Management Component
Next load the Print Management Component by going to Add/Remove Programs | Windows Components | Management and Monitoring tools and selecting Print Management Component, as you can see in Figure 1. Note that next time the Configure Your Server Wizard appears, you’ll see that it’s been installed.
Figure 1 Load the Print Management Console
Figure 1 Load the Print Management Console
Now that the Print Management Component has been loaded, you’re ready to deploy printers to your users or your computers. You can do this either by hand using the regular Group Policy editor snap-in or by using the tools provided in the Print Management console.

Deploy Printers Using GPOs
To push a printer down to your users or computers, you start by creating a GPO and linking it to an organizational unit (OU) containing either users or computers, for example, the Sales Users OU. When you edit your next GPO, you’ll see a Deployed Printers node in both the computer and user half of the GPO along with a new Action called Deploy Printer in the Action menu, as seen in Figure 2.
Figure 2 Manage Printers within the GPO Editor
Figure 2 Manage Printers within the GPO Editor
Note that if you don’t see the Deployed Printers node, it’s likely that you don’t have the updated Adminpak tools on your management station (the computer from which you’re editing this GPO). To get the latest tools, download the Windows Server 2003 Administration Tools. This isn’t one huge file like Adminpak.msi; instead it’s a collection of smaller files for specific updated components, such as the Print Console.
Once you select User Configuration | Deployed Printers | Deploy Printer (as seen in Figure 2) or Computer Configuration | Deployed Printers | Deploy Printer, you’ll be ready to blast new printer assignments down. Just type \\server\printer into the Enter printer name dialog (shown in Figure 3), click Add, and it appears that you’re finished. However, you need to go the extra mile to tell your users or computers to actually accept this newly chosen printer.
Figure 3 Enter the UNC Printer Path
Figure 3 Enter the UNC Printer Path
Pushprinterconnections.exe is a new R2 command that users or computers need to run. If you’re deploying printers to users, the .exe needs to be run in the user’s Login Script. If you’re deploying printers to computers, it needs to be run in the computer’s Startup Script.
The pushprinterconnections.exe program is found on your R2 server in the \windows\PMCSnap directory along with some other bits associated with the Print Management console (which I’ll talk about in a minute). You can see that in Figure 4. However, this isn’t where you run it. Your job is to take the file and plunk it directly into the GPO itself.
Figure 4 pushprinterconnections.exe in /windows/PMCSnap
While editing the GPO, drill down to the script type (User Login or Computer Startup). Click the Show Files button. Next, copy pushprinterconnections.exe into the window that opens. Back at the properties of the script, click Add and locate and select the pushprinterconnections.exe file. Then click OK. If you want to enable the logging of troubleshooting information, type –log in the Script Parameters box. A per-user debug log file will be written to %temp% and a per-machine debug log will be written to %windir%\temp (these are totally different directories). It’s worth noting that you shouldn’t use the –log parameter in a production environment—unless you want the utility filling up your client machine hard disks with megabytes of log files.
If you are using Windows Vista™, you’ll see that this utility isn’t required; the ability to push down printer connections is built in. As a result, the first thing that PushPrinterConnections.exe does when you run it is to check if it is running on Windows Vista. If so, the utility simply exits. Network administrators don’t have to worry if Windows Vista clients accidentally ran pushprinterconnections.exe.
At this point, you should see the printers when you log in as the user or restart the computer. Note that these printers won’t change during background refresh after you’re already logged in because the pushprinterconnections.exe utility only runs at login or startup.

An Easier Method
There is an alternative to deploying printers by hand using the Group Policy editor: the Print Management Console, which displays printers deployed via GPOs. In Figure 5, you can see each of my printers (HPLaser1 and HPLaser2), the GPOs they’re assigned in, and which side—user or computer—is being forced.
Figure 5 Deployed Printers Node
Figure 5 Deployed Printers Node
The Print Management Console also has the ability to push printers directly by creating GPOs of its own. Just drill down to Print Management | Custom Printer Filters | All Printers, locate the printer you want to push down to a computer or user, and select Deploy with Group Policy, as shown in Figure 6. The process begins simply enough, as you can see in the Deploy with Group Policy dialog box in Figure 7; however, after that things get difficult.
Figure 6 Print Management Console
Figure 6 Print Management Console
Figure 7 Deploying Printers via GPOs
Figure 7 Deploying Printers via GPOs
The interface from there on is almost a throwback to the days before the Group Policy Management Console (GPMC), and we all hated those days. The idea is to click Browse and either find a GPO you happen to know is linked to a Site, Domain, or OU or drill down into an OU and choose to create a new GPO that’s linked to the level to which you drilled. You can see this in Figure 8. The icon being pointed to in the figure is used to create a new GPO and link it.
Figure 8 Create a New GPO to Affect Your Target OU
Figure 8 Create a New GPO to Affect Your Target OU
Once you’ve created and linked the GPO, it’s time to deploy the printer. Here you select which group you want to deploy to: users, computers, or both. Make your choice, then click the Add button, and click OK. Just make sure you don’t miss that Add button. I missed it several times and was driving myself completely nuts!
The important point to remember here is that deploying printers via the Print Management Console doesn’t complete the whole job. While it puts the printer in place in the Deployed Printers node, it doesn’t put the pushprinterconnections.exe app into the Logon Script or Startup Script. This means you have to go back in, via the GPMC, edit the GPO, and put in pushprinterconnections.exe. It can be frustrating, but after you’ve done it a few times it will become part of your routine.

Final Thoughts
Clearly, this ability to push printers down to either users or computers is a nice leap forward. But, unfortunately this new magic isn’t built on the client-side extension of Group Policy. Rather, this is a little trick to push printers down to users. I’d like to see the ability for users to get a changed GPO and have the printers change on the fly with the background refresh interval. It’s not there yet, but appears to be coming soon with Windows Vista. Just remember when you’re planning to use this technique that Windows 2000 only supports per-user printer connections, while Windows XP and Windows Server 2003 support per-user and per-computer printer connections.
For more on the Print Management Console, see "Print Management Step-by-Step Guide for Windows Server 2003 R2".

Jeremy Moskowitz, MCSE and MVP in Group Policy, runs GPanswers.com, a community forum on Group Policy. He also runs a two-day Group Policy intensive training course. His latest book is Group Policy, Profiles and IntelliMirror, 3rd edition (Sybex, 2005). Contact Jeremy at www.GPanswers.com.


 From the Editor:
From the Editor

Orlando is not the only place for a conference, you know. We’re just back from Microsoft Tech•Ed in Boston, and let’s just say that it was an absolute pleasure to be in a city where you can take the train to the convention center.
And what a great convention center it is! It’s a brand-new facility in South Boston, and it’s beautiful. The expo hall was as vast as any I can remember, and I go back to the first Tech•Ed in 1992. Of course, feeding over 10,000 hungry geeks is a challenge in and of itself, so perhaps the organizers can be excused for making those flower-shaped hamburgers on Wednesday.
The big buzz at the show this year centered around both Windows Vista and the 2007 Microsoft Office system, and it grew steadily throughout the week. Attendees received the Beta 2 bits of both products. I had already taken the plunge with Windows Vista Beta 2, which has a new Windows Media Player sharing component. As others in the hotel started to play with their CDs, I kept getting notifications that new machines were available for me to share my library with. Judging from this empirical evidence, adoption was brisk in the downtown Sheraton Hotel.
So, you say, what’s it like entrusting your work machine with a beta OS? As you’d expect, some of the features work flawlessly while others are still undergoing fine-tuning. The security in Windows Vista has been cranked up to 11, and it will be a lot harder to accidentally toast your machine. This should be good news for desktop admins out there who spend their days degunking machines assigned to employees who have unwittingly snuck malware past the firewall.
TechNet Magazine also had a great week in Boston. While other booths were giving away squeezy balls, or USB dongles or, if you were lucky, pens bearing logos, we went for broke and gave away an Xbox 360 every day of the show to a lucky reader/attendee who answered a couple of questions correctly. The hordes grew daily—until the booth looked like U2 was about to play on the rooftop or something. Who knew that going monthly could be this exciting?
What if you couldn’t make it to Boston this year—are you out of luck? Not by a long shot! You may have missed out on the Xbox, but you don’t have to miss the rest of the show. Visit TechEd 2006 to watch webcasts of the keynote address and the conference sessions. Check out VirtualTechEd.com to view breakout sessions, demos, as well as news and interviews from the show. Finally, if you don’t want to miss a thing, order the DVDs.
Closer to home, the man behind TechNet Magazine, the guy who puts our content together, Joshua Hoffman, is about to get married. By the time you read this, he will have wed his fiancée Jaclyn, and will be off honeymooning in Europe. We wanted to put a little photo in, but the words "not happening" kept being tossed around the office, so instead we’ll have to settle for the photo of the crowds at our Xbox drawing. We wish Josh and Jaclyn the best of luck; if you’d like to do the same, you can send them best wishes at nuptials@microsoft.com. —J.T.
Thank you to the following Microsoft technical experts: Jason Buffington, Thomas Deml, Paul Donlan, George Holman, Jesper Johansson, Paul Jones, Ayla Kol, Paul Leach, Jon Markarian, Michael Murgolo, Frank Oliver, Steve Riley, Tali Roth, and Liqiang (Larry) Zhu.


 Toolbox: New Products for IT Professionals
Toolbox
New Products for IT Professionals
Greg Steen

Compare and Synchonize SQL Databases
SQL Bundle Standard
www.red-gate.com/products/sql_bundles/index.htm
Whether I am developing data-driven applications, acting as a configuration manager, a QA analyst, or an operations specialist, it seems I am always moving SQL databases around different environments, verifying the integrity of a database, or trying to move data from one environment to another. Red-Gate Software has put together a set of applications that simplifies all those tasks into a few clicks each.
SQL Bundle Standard wraps up three handy applications: SQL Compare, SQL Data Compare, and SQL Packager. All three are fully compatible with both SQL Server™ 2000 and SQL Server 2005, so you should be able to leverage the tools across all your environments. And if you are planning to migrate from SQL Server 2000 to SQL Server 2005, the SQL Packager can greatly simplify the process.
The SQL Compare application compares and synchronizes the structure between two instances of your database. Setting up the baseline for comparison takes only a few clicks and when finished, the application highlights differences within the GUI, color-coding any discrepancies in your data schema.
In addition to comparing two live SQL databases, the application allows you to compare a live database to a snapshot or compare two snapshots taken with the embedded snapshot utility within the application. SQL Compare not only evaluates tables, but also compares database users, stored procedures, and any referential integrity or dependency settings you have in the database. After analyzing the structure of the two databases (or snapshots), you can run the Synchronization Wizard to bring the questionable or out-of-date database to the current version. This is a great way to migrate your code from your development environment to your integration or QA environments.
The application will also generate transaction-based SQL scripts of the operations if you want to synchronize at a later date. After you use SQL Compare to ensure your schemas are in synch, you can head on to SQL Data Compare, which, as the name suggests, helps you synchronize the content of two databases or move data between environments.
Again, with a couple of clicks, the tool automatically scans the data within your source database or database snapshot and compares it to another instance or snapshot. You can synchronize rows, whole tables, or the entire database if you want or, if you must, you can even choose to synchronize data row-by-row. The application allows you to generate SQL scripts of the synchronization tasks in case you want to run the SQL statements at a later point or add the script to your automated build or deployment processes. Another nice feature of both SQL Compare and SQL Data Compare is the ability to switch the directions of the synchronization in case you need to back fill your normal source database.
Finally, the SQL Packager component provides simple tools to help you package both the database structure and data into a SQL script, which makes installation or deployment a breeze. This application also allows you to create a package that will update an existing database's data and structure using the resultant SQL script.
Red-Gate offers SQL Compare, SQL Data Compare, and SQL Packager as standalone purchases, but if you can use one, you will want to use all three and the incremental cost makes the bundle price definitely a worthwhile investment.
Price: $295 direct for individual products.

Capture Screenshots and More
SnagIt
www.techsmith.com
Screenshots may not seem a necessity for the everyday PC user, but for the IT pro, they can be indispensable. TechSmith’s SnagIt 8.0 has developed into a tour-de-force for the screenshot world, delivering tons of features for a pretty reasonable price. OK, $39.95 for a screenshot tool might seem a little steep, but TechSmith put features in that I didn’t even know I might need.
The application has a number of different image capture profiles that allow you to control the behavior of your capture. For example, you can select a specific screen region, grab the active window, the whole screen, or all the elements of a scrolling window including the area outside of your screen. Images can be saved as PNG, GIF, JPG, SWF, BMP, PDF, and even as Windows® cursor or icon files.
SnagIt also has links to send your capture directly to Instant Messenger, FTP, e-mail, and your printer of choice. Even if your capture isn’t quite up to par with your needs, SnagIt gives you a number of tools to tweak and edit the shot before sending it on to its final home. You can edit colors, resize the image, add captions, add perspective and shear, spotlight certain areas, and even add edge effects to give a dramatic effect to your capture.
In addition to standard image capture, there are three other capture modes within the application: text, video, and Web. Text mode allows you to snag text from windows that don’t normally allow copy and paste operations, such as error messages, tool tips, and the file list in Microsoft® Internet Explorer®. Video capture mode will record your on-screen activities (don’t forget to turn off Instant Messenger) and save the video as an AVI file. Starting and stopping recording is as simple as hitting your capture hot-key once to start and once again to stop. Finally, Web capture allows you to grab all the images from a Web page and save them in a folder.
Whether you are developing applications, troubleshooting hard to replicate issues, or just coming up with a good marketing shot for your Web site, SnagIt definitely deserves a place in the IT Toolbox.
Price: $39.95 direct.

Recover Lost Files
Handy Recovery
www.handyrecovery.com
When a file is deleted by regular means (not by a disk-wiping or complete erase utility, for example), the majority of the file remains intact; just the pointers are removed. Assuming your hard disk doesn’t overwrite any of that file area, it is possible to recover deleted files with a file recovery utility and Handy Recovery 3.0 from SoftLogica may be just what you need.
The application is a simple and easy to use program that runs on all Windows® operating systems from Windows 95 to Windows Server™ 2003, and supports FAT12/16/32, NTFS, and NTFS5 file systems. The application supports both hard disks and removable media formatted with one of those file system types. Handy Recovery even recovers EFS-protected files on NTFS. Another nice feature is its ability to create a disk image of a logical drive and save it to a file on a different drive so you can analyze and try to restore files on that image at a later date. As the application analyzes the drive in question, it shows you the probability of a successful recovery for each file. You can also take a binary look at the data in partial files and try to recover files from deleted and quick-formatted partitions.
Price: $30 direct.

Book Review
Intrusion Prevention Fundamentals
www.ciscopress.com
The Intrusion Prevention System (IPS) has finally moved from a catch-phrase thrown around the conference room table to something you can use to help mitigate security threats. IPS evolved from the Intrusion Detection System when the industry realized that just detecting a threat was not enough. Today’s IPS solutions are actually a hybrid of network-based IDS and IPS with some host-based end-point security solutions.
If you’re looking for an introduction to intrusion prevention, pick up Intrusion Prevention Fundamentals: An introduction to network attack mitigation with IPS by Earl Carter and Jonathan Hogue (Cisco Press, 2006). Although the book’s examples are tailored toward Cisco’s line of IPS, it provides a nice overview of the technology and methodology in a general sense.
The book starts with a look at the evolution of computer security threats, including examples of some famous attacks like the Chernobyl virus, and then details the different methods of threat detection. The operational practices section offers insight into what it takes to deploy, monitor, and tune an IPS in your environment. Here you’ll find great information on what to ask a vendor before you invest in an IPS solution. The book then looks at host-based and network-based intrusion prevention—offering a lot of detail on how attackers can exploit your systems and how to configure your solution to protect your infrastructure.
The section on deployment is, not surprisingly, the most Cisco-centric part of the book. Still, IT pros looking to implement an IPS of another brand will find these chapters useful. Overall, this book will benefit any IT professional looking for an introduction to intrusion prevention.
Price: $50 list.

Monitor and Manage Processes
Autoruns
http://www.sysinternals.com/Utilities/autoruns.html
Mark Russinovich and Bryce Cogswell of Sysinternals have released another version of their great start-up monitor Autoruns. Taking start-up monitors to a whole new level, this application tracks much more than your logon start-up programs.
The tab-based GUI shows you everything from LSA providers to Scheduled Tasks, so if you are looking to isolate a start-up issue on your system, this might be your application of choice. You can disable any item by simply unchecking a box. The Autoruns application also gives you a nice view into the details of the Explorer process and its supporting DLLs and applications. Here you could easily disable any problem applications or associations that are wreaking havoc in your desktop environment. You can also see the different device drivers, print monitors, and services currently set to auto-run on your system.
The application also checks for browser helper objects, URL search hooks, toolbars, and extensions to Internet Explorer so you can easily see and deactivate any rogue or misbehaving process. Autoruns lets you save the current view as a text file, so you can easily keep a record of a problem system for reports or documentation.
Another simple but cool feature of Autoruns is that it allows you to right-click on any driver, library, or executable and have the application automatically take you to MSN® Search or Google to search for information regarding the scrutinized process. The download has both a CLI and GUI version of the application included in the ZIP file.
Price: Free.

Greg Steen is a technology professional, entrepreneur, and enthusiast. He is always on the hunt for new tools and methods to help make operations and development easier for IT professionals.


 Utility Spotlight: Scripting Trouble-Free Downloads with BITS
Utility Spotlight
Scripting Trouble-Free Downloads with BITS
Michael Murgolo
Download the code for this article: UtilitySpotlight2006_08.exe (158KB)

Many of us in Microsoft Consulting Services spend lots of time downloading large ISO files and virtual machine images from our primary file servers in Redmond, and it’s so frustrating to have those downloads fail when a wireless or remote access connection is dropped. And though tools like Robocopy and Xcopy can be used in restartable mode, they require manual restart after a machine restart. Fortunately, the Background Intelligent Transfer Service (BITS), which is built into Windows® XP, Windows Server™ 2003, and Windows Vista™ can help you easily manage file downloads. Software and update distribution products like Systems Management Server (SMS) 2003, Windows Server Update Services, and Windows Automatic Updates use BITS to transfer updates and software to computers with limited end-user disruption.
BITS is designed to have minimal impact on resources; it transfers files using leftover network bandwidth. So if you’re using 60 percent of your bandwidth for other tasks, BITS will only use up to the remaining 40 percent and no more. Even better, if you get disconnected or need to restart your machine, the file transfer does not have to be restarted; when the network connection is re-established, BITS continues where it left off.
You can download files with BITS using a command-line tool called the BITS Administration Utility (Bitsadmin), which is found in the Windows Support Tools. With Bitsadmin you can achieve "set-and-forget" downloading. In this column I’ll describe how you can use Bitsadmin with a sample VBScript to automate BITS jobs.

BITS and Bitsadmin Versions
How you use Bitsadmin depends on which version of BITS you have installed on your operating system. Figure 1 lists BITS versions and features as well as the location of the downloads.
Figure 1 BITS Features (Close Figure 1)

Version Description of Features
Version 2.5
  • Added support for certificate-based client authentication for secure HTTP transports and custom HTTP headers.
  • Included in the Windows Vista and Windows Server “Longhorn” operating system. A redistributable will be available for the Windows Server 2003, Windows XP, and Windows 2000 SP3 operating systems from the Microsoft Download Center.
  • The version of %windir%\System32\QMgr.dll is 6.7.xxxx.xxxx.
Version 2.0
  • Added support for performing concurrent foreground downloads, using server message block (SMB) paths for remote names, downloading ranges of a file, changing the prefix or complete name of a remote name, and limiting client bandwidth usage.
  • Included in Windows XP SP2 and Windows Server 2003 SP1. You can also download BITS 2.0 for Windows Server 2003, Windows XP SP1, Windows XP, Windows 2000 SP4, and Windows 2000 SP3. To download BITS 2.0, go to the Microsoft Download Center and install KB 842773.
  • The version of %windir%\System32\QMgr.dll is 6.6.xxxx.xxxx.
Version 1.5
  • Added upload and upload-reply capability, command-line execution for events, explicit credentials, and support for Windows 2000.
  • Included in Windows Server 2003. A redistributable is available for Windows XP and Windows 2000 SP3 from the Microsoft Download Center.
  • The version of %windir%\System32\QMgr.dll is 6.5.xxxx.xxxx.
Version 1.2
  • Same functionality as version 1.0. Contains internal upgrades and improvements.
  • Included in Windows XP SP1 and Windows 2000 SP3.
  • The version of %windir%\System32\QMgr.dll is 6.2.xxxx.xxxx.
Version 1.0
  • Initial release. Provides prioritized, throttled, and asynchronous downloads in the background or foreground. The downloads automatically resume after computer restarts and network disconnects.
  • Included in Windows XP.
  • The version of %windir%\System32\QMgr.dll is 6.0.xxxx.xxxx.
Bitsadmin is versioned based on the BITS version. Figure 2 shows the version of BITS and the source for the corresponding version of Bitsadmin.
Figure 2 Bitsadmin Versions (Close Figure 2)

Version | Bitsadmin Source
Version 2.0 | Windows Server 2003 Service Pack 1 32-bit Support Tools Windows XP Service Pack 2 Support Tools go.microsoft.com/fwlink/?LinkId=68007
Version 1.5 | Windows Server 2003 Support Tools Windows Server 2003 CD, Support\Tools folder
Version 1.2 | Windows XP Service Pack 1 Support Tools Windows XP Service Pack 1 CD, Support\Tools folder
Version 1.0 | Windows XP RTM Support Tools Windows XP RTM CD, Support\Tools folder
If you’re going to use BITS and scripting for your downloading tasks, it’s a good idea to have at least BITS 2.0 installed. BITS 1.5 and later allow an action to be executed after the transfer is complete. BITS 2.0 allows transfers from SMB server shares as well as HTTP transfers, so the source file(s) for a BITS job can be on a file server. BITS 2.0 also includes a new BITS policy setting called Maximum Network Bandwidth, which you access in the Computer Configuration\Administrative Templates\Network\Background Intelligent Transfer Service node of the Group Policy Object Editor. If you have Windows XP Service Pack 2 (SP2), Windows Server 2003 SP1, or later, you have at least BITS 2.0. It is also available as a download for older service packs.

Bitsadmin Commands
Now let’s look at a series of Bitsadmin commands used to create a BITS job, as shown in Figure 3. The example assumes that the client computer is running Windows XP or Windows Server 2003 with BITS 1.5 or higher, and that bitsadmin.exe is located in the %SystemRoot%\system32 folder (generally C:\Windows\system32 by default on a new Windows XP installation).
Figure 3 Bitsadmin Commands (Close Figure 3)
bitsadmin /create myDownloadJob
bitsadmin /addfile myDownloadJob http://www.contoso.com/downloads/test.exe
    C:\Downloads\Test.exe
bitsadmin /SetNotifyCmdLine myDownloadJob "%SystemRoot%\system32\bitsadmin.exe"
    "%SystemRoot%\system32\bitsadmin.exe /complete myDownloadJob "
bitsadmin /resume myDownloadJob
The Create command creates the BITS job in a suspended state. You add files to the job with the AddFile command. The SetNotifyCmdLine switch lets you define a command to launch when the transfer is finished. In this case it runs the Bitsadmin command to Complete the BITS jobs. (A job must be completed after the transfer is finished.) The Resume command starts the download.
For complete Bitsadmin command syntax, at a command prompt you should type: bitsadmin /?

Sample VBScript
I created a sample VBScript—BITSJob.vbs—to avoid having to create command shell scripts manually with Bitsadmin commands. You can use the script with Bitsadmin.exe to create BITS jobs that will download a URL file, a single UNC shared file, or an entire Universal Naming Convention (UNC) shared folder. (There is no scriptable COM interface to BITS for use in VBScript or other languages, so Bitsadmin is currently the only tool for scripting BITS.)
To use BITSJob.vbs, first make sure your computer has BITS 2.0 or higher. Next, you need bitsadmin.exe. If you are using Windows XP or Windows Server 2003, use the version from the Windows Server 2003 SP1 Support Tools. (The Windows XP SP2 Support Tools version has a known issue that does not allow it to be used correctly with the Windows Script Host WshShell.Exec method to capture StdOut.) Windows Vista already has bitsadmin.exe in the OS. Due to the quirky syntax of the Bitsadmin SetNotifyCmdLine command syntax, the location of bitsadmin.exe must not have any spaces in its path. So for this version of the script, the bitsadmin.exe location is hardcoded to %SystemRoot%\system32. You must copy it there on Windows XP or Windows Server 2003. (It’s already in this location on Windows Vista.)
BITSJob.vbs works in two stages. You use it first to create a CMD file, then the CMD file is used to actually execute the bitsadmin commands. The remote file or folder needs to be available when you run the script or CMD file. The advantage of this is that you can look at the total file count in a remote folder (shown in the CMD file) to see if there are too many files for BITS to handle in one job. Due to limitations of BITS, it is not recommended for jobs with more than 100 files. You can comment out any files you decide not to add to the job. (An optional switch for BITSJobs.vbs lets you execute the CMD file immediately after it is created.) Use bitsadmin’s /list or /info switches to check on the status of your BITS jobs.
BITSJobs.vbs is available at Code Downloads. I’ve included a small CMD file along with the download that shows examples of the BITSJobs.vbs switches. You can also get help by executing
cscript BITSJobs.vbs /?
at a command prompt.
For more on BITS, see Background Intelligent Transfer Service in Windows Server 2003. Also check out technet.microsoft.com and msdn.microsoft.com.

Michael Murgolo is a Senior Infrastructure Consultant for Microsoft Consulting Services. He is focused in the areas of operating systems, deployment, network services, active directory, systems management, automation, and patch management. He is a subject-matter expert in the area of desktop deployment and migration.


 SQL Q&amp;A: Clusters, Snapshots, Log Shipping, and More
SQL Q+A
Clusters, Snapshots, Log Shipping, and More
Edited by Nancy Michell

Cluster vs. Standby Server
Q I have a SQL Server 2000 failover cluster. My system and user databases are replicated to a single machine in another building because I don’t have enough machines to create a second cluster there. I need to manage disaster recovery with a single machine. What is the best way to configure that machine: use a SQL Server failover cluster with only one node or a standalone SQL Server (standby server) with the original virtual server name?

A There are a few things to keep in mind when making your decision. If in the future you acquire more machines you can add additional nodes relatively easily to a one-node failover cluster, but you’d have to upgrade a standalone instance to a failover cluster before you could add more nodes, which is not as easy (though it is supported in Microsoft® SQL Server™ 2000). See the following SQL Server Books Online articles for information on this topic: "How to: Create a New SQL Server 2005 Failover Cluster" and "How to upgrade from a default instance to a default clustered instance of SQL Server 2000".
A standalone SQL Server 2005 instance cannot be upgraded to a failover cluster without reinstalling the SQL Server instance as a failover clustered instance, so if it were a version 2005 instance and there was a chance you’d want to eventually add more nodes you should start with a single-node cluster.
For your storage-level replication be sure the data and log being replicated on the disaster recovery site actually represent the state of the data and log that appeared at the primary site at some point in time. You don’t want your data files to get too far ahead of your log files, or vice versa. There is a relationship and ordering between the files that must be preserved (via consistency groups or some other mechanism) so that the database will be recoverable.
Also be sure you’re testing the disaster recovery site periodically, and be aware that there is other state information that is outside of the data and log files. For example, changes to the registry will not magically appear on the remote site like they would with a geographically dispersed cluster.
Finally, remember that for SQL Server 2000, the standalone will be easier to rename. See Knowledge Base article 307336 "The SQL Server Network Name resource cannot be renamed". Also note that renaming the SQL Server virtual or clustered servers is not supported in SQL Server 2000.

Database Access During Snapshot
Q If my application tries to access a snapshot during the drop/creation of a new snapshot, I presume it will get an error and I will have to resubmit my query. With log shipping, when the database is unavailable due to the application of a new log, will my application also get an error, or will it just run slower while the log is being applied?

A With regard to your question about what happens to your client application when a log is being restored, there are two possibilities. If the application is connected when you attempt the restore, then the restore will fail since it’s an exclusive-use operation. You have the option to break the connection before the restore, but if you do that then your application will see the disconnect. The second possibility is that the client attempts to connect while the restore is in process. If this happens, then either the connection attempt will time out, or will successfully wait until the restore is complete.
The important concept to understand is that you can’t read the database while a log is being restored; restore is an exclusive-use operation.
You have two options for dealing with this: one is to prevent logs from being applied when users are connected to the database, and the other is to close user connections to the database when logs are being applied.
On a practical level, you will find that there will be times when users can’t access the log shipping standby regardless of which option you choose. Log shipping was never designed for a reporting solution despite the fact that you are technically able to do it if you restore the main database using WITH STANDBY. The main purpose of log shipping is for use in availability and disaster recovery. This question has come up quite frequently and the answer is always the same. If you need a reporting solution, use something else. Log shipping needs exclusive access while loading transaction logs (TLogs) for however long that takes.

Drive Space During Installation
Q I need to upgrade a Systems Management Server (SMS) 2003 SP2 server, which is running SQL Server 2000 SP4. The server has a C drive with about 1.2GB of free space and a D drive with about 10GB free. SQL Server 2000 is installed on the D drive. During the setup for SQL Server 2005, I selected Advanced and gave it a path of D:\Program Files\Microsoft SQL Server, yet I can’t install because I don’t have enough free space. How can I install everything on D as with SQL Server 2000?
A To successfully install SQL Server 2005 you’ll need about 2GB of free space on the system drive even if you’ve elected to install all SQL Server components to a non-system drive. There are SQL Server files that must go to %Program Files%\Microsoft SQL Server\90. In addition, Windows® Installer caches the installation package to the system drive until the installation is complete; the cache is temporary and is deleted when setup finishes.
To install SQL Server components to another drive, choose an installation location for each feature in the Advanced Feature Selection dialog of SQL Server setup. Another option is to use the command-line utility written by Gert Drapers to temporarily divert "Program Files" installs to an alternative drive.

Hardware Recommendations
Q I need some advice on hardware for my SQL Server 2005 migration. The vast majority of my line-of-business applications currently run on SQL Server 2000. What are your recommendations?
A It sounds like you want a prescriptive guide to moving from SQL Server 2000 running on server A to SQL Server 2005 running on server B. Unfortunately, it’s not that simple as servers have changed a lot since the introduction of SQL Server 2000, as has the capabilities of SQL Server itself. You probably don’t want the new server to just run the same old apps at the same old speed. Chances are you want a boost in throughput or performance, or want to buy less hardware to consolidate some servers.
There are a couple of SQL Server 2000 sizing tools from Microsoft, HP, and Unisys (look under the Build section). Another resource to try is the Windows Server™ System Reference Architecture (WSSRA) site which includes hardware recommendations for each piece of hardware in the reference architecture. If you are considering server consolidation, then the following whitepaper is a good place to start: "Planning for Consolidation with Microsoft SQL Server 2000".

SQL Server Security
Q I need to deploy SQL Server Express and a Microsoft .NET Framework 2.0-based smart client application to many users. I need to seal the database (especially the application data) so none of the users can see or alter the data. The systems administrator should not be able to see or change the application data, but he should be able to drop tables or attach and detach databases. I also need to prevent anybody from seeing the data if they detach the database and attach it to another server. How can I accomplish this?
A One solution is to encrypt the data in the application before it is written to the database. Alternatively, you could use column-level encryption, a new feature in SQL Server 2005. The big issue is that once a column is encrypted at the SQL Server level, performance during data modification or lookup suffers. If you are going to encrypt all the columns, a simple lookup will require an on-the-fly decryption of all the data in the table to find what you are looking for. If the encryption is done at application level, the client application will have to decrypt the data and then find the rows it’s looking for. Either way, you may want to consider a partial encryption, leaving the primary key and foreign keys unencrypted.

Log Shipping Shrinks TLog
Q I have configured log shipping between two SQL Server 2000 installations. The problem is that the TLog of the database that is configured for log shipping, after being shipped on the standby SQL Server, remains shrunken on the source server.
The TLog impacts performance by growing quite large on a regular basis, until it shrinks again at the next shipping. Is it possible to avoid the shrinking action on the source SQL Server?

A A log shipping TLog backup will not shrink the physical log file. The used space in the files will go down as logical log files are released and truncated, but the physical space will remain.
Shrinking the database/log file with DBCC SHRINKDATABASE or SHRINKFILE will queue deferred log file shrinking and it will not take effect until the tail of the file has been backed up, which sounds exactly like what you’re experiencing. If you do not want the log file to return to the allocation size you can create the file at the size you want and the shrink action will not shrink the log file to smaller than the size at which the file was created.
To avoid shrinking the log files, turn off autoshrink on the database and use DBCC SHRINKFILE on the data files instead of DBCC SHRINKDATABASE if you are manually shrinking the database.

Log Shipping and Data Loss
Q Let’s say I have a log shipping job that backs up TLogs set up to run every 15 minutes. The log shipping job executes at 10:00 A.M. and the database server goes down at 10:05 A.M. Let’s also assume that I have data that comes in between 10:00 and 10:05 and I lose the .mdf file on the primary server, but still have the .ldf file. Would I be able to restore the secondary server up to 10:05 A.M.? How can I minimize the possibility of data loss when using log shipping?

A To minimize the possibility of data loss, you can increase the frequency of the backup job and implement better logic to do the failover when the primary server goes down. But as you can see, no matter what you do, there will always be a window during which data loss can occur. So this should be part of your deployment plan.
You can also back up the data in the .ldf file without the .mdf file (in SQL Server 2000). The challenge is that you need to run the BACKUP LOG WITH CONTINUE_AFTER_ERROR (SQL Server 2005) or BACKUP LOG WITH NO_TRUNCATE on the primary server.
Some people use hardware mirroring on the log volumes to ensure that the log is available on a remote site in order to cover the case where the primary server is unavailable.

Bulk Insert with XML Formatfile
Q I’m trying to load ST_ENDERECO.txt into the table tb_endereco using the format file fmt_endereco.xml.
BULK INSERT dbo.tb_endereco
FROM ‘C:\BulkInsert\ST_ENDERECO.TXT’
WITH TABLOCK
     (formatfile = ‘C:\BulkInsert\fmt_     endereco.xml’,
)
When I do this via a bulk insert, I get an error stating that string or binary data would be truncated. However, when I do it using openrowset (see Figure 1), it works fine. Can you tell me what I am doing wrong in bulk insert process?
Figure 1 Using OPENROWSET (Close Figure 1)
INSERT dbo.tb_endereco
  (id_integracao, tp_logradouro, no_logradouro,
  nu_endereco, no_complemento, no_bairro, no_cidade, sg_estado, cd_cep)
    SELECT t1.*
        FROM  OPENROWSET(BULK  ‘C:\BulkInsert\ST_ENDERECO.TXT’,
        FORMATFILE=‘C:\BulkInsert\fmt_endereco.xml’
        AS t1

A First of all you need to know that the bulk load statement and the INSERT...SELECT...FROM OPENROWSET statement are not equivalent.
With the format file provided, bulk insert would map the first nine columns (including the identity column) to the nine source fields in the data file. Since you haven’t specified the KEEP_IDENTITY option the data for the identity column would be automatically omitted.
The equivalent INSERT...SELECT...FROM OPENROWSET would look like Figure 2, and it would fail with exactly the same error.
Figure 2 Equivalent Code (Close Figure 2)
INSERT dbo.tb_endereco
  (id_integracao, tp_logradouro, no_logradouro,
  nu_endereco, no_complemento, no_bairro, no_cidade, sg_estado)
    SELECT
  tp_logradouro, no_logradouro,
  nu_endereco, no_complemento, no_bairro, no_cidade, sg_estado, cd_cep
        FROM  OPENROWSET(BULK  ‘C:\BulkInsert\ST_ENDERECO.TXT’,
        FORMATFILE=‘C:\BulkInsert\fmt_endereco.xml’
       ) AS t1
Unfortunately, this is one case where the new style format file does not allow you to express your intention (you want to skip the first target column and map the rest of the target columns to the source fields). In order to work around the issue you have two options.
You could create a view that would skip the identity column:
CREATE VIEW v_tb_endereco
AS
SELECT
id_integracao, tp_logradouro, no_logradouro,
   nu_endereco,
   no_complemento, no_bairro, no_cidade,
   sg_estado, cd_cep
FROM tb_endereco
And then issue the original bulk insert statement, like so:
BULK INSERT v_tb_endereco
FROM ‘C:\BulkInsert\ST_ENDERECO.TXT’
     WITH (formatfile = ‘C:\BulkInsert\fmt_
     endereco.xml’,tablock)
Alternatively, you could use the equivalent older style format file with the same bulk insert statement.
The XML format file is geared more towards generating a rowset from datafile. So for example, you can do this:
SELECT *
FROM OPENROWSET (BULK,  ....)
You will notice that the XML format file provides you with an explicit type for the column while in the old format file it is derived from the type of the field. Now, you typically load a rowset into a target table, using INSERT and SELECT statements like the ones in this code snippet:
INSERT INTO <target table> (<col-list>)
SELECT *
FROM <a rowset>
In this command, you need to explicitly provide that mapping of columns as generated by the select statement to the columns in the target table.
The old format file was meant for bulk import, so it also had the target table column mapping. But as indicated, you can use view to get around this problem with XML format file. In this case, the view definition is actually doing the column mapping for you. SQL Server Books Online has more information on these topics.
Thanks to the following Microsoft IT pros for their technical expertise: Sunil Agarwal, Yunwen Bai, Batu Balkanli, Byron Boudreaux, Patrick Conlan, Gert Drapers, Michael Epprecht, Fernando Ferreira, Padma Ganapathy, Alessandro Gregnanin, Mucahit Gundebahar, Allan Hirt, Dan Jones (SQL), Tengiz Kharatishvili, Steve Lindell, Saverio Lorenzini, Jeff Mayer, Ivan Penkov, Steve Schmidt, Margi Showman, Chris Skorlinski, Stephen Strong, Derek Tan, Don Vilen, Steven Wort, and Sedat Yogurtcuoglu.

Edited by Nancy Michell


 How IT Works: Certificate Services
How IT Works
Certificate Services
Randy Muller

The term PKI (or public key infrastructure) strikes fear into the hearts of many network administrators. But, in reality, network administrators should draw comfort from the thought of PKI, knowing that it provides a secure means of communication on their network.
Essentially, PKI provides credentials (in the form of certificates) which can be used by applications to sign and/or encrypt data and communications. In public key cryptography, a pair of associated keys—one public and one private—are used for encrypting and signing data. Simply put, the data is encrypted with one of these keys and decrypted with the other. If, for example, I want to send a signed e-mail, my mail program creates a one-way hash of my message, which I encrypt using my private key. The recipient then uses my public key to verify the signature, ensuring that the e-mail was actually sent by me. I can send my public key to any individuals who need it, while I keep my private key well guarded.
Certificate Services is the essential component of a Windows®-based PKI. If you deploy an application that is PKI aware and you want to make use of the security capabilities offered by PKI, you will need Certificate Services. This is used to manage the lifecycle of certificates for the applications and accounts that require them. It covers the request, issuance, enrollment, publication, maintenance, revocation, and expiration of certificates. Certificate Services also provides information assurance, meaning that measures are taken to safeguard aspects of information and information systems. This is done by ensuring authentication, confidentiality, integrity, and nonrepudiation.
After you decide what types of PKI-enabled applications you are going to deploy, you have to decide which accounts will make use of these applications—thus, determining which accounts (or security principals) will be issued certificates. A security principal is the subject of the issued certificate. It can consist of computers, services, and users. A computer or user receives a certificate to establish its unique identity with PKI-aware applications. Services, meanwhile, use certificates to identify themselves on the network and authenticate themselves to other services, computers, or users. (For more information about when you might use PKI, see the sidebar "Common PKI Uses.")
Common PKI Uses
There are a variety of applications and technologies that are PKI compatible. Here are some common uses of PKI and certificates.
  • 802.1X Controls access to wired and wireless networks, ensuring that only authorized users and computers are granted access.
  • Encrypting File System (EFS) Allows users to encrypt data and share encrypted files, as well as provides for the establishment of Recovery Agents.
  • Secure Internet Used for authentication between a client and server, while encrypting the communication between the client and server.
  • IPSec Enables the creation of an authenticated communications channel between two systems, as well as the encryption of traffic once the session is established.
  • Secure E-Mail Verifies the sender and ensures the message has not been tampered with in transit.
  • Smart Card Logon Provides two-factor authentication to provide an additional layer of security in which the user must have the smart card and know the PIN associated with it.
  • Software Code Signing Provides signing and authorization of code (both drivers and applications).
  • Software Restriction Policy Provides a mechanism that can be used to prevent programs that are not authorized from being executed.

The Underlying Certificate
A digital certificate is an electronic credential that is based on a standard schema (such as X.509 or XRML). It consists of a variety of properties, such as who the certificate is granted to (Subject), the expiration date (Valid to), and what certificate authority it was granted by (Issuer).
Certificates can be used for several purposes. In some instances, you may want a single certificate to map to an individual user account—this is known as one-to-one mapping. In this case, the certificate might be issued to a user for Encrypting File System (EFS) or secure e-mail purposes. Alternatively, you might want a many-to-one mapping, in which multiple certificates are issued to a single user account for several purposes.
A certificate server issues a certificate based on who or what requested the certificate. The client computer generates the public and private keys. It keeps the private key on the local machine (or in the user’s profile) and sends the public key to the certificate authority (CA). The CA verifies the identity of the requestor and that the requested certificate is valid for the given requestor. If the requested certificate is permitted, the CA signs the public key and binds it to a certificate. Finally, the CA issues and manages the certificate until it is revoked or expires.
An actual certificate can be stored in one of several locations depending on how it is going to be used. If, for instance, a certificate is being used for a smart card, then it is stored right on the smart card. If, on the other hand, a certificate is issued for a user, then it is stored either on the user’s local hard drive or in another secure location accessible by the user.

The Infrastructure
The first step in planning an implementation is to determine its scope and purpose. Then you can decide on the types of certificates it will require and how your CA hierarchy should be structured. Generally, certificate hierarchies are three levels deep, consisting of an offline root CA, a policy (or intermediate) CA, and the issuing CAs (see Figure 1). The specifics, however, depend on the details of your overall deployment.
Figure 1 Certificate Authority Hierarchy
Figure 1 Certificate Authority Hierarchy
CAs can be one of either two types: enterprise or standalone. An enterprise CA is connected to the network and requires communication with Active Directory®. A standalone CA, on the other hand, does not require communication with Active Directory and can be removed from the network. An environment can contain a mixture of enterprise and standalone CAs. The root CA is often standalone while the issuing CAs are enterprise to facilitate user requests for certificates (the policy CAs are frequently either standalone or enterprise).
The root CA is the first server, or top level, in the CA hierarchy. A root CA can self-sign its own root certificate or use a certificate from a third party. Requests made to the root CA are automatically approved or denied based on the requestor and the permissions associated with the certificate template. Certificate requests to an enterprise root CA can be made by the Certificate Request Wizard or by the certification authority Web page.
A standalone CA does not require Active Directory and is typically removed from the network to become an offline root CA to maintain a higher level of security. If Active Directory is present, a standalone CA will use it to publish certificates and the certificate revocation list (CRL). All certificate requests made to a standalone CA must be approved or denied manually by the designated certificate manager. (With standalone CAs, certificate templates are not used and there are restrictions with smartcards.)
The next level is the policy CA. This is subordinate to a root CA and is certified (by the root CA) to issue and authorize certificates for other requesting CAs (the issuing CAs).
Below the intermediate CAs are the issuing CAs. This is the level with which users interact. Issuing CAs are authorized by a higher level CA (usually a policy CA) and are responsible for issuing certificates to the applications and subjects that request them. (In some scenarios, each issuing CA is responsible for just one type of certificate.)

Installing Certificate Services
After you’ve identified your certificate requirements and planned your CA hierarchy, you need to install the actual Certificate Services. To do this, open Add or Remove Programs and select Windows Components. Choose Certificate Services, and follow the wizard to select what type of server you wish to install, specify a common name for your CA (see Figure 2), and indicate the location where the database and log file will be installed. Once you’ve installed Certificate Services, you are ready to issue certificates.
Figure 2 Naming the CA
Figure 2 Naming the CA
Another method for creating a CA is through the use of a CAPolicy.inf file. This is an optional file that you create manually and store in the %windir% folder. It is used to configure Certificate Services during installation and for the renewal of the CA’s own certificate. You do not have to use a CAPolicy.inf for creating a CA, but it is required for an offline root CA. The CAPolicy.inf contains configuration information for distribution points, CRLs, key usage, Certificate Practice Statements (CPS), key validity period, and so on.

Issuing and Revoking Certificates
When security principals and applications request certificates, the requests can take many forms depending on who is requesting the certificate and what type of certificate is being requested. There are five methods used for requesting certificates: Autoenrollment, Certificates Console, Certreq.exe, enrollment agents, and Web-based requests. Here’s a simple overview of how they differ:
  • Autoenrollment allows Windows XP and Windows Server™ 2003 clients to automatically request and renew user and computer certificates. Group Policy and version 2 certificate templates are used to enable autoenrollment.
  • A user or a computer can use the Certificates Console to request a certificate from an enterprise CA. You use the Microsoft Management Console (MMC) to access the Certificates Console, which allows you to access the Certificates Request wizard.
  • A user can submit a request to a CA through the use of Certreq.exe. This provides for the management of requesting, retrieving, installing, and signing certificates from a command line.
  • An enrollment agent can request a certificate on behalf of a user for a Smart Card Logon certificate or a Smart Card User certificate.
  • The Web-based request form lets a user request a certificate from an enterprise CA. It can also be used to verify requests made to the CA.
To use the Web-based enrollment method, a user must open Microsoft® Internet Explorer® and type http://servername/certsrv in the address line (where servername is the name of the server on which Certificate Services is installed). The user can choose the type of certificate. For example, a user can request an advanced certificate by selecting Create and submit a request to this CA. In the Advanced Certificate Request window, the user must choose which certificate template he wants to use (see Figure 3). Here, the user can also specify the key size used for encryption, whether the keys are exportable, and so on. The user can then click the Submit button and await approval from the enterprise CA. Once approval is granted, the user can install the certificate on his system.
Figure 3 Making an Advanced Certificate Request
Figure 3 Making an Advanced Certificate Request
At some point a certificate will no longer be valid. This could occur because the certificate reached its expiration time or the certificate was revoked due to a compromise. Whatever the situation, a list of revoked certificates is maintained and published in a CRL. This list is automatically published at specified intervals and used by applications to ensure that the certificate presented is valid and in the certificate chain (the chain of certificates issued by higher level CAs until a self-signed root CA certificate is found and validated). A certificate manager can revoke a certificate through the CA console (see Figure 4). In doing so, the manager has the option of specifying a reason for revoking the certificate.
Figure 4 Revoking a Certificate
Figure 4 Revoking a Certificate

Wrap Up
PKI comprises a number of parts, such as CAs and digital certificates, which work together to enable secure communications. When a user requests a certificate—either directly (using a Web page) or indirectly (using, for instance, EFS)—the request is verified by an issuing CA. The issuing CA has been verified by a policy CA, which in turn has been verified by a root CA. Ultimately, this verification ensures that the requestor is authorized to be issued the certificate.
TechNet Online Resource
For more information about Certificate Services, see:
Best Practices for Implementing a Windows Server 2003 Public Key Infrastructure

Randy Muller, MCT, MCSE, MCSA, MCDST, teaches a variety of networking, security, and other computer classes. He is a former Army Signal Corp Officer and has been teaching since 2000. You can contact Randy at randy@randymuller.org.


 Hey, Scripting Guy!: How Can I Delete a Key and its Subkeys from the Registry?
Hey, Scripting Guy!
How Can I Delete a Key and its Subkeys from the Registry?
The Microsoft Scripting Guys
Download the code for this article: ScriptingGuys2006_08.exe (150KB)

A few years ago, Aaron, a neighbor kid who lived a couple of houses away, would come by every day to play with one of the Scripting Sons (come to think of it, the only Scripting Son). Aaron was a nice enough kid, but he had one very annoying habit: he was always munching away on treats. The treats themselves weren’t a problem, but whatever he was eating—Popsicles, Hostess cupcakes, or what-have-you—Aaron would simply drop his wrappers where he stood, which was invariably in the Scripting Guy’s yard. At first the Scripting Guy simply picked up the wrappers and threw them away. As time went on, however, and as the wrappers started to pile up, the Scripting Guy suggested to Aaron that maybe he ought to throw his wrappers in the garbage can. As time continued to go on, these suggestions became a bit stronger. Eventually the Scripting Guy reached the breaking point. "Aaron," he said, "I’m tired of picking up your garbage every single day. From now on you are banished from our yard. Scripting Son can go over to your house to play, but you are not allowed to set foot in our yard. Ever."
The next day, when the Scripting Guy arrived home from work, he found Aaron standing in the yard, eating a Popsicle. "Hey, Aaron," said the Scripting Guy, "didn’t I tell you yesterday that you were never allowed to come in our yard again?"
Aaron was nonplussed by such a silly question. "Yes," he said, and he went back to eating his Popsicle.
If you’re on the edge of your seat, dying to know how the Scripting Guy reacted, you might be disappointed: he simply stood there for a moment and then walked into the house without saying a word. It wasn’t that the Scripting Guy wasn’t upset, it’s just that he had had a sudden epiphany: there are some things in this world that you simply can’t get rid of—Aaron was one of those things. The Scripting Guy might have considered himself an irresistible force, but he suddenly realized that, like dandelions and registry keys, Aaron (and his Popsicle wrappers) was an unremovable object.
And yes, you read that correctly: in addition to Aaron and his Popsicle wrappers, two other things you can’t get rid of are dandelions and registry keys. We know what you’re thinking: "OK, dandelions I agree with; after all, one of the Scripting Guys once found a dandelion growing in a storage room in which his son had tossed his muddy shoes. But registry keys? Registry keys are easy to get rid of; heck I can even write a script that deletes registry keys."
To tell you the truth, you’re right—sort of. Suppose you have a single registry key in the registry, something along the lines of Figure 1.
Figure 1 Dandelion? No, Registry Key
Figure 1 Dandelion? No, Registry Key
Can you delete this single registry key using a script? You bet you can; in fact, this script right here will do that for you:
Const HKEY_CURRENT_USER = &H80000001

strComputer = "."

Set objReg = GetObject("winmgmts:\\" & _
    strComputer & "\root\default:StdRegProv")
strKeyPath = "SOFTWARE\Test"

objReg.DeleteKey HKEY_CURRENT_USER, strKeyPath
"Wait a second," you say. "First you tell us that registry keys are unremovable, then you show us a script that removes registry keys. What’s the deal here?"
Well, the deal is that we showed you a script that removes a registry key; unfortunately, it won’t remove registry keys. Suppose the Test key has a couple of subkeys underneath it, as in Figure 2.
That’s a problem. The Windows Management Instrumentation (WMI) DeleteKey method can delete a single registry key, but it can’t delete multiple registry keys. If the Test key has any subkeys, the script we just showed you won’t remove a thing—try it and you’ll see what we mean.
Figure 2 Ah, They Multiply
Figure 2 Ah, They Multiply
You might not realize it, but we’re in a spot of trouble here—after all, we can’t delete the Test key without deleting Subkey 1 and Subkey 2. Even worse, we can’t delete Subkey 1 until we first delete Subkey A. And if Subkey A happens to have any subkeys, well.... Is this one of those situations where we have to just walk quietly into the house, recognizing that some things can’t be removed?
Fortunately, no. Our Scripting Guy never did figure out what to do about Aaron (other than hope that he’d eventually grow up and move away). As it turns out, though, it is possible to remove registry keys with a script, even registry keys that have subkeys. All you have to do is act recursively.
We don’t have time to explain recursion in detail today; for more information you might want to take a look at the "Microsoft Windows 2000 Scripting Guide". For now we’ll just note that a recursive subroutine is simply a subroutine that can call itself. How does that help us? Take a look at the accompanying script and then we’ll try to explain how it all works.
On Error Resume Next

Const HKEY_CURRENT_USER = &H80000001

strComputer = "."
strKeyPath = "Software\Test"

Set objRegistry = GetObject("winmgmts:\\" & _
    strComputer & "\root\default:StdRegProv")

DeleteSubkeys HKEY_CURRENT_USER, strKeypath

Sub DeleteSubkeys(HKEY_CURRENT_USER, strKeyPath)
    objRegistry.EnumKey HKEY_CURRENT_USER, strKeyPath, arrSubkeys

    If IsArray(arrSubkeys) Then
        For Each strSubkey In arrSubkeys
            DeleteSubkeys HKEY_CURRENT_USER, strKeyPath & "\" & strSubkey
        Next
    End If

    objRegistry.DeleteKey HKEY_CURRENT_USER, strKeyPath
End Sub
Our script starts off simply enough, defining a constant named HKEY_CURRENT_USER and setting the value to &H80000001. This tells the script which registry hive to work with. For more information on writing WMI scripts that deal with the registry, take a peek at this chapter in the "Scripting Guide".
We then assign the name of the computer (using a dot to represent the local computer) to a variable named strComputer, and we assign the path to the registry key we want deleted (Software\Test) to a variable named strKeyPath.
After that we connect to the WMI service, taking care to bind to the root\default namespace. (Although you typically find WMI classes in root\cimv2, you won’t find the System Registry provider there.)
Now it gets interesting. Notice that we don’t try to delete the registry key at this point. Why not? Well, as we’ve already learned, if this registry key has any subkeys we can’t delete it. Therefore, we call a subroutine named DeleteSubkeys, passing as parameters the constant HKEY_CURRENT_USER and the variable strKeyPath.
Once we’re inside the subroutine, we delete the registry key, right? Nope, sorry; remember that operation will fail if the key has any subkeys. Instead, the first thing we do is call the EnumKey method, a method that returns a list of all the subkeys (if any) found in the registry key Test. As you can see, we pass EnumKey three parameters: HKEY_CURRENT_USER, strKeyPath, and an "out" parameter named arrSubkeys. If you’re thinking, "Hey, we didn’t even assign a value to arrSubkeys," well, you’re right, we didn’t. That’s because with an out parameter, all we do is supply a variable name and the method assigns it a value. In this case, that value will be a list of all the subkeys found in Test.
EnumKey will end up telling us that the Test key has two subkeys: Subkey 1 and Subkey 2. And, no, we still can’t start deleting registry keys. Why not? Well, suppose either Subkey 1 or Subkey 2 has subkeys of their own. (And one of them does: Subkey 1 has a subkey named Subkey A.) Because we can’t delete registry keys that have subkeys, the presence of Subkey A will cause the script to fail. (And, yes, everything seems to cause this script to fail doesn’t it? But don’t lose hope.)
So what do we do now? Well, brace yourself, because this is where it really gets confusing. The first thing we do is use the IsArray function to determine whether the variable arrSubkeys is an array. If it is, the Test key does have at least one subkey (otherwise arrSubkeys would have no value at all). Because IsArray comes back True, we set up a For Each loop, one that loops through all the subkeys stored in the variable arrSubkeys (which, just to remind you, contains the values Subkey 1 and Subkey 2).
Note that, even though we’re already in the DeleteSubkeys subroutine, we go ahead and call that subroutine again. (That’s what recursion is all about.) This time we call the subroutine using the current registry path (stored in the variable strKeyPath) plus a \ and the name of the subkey we’re looking at. In other words, the first time through this loop we’ll be looking at the subkey Subkey 1. Therefore, we call DeleteSubkeys a second time using strKeyPath (Software\Test) plus \Subkey 1; in other words, Software\Test\Subkey 1. As you know, this happens to be the path to the first subkey found in the Test key.
So now what happens? Well, the DeleteSubkeys subroutine will call the EnumKey method to determine whether Subkey 1 has any subkeys of its own. As it turns out, it does. Because of that we call DeleteSubkeys a third time, this time passing the value Software\Test\Subkey 1\Subkey A.
We told you it would get confusing.
Fortunately (for the sake of our sanity), Subkey A has no subkeys of its own; that means the IsArray function comes back False. That also means that we’ll be able to skip the line of code that calls the DeleteSubkeys subroutine and, instead, call objRegistry.DeleteKey to delete Subkey A—and only Subkey A. Still with us? What we’ve done so far is follow the path down to Software\Test\Subkey 1\Subkey A and finally found a registry key with no subkeys. Because of that, we go ahead and delete Subkey A.
Good point. That’s nice, but it still leaves us with a bunch of other subkeys to delete (not to mention the Test key that we started with). How the heck do we get rid of those other registry keys?
To be honest, this is where it helps to have a little blind faith in VBScript. Each time VBScript calls the recursive subroutine, it makes a note to itself as to what’s going on. In other words, when DeleteSubkeys checks the subkey Software\Test\Subkey 1\Subkey A VBScript is well aware that the other keys exist, too.
After deleting Subkey A, VBScript automatically goes back to Subkey 1 to see if it has any other subkeys. Because it doesn’t, the script then deletes Subkey 1. The script then runs this check on Subkey 2, the other subkey found in Test. Because Subkey 2 has no subkeys, it gets deleted as well. (This will continue based on how many subkeys and sub-subkeys you might have.)
After deleting Subkey 2 VBScript goes back and checks the Test key. Because this key no longer has any subkeys, Test gets deleted, and our work is done. Try it and see.
If you feel like your head is about to explode, don’t worry (well, not unless your head actually does explode). If you’d rather not delve into the intricacies of recursion, then just use the script as is and don’t give it a second thought. Of course, having said that we recommend that you try walking through this scenario a few times until it begins to make sense. Why? Well, if you ever can get a grasp on how this script works then you’ll be able to write recursive scripts of your own. And that means being able to write scripts that, for example, enumerate all the values in all the subkeys of a registry key or list all the files in all the subfolders of a folder.
The main thing is that this scenario has a happy ending: we figured out a way to remove hitherto unremovable registry keys. That doesn’t mean this approach will work for everything: dandelions, to name one, seem immune to recursive subroutines. As for Aaron, well, as he and the Scripting Son grew older their interests began to diverge, and the two don’t spend much time together anymore. Nevertheless, the other day, when our Scripting Guy returned home, he found an empty Big Gulp cup lying in the front yard. Like we said, there are some things you just can’t get rid of.
A Note from the Scripting Guys
Weekly Scripting Puzzle
Everyone knows that if you don’t use a skill regularly you become, well, a little less skilled. (How many of us really remember our high school algebra?) Don’t let that happen to your scripting skills. The Scripting Puzzle is a fun way to hone your skills. You’ll find a new puzzle on the Script Center every Friday.

Scripting Newswire
Be the first in your neighborhood to get the scoop on everything going on in the Script Center. Or if it’s too late to be the first, don’t be left out—the peer pressure is getting to you isn’t it? Sign up now for the Scripting Newswire, delivered directly to you inbox every month. Send e-mail to scripter@microsoft.com with the subject line "Subscribe" and you’ll start receiving your newsletters in no time.

The Microsoft Scripting Guys work for—well, are employed by—Microsoft. When not playing/coaching/watching baseball (and various other activities) they run the TechNet Script Center. Check it out at www.scriptingguys.com.


 Inside Microsoft.com: Analyze Web Stats with Log Parser
Inside Microsoft.com
Analyze Web Stats with Log Parser
Jeffrey A. Johnson
Download the code for this article: InsideMSCOM2006_08.exe (151KB)

When it comes to mining data there are few tools more versatile than Log Parser. First released as part of the IIS Resource Kit, it quickly found its way into many IT pros’ toolboxes. Log parser’s main strength is that it provides universal query access to text-based data such as log files, XML files, and CSV files, as well as a number of data sources in Microsoft® Windows® including the Event Log, the Registry, the file system, and Active Directory®. Using Log Parser you can execute queries to mine data that you wouldn’t easily be able to manipulate otherwise. In this column I will explain how you issue queries using Log Parser, how it outputs data, and finally how to pivot the data you have mined.
Log Parser lets you run SQL-like commands against a large number of different input types and send the results to a variety of outputs. Out of the box, Log Parser supports the input types shown in Figure 1.
While the list of directly supported input types is impressive, you can also add your own input types. The output types supported are also shown in Figure 1. While it might not be considered an output type, the fact that Log Parser exposes a COM interface to execute queries means that you can create an application that can output pretty much any format you want.
With all of this, there are still a few things that Log Parser can’t do. It currently doesn’t support the SQL Join operation (though rumor has it that future versions will). Log Parser also doesn’t currently have a way to pivot data so I’ve decided to add that functionality here.
Figure 1: Input and Output Types
Input Types
  • All IIS and https.sys log formats
  • NCSA
  • URLscan logs
  • Comma delimited
  • Tab delimited
  • Xml files
  • Event viewer logs (including current data)
  • Event tracing for Windows logs
  • Network monitor capture files
  • The Windows registry
  • ADS sources
  • Generic text files
  • The file system
Output Types
  • Comma- or tab-delimited files
  • XML
  • Datagrid
  • GIF files using the Microsoft Office chart Web component
  • Syslog
  • W3C
  • SQL (allowing the creation of new tables)
  • A template format that allows you to define a text template to format the output

Building Queries
Now let’s first get familiar with Log Parser by implementing a few simple queries that will illustrate how to extend the functionality and usefulness of the tool. I won’t go into too much depth about how to write queries; for more information on that subject, download Log Parser and read through the help file or go to the unofficial homepage.
I’ll be writing the queries in a format that can be copied into a .SQL query file for Log Parser to execute. Assuming you save the query as a file called query.sql, the command line to execute it would be similar to this:
logparser file:query.sql?inputfile=ex*.log
The ex*.log would be replaced with the logs you want to use. I’ll be writing the scripts to parse from the IISW3C formatted input; however, modifying these to work with other inputs is as simple as changing the field names.
The first example looks at the log file to determine which HTTP status codes were sent most frequently for a given time period. The algorithm to determine this is simple: get all the status codes from the log file, group them by the value of the returned code, count the number of items in each group, and sort them in descending order by the number of items. The actual query looks like this:
SELECT sc-status, COUNT(*)
FROM %inputfile%
GROUP BY sc-status
ORDER BY COUNT(*) DESC
While this query will work fine, you may want to refine it a little, reading the various results into the variables StatusCode and NumberOfHits:
SELECT sc-status AS StatusCode,
COUNT(*) AS NumberOfHits
FROM %inputfile%
GROUP BY StatusCode
ORDER BY NumberOfHits DESC
Take a look at the output from the query, which is very useful information (see Figure 2). You could filter it a bit more by using functions such as PROPCOUNT to get the percentage of hits, or you could try to determine which URLs seem to be most problematic by selecting both cs-uri-stem and sc-status. However, I’m going to focus on a slightly different question: do my servers behave differently at different times of day? The first attempt at this simply adds the time component. The new Log Parser query looks like this:
SELECT
TO_LOCALTIME(TO_TIMESTAMP(date,time))
as LocalTime, sc-status AS StatusCode,
COUNT(*) AS NumberOfHits
FROM %inputfile%
GROUP BY LocalTime, StatusCode
ORDER BY LocalTime, StatusCode
Take a look at the output in Figure 3. While the data is all there, it is a bit hard to read as you have to match up the status codes, and they are not guaranteed to be in the same order (for example, the 301 responses pushed the other status codes down a row during the 10:00:00 time period). To solve this problem, you need to pivot the data—take row values and turn them into columns.
Figure 2 Query Output (Close Figure 2)
StatusCode NumberOfHits
---------- ------------
200        167262
304        6527
302        2998
404        2980
301        993
500        53
403        27
501        27
206        18
400        1

Pivoting the Data
As I mentioned, pivoting is not currently supported by Log Parser. However with a bit of ingenuity, you can do it yourself. The trick is to use the CASE statement in Log Parser. The Log Parser help file explains the CASE syntax like so:
CASE <field_expression>
  WHEN <field_expression>
  THEN <field_expression>
  [ ... ]
  [ ELSE <field_expression> ]
 END
As an example, I could write a badly implemented version of the COUNT functionality using the CASE statement, like I have done in the top of Figure 4 (the resulting output is shown in the bottom of the figure).
Figure 4 Query Using CASE Statement (Close Figure 4)
SELECT
TO_LOCALTIME(TO_TIMESTAMP(date,time)) AS LocalTime,
SUM(_200) AS 200s
USING
CASE sc-status WHEN 200 THEN 1 ELSE 0 END AS _200
FROM
%inputfile%
GROUP BY
LocalTime
ORDER BY
LocalTime

Output
LocalTime           200s
------------------- -----
2005-11-16 09:59:59 68
2005-11-16 10:00:00 153
2005-11-16 10:00:01 157
2005-11-16 10:00:02 184
2005-11-16 10:00:03 189
2005-11-16 10:00:04 172
By extrapolation, I add a few more CASE statements (one for each sc-status I want to support). The query in Figure 5 implements this for the sc-status codes you saw before, and the output is shown in Figure 6 in a format that is much easier for a human to analyze.
Figure 6 East-to-Read Format (Close Figure 6)
LocalTime           200s 206s 301s 302s 304s 400s 403s 404s 500s 501s
------------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2005-11-16 09:59:59 68   0    0    1    3    0    0    2    0    0
2005-11-16 10:00:00 153  0    2    2    7    0    0    2    0    0
2005-11-16 10:00:01 157  0    0    6    8    0    0    5    0    0
2005-11-16 10:00:02 184  0    3    3    8    0    0    2    0    0
2005-11-16 10:00:03 189  0    1    4    5    0    0    4    0    0
2005-11-16 10:00:04 172  0    3    5    3    0    1    5    0    0
2005-11-16 10:00:05 182  0    2    2    6    0    0    1    0    0
2005-11-16 10:00:06 214  0    0    1    10   0    0    3    0    1
2005-11-16 10:00:07 192  0    1    1    6    0    0    2    0    0
2005-11-16 10:00:08 189  0    0    6    4    0    0    1    0    0
Figure 5 New CASE Query (Close Figure 5)
SELECT
TO_LOCALTIME(TO_TIMESTAMP(date,time)) AS LocalTime,
SUM(_200) AS 200s,
SUM(_206) AS 206s,
SUM(_301) AS 301s,
SUM(_302) AS 302s,
SUM(_304) AS 304s,
SUM(_400) AS 400s,
SUM(_403) AS 403s,
SUM(_404) AS 404s,
SUM(_500) AS 500s,
SUM(_501) AS 501s
USING
CASE sc-status WHEN 200 THEN 1 ELSE 0 END AS _200,
CASE sc-status WHEN 206 THEN 1 ELSE 0 END AS _206,
CASE sc-status WHEN 301 THEN 1 ELSE 0 END AS _301,
CASE sc-status WHEN 302 THEN 1 ELSE 0 END AS _302,
CASE sc-status WHEN 304 THEN 1 ELSE 0 END AS _304,
CASE sc-status WHEN 400 THEN 1 ELSE 0 END AS _400,
CASE sc-status WHEN 403 THEN 1 ELSE 0 END AS _403,
CASE sc-status WHEN 404 THEN 1 ELSE 0 END AS _404,
CASE sc-status WHEN 500 THEN 1 ELSE 0 END AS _500,
CASE sc-status WHEN 501 THEN 1 ELSE 0 END AS _501
FROM
%inputfile%
GROUP BY
LocalTime
ORDER BY
LocalTime
Now that you can pivot the data and get it into the structure that is most meaningful for your purposes, it is not terribly difficult to apply this technique to other data problems, like finding the most frequently used virtual directories over a given period of time. Given that you’re doing nothing more than looking for column names within rows of a data set, Log Parser should certainly be able to help automate this task. While it is possible to use Log Parser’s template output format to get close (you need to edit a couple of characters manually), the most hands-off approach would be to use a script.

Script-Generated Queries
The logic for a script to automatically generate these pivot queries is fairly simple. You need a few pieces of input: the column you want to pivot on and the series you want to pivot across. In the example, I want to pivot on the sc-status column, while the series is the LocalTime. I then use Log Parser’s scripting interface to parse the input log (or data you want pivoted) and build the new query based on the values in that input. This requires two parses of the input to get the pivoted data, though the resulting query can be saved and used at a later date. Figure 7 shows an example called PivotBuilder.js that does exactly that, though it is slightly expanded to allow you to add TOP and INTO statements to the query that is produced.
Figure 7 PivotBuilder.js (Close Figure 7)
var Matches = new Array();
var querySnippet = null;
var seriesQuery = null;
var seriesTitle = null;
//Parse arguments first
var szInputLogFilename=null;
var szArgs = WScript.Arguments;
var topVal = null;
var intoVal = null;

if(szArgs.length<1)
{
    WScript.Echo("Usage:");
    WScript.Echo(" -log:<log_filename> -query:<queryToUse>
-series:<seriesQuery> -sTitle:<SeriesTitle> [-top:<MaxNumColumns>
-into:<Value to set as INTO in Generated query>]");
    WScript.Quit(-2);
}

for (var i=0; i < szArgs.length; i++)
{
    if(szArgs(i).substr(0,5).toLowerCase()=="-log:")
    {
        szInputLogFilename=szArgs(i).substr(5);
    }
    else if(szArgs(i).substr(0,7).toLowerCase()=="-query:")
    {
        querySnippet=szArgs(i).substr(7);
    }
    else if(szArgs(i).substr(0,5).toLowerCase()=="-top:")
    {
        topVal=szArgs(i).substr(5);
    }
    else if(szArgs(i).substr(0,6).toLowerCase()=="-into:")
    {
        intoVal=szArgs(i).substr(6);
    }
    else if(szArgs(i).substr(0,8).toLowerCase()=="-series:")
    {
        seriesQuery=szArgs(i).substr(8);
    }
    else if(szArgs(i).substr(0,8).toLowerCase()=="-stitle:")
    {
        seriesTitle=szArgs(i).substr(8);
    }
    else
    {
        WScript.Echo("Unrecognized argument: " + szArgs(i));
        WScript.Quit(-1);
    }
}

if( szInputLogFilename==null || querySnippet == null || seriesTitle == null)
{
    WScript.Echo("Not all the required parameters have been specified");
    WScript.Quit(-1);
}
//Create the main Log Parser Query object
var myQuery=new ActiveXObject("MSUtil.LogQuery");

//Create the text of the query
//Note we assume that querySnippet is a TO_STRING-able thing, if not this
//will fail

var szQuery =     "SELECT "+((topVal==null)?"":"TOP "+topVal)+ " "+((casesensative)?"":"TO_LOWERCASE(") + "TO_STRING("+querySnippet+")" + ((casesensative)?"":")") +" AS columnQuery, COUNT(*) FROM " + szInputLogFilename +" GROUP BY columnQuery ORDER BY COUNT(*) DESC";

outputQuery = "";
var recordSet=myQuery.Execute(szQuery);

for(; !recordSet.atEnd(); recordSet.moveNext())
    {
    record=recordSet.getRecord();
    Matches.push(record.Getvalue(0));
    }

outputQuery = outputQuery +"SELECT "+seriesQuery+" AS "+seriesTitle+", "+ "\r\n";
for(i=0;i<Matches.length;i++)
{
    outputQuery = outputQuery +"    SUM([_"+Matches[i]+"]) AS ["+Matches[i]+"s]" + (((i<Matches.length - 1)?",":"")+ "\r\n");
}
outputQuery = outputQuery +"USING "+ "\r\n";
for(i=0;i<Matches.length;i++)
{
    outputQuery = outputQuery +"    CASE "+((casesensative)?"":"TO_LOWERCASE(") + "TO_STRING("+querySnippet+")"+((casesensative)?"":")") + " WHEN ‘"+Matches[i]+"’ THEN 1 ELSE 0 END AS [_"+Matches[i]+"]"+((i<Matches.length - 1)?",":"") + "\r\n";
}
outputQuery = outputQuery + ((intoVal==null)?"":" INTO "+intoVal) + " FROM %inputfile% GROUP BY " + seriesTitle + " ORDER BY "+ seriesTitle+ "\r\n";
WScript.Echo(outputQuery );
To run the script (PivotBuilder.js), you have to issue the following command line, which takes a number of important parameters. This particular command will cause PivotBuilder.js to run the query to produce the same output as you saw in Figure 6:
cscript /nologo PivotBuilder.js
-log:ex*.log -query:sc-status -series:
"TO_LOCALTIME(TO_TIMESTAMP(date,time))"
-stitle:LocalTime
Figure 8 shows the Log Parser query that the script generated when run with the aforementioned parameters, which comes out to be pretty similar to the hand crafted one in Figure 5. And the output from this query looks something like Figure 9.
Figure 9 Output Generated from Scripting Query (Close Figure 9)
LocalTime           200s 304s 302s 404s 301s 500s 403s 501s 206s 400s
------------------- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----
2005-11-16 09:59:59 68   3    1    2    0    0    0    0    0    0
2005-11-16 10:00:00 153  7    2    2    2    0    0    0    0    0
2005-11-16 10:00:01 157  8    6    5    0    0    0    0    0    0
2005-11-16 10:00:02 184  8    3    2    3    0    0    0    0    0
2005-11-16 10:00:03 189  5    4    4    1    0    0    0    0    0
2005-11-16 10:00:04 172  3    5    5    3    0    1    0    0    0
2005-11-16 10:00:05 182  6    2    1    2    0    0    0    0    0
2005-11-16 10:00:06 214  10   1    3    0    0    0    1    0    0
2005-11-16 10:00:07 192  6    1    2    1    0    0    0    0    0
2005-11-16 10:00:08 189  4    6    1    0    0    0    0    0    0
Figure 8 Query Generated by PivotBuilder.js (Close Figure 8)
SELECT TO_LOCALTIME(TO_TIMESTAMP(date,time)) AS LocalTime,
    SUM([_200]) AS [200s],
    SUM([_304]) AS [304s],
    SUM([_302]) AS [302s],
    SUM([_404]) AS [404s],
    SUM([_301]) AS [301s],
    SUM([_500]) AS [500s],
    SUM([_403]) AS [403s],
    SUM([_501]) AS [501s],
    SUM([_206]) AS [206s],
    SUM([_400]) AS [400s]
USING
    CASE TO_STRING(sc-status) WHEN ‘200’ THEN 1 ELSE 0 END AS [_200],
    CASE TO_STRING(sc-status) WHEN ‘304’ THEN 1 ELSE 0 END AS [_304],
    CASE TO_STRING(sc-status) WHEN ‘302’ THEN 1 ELSE 0 END AS [_302],
    CASE TO_STRING(sc-status) WHEN ‘404’ THEN 1 ELSE 0 END AS [_404],
    CASE TO_STRING(sc-status) WHEN ‘301’ THEN 1 ELSE 0 END AS [_301],
    CASE TO_STRING(sc-status) WHEN ‘500’ THEN 1 ELSE 0 END AS [_500],
    CASE TO_STRING(sc-status) WHEN ‘403’ THEN 1 ELSE 0 END AS [_403],
    CASE TO_STRING(sc-status) WHEN ‘501’ THEN 1 ELSE 0 END AS [_501],
    CASE TO_STRING(sc-status) WHEN ‘206’ THEN 1 ELSE 0 END AS [_206],
    CASE TO_STRING(sc-status) WHEN ‘400’ THEN 1 ELSE 0 END AS [_400]
 FROM %inputfile% GROUP BY LocalTime ORDER BY LocalTime
Obviously this is not a perfect world, since if your values change from input to input you would have to rerun this script to produce the correct query for the log, but this is a lot easier than rewriting the script for each one by hand. If you want, you could add or edit a few lines of PivotBuilder.js to have it call Log Parser with the query that you built up (instead of using WScript.Echo to let you see the query, you would save the query into a variable and then call Log Parser to execute the query). You can download a copy of this script from the TechNet Magazine site.

Wrap-Up
I have used Log Parser when working with more than a hundred values (so more than one hundred columns in the output) and it has worked quite well. For fun I took the number of columns in a particular query to 1,000 and while it worked, I didn’t completely verify all the output (it also took a while to run). However in most cases, you will only need a few columns and can use the "–top:" parameter with PivotBuilder.js.
While this works great for pivoting certain kinds of data (specifically counts), it doesn’t work as well when the content of the field you want is a string or other value. In that case you need to write a custom input parser COM object. While this does involve writing some code, I’ve found that it isn’t really a daunting task if I take advantage of the simplicity involved in using a scripting language to produce COM objects. QFE.wsc was included in the sample scripts for Log Parser. It is a good example of how to implement a custom input type using a scripting language. By using the versatility and extendibility inherent in Log Parser, it is possible to produce more readable outputs to allow for easier data mining.
Log Parser Version 2.2 Sample Commands
Here’s a sampling of LogParser functions to give you an idea of what kind of data you can extract using LogParser.

Print the 10 largest files on the C: drive:
LogParser "SELECT TOP 10 *
FROM C:\*.* ORDER BY Size DESC" -i:FS
Create an XML report file containing logon account names and dates from the Security Event Log messages:
LogParser "SELECT TimeGenerated AS LogonDate,
EXTRACT_TOKEN(Strings, 0, ‘|’)
AS Account INTO Report.xml FROM Security
WHERE EventID NOT IN (541;542;543)
AND EventType = 8 AND EventCategory = 2"
Load a portion of the registry into a SQL table:
LogParser "SELECT * INTO MyTable
FROM \HKLM" -i:REG -o:SQL -server:MyServer
-database:MyDatabase
-driver:"SQL Server"
-username:TestSQLUser
-password:TestSQLPassword -createTable:ON
Parse the output of a netstat command:
netstat | LogParser "SELECT * FROM STDIN"
 -i:TSV -iSeparator:space -nSep:2
-fixedSep:off -nSkipLines:3
Display users’ job title breakdown from Active Directory:
LogParser "SELECT title,
MUL(PROPCOUNT(*), 100.0) AS Percentage INTO
DATAGRID FROM
‘LDAP://myusername:mypassword@mydomain/CN=Users,
DC=mydomain, DC=com’ WHERE title IS NOT NULL
GROUP BY title ORDER BY Percentage DESC"
-objClass:user
Get the distribution of EventID values for each Source:
LogParser "SELECT SourceName,
EventID, MUL(PROPCOUNT(*) ON (SourceName),
100.0) AS Percent FROM System
GROUP BY SourceName, EventID ORDER BY
SourceName, Percent DESC"
Display total network traffic bytes per second:
LogParser "SELECT QUANTIZE(DateTime, 1)
AS Second, SUM(FrameBytes) INTO
DATAGRID FROM myCapture.cap GROUP BY Second"
Display the distribution of registry value types:
LogParser "SELECT ValueType, COUNT(*)
INTO DATAGRID FROM \HKLM GROUP BY
ValueType"
Display titles of current channels on MSDN blogs:
LogParser "SELECT title INTO MyOutput.txt FROM
http://blogs.msdn.com/MainFeed
.aspx#/rss/channel/item" -i:XML
-fMode:Tree -o:tpl -tpl:mytemplate.tpl

Jeffrey A. Johnson is a Senior Systems Engineer with Microsoft.com and spends his days analyzing and debugging Web server-related issues. As a veteran of the IIS test team, he was the unofficial tester for Log Parser and has been using it since before its public release. For more information contact mscomblg@microsoft.com.


 Security Watch: The Most Misunderstood Windows Security Setting of All Time
Security Watch
The Most Misunderstood Windows Security Setting of All Time
Jesper Johansson

Almost everyone who runs a network on Windows® has heard of NTLM version 2 (NTLMv2) and the LMCompatibilityLevel setting that governs it. The setting first became available in Windows NT® 4.0 Service Pack 4 (SP4), and has been in every version of Windows based on Windows NT since then. LMCompatibilityLevel has been recommended in every security guide for Windows since 1998.
A few years ago I thought I really knew how LMCompatibilityLevel worked. I was wrong. Then for a couple of years I thought I knew how it worked again. This has happened a few times, and recently I found out that I still didn’t completely understand it. This prompted me to finally go back to the code, tear through it, and figure out what really was going on. This article is about what I found. Anyone who runs Windows probably should understand this setting and how it impacts network security and stability.

Background
Windows NT-based operating systems up through and including Windows Server™ 2003 store two password hashes, the LAN Manager (LM) hash and the Windows NT hash. Starting in Windows Vista™, the capability to store both is there, but one is turned off by default.
The NT hash is a straight MD4 hash of the plaintext password. It supports all Unicode characters and passwords up to 256 characters long.
The LM hash, originally invented for use in LAN Manager over 20 years ago, is not actually a hash at all (see Figure 1). A hash is a mathematical function used to summarize or probabilistically identify data. LM instead uses a cryptographic one-way function (OWF). Instead of encrypting the password with some other key, the password itself is the key. This is why you will sometimes see the LM hash referred to as the LM OWF. Conversely, the NT hash is typically referred to as the NT OWF internally.
Figure 1 LM Hash Generation
Figure 1 LM Hash Generation
The algorithm introduces several weaknesses that attackers can exploit. First, all lowercase characters are set to uppercase, reducing the number of possible characters. Second, it splits a long, strong, password into two seven-character chunks. Length has been shown to be the most important factor in password strength by far (see "Frequently Asked Questions About Passwords" for more on this topic). This operation reduces the strength of the password by many orders of magnitude.
Windows uses the LM OWF for the LM authentication protocol and the NT OWF for everything else. One common misconception is that the system stores a specific NTLMv2 OWF. There is not even an NTLM OWF for the NTLM protocol. The NT OWF is used for all other Windows authentication protocols in one way or another. The two OWFs are used in the authentication sequence as shown in Figure 2.
Figure 2 LM and NTLM Response Computation
Figure 2 LM and NTLM Response Computation
Both the LM and NTLM protocols operate essentially the same way; the only difference is the password hash. The actual challenge-response computation is fairly simple. The client requests authentication and the authentication server responds with an 8-byte challenge. The challenge is just a piece of random data. The client encrypts the challenge using the OWF as the key and returns the result as the response to the server. The LM and NTLM responses are each always 24 bytes long. The server has the NT OWF and usually the LM OWF as well. It starts by computing the NTLM response using the same algorithm the client used, and then compares that to the client result. If the two match, they used the correct password hash and—by an extension of logic that is not always correct—had the correct password, a successful logon results. If the results do not match, the server computes the LM response and checks if that matches what the client sent. If that fails too, the client is denied access due to a bad username or password error.
NTLMv2 was developed in response to attacks against the LM authentication protocol. The LM protocol, as the name implies, was originally used in the old LAN Manager network operating system in the mid-1980s. It was developed for the security requirements of the day, which included mostly floppy-based viruses, and hence was no match for a late 90s-style cracking attack. When the L0phtCrack password cracker from L0pht Heavy Industries, later purchased by @Stake and finally by Symantec, included a cracker against a captured LM authentication sequence, Microsoft implemented the new NTLMv2 authentication protocol to defeat L0phtCrack. (For a synopsis of the protocol and the attack that caused it to be created, see this Windows IT Pro article. Keep in mind, however, that this article was published in 1999 and is now fairly dated and not entirely accurate in some respects.)
NTLMv2 is turned on using the LMCompatibilityLevel switch (known as some variant on "LAN Manager authentication level" in Group Policy). LMCompatibilityLevel takes six different values, from 0 to 5. The levels are shown in Figures 3 and 4.
Figure 4 Server-Side LMCompatibilityLevel Impact (Close Figure 4)

Level | Group Policy Name | Sends | Accepts | Prohibits Sending
4 | Send NTLMv2 response only/refuse LM | NTLMv2 Session Security | NTLM, NTLMv2 | LM
5 | Send NTLMv2 response only/refuse LM and NTLM | NTLMv2, Session Security | NTLMv2 | LM and NTLM
Figure 3 Client-Side LMCompatibilityLevel Impact (Close Figure 3)

Level | Group Policy Name | Sends | Accepts | Prohibits Sending
0 | Send LM and NTLM Responses | LM, NTLM NTLMv2 Session Security is negotiated | LM, NTLM, NTLMv2 | NTLMv2 Session Security (on Windows 2000 below SRP1, Windows NT 4.0, and Windows 9x)
1 | Send LM and NTLM—use NTLMv2 session security if negotiated | LM, NTLM NTLMv2 Session Security is negotiated | LM, NTLM, NTLMv2 | NTLMv2
2 | Send NTLM response only | NTLM NTLMv2 Session Security is negotiated | LM, NTLM, NTLMv2 | LM and NTLMv2
3 | Send NTLMv2 response only | NTLMv2 Session Security is always used | LM, NTLM, NTLMv2 | LM and NTLM
Admittedly, this is a brief introduction to Windows and passwords. Interested readers should see chapter 11 in Protect Your Windows Network, which covers the topic in much more detail.

How NTLMv2 Works
The NTLMv2 response is very different. First, the client concatenates the user name and the logon domain name and computes an HMAC-MD5 message authentication code of those using the NT OWF as a key. The result of this keyed hash is sometimes referred to as the NTLMv2 OWF. The logon domain name is the account the client is attempting to log on to. In the case where a workstation is trying to contact a system that is not in the client’s domain, this would be the server name.
The client then generates its own 8-byte challenge, which it puts into a blob that also contains a timestamp, information on the target the client is attempting to connect to, and some other data. This blob is then concatenated with the server challenge and another HMAC-MD5 message authentication code is computed on this combined challenge using the NTLMv2 OWF as the key. The result of that computation is concatenated with the blob and returned as the NTLMv2 response buffer. The client also computes a response based only on the NTLMv2 OWF, the server challenge, and the client challenge. This response is called the LM 2 Response and is returned in the LM response field along with the client challenge, as shown in Figure 5.
The entire NTLMv2 response buffer is not documented publicly, but many sources have inferred a lot about it. Essentially, it contains version information, room for flags in a future implementation, room for a message from the client to the server (which is not used today), the timestamp, the client challenge, and some information about the server and share name the client is connecting to.
Figure 5 LMv2 and NTLMv2 Response Computation
Figure 5 LMv2 and NTLMv2 Response Computation
As noted earlier, when NTLMv2 is used, the LM response changes as well. This is done for compatibility with servers that are domain members but do not understand NTLMv2. The problem is that the NTLMv2 response is variable in length, including items such as the server name, the share name, and so on. Older domain-joined servers, such as Windows NT 4.0 prior to SP4 and Windows 9x cannot pass through a variable-length response to the domain controller; they assume that both the LM and NTLM responses are exactly 24 bytes. By luck or very smart design, an HMAC-MD5 computation yields a 128-bit value which, when combined with the 8-byte client challenge, gives us 192 bits or 24 bytes—exactly the same as the three Data Encryption Standard (DES) computations used in the LM response. The older machines can thus pass the LMv2 response through to the domain controller without needing to understand the exact meaning of the bits within it. The domain controller will do the same computation it always does, checking first if the response is an NTLMv2 response, and if that fails falling back to LMv2, and then to the older protocols.
Until recently, I thought this was all there was to LMCompatibilityLevel. I conveniently pushed aside the feeling that I still did not understand the thing referred to as NTLMv2 Session Security.

Years of Confusion
As did Randy Franklin Smith in the article mentioned earlier, I believed for years that at level 1, the system was supposed to send the LM response (which is correct) and would negotiate and send the NTLMv2 response if it were possible. This is incorrect. When you set LMCompatibilityLevel to 1, it does not actually send NTLMv2 authentication.
Then another epiphany hit me. The documentation, which says that "DCs will accept" and "clients will send" is really misleading. The simpler way to think about the settings is that levels 0-3 affect the computer when acting as a client, while levels 4 and 5 affect the machine when acting as the authentication server. It is not just domain controllers that authenticate users. All systems can authenticate users against their own SAM database, so the settings really affect all of them.
Once again, I thought I knew exactly how it worked, and I wrote another article on it. Shortly after that, Seki Hidenobu wrote to me, prodding about something he called "NTLM2 Session Response," a term he picked up from Eric Glass. In fact, not only does the Ethereal network sniffer filters identify NTLM2 Session Response, it has even been documented as part of the Samba (the UNIX implementation of SMB) documentation. In addition, Christopher R. Hertel’s book on the Common Internet File System (CIFS—the standardized version of the Microsoft Server Message Block, or SMB, protocol) contains a very technical and detailed description of all the protocols involved and discusses the NTLM2 Session Response. More information on this book is at www.ubiqx.org/cifs. These documents were largely developed while adding NTLMv2 support to Samba. Luke K. C. Leighton, who wrote most of the NTLMv2 code for Samba, also wrote a book on the protocols, DCE/RPC over SMB: Samba and Windows NT Domain Internals (SAMS, 1999). That book even includes source code for an implementation of NTLMv2.

NTLMv2 Session Security
Hidenobu did not know exactly what the NTLM2 Session Response was for and why it appeared, but as he pointed out, it can be cracked. Basically, when it is used, you do not pass the LM response. Instead, a client challenge is created and mixed with the server challenge using HMAC-MD5. Then the NTLM response to this combined challenge is computed the normal way, as shown in Figure 2, by using three DES encryptions of the challenge with pieces of the nonce, and returned to the server in the NTLM response field. The client challenge is returned in the now unused LM response field. This modified NTLM response can be cracked using a modified precomputed hash attack even though there is now a client challenge involved. Seki exploited the fact that only two bytes of the challenge are used in the third component of the response. Once he identifies which bytes those are, he can use them as an index into a lookup table to determine all the passwords that could have possibly generated an OWF with those two bytes at the end.
Eric Glass, who did not have the luxury or the curse (depending on how you look at it) of Windows source code access, stated that the "NTLM2 Session Response can be used in conjunction with NTLM2[sic] Session Security...." This is, in fact, almost exactly the correct description. To produce an answer for Hidenobu I ended up having to go back to the source code. It turns out that what the Samba documents call the "NTLM2 Session Response" is actually a slightly modified version of the NTLM response, shown in Figure 6.
Figure 6 Response Computation for LMCompatibilityLevel 1 or 2
Figure 6 Response Computation for LMCompatibilityLevel 1 or 2
This modified response is used when the client knows it is talking to an uplevel server that can handle this new response. This means that, as Glass pointed out, it is seen when you use NTLMv2 Session Security. It was created to make certain man-in-the-middle attacks more difficult by introducing a client-side challenge, while still retaining the ability for downlevel clients to pass the authentication through to an upgraded domain controller. This new response does not really even have a name, and has never been documented by Microsoft. In the source code the term "NTLM2" sometimes shows up, contrasted with "NTLM3," which is really NTLM version 2. It is possible that the use of the term NTLM2 Session Response originated from someone who was looking at debug symbols and saw that term.
NTLMv2 Session Security has been another enigma. In fact, "NTLMv2 Session Security" is really a misnomer. It really should be referred to as "NTLMv2-style session key computation," although one can easily understand why that term never caught on. NTLMv2 Session Security is related to how session keys are computed. When using the original NTLM protocol, the session key is based on the user’s NT hash (the NT OWF). When NTLMv2 Session Security is used, the session key is based on not only the NT OWF but also on the client and server challenges. Those session keys are not used during the actual authentication sequence, but when an application requests security by calling the EncryptMessage or SignMessage APIs.
In other words, you can think of NTLMv2 Session Security as having two components: a stronger response computation and a better session key generation algorithm. The response computation, however, can be either the mysterious NTLM2 Session Response or a real NTLMv2 response.
NTLMv2 Session Security can be forced by setting the LMCompatibilityLevel setting, but it is also negotiated between the client and the server. Starting with Windows 2000 Security Rollup Pack 1 (SRP1) all systems will attempt to use NTLMv2 Session Security if both sides agree. If either side disagrees they will fall back to the old protocols if they are configured to allow that.

LMCompatibilityLevel and Security
As mentioned earlier, starting with Windows 2000 SRP1, including Windows 2000 SP3, all versions of Windows XP and all versions of Windows Server 2003, NTLMv2 Session Security is always negotiated. Therefore, there is no longer any functional difference between LMCompatibilityLevel 0 and 1. They generate exactly the same results on all supported platforms.
The difference between LMCompatibilityLevel set to 0 or 1 and setting it to 2 is that with level 2 the client refuses to send the LM response. This is why level 2 is the default setting on Windows Server 2003. It also means that Windows Server 2003 cannot be a client to a Windows 9x system unless it passes the authentication through to a domain controller. When acting as the authentication server, however, Windows Server 2003 accepts LM responses and authenticates against the LM hashes stored locally or in Active Directory®.
This meant I had to modify my tables showing how the various levels of LMCompatibilityLevel work. Figure 3 shows the settings that affect how the system acts as a client. Figure 4 shows how the settings affect the system when acting as an authentication server.
Levels 0 and 1 are identical on systems with Windows 2000 SRP1 or higher. The Security Rollup Pack is included in Windows 2000 SP3 and, of course, the same functionality with respect to NTLMv2 Session Security is turned on by default in Windows XP and Windows Server 2003.
In terms of actual security impact, negotiating NTLMv2 Session Security ranges from significant to minimal, depending on the environment. Theoretically, NTLMv2 Session Security can be used with any protocol. But used with NTLM version 1, it also improves the challenge response computation by including the client-side challenge, similar to how NTLMv2 itself works. This makes a precomputed hash attack against captured challenge-response pairs more difficult, but certainly not impossible. More significantly, it makes man-in-the-middle attacks much more difficult.
NTLMv2 Session Security is still negotiated. Therefore, if the attacker has the ability to act as the server itself or to modify the transaction in any way, the attacker can downgrade the authentication protocols to the older versions, enabling both types of attack once again. Only if LMCompatibilityLevel is set to 3 on the client are these attacks actually stopped.

Recommendations
Although Windows Vista has not been released yet, it is worthwhile to point out some changes in this operating system related to these protocols. The most important change is that the LM protocol can no longer be used for inbound authentication—where Windows Vista is acting as the authentication server. Windows Vista will no longer store the LM hash by default. Acting as a client, Windows Vista also makes a change to outbound protocols by setting LMCompatibilityLevel to 3 by default. In other words, NTLMv2 will finally be the default protocol for non-domain authentication. In the next scheduled release of the Windows Server platform, code-named "Longhorn Server," a lot of work has been done to reduce the need for NTLM altogether. In Windows Server 2003, NTLM, and sometimes even LM, is used in many cases, such as in clusters. In the next version of the operating systems many of these protocols will finally be turned off by default.
Please note that these statements are based on prerelease versions of the operating systems. It is possible that compatibility testing during the beta cycle will force these settings to change.
So what should you do now? In all environments you should consider setting the LMCompatibilityLevel as high as possible. The decision needs to take into account the types of systems used in the environment and how high the threat of attack is. For instance, inside a datacenter where all communication lines are secured, the threat of man-in-the-middle attacks is relatively small. In an unsecured or WEP-secured wireless network the risk is significant. Attackers commonly sniff these networks in the hopes of catching the authentication sequence and cracking it.
Generally, the setting should be as high as the needs of the environment allows. In a network that consists of only Windows 2000 SP3 and higher systems, and has no third-party devices that require LM or NTLM authentication, LMCompatibilityLevel can be safely set to 3 or higher on all systems. Problems may occur when level 5 is used on systems running operating systems prior to Windows Server 2003 SP1. Routing and Remote Access Services will fail if the RRAS server or the domain controller is running with LMCompatibilityLevel 5.
It is also important to consider the error messages you get when you have a conflict in the LMCompatibilityLevel capabilities of different systems. Effectively, the only negotiation in this entire set of protocols is NTLMv2 Session Security. Downlevel systems will ignore that flag. Therefore, the only error message you get if there is a settings conflict between client and server is that access was denied due to a bad username or password. Neither side to the transaction has the ability to determine that the issue was due to unsupported authentication protocols. This makes troubleshooting somewhat challenging.
Hardening the NTLM SSP
The NtlmMinClientSec and NtlmMinServerSec settings, which are known as some variant of "Minimum session security for NTLM SSP based (including secure RPC) clients" in Group Policy, govern which behaviors are allowed for applications using the NTLM Security Support Provider (SSP). The SSP Interface (SSPI) is used by applications that need authentication services.
The NtlmMinClientSec and NtlmMinServerSec settings do not modify how the authentication sequence works. Rather, they are used to require certain behaviors in applications that use the SSPI. Each setting contains a bitmask that can be used to require some combination of up to four different types of behaviors, shown in Figure A.
The easiest way to differentiate between the NTLM SSP settings and LMCompatibilityLevel setting is by just considering the items they affect. The SSPI settings govern the behavior of applications that use authentication, while LMCompatibilityLevel governs which authentication protocols the operating system can use.

Jesper Johansson, a senior security strategist in the Microsoft Security Technology Unit and contributing editor for TechNet Magazine, focuses on how customers should best deploy Microsoft products more securely. He has a PhD in IS and has delivered speeches on security at conferences all over the world.


 How IT Works: Roaming in SMS 2003
How IT Works
Roaming in SMS 2003
Steve Rachui

Systems Management Server (SMS) 2003 is a comprehensive solution that allows you to roll out relevant software and updates to your users easily and cost effectively. SMS lets you designate "sites" that represent groups of computers. SMS sites give you very granular control over software distribution conditions. Sites have boundaries and when you define the boundaries of an SMS site, either by IP address or through association with Active Directory®, you are defining the groups of clients a particular site should own and manage.
For software distribution, boundaries are used to determine a client’s location relative to the site—either local or remote. This is important because SMS assumes that a client that is within a local boundary of the site is on a fast LAN connection, while a client within a remote boundary is on something less—a VPN or a slower WAN link. These designations help you determine which groups will get updates and when. If a particular software package is very large, for example, you may not want to push it out to clients residing in remote boundaries of a site.
The way machines move around to different SMS sites is called roaming, and understanding it is crucial to getting the most out of SMS 2003 software distribution. Because the topic has led to confusion for many SMS administrators, I’ll attempt to explain key concepts here.

Local Roaming Boundaries
By default, all boundaries defined on the site boundaries list are included in the Local Roaming Boundaries category. Any clients that reside within the defined boundaries or that are within the boundaries while visiting from another site are considered local to the SMS distribution points defined at this site for software distribution—regardless of whether the client is actually assigned to the site or not. Figure 1 shows the Roaming Boundaries dialog.
Figure 1 Roaming Boundaries
Figure 1 Roaming Boundaries

Remote Roaming Boundaries
An SMS site can include boundaries that define clients the site should manage but that may be across a slow WAN link and considered remote from the installed SMS site server and distribution points. Such boundaries are designated as "remote roaming" boundaries, which means that clients residing within or traveling to the designated boundary have no local distribution point and should be assumed remote for the purposes of software distribution. You add a new boundary using the dialog shown in Figure 2.
Figure 2 New Roaming Boundary
Figure 2 New Roaming Boundary
It is not uncommon for an administrator to install a local distribution point within a boundary that is listed as remote roaming with the expectation that clients residing within that remote roaming boundary will begin to use the new distribution point and treat it as local. It is important to remember that all distribution points installed as part of a particular site are considered local to that site, but not necessarily to the clients of that site. In the described scenario, the clients of the remote roaming boundary will treat this newly installed local distribution point just as they would every other distribution point (unless it’s protected—more on that in a moment) because the designation as local or remote is controlled not by the location of the distribution point but rather by the specification of the client’s boundary in relation to the site. If a client is within a boundary that is designated as remote roaming, that client will treat every distribution point of the site as remote.
When a boundary is designated as remote roaming, an administrator can control whether an advertisement should execute or be delayed until the client comes back into a local roaming boundary. This is particularly useful when distributing large packages—such as Microsoft® Office—which may not be appropriate to distribute across slower WAN connections. The options that control whether the distribution will continue are found on the Advanced Client tab (see Figure 3).
Figure 3 Advertisement Properties
Figure 3 Advertisement Properties
There you can choose to avoid running the advertisement if there are no local distribution points, download the advertisement to the local cache using Background Intelligent Transfer System (BITS) before running, or simply proceed with the execution of the advertisement.
The SMS 2003 Advanced Client is designed to accommodate travel. Each client will have an assigned site to which it reports. The assigned site will have a list of boundaries—both local and remote—to define clients it should manage. If the client travels to an SMS site that is not its assigned site, it will attempt to use the local site for software distribution and the concepts discussed earlier will apply. If the client travels to a location where there is no local SMS site controlling the boundary, it will default back to its assigned site for management and software distribution.

Global and Regional Roaming
As I noted earlier, roaming is a concept that applies to software distribution and is designed to allow the SMS Advanced Client to use local distribution points—potentially at any site in the hierarchy—to facilitate local software distribution (assuming the distribution point has a copy of the package). Regional roaming involves a client traveling to any branch of the SMS hierarchy that is below its assigned SMS site. Any SMS advanced client may participate in regional roaming. Global roaming, in contrast, is defined as a client traveling to any SMS site in the hierarchy—regardless of location within the hierarchy. Global roaming requires Active Directory integration and is not supported across different forests.

Protected Distribution Points
The final piece of the picture you need to understand involves protected distribution points. These are important because they extend the control of software distribution by allowing the SMS administrator to choose which distribution points should be used by clients residing in a particular subnet. When a distribution point is configured as protected, as in Figure 4, it is not available for use by any client outside the boundary range configured specifically on the protected distribution point.
As you can see, with roaming boundaries SMS advanced clients can move from one location to another and still receive software packages from SMS—without clogging slow network links.
Figure 4 Protected Distribution Point
Figure 4 Protected Distribution Point

Steve Rachui is a Manageability Support Escalation Engineer in the Product Support Services group at Microsoft. He has supported SMS since version 1.2. Steve can be reached at steverac@microsoft.com.


 Field Notes: TLC for Your Server
Field Notes
TLC for Your Server
Jay Shaw

In the July issue I wrote about the care and feeding of your server. I beseeched you to treat your servers as if they were living beings. Do so, and they’ll reward you with steady service. Don’t, and you may have to start thinking about disaster recovery. It’s far better to build disaster prevention into your daily maintenance schedule.
It doesn’t take the power of a lightning bolt to destroy your server. A simple brownout can do lots of damage. Yes, yes, I know you have a mondo UPS with line conditioning, but when was the last time you looked at the UPS itself? They put those little lights on the front for a reason. Over time, batteries fail. The little lights turn red in warning, often months or even years before the batteries quit. Then their services are needed one day and—BAM. Now you have a dead server.
You should also have a look at the load distributed on each UPS. Good ones have a meter to show you whether they are overtaxed. I usually keep each one at no more than 50 percent capacity, just to be on the safe side. Of course, you hooked up the USB cable and loaded the necessary software to shut the server down in the event of a power outage, right?
Recently I had a UPS battery fail. This isn’t usually an emergency but I made sure to order a replacement right away. When my new battery arrived, I went to replace the old one and it was a mess. The battery had swelled from heat on the inside of the steel case. It didn’t rupture but it was very close. It was so bad I had to use a hammer and a screwdriver in order to remove the old battery. I hate to think what might have happened if I hadn’t noticed the problem.
Most of the time your server doesn’t ask for much—it just sits there humming along. But it does try to give you advance warning when things are not as they should be. It speaks its own special language known as the Event Log. I know many of you ignore the Event Log, but this is exactly where integrating prevention into your regular schedule will do you the most good. Most problems will be identified with a warning in the log before they reach a critical state. The Event Log can identify problems with power and heat, register an alert that you’re running out of drive space, or note a failed hard drive or other peripheral. Failed backups will show up in the Event Log, as will all attempts at unauthorized access. Look at the Event Log daily and look up anything that isn’t clear.
Of course, we’ve all seen servers get sick with a virus. Just as with people, servers can be cured most of the time, but the longer they’re left untreated the more damage they’ll suffer. I know you have virus protection and scans scheduled to run on a regular basis. Do you actually look at the scan results, clean things up if necessary, and then make sure the server is clean by running an additional scan? If you don’t find and eliminate a virus early enough, it can spread to other servers and PCs.
Certainly human error is one of the most dangerous of all threats to your server. How many times have you done something you know is wrong but you’re just too impatient to take proper precautions? Or maybe you haven’t done something you should have. I was once working with a client who was having trouble with his Exchange Server information store. I was remote, helping him over the phone. I asked him to make sure his backup was current before running eseutil.exe and he told me he had it covered. I wanted to run the utility on a copy of the database rather than the original file, but making a copy would have required extra time. Of course Mr. Murphy (of Murphy’s law) reared his ugly head. The database was not repairable and we found that, in fact, there was no backup done the night before. Two days of e-mail were lost, and lots of users were screaming.
You can also kill the server with kindness. Even if you’re diligent about preventive maintenance, you still need to be on guard. One of my customers has a single server sitting in a closet that is shared. Every few months the cleaning crew vacuums in and around the closet, and almost every time they do, I get a call the next morning. Usually I find that the cables were removed and not put back correctly or the server was just simply turned off by accident and not restarted. Sure, you want to keep dust away from your server, but exercising a little caution when cleaning near the machines is pretty important. It may even be necessary to restrict physical access to the server area.
I’m sure you see the common thread here. Too often we become unwitting collaborators in our server’s destruction, just for lack of a few logical precautions. If you take care of problems while they’re small, they’ll never grow into disasters. To do this, though, you need to be aware that the problems exist. If your daily maintenance schedule makes checkups routine, your server will live a long, happy life.

Jay Shaw is an independent network consultant. His company, Network Consulting Services, is located in Long Island, New York. He can be reached at info@ncservices.net.


 What is TechNet?: Resources for the IT Professional
What is TechNet?
Resources for the IT Professional

The complete Microsoft TechNet Program provides comprehensive tools and authoritative information to help IT professionals get the most out of Microsoft products and technologies. It’s where you turn for the latest security alerts, live discussions, virtual testing labs, free tools, and in-depth articles that allow you to be more responsive, more informed, and more effective. Take a look at all the resources that Microsoft TechNet has to offer.

TechNet Online
The TechNet Web site is the place for IT professionals to get the information they need to plan, deploy, troubleshoot, maintain, secure, and support their Microsoft technology investments. You can access the latest security bulletins, tools, product documentation, TechCenters, Microsoft Knowledge Base articles, the Security Center, and the Script Center. You can also find learning resources, including e-learning, virtual labs, webcasts, and classroom training information, all at TechNet Online.

TechNet Flash
The biweekly TechNet Flash newsletter delivers the latest IT news, security bulletins, software updates, and event announcements right to your inbox—making it easy for you to stay up-to-date with the latest technological developments from Microsoft. You can customize your own version to receive the information that’s most important to you. Register at TechNet Flash.

TechNet Magazine
Improve SQL Server™ performance, tune up your Windows servers, simplify your life with scripts, and learn how to best protect your environment against the latest security threats. TechNet Magazine is a monthly print publication, also available online, that features in-depth articles from experts to help you do all this and more. But don’t just take our word for it. Flip through this issue and see for yourself, or read online at TechNet Magazine.

TechNet Events
Microsoft TechNet Events are free local events for IT professionals that explain how to use Microsoft products and technologies. You’ll see live demos and hear directly from specialists who live with real-world implementations of Microsoft products and technologies every day. Schedule and registration are available at TechNet Events.

TechNet Webcasts
TechNet webcasts are live broadcasts that feature interactive technical presentations, product demonstrations, and question-and-answer sessions. Up to 30 new webcasts are broadcast every month, each presented by an expert in their field. All content is also recorded and made available on demand. TechNet Support webcasts focus on support issues like troubleshooting, configuration, diagnostics, and fixes. Schedule and registration are available at TechNet Webcasts.

TechNet Virtual Labs
TechNet Virtual Labs let you evaluate Microsoft software in a risk-free environment. You can explore apps through guided tours that highlight new features, without the hassle of downloads or complicated installations. You can gain real-world insight into business issues by working hands-on with Microsoft products, including Windows Server™ 2003, Exchange Server, and SQL Server 2005. TechNet Virtual Labs are free at TechNet Virtual Labs.

TechNet Radio
TechNet Radio features interviews and commentary from Microsoft insiders and industry luminaries, as well as profiles of leading companies. We bring you the trends and technologies that are shaping the industry, and offer perspectives from your peers. Take a listen at TechNet Radio.

TechNet Plus Subscriptions
With a TechNet Plus Subscription, you get two free professional support incidents to use on any Microsoft technology, access to full-version software with no time limits for evaluation purposes, early access to beta software, newsgroups monitored by support professionals (guaranteeing next-business day responses to your technical questions), free training, and much more. Sign up at TechNet Plus Subscriptions


 Windows Confidential: A Directory by Any Other Name
Windows Confidential
A Directory by Any Other Name
Raymond Chen

What’s in a name? Many people have said that the "Program Files" directory introduced in Windows® 95 should really be named "Programs." But programs and program files are different. Users like programs. Programs do things. If you have a list of programs, you can browse through and choose which program you want to use. Programs are fun to play with.
Program files, on the other hand, are scary. These are the nuts and bolts that make the computer go: data files, DLLs, configuration files, helper processes, and other things that users shouldn’t be messing with.
This distinction was preserved in the user interface. Programs appeared on the Start menu. This offered a friendly, inviting place where you could see all the programs on your computer and choose one to run. Program files, on the other hand, were squirreled away in a directory with a scary name that discouraged exploration. A directory named Programs would be an attractive nuisance. A user who wanted to run a program might open My Computer, see a directory named Programs, and wander into it, unwittingly entering a frightening world of DLLs and executables with baffling names. If this user were lucky, she might discover that this thing called WinWord.exe launches Microsoft® Word. She might even conclude that, in order to run Word, she needs to open My Computer, click her C drive, click Programs, go into the Microsoft Office directory, then to Office11, and then in the list of 200 files (all of which have nonsense names) find the file called WinWord.exe and double-click it. This is so complicated. Why can’t Microsoft make a computer that’s easy to use?
The Documents and Settings directory arrived on the scene in Windows 2000. Windows NT® kept user profiles in a subdirectory of the Windows directory. This turned out to be poor planning, because it made it difficult to adjust the access control list (ACL) on the Windows directory without accidentally messing up all the user profiles. Moving the user profiles out of the Windows directory solved that problem.
But why "Documents and Settings"? The original name for the directory was actually "Documents & Settings". That ampersand caused a good deal of application compatibility problems. Programs misinterpreted the ampersand as a menu accelerator and ended up displaying the directory name as "Documents _Settings".
In retrospect, the name "Documents and Settings" was a poor choice. Although this new directory met the immediate need of moving profiles out of the Windows directory and it had a highly descriptive name, the spaces didn’t make the directory name particularly easy to type, and its whopping twenty-two character length consumed nearly ten percent of the MAX_PATH characters available for file names.
In Windows Vista™, the name of the user profile directory is simply "Users." It gets the point across, it’s easy to type, it doesn’t contain spaces, and it’s short. Of course, programs should continue to use functions such as SHGetFolderPath to obtain the locations of these special directories.
Another security change in Windows Vista is that the All Users profile has been moved out of the profiles directory and into the root of the drive (though a symbolic link has been left in the profiles directory for compatibility purposes). Some administrators share out the root of the user profiles directory as a cheap way to allow users to access their user profile from another machine. The Application Data directory inside the All Users profile can be written to by any user. This has two serious consequences.
Anybody who has access to the share can create files in the All Users\Application Data directory, thereby consuming disk space without even having an account on the machine. Suddenly, your server has become a warez server—illegally hosting cracked software and bootleg movies—without your even realizing it. And when you go through the audit logs, all you see is that the files were uploaded by "Anonymous" or "Guest."
A second (and possibly more dangerous) consequence of sharing out the profile root is that some applications may create a subdirectory of All Users\Application Data and override the default security settings on that directory with a much less secure ACL, allowing any local user to modify global configuration settings or, worse, modify executable code. Sharing out the All Users profile turns that local vulnerability into a remote one, raising the security stakes considerably. When Shakespeare wrote "What’s in a name?" he probably didn’t have security and software compatibility in mind. But then he wasn’t managing a Windows environment.
(I’d like to give many thanks to Eric Flo, Piero Sierra, and Chris Guzak for their historical assistance.)

Raymond Chen’s Web site deals with Windows history and Win32 programming. He occasionally wonders why so many Italian cities are named after cheeses and restaurants.