August 2008 August 2008
SQL Server: Top Tips for Effective Database Maintenance
Far too often, people who are not trained database administrators somehow end up responsible for a database. They lack the proper training and knowledge to maintain their database, and problems begin to develop. Here’s a primer for all those involuntary DBAs who need a crash course in database maintenance best practices. Paul S. Randal
Windows Administration: Taking Your Server’s Pulse
Your users are complaining that a server is running poorly—do you know where to look to diagnose the problem? PerfMon can be an indispensible tool for this as it has numerous diagnostic capabilities. Get an overview of the key indicators you should use to diagnose a variety of common bottlenecks that can slow down your servers. Steven Choy
System Center: Windows PowerShell in System Center Operations Manager
Windows PowerShell has been integrated into System Center Operations Manager 2007, offering a powerful way to perform and automate common administrative tasks. Get an overview of how you can use Windows PowerShell in OpsMgr to perform routine maintenance, manage agents, and more. Marco Shaw
Internet Information Services: Scaling ASP.NET Applications: Lessons Learned
The key to successfully scaling an ASP.NET application is having a collaborative effort between developers and network administrators, starting at the beginning of the application’s lifecycle. Find out what factors are necessary to scale an application, and see how this collaboration can help ensure the application will run as intended. Richard Campbell
Columns
From the Editor: Live from Tech•Ed: | Letters: Readers Speak Out: | Toolbox: New Products for IT Pros:
SQL Q&A: Partitioning, Consistency Checks, and More: Paul Randal answers your SQL Server questions, showing you how to perform consistency checks and other maintenance on very large databases, how to handle an accidental upgrade, and more.Paul S. Randal | Utility Spotlight: Windows Installer CleanUp Utility: Leftover Windows Installer configuration information can cause problems when you try to update or uninstall an application. You can solve the problem using the Windows Installer CleanUp utility, which can help you remove an application’s Windows Installer information and then reinstall the app.Lance Whitney | Inside SharePoint: Troubleshoot Messaging Integration: Integrating SharePoint into a secure messaging environment is no trivial task, but the positive impact on productivity will make the project worth all the effort. Here is some guidance on how to troubleshoot issues that may arise when you’re setting up the environment.Pav Cherny
Windows PowerShell: A Sneak Peek at Remote Management in Version 2.0: Windows PowerShell 2.0 is in the works and the latest CTP provides some great new capabilities. Here’s a look at remoting—one of the most exciting new features you'll find in Windows PowerShell 2.0 CTP2.Don Jones | Hey, Scripting Guy!: Taking the Express Route: With Windows Vista, the UserAccounts.CommonDialog ActiveX control for opening a file open dialog box no longer exists. Here's a look at how you can solve that problem using Visual Basic Express Edition.The Microsoft Scripting Guys | The Desktop Files: Windows Deployment Services 101: Explore the different versions of Windows Deployment Services, see how you can capture a custom OS image, and find out what you need for performing more automated installations.Wes Miller
Security Watch: Passwords and Credit Cards, Part 2: In the second part of this three part series, Jesper Johansson looks at how software vendors, security suites, and marketing efforts hinder true security efforts and he warns us about how these practices can lead to the collapse of the technology security ecosystem.Jesper M. Johansson | Sustainable Computing: Filtering the Greenwashing: Understanding your power consumption is now an essential part of managing an IT infrastructure. Learn about the strategies you can employ to implement power monitoring in your environment to better understand the power consumption of devices throughout your organization.Dave Ohara | Field Notes: IMAP Saves the World: Do you have e-mail saved in many different places, in different formats, and on different operating systems? Dara Korra’ti explains how you can access all your mail from one place using IMAP4.R’ykandar Korra’ti
Windows Confidential: Building on the Past: Raymond explains how the one-time powerful Alpha AXP system came out of retirement to help develop a new generation of 64-bit systems.Raymond Chen


SQL Server
Top Tips for Effective Database Maintenance
Paul S. Randal
At a Glance:
  • Managing data and transaction log files
  • Eliminating index fragmentation
  • Ensuring accurate, up-to-date statistics
  • Detecting corrupted database pages
  • Establishing an effective backup strategy

Contents
Data and Log File Management
Index Fragmentation
Statistics
Corruption Detection
Backups
Wrap-Up
Several times a week I'm asked for advice on how to effectively maintain a production database. Sometimes the questions come from DBAs who are implementing new solutions and want help
fine-tuning maintenance practices to fit their new databases' characteristics. More frequently, however, the questions come from people who are not professional DBAs but for one reason or another have been given ownership of and responsibility for a database. I like to call this role the "involuntary DBA." The focus of this article is to provide a primer of database maintenance best-practices for all the involuntary DBAs out there.
As with the majority of tasks and procedures in the IT world, there isn't an easy one-size-fits-all solution for effective database maintenance, but there are some key areas that nearly always need to be addressed. My top five areas of concern are (in no particular order of importance):
  • Data and log file management
  • Index fragmentation
  • Statistics
  • Corruption detection
  • Backups
An unmaintained (or poorly maintained) database can develop problems in one or more of these areas, which can eventually lead to poor application performance or even downtime and data loss.
In this article, I'll explain why these issues matter and show you some simple ways to mitigate the problems. I will base my explanations on SQL Server® 2005, but I'll also highlight the major differences that you'll find in SQL Server 2000 and the upcoming SQL Server 2008.

Data and Log File Management
The first area I always recommend checking when taking over a database concerns the settings related to data and (transaction) log file management. Specifically, you should make sure that:
  • The data and log files are separated from each other and isolated from everything else as well
  • Auto-growth is configured correctly
  • Instant file initialization is configured
  • Auto-shrink is not enabled and shrink is not part of any maintenance plan
When data and log files (which ideally should be on separate volumes altogether) share a volume with any other application that creates or expands files, there is the potential for file fragmentation. In data files, excessive file fragmentation can be a small contributing factor in poorly performing queries (specifically those that scan very large amounts of data). In log files, it can have a much more significant impact on performance, especially if auto-growth is set to increase each file size only by a very small amount each time it is needed.
Log files are internally divided into sections called Virtual Log Files (VLFs) and the more fragmentation there is in the log file (I use the singular here because there is no gain from having multiple log files—there should only be one per database), the more VLFs there are. Once a log file has more than, say, 200 VLFs, performance can be negatively impacted for log-related operations such as log reads (for transactional replication/rollback, for example), log backups, and even triggers in SQL Server 2000 (the implementation of triggers changed in SQL Server 2005 to the row versioning framework instead of the transaction log).
The best practice regarding the sizing of data and log files is to create them with an appropriate initial size. For data files, the initial size should take into account the potential for additional data being added to the database in the short-term. For instance, if the initial size of the data is 50GB, but you know that over the next six months an additional 50GB of data will be added, it makes sense to create the data file to be 100GB right away, rather than having to grow it several times to reach that size.
It's a little more complicated for log files, unfortunately, and you need to consider factors like transaction size (long-running transactions cannot be cleared from the log until they complete) and log backup frequency (since this is what removes the inactive portion of the log). For more information, see "8 Steps to Better Transaction Log Throughput", a popular blog post on SQLskills.com written by my wife, Kimberly Tripp.
Once set up, the file sizes should be monitored at various times and proactively grown manually at an appropriate time of day. Auto-grow should be left on as a just-in-case protection so the files can still grow if they need to if some abnormal event occurs. The logic against leaving file management entirely to auto-grow is that auto-grow of small amounts leads to file fragmentation, and that auto-grow can be a time-consuming process that stalls the application workload at unpredictable times.
The auto-grow size should be set to a specific value, rather than a percentage, to bound the time and space needed to perform the auto-grow, if it occurs. For instance, you may want to set a 100GB data file to have a fixed 5GB auto-grow size, rather than, say 10 percent. This means it will always grow by 5GB, no matter how large the file ends up being, rather than an ever-increasing amount (10GB, 11GB, 12GB, and so on) each time the file gets bigger.
When a transaction log is grown (either manually or through auto-grow), it is always zero-initialized. Data files have the same default behavior in SQL Server 2000, but starting with SQL Server 2005, you can enable instant file initialization, which skips zero-initializing the files and hence makes growth and auto-growth virtually instantaneous. Contrary to popular belief, this feature is available in all editions of SQL Server. For more information, enter "instant file initialization" in the index of Books Online for SQL Server 2005 or SQL Server 2008.
Finally, care should be taken that shrink is not enabled in any way. Shrink can be used to reduce the size of a data or log file, but it is a very intrusive, resource-heavy process that causes massive amounts of logical scan fragmentation in data files (see below for details) and leads to poor performance. I changed the SQL Server 2005 Books Online entry for shrink to include a warning to this effect. Manual shrinking of individual data and log files, however, can be acceptable under special circumstances.
Video
Watch Paul Randal demonstrate how shrink and auto-shrink can cause serious fragmentation problems for your database.

Auto-shrink is the worst offender as it starts every 30 minutes in the background and tries to shrink databases where the auto-shrink database option is set to true. It is a somewhat unpredictable process in that it only shrinks databases with more than 25 percent free space. Auto-shrink uses lots of resources and causes performance-dropping fragmentation and so is not a good plan under any circumstances. You should always switch off auto-shrink with:
ALTER DATABASE MyDatabase SET AUTO_SHRINK OFF;
A regular maintenance plan that includes a manual database shrink command is almost as bad. If you find that your database continually grows after the maintenance plan shrinks it, that's because the database needs that space in which to run.
The best thing to do is allow the database to grow to a steady-state size and avoid running shrink altogether. You can find more information on the downsides of using shrink, plus some commentary on the new algorithms in SQL Server 2005 on my old MSDN® blog at blogs.msdn.com/sqlserverstorageengine/archive/tags/Shrink/default.aspx.

Index Fragmentation
Apart from fragmentation at the file-system level and within the log file, it's also possible to have fragmentation within the data files, in the structures that store the table and index data. There are two basic types of fragmentation that can occur within a data file:
  • Fragmentation within individual data and index pages (sometimes called internal fragmentation)
  • Fragmentation within index or table structures consisting of pages (called logical scan fragmentation and extent scan fragmentation)
Internal fragmentation is where there is a lot of empty space in a page. As Figure 1 shows, each page in a database is 8KB in size and has a 96-byte page header; as a result, a page can store roughly 8096 bytes of table or index data (specific table and index internals for data and row structures can be found on my blog at sqlskills.com/blogs/paul in the Inside The Storage Engine category). Empty space can occur if each table or index record is more than half the size of a page, as then only a single record can be stored per-page. This can be very hard or impossible to correct, as it would require a table or index schema change, for instance by changing an index key to be something that doesn't cause random insertion points like a GUID does.
Figure 1 The structure of a database page
More commonly, internal fragmentation results from data modifications, such as inserts, updates, and deletes, which can leave empty space on a page. Mismanaged fill-factor can also contribute to fragmentation; see Books Online for more details. Depending on the table/index schema and the application's characteristics, this empty space may never be reused once it is created and can lead to ever-increasing amounts of unusable space in the database.
Consider, for instance, a 100-million-row table with an average record size of 400 bytes. Over time, the application's data modification pattern leaves each page with an average of 2800 bytes of free space. The total space required by the table is about 59GB, calculated as 8096-2800 / 400 = 13 records per 8KB page, then dividing 100 million by 13 to get the number of pages. If the space wasn't being wasted, then 20 records would fit per page, bringing the total space required down to 38GB. That's a huge savings!
Wasted space on data/index pages can therefore lead to needing more pages to hold the same amount of data. Not only does this take up more disk space, it also means that a query needs to issue more I/Os to read the same amount of data. And all these extra pages occupy more space in the data cache, thus taking up more server memory.
Logical scan fragmentation is caused by an operation called a page split. This occurs when a record has to be inserted on a specific index page (according to the index key definition) but there is not enough space on the page to fit the data being inserted. The page is split in half and roughly 50 percent of the records moved to a newly allocated page. This new page is usually not physically contiguous with the old page and therefore is called fragmented. Extent scan fragmentation is similar in concept. Fragmentation within the table/index structures affects the ability of SQL Server to do efficient scans, whether over an entire table/index or bounded by a query WHERE clause (such as SELECT * FROM MyTable WHERE Column1 > 100 AND Column1 < 4000).
Figure 2 shows newly created index pages with 100 percent fill-factor and no fragmentation—the pages are full and the physical order of the pages matches the logical order. Figure 3 shows the fragmentation that can occur after random inserts/updates/deletes.
Figure 2 Newly created index pages with no fragmentation; pages 100% full
Figure 3 Index pages showing internal and logical scan fragmentation after random inserts, updates, and deletes
Fragmentation can sometimes be prevented by changing the table/index schema, but as I mentioned above, this may be very difficult or impossible. If prevention is not an option, there are ways to remove fragmentation once it has occurred—in particular, by rebuilding or reorganizing an index.
Rebuilding an index involves creating a new copy of the index—nicely compacted and as contiguous as possible—and then dropping the old, fragmented one. As SQL Server creates a new copy of the index before removing the old one, it requires free space in the data files approximately equivalent to the size of the index. In SQL Server 2000, rebuilding an index was always an offline operation. In SQL Server 2005 Enterprise Edition, however, index rebuilding can take place online, with a few restrictions. Reorganizing, on the other hand, uses an in-place algorithm to compact and defragment the index; it requires only 8KB of additional space to run—and it always runs online. In fact, in SQL Server 2000, I specifically wrote the index reorganize code as an online, space-efficient alternative to rebuilding an index.
In SQL Server 2005, the commands to investigate are ALTER INDEX … REBUILD to rebuild indexes, and ALTER INDEX … REORGANIZE to reorganize them. This syntax replaces the SQL Server 2000 commands DBCC DBREINDEX and DBCC INDEXDEFRAG, respectively.
There are many trade-offs between these methods, such as the amount of transaction logging generated, the amount of free space in the database required, and whether the process is interruptible without loss of work. You'll find a white paper that discusses these trade-offs, and more, at microsoft.com/technet/prodtechnol/sql/2000/maintain/ss2kidbp.mspx. The paper is based on SQL Server 2000 but the concepts translate well to later versions.
Some people simply choose to rebuild or reorganize all indexes every night or every week (using a maintenance plan option, for instance) rather than figuring out which indexes are fragmented and whether any benefit will come from removing the fragmentation. While that can be a good solution for an involuntary DBA who just wishes to put something in place with minimal effort, note that it can be a very poor choice for larger databases or systems where resources are at a premium.
A more sophisticated approach involves using the DMV sys.dm_db_index_physical_stats (or DBCC SHOWCONTIG in SQL Server 2000) to periodically determine which indexes are fragmented, and then choosing whether and how to operate on those. The whitepaper also discusses using these more targeted choices. Additionally, you can see some example code to do this filtering in Example D of the Books Online entry for the DMV sys.dm_db_index_physical_stats in SQL Server 2005 (msdn.microsoft.com/library/ms188917) or Example E in the Books Online entry for DBCC SHOWCONTIG in SQL Server 2000 and later (at msdn.microsoft.com/library/aa258803).
Whichever method you use, it is highly advisable to investigate and fix fragmentation on a regular basis.

Statistics
The Query Processor is the part of SQL Server that decides how a query should be executed—specifically, which tables and indexes to use, and which operations to perform on them to obtain the results; this is called a query plan. Some of the most important inputs into this decision-making process are statistics that describe the distribution of data values for columns within a table or index. Obviously, statistics need to be accurate and up-to-date to be useful to the Query Processor, otherwise poorly performing query plans may be chosen.
Statistics are generated by reading the table/index data and determining the data distribution for the relevant columns. Statistics can be built by scanning all the data values for a particular column (a full scan) but they can also be based on a user-specified percentage of the data (a sampled scan). If the distribution of values in a column is pretty even, then a sampled scan should be good enough, and this makes creating and updating the statistics faster than with a full scan.
Note that statistics can be automatically created and maintained by turning on the AUTO_CREATE_STATISTICS and AUTO_UPDATE_STATISTICS database options, as shown in Figure 4. These are on by default but if you've just inherited a database, you may want to check to make sure. Sometimes statistics can become out-of-date, in which case manually updating them is possible using the UPDATE STATISTICS operation on specific sets of statistics. Alternatively, the sp_updatestats stored procedure can be used, which updates all statistics that are out of date (in SQL Server 2000, sp_updatestats updates all statistics, regardless of age).
Figure 4 Changing database settings through SQL Server Management Studio
If you want to update statistics as part of your regular maintenance plan, there is a catch you should be aware of. Both UPDATE STATISTICS and sp_updatestats default to using the previously specified level of sampling (if any)—and this may be less than a full scan. Index rebuilds automatically update statistics with a full scan. If you manually update statistics after an index rebuild, it's possible to end up with less accurate statistics! This can happen if a sampled scan from the manual update overwrites the full scan generated by the index rebuild. On the other hand, reorganizing an index does not update statistics at all.
Again, many people have a maintenance plan that updates all statistics at some point before or after rebuilding all indexes—and so unknowingly they end up with potentially less accurate statistics. If you do choose to simply rebuild all indexes every so often, that will take care of statistics too. If you choose to go a more complex route with fragmentation removal, you should also do that for statistics maintenance. Here's what I suggest:
  • Analyze indexes and determine which indexes to operate on and how to do the fragmentation removal.
  • For all indexes that were not rebuilt, update the statistics.
  • Update statistics for all of the non-indexed columns.
For more information on statistics, see the white paper "Statistics Used by the Query Optimizer in Microsoft® SQL Server 2005" (microsoft.com/technet/prodtechnol/sql/2005/qrystats.mspx).

Corruption Detection
I have discussed performance-related maintenance. Now I want to switch gears and discuss corruption detection and mitigation.
It's very unlikely that the database you're managing contains totally useless information that no one cares about—so how do you go about making sure that the data remains uncorrupted and recoverable in the event of a disaster? The ins-and-outs of putting together a full disaster-recovery and high-availability strategy are beyond the scope of this article, but there are a few simple things you can do to get started.
The overwhelming majority of corruptions are caused by "hardware." Why do I put it in quotes? Well, hardware here is really shorthand for "something in the I/O subsystem underneath SQL Server." The I/O subsystem consists of elements such as the operating system, file-system drivers, device-drivers, RAID controllers, cables, networks, and the actual disk drives themselves. That's a lot of places where problems can (and do) occur.
One of the most common problems is when a power failure occurs and a disk drive is in the middle of writing out a database page. If the drive cannot complete the write before it runs out of power (or write operations are cached and there isn't enough battery backup to flush the drive's cache) the result could be an incomplete page image on the disk. This can happen because an 8KB database page is actually comprised of 16 contiguous 512-byte disk sectors. An incomplete write could have written some of the sectors from the new page but leave some of the sectors from the previous page image. This situation is called a torn page. How can you detect when this happens?
SQL Server has a mechanism to detect this situation. It involves storing a couple of bits from every sector of the page and writing a specific pattern in their place (this happens just before the page is written to disk). If the pattern is not the same when the page is read back in, SQL Server knows the page was "torn" and raises an error.
In SQL Server 2005 and later, a more comprehensive mechanism called page checksums is available that can detect any corruption on a page. This involves writing a whole-page checksum on the page just before it is written out and then testing it when the page is read back in, just as for torn-page detection. After enabling page checksums, a page needs to be read into the buffer pool, changed in some way, and then written out to disk again before it is protected by a page checksum.
So, it is a best practice to have page checksums enabled for SQL Server 2005 onwards, with torn-page detection enabled for SQL Server 2000. To enable page checksums, use:﻿
ALTER DATABASE MyDatabase SET PAGE_VERIFY CHECKSUM;
To enable torn-page detection for SQL Server 2000, use this:
ALTER DATABASE MyDatabase SET TORN_PAGE_DETECTION ON;
These mechanisms allow you to detect when a page has a corruption on it, but only when the page is read. How can you easily force all the allocated pages to be read? The best method for doing this (and finding any other kinds of corruption) is to use the DBCC CHECKDB command. Regardless of the options specified, this command will always read all pages in the database, thus causing any page checksums or torn-page detection to be verified. You should also set up alerts so you know when users encounter corruption problems when running queries. You can be notified of all the problems described above using an alert for Severity 24 errors (Figure 5).
Figure 5 Setting up an alert for all Severity 24 errors
So another best practice is to regularly run DBCC CHECKDB on databases to verify their integrity. There are many variations on this command and questions around how often to run it. Unfortunately, there is no white paper available that discusses this. However, as DBCC CHECKDB was the main piece of code I wrote for SQL Server 2005, I have blogged extensively about it. See the "CHECKDB From Every Angle" category of my blog (sqlskills.com/blogs/paul) for many in-depth articles on consistency checking, best practices, and how-to advice. For involuntary DBAs, the rule of thumb is to run a DBCC CHECKDB as often as you take a full database backup (more on this below). I recommend running the following command:
DBCC CHECKDB ('MyDatabase') WITH NO_INFOMSGS,
  ALL_ERRORMSGS;
If there is any output from this command, DBCC has found some corruptions in the database. The question then becomes what to do if DBCC CHECKDB finds any corruptions. That's where backups come in.

Backups
When a corruption or other disaster occurs, the most effective way to recover is to restore the database from backups. Now, this assumes that you have backups in the first place, and that they're not corrupt themselves. All too often, people want to know how to get a badly corrupted database running again when they don't have a backup. The simple answer is that you can't, not without experiencing some form of data loss that could play havoc with your business logic and relational data integrity.
So there is a very strong case for taking regular backups. The intricacies of using backup and restore are well beyond the scope of this article, but let me give you a quick primer on how to establish a backup strategy.
First, you should take regular full-database backups. This gives you a single point-in-time to which you can later restore. You can take a full-database backup using the BACKUP DATABASE command. Look in Books Online for examples. For added protection, you can use the WITH CHECKSUM option, which verifies the page checksums (if present) of pages being read and calculates a checksum over the entire backup. You should choose a frequency that reflects how much data or work your business is comfortable losing. For example, taking a full database backup once per day means you may lose up to a day's worth of data in the event of a disaster. If you are only using full database backups, you should be in the SIMPLE recovery model (commonly called recovery mode) to avoid complexities relating to transaction log growth management.
Second, always keep the backups around for a few days in case one becomes corrupt—a backup from a few days ago is better than no backup at all. You should also verify the integrity of your backups using the RESTORE WITH VERIFYONLY command (again, see Books Online). If you used the WITH CHECKSUM option when the backup was created, running the verification command will check that the backup checksum is still valid, as well as re-check all the page checksums of pages within the backup.
Third, if a daily full database backup does not allow you to meet the maximum data/work loss your business can sustain, you may want to investigate differential database backups. A differential database backup is based on a full database backup and contains a record of all the changes since the last full database backup (a common misconception is that differential backups are incremental—they are not). A sample strategy might be to take a daily full database backup, with a differential database backup every four hours. A differential backup provides a single extra point-in-time recovery option. If you are only using full database and differential database backups, you should still be using the SIMPLE recovery model.
Finally, the ultimate in recoverability comes with using log backups. These are only available in the FULL (or BULK_LOGGED) recovery models and provide a backup of all the log records generated since the previous log backup. Maintaining a set of log backups with periodic full database (and maybe differential database) backups gives an unlimited number of points-in-time to recover to—including up-to-the-minute recovery. The trade-off is that the transaction log will continue to grow unless it is "freed" by taking a log backup. A sample strategy here would be a full database backup every day, a differential database backup every four hours, and a log backup every half hour.
Deciding on a backup strategy and setting it up can be complicated. At the very least, you should have a regular full-database backup to ensure you have at least one point-in-time to recover from.

Wrap-Up
As you can see, to ensure your database stays healthy and available there are a few "'must do" tasks. Here's my final checklist for an involuntary DBA taking over a database:
  • Remove excessive transaction log file fragmentation.
  • Set auto-growth correctly.
  • Turn off any scheduled shrink operations.
  • Turn on instant file initialization.
  • Put a regular process in place to detect and remove index fragmentation.
  • Turn on AUTO_CREATE_STATISTICS and AUTO_UPDATE_STATISTICS, plus have a regular process in place to update statistics.
  • Turn on page checksums (or least torn-page detection on SQL Server 2000).
  • Have a regular process to run DBCC CHECKDB.
  • Have a regular process in place to take full database backups, plus differential and log backups for point-in-time recovery.
I've given T-SQL commands within the article, but you can do a lot from Management Studio too. Hopefully, I have given you some useful pointers for effective database maintenance. If you have any feedback or questions, drop me a line—paul@sqlskills.com.

Paul S. Randal is the Managing Director of SQLskills.com and is a SQL Server MVP. He worked on the SQL Server Storage Engine team at Microsoft from 1999 to 2007. Paul is an expert on disaster recovery, high availability, and database maintenance. He blogs at SQLskills.com/blogs/paul.

© 2008 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Windows Administration
Taking Your Server's Pulse
Steven Choy
At a Glance:
  • Customizing Performance Monitor
  • Guidance on what and how often to measure
  • An overview of key counters and what to look for

Contents
Making the Results More Readable
What and When to Measure
Hard Disk Bottleneck
Memory Bottleneck
﻿Processor Bottleneck
Network Bottleneck
Process Bottleneck
Wrapping Up
Imagine you just arrived at the office on a Monday morning and you’re greeted by an eager user who is complaining that his server is running too slow. How do you even begin to help him? Performance
Monitor, a handy tool built into Windows®, can assist you in diagnosing the problem.
You can access Performance Monitor by typing perfmon at the command prompt or by selecting the Performance or Reliability and Performance Monitor (in Windows Vista® and Windows Server® 2008) from the Administrative Tools menu. To add performance counters and objects to be monitored, you simply click the plus sign and select from a host of possible choices.
So how do you measure the pulse of a server? There are more than 60 basic performance objects, and each object contains multiple counters. In this article, I will discuss the counters that reveal the vital signs of a server, and I will describe the typical sampling intervals that Microsoft® Service Support engineers use most often to troubleshoot performance-related issues.
Of course, a baseline provides a critical reference point when troubleshooting. Since the server load depends on the business requirements and also varies from time to time depending on the business cycle, it is important to establish a baseline determined by the normal workload over a specified period of time. That allows you to observe changes and identify trends.

Making the Results More Readable
Before I dive into an analysis of the counters that represent the vital signs of servers, I'll tell you about two tricks that will make it easier for you to measure the vital signs of servers using Performance Monitor. Note that these tricks are not needed in Windows Vista and Windows Server 2008, but if you are running Performance Monitor on earlier versions of Windows, these two tweaks can come in very handy.
First, you can remove all the distracting sample noise that obscures the graphical view of trend lines. In Windows Vista and Windows Server 2008, Performance Monitor can display up to 1000 data points in graphical view. In previous versions of Windows, the limit is only 100 data points. When there are more than 100 points, Performance Monitor "buckets" the data points. A bucket is represented by a vertical line, indicating the minimum, average, and maximum of the sample points included in the bucket.
As you can tell by looking at the graph in Figure 1, it is difficult to spot the trend line when so much data is displayed at the same time. The Figure 2 graph shows how much easier it is to grasp the data quickly when all the extraneous visual information has been turned off. For details on how you can turn off these vertical lines, see the Knowledge Base article that is available at support.microsoft.com/kb/283110.
Figure 1 Performance data shown with distracting buckets and no commas
Figure 2 A cleaner view of data with comma separators
The second trick is to add comma separators in the numbers, making it much easier to read the values shown in the counters. Windows Vista and Windows Server 2008 have comma separators enabled by default. In previous versions of Windows, however, Performance Monitor does not enable commas by default.
This may not sound like it would make a huge difference, but take a look at Figure 1, which shows the performance counters without commas, and then look at Figure 2, which shows the counters with commas. I find the latter much more readable. For some simple instructions on adding comma separators to your performance counters in Windows XP, take a look at the Knowledge Base article at support.microsoft.com/kb/300884.

What and When to Measure
Bottlenecks occur when a resource reaches its capacity, causing the performance of the entire system to slow down. Bottlenecks are typically caused by insufficient or misconfigured resources, malfunctioning components, and incorrect requests for resources by a program.
There are five major resource areas that can cause bottlenecks and affect server performance: physical disk, memory, process, CPU, and network. If any of these resources are overutilized, your server or application can become noticeably slow or can even crash. I will go through each of these five areas, giving guidance on the counters you should be using and offering suggested thresholds to measure the pulse of your servers.
Since the sampling interval has a significant impact on the size of the log file and the server load, you should set the sample interval based on the average elapsed time for the issue to occur so you can establish a baseline before the issue occurs again. This will allow you to spot any trend leading to the issue.
Fifteen minutes will provide a good window for establishing a baseline during normal operations. Set the sample interval to 15 seconds if the average elapsed time for the issue to occur is about four hours. If the time for the issue to occur is eight hours or more, set the sampling interval to no less than five minutes; otherwise, you will end up with a very large log file, making it more difficult to analyze the data.

Hard Disk Bottleneck
Since the disk system stores and handles programs and data on the server, a bottleneck affecting disk usage and speed will have a big impact on the server's overall performance.
Please note that if the disk objects have not been enabled on your server, you need to use the command-line tool Diskperf to enable them. Also, note that % Disk Time can exceed 100 percent and, therefore, I prefer to use % Idle Time, Avg. Disk sec/Read, and Avg. Disk sec/write to give me a more accurate picture of how busy the hard disk is. You can find more on % Disk Time in the Knowledge Base article available at support.microsoft.com/kb/310067.
Following are the counters the Microsoft Service Support engineers rely on for disk monitoring.
LogicalDisk\% Free Space This measures the percentage of free space on the selected logical disk drive. Take note if this falls below 15 percent, as you risk running out of free space for the OS to store critical files. One obvious solution here is to add more disk space.
PhysicalDisk\% Idle Time This measures the percentage of time the disk was idle during the sample interval. If this counter falls below 20 percent, the disk system is saturated. You may consider replacing the current disk system with a faster disk system.
PhysicalDisk\Avg. Disk Sec/Read This measures the average time, in seconds, to read data from the disk. If the number is larger than 25 milliseconds (ms), that means the disk system is experiencing latency when reading from the disk. For mission-critical servers hosting SQL Server® and Exchange Server, the acceptable threshold is much lower, approximately 10 ms. The most logical solution here is to replace the current disk system with a faster disk system.
PhysicalDisk\Avg. Disk Sec/Write This measures the average time, in seconds, it takes to write data to the disk. If the number is larger than 25 ms, the disk system experiences latency when writing to the disk. For mission-critical servers hosting SQL Server and Exchange Server, the acceptable threshold is much lower, approximately 10 ms. The likely solution here is to replace the disk system with a faster disk system.
PhysicalDisk\Avg. Disk Queue Length This indicates how many I/O operations are waiting for the hard drive to become available. If the value here is larger than the number of spindles plus two, that means the disk itself may be the bottleneck.
Memory\Cache Bytes This indicates the amount of memory being used for the file system cache. There may be a disk bottleneck if this value is greater than 200MB.

Memory Bottleneck
A memory shortage is typically due to insufficient RAM, a memory leak, or a memory switch placed inside the boot.ini. Before I get into memory counters, I should discuss the /3GB switch.
More memory reduces disk I/O activity and, in turn, improves application performance. The /3GB switch was introduced in Windows NT® as a way to provide more memory for the user-mode programs.
Windows uses a virtual address space of 4GB (independent of how much physical RAM the system has). By default, the lower 2GB are reserved for user-mode programs and the upper 2GB are reserved for kernel-mode programs. With the /3GB switch, 3GB are given to user-mode processes. This, of course, comes at the expense of the kernel memory, which will have only 1GB of virtual address space. This can cause problems because Pool Non-Paged Bytes, Pool Paged Bytes, Free System Page Tables Entries, and desktop heap are all squeezed together within this 1GB space. Therefore, the /3GB switch should only be used after thorough testing has been done in your environment.
This is a consideration if you suspect you are experiencing a memory-related bottleneck. If the /3GB switch is not the cause of the problems, you can use these counters for diagnosing a potential memory bottleneck.
Memory\% Committed Bytes in Use This measures the ratio of Committed Bytes to the Commit Limit—in other words, the amount of virtual memory in use. This indicates insufficient memory if the number is greater than 80 percent. The obvious solution for this is to add more memory.
Memory\% Available Mbytes This measures the amount of physical memory, in megabytes, available for running processes. If this value is less than 5 percent of the total physical RAM, that means there is insufficient memory, and that can increase paging activity. To resolve this problem, you should simply add more memory.
Memory\Free System Page Table Entries This indicates the number of page table entries not currently in use by the system. If the number is less than 5,000, there may well be a memory leak.
Memory\Pool Non-Paged Bytes This measures the size, in bytes, of the non-paged pool. This is an area of system memory for objects that cannot be written to disk but instead must remain in physical memory as long as they are allocated. There is a possible memory leak if the value is greater than 175MB (or 100MB with the /3GB switch). A typical Event ID 2019 is recorded in the system event log.
Memory\Pool Paged Bytes This measures the size, in bytes, of the paged pool. This is an area of system memory used for objects that can be written to disk when they are not being used. There may be a memory leak if this value is greater than 250MB (or 170MB with the /3GB switch). A typical Event ID 2020 is recorded in the system event log.
Memory\Pages per Second This measures the rate at which pages are read from or written to disk to resolve hard page faults. If the value is greater than 1,000, as a result of excessive paging, there may be a memory leak.

﻿Processor Bottleneck
An overwhelmed processor can be due to the processor itself not offering enough power or it can be due to an inefficient application. You must double-check whether the processor spends a lot of time in paging as a result of insufficient physical memory. When investigating a potential processor bottleneck, the Microsoft Service Support engineers use the following counters.
Processor\% Processor Time This measures the percentage of elapsed time the processor spends executing a non-idle thread. If the percentage is greater than 85 percent, the processor is overwhelmed and the server may require a faster processor.
Processor\% User Time This measures the percentage of elapsed time the processor spends in user mode. If this value is high, the server is busy with the application. One possible solution here is to optimize the application that is using up the processor resources.
Processor\% Interrupt Time This measures the time the processor spends receiving and servicing hardware interruptions during specific sample intervals. This counter indicates a possible hardware issue if the value is greater than 15 percent.
System\Processor Queue Length This indicates the number of threads in the processor queue. The server doesn't have enough processor power if the value is more than two times the number of CPUs for an extended period of time.

Network Bottleneck
A network bottleneck, of course, affects the server's ability to send and receive data across the network. It can be an issue with the network card on the server, or perhaps the network is saturated and needs to be segmented. You can use the following counters to diagnosis potential network bottlenecks.
Network Interface\Bytes Total/Sec This measures the rate at which bytes are sent and received over each network adapter, including framing characters. The network is saturated if you discover that more than 70 percent of the interface is consumed. For a 100-Mbps NIC, the interface consumed is 8.7MB/sec (100Mbps = 100000kbps = 12.5MB/sec* 70 percent). In a situation like this, you may want to add a faster network card or segment the network.
Network Interface\Output Queue Length This measures the length of the output packet queue, in packets. There is network saturation if the value is more than 2. You can address this problem by adding a faster network card or segmenting the network.

Process Bottleneck
Server performance will be significantly affected if you have a misbehaving process or non-optimized processes. Thread and handle leaks will eventually bring down a server, and excessive processor usage will bring a server to a crawl. The following counters are indispensable when diagnosing process-related bottlenecks.
Process\Handle Count This measures the total number of handles that are currently open by a process. This counter indicates a possible handle ﻿leak if the number is greater than 10,000.
Process\Thread Count This measures the number of threads currently active in a process. There may be a thread leak if this number is more than 500 between the minimum and maximum number of threads.
Process\Private Bytes This indicates the amount of memory that this process has allocated that cannot be shared with other processes. If the value is greater than 250 between the minimum and maximum number of threads, there may be a memory leak.

Wrapping Up
Now you know what counters the Service Support engineers at Microsoft use to diagnose various bottlenecks. Of course, you will most likely come up with your own set of favorite counters tailored to suit your specific needs. You may want to save time by not having to add all your favorite counters manually each time you need to monitor your servers. Fortunately, there is an option in the Performance Monitor that allows you to save all your counters in a template for later use.
You may still be wondering whether you should run Performance Monitor locally or remotely. And exactly what will the performance hit be when running Performance Monitor locally? This all depends on your specific environment. The performance hit on the server is almost negligible if you set intervals to at least five minutes.
You may want to run Performance Monitor locally if you know there is a performance issue on the server, since Performance Monitor may not be able to capture data from a remote machine when it is running out of resources on the server. Running it remotely from a central machine is really best suited to situations when you want to monitor or baseline multiple servers.

Steven Choy is a Senior Premier Field Engineer with Microsoft Premier Field Engineering. He has supported overnment agencies and commercial clients with their projects on desktop deployment and migration, patch management, and group policy. He is currently focused on server virtualization and server health.


System Center
Windows PowerShell in System Center Operations Manager
Marco Shaw
At a Glance:
  • The OpsMgr Command Shell
  • The OpsMgr monitoring provider
  • Automating common administrative tasks
  • Some real-world Windows PowerShell examples

Contents
Operations Manager Command Shell
The OpsMgr Monitoring Provider
Automating Common Tasks
The Real World
Wrapping Up
System Center Operations Manager 2007 (OpsMgr) has given administrators access to Windows PowerShell, a powerful new scripting language for automating tasks. Released to the public in November 2006, it has been downloaded more than two million times since then.
In this article, I am going to talk about Windows PowerShell® and discuss how it applies to Operations Manager. I'll cover some of the common tasks that Windows PowerShell can help you accomplish in a far easier and more automated manner, and refer you to some Web sites that provide useful scripts and explanations. I've gathered insights and blog posts from many of the experts using Windows PowerShell today.
Although Microsoft has started releasing the Community Technology Preview (CTP) versions of Windows PowerShell 2.0, these versions are not production-ready, have not been tested with OpsMgr, and should not be installed on a production system.

Operations Manager Command Shell
In OpsMgr, you access Windows PowerShell through the Command Shell, which is similar to the default Windows PowerShell environment except it loads a console file as well as a script that initializes the environment with OpsMgr cmdlets, functions, and a default connection.
You can start the Command Shell from an icon on the OpsMgr Start menu or by right-clicking on a computer name in the OpsMgr UI console (see Figure 1). This places you directly in the OpsMgr Monitoring drive path (which I will discuss shortly).
Figure 1 Opening the Command Shell from the OpsMgr UI
Windows PowerShell interfaces with OpsMgr through the OpsMgr SDK. Luckily for administrators, cmdlets have already been provided for many of the tasks you would typically want to automate or complete from a command line. If there's no cmdlet for a particular task, you can use Windows PowerShell to interact with the SDK.
The commands provided by the Operations Manager Command Shell are contained in a snap-in—a DLL that gets loaded by Windows PowerShell and contains cmdlets for OpsMgr administration. The snap-in also includes the OperationsManagerMonitoring Windows PowerShell provider. Also known as the Monitoring provider, this tool allows for navigation on connections, groups, and monitoring objects, much like navigating the file system.
You can get a list of all the cmdlets specific to OpsMgr using the Get-OperationsManagerCommand cmdlet, as shown in Figure 2. (In the first version, this was a function, which did not support tab completion; it became a cmdlet in SP1.) The original release of Operations Manager included 74 cmdlets, while OpsMgr SP1 has 87.
Figure 2 Getting a list of the OpsMgr cmdlets

The OpsMgr Monitoring Provider
By using the cmdlet Set-Location, or the alias cd, you can navigate through the layout of groups and computers. The base layout for the default Monitoringdrive is something like the following:
Monitoring:\->RMS->Groups (as defined in OpsMgr)
  ->Computers(as defined in OpsMgr)
From here, you can get to more specific objects. Note that I am only dealing with a simple environment in this case, where there is only a single management server. The first management server installed in a management group is known as a Root Management Server (RMS).
When the Command Shell is started, it creates a drive named Monitoring, maps the drive to the root of OperationsManagerMonitoring provider, and finally sets the current location or path to the root of the Monitoring drive. The Command Shell then searches the registry for the name of the default RMS with which to connect. If the connection to the RMS succeeds, the current location or path is set to the name of the connection, or RMS, as shown in Figure 3.
Figure 3 The Command Shell location is set to the RMS

Automating Common Tasks
Let's see how Windows PowerShell can handle some of the most common administrative tasks.
Controlling Maintenance Mode No matter what task you're dealing with, you generally will want to specify the date and time it should occur. I'll take a brief look at the Get-Date cmdlet to show you how easy this can be. Figure 4 shows a few examples.
Figure 4 Using the Get-Date cmdlet
As you can see, I created a $date variable that contains an object representing the current time. I then used some documented methods supported by the object to show how you can easily get the date and time five minutes later, and then five hours later. If I wanted to get values from the past, I would simply use (-5) instead of (5).
When you need to block all alerts coming from a computer, you can enable maintenance mode. OpsMgr 2007 allows you to put a Windows® service, or even a particular database, into maintenance mode instead of an entire computer or group. There are three cmdlets that deal specifically with maintenance mode tasks: Get-MaintenanceWindow, New-MaintenanceWindow, and Set-MaintenanceWindow.
To put a computer into maintenance mode from within the Command Shell, navigate to the desired computer or monitoring object using the Monitoring provider and invoke the New-MaintenanceWindow cmdlet, as shown in Figure 5. As you can see, this action places the computer called Denver.contoso.com into maintenance mode. I have also defined the start time for the maintenance window to take effect immediately and the end time to take effect exactly one hour later. Note that putting a computer into maintenance mode using this method does not stop all alerts, as the HealthService and HealthServiceWatcher instances for this object are still enabled.
Figure 5 Using the New-MaintenanceWindow cmdlet
Boris Yanushpolsky, Program Manager on the Microsoft OpsMgr team, has provided some very handy Windows PowerShell code that can be used to set all objects that refer to a computer into maintenance mode, and he has explained how to use this once you've created a script. To read more about this, see his blog at blogs.msdn.com/boris_yanushpolsky/archive/2007/07/25/putting-a-computer-into-maintenance-mode.aspx.
Sometimes you need to determine if objects are in maintenance mode that shouldn't be. Cycling through all of the objects to try to figure this out could be a pretty big undertaking, though. Fortunately, Boris Yanushpolsky comes to the rescue again with a Windows PowerShell script that uses the OpsMgr SDK. You can take the code directly from his blog post (blogs.msdn.com/boris_yanushpolsky/archive/2007/08/06/so-what-is-in-maintenance-mode.aspx) and paste it into a Command Shell window to get a listing of all objects in maintenance mode.
When an object is in maintenance mode, you may want to end the maintenance period before the originally specified end time. If you're familiar with Windows PowerShell, you may expect a cmdlet with a stop or remove verb, but you must actually use Set-MaintenanceWindow, as in Figure 6.
Figure 6 Using Set-MaintenanceWindow to change the end time
Managing Agents Administrators very often work with agents, and there are six cmdlets and one function (as of the release) that deal with various agent-related tasks. You can get a list of them with this command:
Get-Command *-agent*
As of the SP1 release, Install-AgentByName is packaged as a cmdlet instead of a function. It is recommended that you use the Install-AgentByName cmdlet, as it provides a better basis for support and consistency.
The built-in help included with the Command Shell provides some good examples of using the Install-Agent and Uninstall-Agent cmdlets. Roger Sprague, Senior Software Design Engineer on the Microsoft OpsMgr team, posted an alternative method on his blog, which is reproduced in Figure 7 (see his original post at blogs.msdn.com/scshell/archive/2006/09/28/getting-started.aspx).
This script works fine with the OpsMgr RTM (you must be located in the root of the monitoring provider—in this article it is monitoring:\oxford.contoso.com), but it fails with OpsMgr SP1. To make it work with OpsMgr SP1, the first command in Figure 7 should be changed to the following:
$managementServer = Get-RootManagementServer
Figure 7 Installing an agent
# Get the Root Management Server.
$managementServer = Get-ManagementServer -Root: $true

# Create the discovery configuration for computer2 and computer3.
$discoConfig = New-WindowsDiscoveryConfiguration -ComputerName: computer2, computer3

# Discover the computers.
$discoResult = Start-Discovery -ManagementServer: $managementServer -WindowsDiscoveryConfiguration: $discoConfig

# Install an agent on each computer.
Install-Agent -ManagementServer: $managementServer -AgentManagedComputer: $discoResult.CustomMonitoringObjects
At this point, the agent is installed on the remote system that should be monitored, but there is still one last step where the management server must actually accept the new agent before it is completely monitored. If no further action is taken, the monitoring server will automatically accept the new agent for monitoring. But this acceptance process can also be fast-tracked by using the Get-AgentPendingAction cmdlet. This one-liner will fast-track the agent acceptance process:
Get-AgentPendingAction | Where Object {$_.AgentName –like
'computer*'} | Approve-AgentPendingAction
By piping this to Reject-AgentPendingAction, instead of Approve-AgentPendingAction, you can block the acceptance of this agent by the OpsMgr server, if the action is still outstanding. If it is not outstanding, use the Uninstall-Agent cmdlet instead.
As I mentioned, you can also use Install-AgentByName for specifying a computer directly at the command line where the agent is to be installed.
Working with Management Packs There are four cmdlets to help you deal with various management pack tasks. You can list them using this:
Get-Command –noun ManagementPack
This simple command provides the currently installed management packs and their version numbers:
Get-ManagementPack | Format-Table –autosize
Now I'll use the Command Shell to install two common management packs using these installers:
  • Internet Information Services System Center Operations Manager2007 Management Pack.msi
  • Windows Server® Base OS System Center Operations Manager2007 Management Pack.msi.
Since my goal here is to show how the Command Shell can make regular tasks easier, I am going to do this using the fewest commands possible (as shown in Figure 8). I could have changed this installation procedure to pass the quiet flag to the installer (the .msi files), but I wanted to select the location where the files would be extracted manually.
Figure 8 Installing management packs
Next, I need to install the common libraries, which I can do with the following:
Get-ChildItem –Path C:\MPs –filter *Library.mp |
ForEach-Object
  {Install-ManagementPack –filePath $_.FullName}
Then I install the other required management packs:
Get-ChildItem –Path C:\MPs –filter *200?.mp |
ForEach-Object
  {Install-ManagementPack –filePath $_.FullName}
As Figure 9 shows, the built-in Command Shell help provides an excellent example of using the Export-ManagementPack cmdlet to export unsealed management packs. If you want to export all management packs, change this line
$mps=Get-ManagementPack |
Where-Object {$_.Sealed –eq $false}

$mps=Get-ManagementPack
Figure 9 Exporting a management pack
Manipulating User Roles The Get-UserRole cmdlet provides some functionality for administering users but, oddly, doesn't come with a complementary Set cmdlet, and you don't typically use it to make edits or updates (according to the Windows PowerShell SDK documentation). As you can see in Figure 10, first I get a listing of the current user roles, and then I add a user to the Read-Only Operators group (see Figure 11).
Figure 10 Displaying user roles
Figure 11 Adding a user
Enabling Audit Collection Services (ACS) ACS is a new optional feature in Operations Manager 2007 that, in brief, provides a centralized way to deal with security audit information. ACS isn't enabled by default and typically might be configured later on in a future phase of an OpsMgr deployment.
When it comes time to enable a large number of agents for ACS, Windows PowerShell comes to the rescue by helping to automate the setup. During the OpsMgr beta, Microsoft provided a script to enable ACS on all agents. Neale Browne, a contributor and blogger for SystemCenterForum.org, took this a step further and added support for additional parameters.
The SystemCenterForum.org site (a community site that provides Microsoft® System Center solutions) has made two different Windows PowerShell scripts available for automating the setup of ACS. To set up all the agents in a particular group, use systemcenterforum.org/wp-content/uploads/ACSBulkEnableGroupDisplayName.zip. To set up all monitored agents, download systemcenterforum.org/wp-content/uploads/ACSBulkEnableAllAgents.zip.
Enabling Agent Proxying Your OpsMgr environment may include agentless monitored devices. These devices must be assigned to a management server or to an agent-managed device that will provide remote monitoring. You can find a detailed description of using a Windows PowerShell script to configure a large number of agents at systemcenterforum.org/enable-agent-act-as-a-proxy-in-bulk-via-powershell. An updated version of the script is available at systemcenterforum.org/wp-content/uploads/SetAgentProxyBulk_FQDN.zip.
There are other conditions where particular management packs require that an agent be set to act as a proxy also. Please consult the management pack documentation for more details.

The Real World
Here are some real-world examples to demonstrate further into how Windows PowerShell can help with automation.
Resolving Alerts Have you ever had to delete several alerts for a particular computer? Perhaps something went wrong with an application or the alerts weren't being actively resolved. Here's a one-line command that will resolve all the alerts that have a resolution state of zero:
Get-Alert –criteria 'ResolutionState = ''0''' |
Resolve-Alert | Out-Null
This next example accomplishes the same thing as the first one, but it will run much faster in a bigger environment with more outstanding alerts:
Get-Alert | Where-Object {$_.ResolutionState -eq 0} |
Resolve-Alert | Out-Null
The reason behind the performance difference is that when the criteria parameter is used, the value passed is provided directly to the SQL Server® database, and only the relevant data is returned. This reduces the objects that must be passed all the way back to the Windows PowerShell console.
Now you have a quick way to remove all the outstanding alerts for one computer. Depending on your requirements, you can set this to run automatically.
Finally, here's a quick command that lets you view all of the alerts for a specific day:
Get-Alert -criteria 'TimeRaised >= ''4/25/2008'''
You can very easily change the date value, and you can pipe the output to the Resolve-Alert cmdlet.
Test Alerts Sometimes you want to be able to monitor certain events in the Windows Event Viewer and test for them. These two lines will create a quick event log entry:
$api=New-Object -comObject MOM.ScriptAPI
$api.logscriptevent("API test",100,0,
  "Test using PowerShell")
By default, however, this will write to the Operations Manager event log, which probably isn't the place where you'll be watching for a particular event to be logged. Fortunately, Stefan Stranger, Premier Field Engineer with Microsoft, has written a script to create events in the Windows Event Viewer, and it provides more flexibility on being able to write to a particular log. You'll find the script at go.microsoft.com/fwlink/?LinkId=120308. (Stefan has packaged the script in a .cab file. You will need to download the file, then right-click it to extract his script.)
The only thing to note with Stefan's script is that the value entered for the event source determines to which log the entry will be written. Perhaps the easiest way to make sure an alert is directed toward the proper log is to open the Windows Event Viewer and find the source of the latest entry.
Set the Owner It may sometimes be useful to set the owner of an alert automatically. Here's a simple way to do it from the Command Shell:
$alert = Get-Alert
  -id f3f73d62-37ab-45ce-a7ff-2bdda0dfaeb4
$alert.set_owner("Administrator")
$alert.update("Updated owner")
The first line of this code gets the ID of a particular alert object and passes it to the $alert variable. In the second line, that variable is used to set the owner, and, finally, the update is applied to the OpsMgr database. If you check the owner of the alert with the following command, you'll see it changed:
Get-Alert -id f3f73d62-37ab-45ce-a7ff-2bdda0dfaeb4 |
Select-Object Owner
To set the owner for an entire set of alerts, you could simplify the code like this:
Get-Alert | ForEach-Object {$_.Set_
  Owner("Administrator");
  $_.Update("Owner set")}
Restore Monitoring State Chances are you have come across agents in a "not monitored" state. When this happens, you may need to restore all the agents to a full monitored state quickly, and then attempt to determine what happened. Running the script in Figure 12 directly from the Command Shell should restore full monitoring to any agents affected.
Figure 12 Resetting the Health Service Store
$all="Microsoft.SystemCenter.AllComputersGroup"
$agents = Get-ChildItem Microsoft.SystemCenter.AllComputersGroup | `
  Where-Object {$_.HealthState -eq 'Uninitialized'}
foreach ($agent in $agents)
{
$agent.DisplayName
Push-Location $all\$agent\Microsoft.SystemCenter.HealthService
Get-Task | Where-Object {$_.Name -eq "Microsoft.SystemCenter.ResetHealthServiceStore"} | `
    Start-Task -Asynchronous
Pop-Location
}
Take a look at the code in Figure 12. After declaring a variable, I get a list of all the agents on this RMS and filter on the ones that appear to be in trouble. Then I call a task asynchronously that resets the Health Service Store on all agents in the filtered list.
Associate Alerts with Management Packs Figure 13shows how to get a list of all new alerts as well as the management pack with which each is associated.
The code required a few tricks to make sure all of the management pack names could be resolved. The cmdlets used will vary, depending on whether the alert is from a rule or a monitor. (Note that in Figure 13 I needed to implement a small workaround for the Get-Monitor cmdlet. As of OpsMgr SP1, the cmdlet now supports a new ID parameter, which would slightly simplify the code.)
Figure 13 Finding the management pack
Get-Alert | Where-Object {($_.PrincipalName -ne $null) -and ($_.ResolutionState = '0')}| `
Format-Table –autosize PrincipalName,Severity, `
  @{Label="MP"
      Expression={If(!($_.IsMonitorAlert)){
        ForEach-Object {
          ((Get-Rule $_.MonitoringRuleId).GetManagementPack()).DisplayName}
       }
       Else{
         ForEach-Object {
          $id=$_.ProblemId
          ((Get-Monitor -criteria "Id='$id'").GetManagementPack()).DisplayName}
         }
   }
}
Easy Reporting When you upgrade your environment, it may be useful to audit the versions of all installed agents. It takes just a simple script to print out a nice report:
Get-Agent| `
Format-Table DisplayName,@{ `
  Label="Version"
  Expression={ `
    switch ($_.Version){
    "6.0.5000.0" {"RTM"}
    "6.0.6246.0" {"SP1 (RC)"}
    "6.0.6278.0" {"SP1 (RTM)"}
    }
  }
}
Figure 14 shows output from the script. This script is an expanded version of examples provided at systemcenterforum.org/checking-operations-manager-2007-agent-and-server-versions-via-powershell.
Figure 14 Output showing the versions of some agents
Scheduling a Task You may want to run a Windows PowerShell script on a regular basis. Let's say you use a client computer to connect to and manage your OpsMgr server, and you have the OpsMgr admin tools installed locally. And suppose, for example, you want to run the agent version report on a daily basis and have the output saved to a file with the current date used as the name. You can use the Windows built-in task scheduler to run the script from your local machine.
To do this, you simply set up a new task with the program set as powershell.exe (typically located in C:\Windows\System32\WindowsPowerShell\v1.0). After you have created the task, edit it and set the command to run as something like this:
C:\Windows\System32\WindowsPowershell\v1.0\powershell.exe
  –Command "& {& 'c:\agent_report2.ps1'}"
It turned out that I had to make quite a few changes to my original Windows PowerShell code, as you can see in Figure 15. I had to add the OpsMgr PowerShell snap-in, create the Monitoring drive mapped to the OperationsManagerMonitoring provider, and create a connection to the RMS. I also load the OpsMgr custom Windows PowerShell script (Microsoft.EnterpriseManagement.OperationsManager.ClientShell.Startup.ps1) to load OpsMgr-specific functions.
Figure 15 Scheduling the agent-version script
Add-PSSnapin Microsoft.EnterpriseManagement.OperationsManager.Client
New-PSDrive Monitoring Microsoft.EnterpriseManagement.OperationsManager.Client\
  OperationsManagerMonitoring ""
New-ManagementGroupConnection Oxford.contoso.com

Set-Location 'C:\Program Files\System Center Operations Manager 2007'
./Microsoft.EnterpriseManagement.OperationsManager.ClientShell.NonInteractiveStartup.ps1

$file="C:\$(Get-Date -f `"MMddyyyy`").rpt"

Get-Agent -Path Monitoring:\Oxford.contoso.com | `
Format-Table DisplayName,@{ `
  Label="Version"
  Expression={ `
    switch ($_.Version){
    "6.0.5000.0" {"RTM"}
    "6.0.6246.0" {"SP1 (RC)"}
    "6.0.6278.0" {"SP1 (RTM)"}
    }
  }
} | Out-File $file
But that's not all. You might notice that every time the task is run, a black console window pops up on the screen if someone is logged onto the system where the script is being run. You'll find a little trick that takes care of this, provided by Don Jones and Jeffery Hicks from Sapien, at blog.sapien.com/index.php/2006/12/26/more-fun-with-scheduled-powershell.
Basically, you need to wrap the script within a VBScript. To accomplish this, you'd use code like that in Figure 16 instead of calling the following from your scheduled task:
C:\Windows\System32\WindowsPowershell\v1.0\powershell.exe
  –Command "& {& 'c:\agent_report2.ps1'}"
Figure 16 Hiding the pop-ups
Dim objShell
Set objShell=CreateObject("WScript.Shell")

'enter the PowerShell expression you need to use short filenames and paths
strExpression="'c:\agent_report2.ps1'"

strCMD="powershell -nologo  -command " & Chr(34) & _
"&{&" & strExpression &"}" & Chr(34)

'Uncomment next line for debugging
'WScript.Echo strCMD

'use 0 to hide window
objShell.Run strCMD,0
The task would now use the wscript.exe program, and the call would look similar to this:
C:\WINDOWS\System32\wscript.exe C:\agent_report.vbs
Finally, I can create automated reports, and the process is hidden from logged-on users.

Wrapping Up
This has been a whirlwind tour of the new automation features available with OpsMgr 2007, and it barely scratches the surface of how you can use Windows PowerShell to take care of OpsMgr administration.
I'd like to thank all the people mentioned in the article, as well as Pete Zerger, MOM MVP and SystemCenterForum.org founder, for their valuable help.

Marco Shaw is an IT System Analyst for a Canadian telecommunications company. He has been working in the IT industry for more than 10 years, and he recently received a Windows PowerShell MVP award. Marco is also the Assistant Community Director of the new PowerShell Community Web site at powershellcommunity.org. He blogs at marcoshaw.blogspot.com.


Internet Information Services
Scaling ASP.NET Applications: Lessons Learned
Richard Campbell
At a Glance:
  • The importance of developer/IT collaboration
  • Understanding the principal scaling strategies
  • Sharing knowledge across teams
  • Agreeing on a core set of knowledge

Contents
The Fundamentals of Scaling
The Meeting of the Minds
What Networking Needs to Know from Development
What Development Needs to Know from Networking
Getting Back to Scaling
The Cooperative Firefight
As a consultant, every successfully scaling ASP.NET application I've ever worked with has been the result of a collaborative effort between the developers (who built it) and the network administrators (who actually run it). Unfortunately, it's not always apparent at the beginning of an
application's lifecycle that this collaboration is essential. As a result, I almost never have a chance to get involved at the beginning of an application's lifecycle—only at the bitter end when there is trouble.
The simple truth is that no application scales effectively the first time around—it's impossible to get everything correct right out of the gate. To successfully scale an application, you need to understand how the application was built as well as how the operating environment it runs in works. In other words, you need information from both the development folks and the network folks. Without that shared knowledge, you can not succeed.

The Fundamentals of Scaling
Before diving in, let's set the stage for what it takes to actually scale an ASP.NET application. There are two fundamental strategies that are typically employed—specialization and distribution—and most large, scaling ASP.NET applications apply both strategies. Furthermore, every trick in the book to help your ASP.NET application scale breaks down into one or the other.
Specialization entails separating elements of the application so they can be scaled independently. For example, you might build dedicated image servers rather than using the same servers that render the ASP.NET pages. The optimal configuration of an image server is quite different from an ASP.NET server. Also, separating your image requests from the rest of the application opens up the possibility of using third-party resources for serving images. The same approach can apply to other resource files.
Distribution, on the other hand, involves spreading the application symmetrically across multiple servers, typically called a Web farm. ASP.NET applications are particularly suited to distribution because each individual page request is relatively small, and the interactions of a given user are largely independent of other users. Distribution is really the manifestation of the "scale out" philosophy, where multiple mid-performance servers work together to service the users, rather than the "scale up" approach that has one massive server doing it all.
Combining specialization and distribution increases efficiency—you can distribute only the elements of the application that need additional performance. For example, if you've created specialized image servers but image serving is still not adequate, you can add image servers rather than adding servers for the entire application. It's important to keep these strategies in mind as you dig into improving the performance and scalability of your ASP.NET application.

The Meeting of the Minds
At some point during the lifecycle of every ASP.NET application, the developers and the network/IT personnel are likely to meet. With any luck, this will take place before the application is deployed, but sometimes it happens only after there is a crisis—for example, when the application runs great in the test environment, then ships to the users and slows to a crawl. (This is when consultants like me are called in.)
When developers and network people meet, the primary goal is to exchange information. The developers possess key knowledge about the application, as do the network personnel about the operating environment. Each group needs to understand the other group's data, and the earlier in the application lifecycle they meet, the better.
This meeting should not occur for the first time during a crisis. Without a core understanding between the two teams, it's extremely hard to figure out why the application isn't performing up to requirements. Furthermore, it's all too easy to defensively decide that the problem lies solely with the other team. And that's simply never true—invariably it takes both parties to solve any significantly complex problem.
Even in good times, however, such a meeting can be a very challenging. Those I have orchestrated typically start with the network folks on one side of the table, the development folks on the other. Staring ensues. To get the conversation rolling, I outline the goal of the meeting, specifically a trading of knowledge. It doesn't matter who goes first. I'm going to start with what networking needs to know from development.

What Networking Needs to Know from Development
Every ASP.NET application has its own quirks, but there are key elements that apply in every case. The web.config file is one of these (see Figure 1). Web.config is an intersection point between development and networking. Sometimes it's configured by the developers, sometimes by networking folks. Either way, what's in web.config puts limits on the hardware and network configuration and directly affects how the application operates. Exploring every little aspect of the web.config file in detail would easily fill an entire book; here the point is that both groups need to study the web.config file and agree on what the configuration it represents means and also on the impact the various settings will have on the application and on the environment.
Figure 1 A basic web.config showing some application settings and custom error configuration
For example, the <authorization> section of web.config specifies how users will be authenticated in the application, and thus defines a dependency. If the application uses Windows® authentication, it may be dependent on Active Directory®. If it uses forms-based authentication, the dependency is on a data store of user accounts. That's certainly worth a conversation.
The <customErrors> section is worth noting because it affects how failure looks to users. This is not a complex setting, but it's worth discussing just to understand what the error pages will be. Early on in the collaborative cycle there probably are no custom error pages—that too is worth a conversation.
The <appSettings> section of web.config can be especially significant. This is typically where developers stash global values, like connection strings for databases. It's a great source of dependencies and a key part of planning for failover, migration, and so on. But since the <appSettings> section is entirely custom, it can contain almost anything and may require a lot of explanation to understand what's there. Often orphans are left in this section—values that don't actually get used anywhere in the application.
Even if your developers aren't using <appSettings>, the network/operations folks may want them to—having all database connection strings there is an effective way to create a simple failover strategy. If the database server dies, a replacement string can be inserted to point to a different database server. Catching this opportunity early in the development cycle can increase the reliability and maintainability of the application.
Finally, you should note that an absolutely key value in web.config from a scaling perspective is the <sessionState> tag, which determines where session data will be stored for the application. By default, session data is stored "InProc," or in the process space of the ASP.NET application. If the session data is configured for in-process, it means all load balancing must be "sticky"—a given user must always be served back to the same server, where the session data resides. This is a huge conversation between developers and network folks because it directly impacts how you scale and how you failover. Talking early about this can save a lot of heartache when trying to debug the application.
Once the conversation gets to session state, it's easy to segue into the load-balancing requirements of the application in general. Some environments have dedicated load-balancing hardware with specific features—but if the application can't handle these features, they aren't going to mean much. I have encountered situations where hardware load balancing was being used with an ASP.NET application that had in-process session data. The consequence was occasional lost sessions with no explanation. It looked like a bug in the application, but it wasn't—it was a configuration mistake.
The network folks need to know clearly from development exactly what load balancing schemes will work for the application. This is very much a two-way conversation, about what load-balancing schemes are available as well as what the application can tolerate. A powerful advantage to having this conversation early in the lifecycle of the application is that you could plan in advance to go to an out-of-process, non-affinity load balancing scheme.
By the time an ASP.NET application is ready for deployment (or is already in its initial deployment), the development staff will have a pretty good idea of the areas of the application that are fast and slow. Most importantly, they'll have a sense of the bottlenecks or potential crisis points in the system. If the network/operations team knows about these bottlenecks, they can prepare for problems ahead of time, and perhaps even avert them.
For example, I once worked with an application that had a massive data-load process every night, during the application's downtime. The developers built the data-loading mechanism, tested it, and understood how it worked. They knew it was very stressful for the database, but it was effective and a good way to solve the problem.
But they never told the networking team that this process took place, or that they scheduled it to run at 1 A.M.—exactly the time the database backups ran. The database remained online, but it ran much more slowly during backup time as all transactions being sent to the database would be held in the transaction log.
It wasn't until the combination of simultaneous dataload and backup caused the database transaction log volume to run out of disk space that the conflict was identified. Moving the data load to start at 3 A.M. completely resolved the problem—but only after several days of crisis.
Having a conversation early about these kinds of workload items in your application can save a ton of grief down the road.

What Development Needs to Know from Networking
My favorite starting point for the part of the meeting where the networking folks explain their world to the development folks is the network diagram. All too often, the developers see the network as in Figure 2—that is, no network, merely just Web servers, browsers, and the Internet.
Figure 2 The simple network vision of the developer
The reality, of course, is a lot more complex. A diagram like Figure 3 is closer to the truth, though even that is simplified. When you look at Figure 3, you immediately have more questions, such as "how will the Virtual Private Network (VPN) users work with the application," or "what is the authentication difference between an internal user and a public user," and so on.
Figure 3 The real network that the developers need to understand
Obviously, it's not enough to simply provide a network diagram. It's important to also explain the network in detail because it's impossible to know, just by looking at a diagram, exactly what elements will impact the application. You need to talk through the actual connect process of a public user, a VPN user, and an internal user. And discussing the various firewall rules that exist, especially in more complex DMZ-style network architectures, will naturally bring up potential issues for the application.
Obviously, it would be advantageous to have all these discussions before the application was deployed, even before development started—but no matter what, the discussion has to happen eventually, and having everyone at the table understand the entire network diagram is key.
Networking is also the arena of failover and redundancy models, but without some software support or at least awareness of behavior, these models rarely work as planned. A detailed discussion of what various scenarios of failure look like has to take place. If there is clustering in place on the database, this will affect code relating to accessing the database. For example, can the queries retry after a server switchover? If there is a redundant site available, how is data going to be replicated to the site? How does a switchover take place from one site to the other?
Once again, having the conversation early helps, but better late than never—all the folks involved with the application need to understand how these things work. There's nothing more frustrating than having a failover solution that doesn't actually work when it's needed.
Finally, there's one other key network resource that needs to be shared—production logs. I always recommend that production logs be made available to the development team. The usual reaction from network folks to this request is "Ask me for them and I'll send them to you." I don't think this is adequate—it's much more effective to give developers the ability to retrieve the logs themselves, typically from a backup site.
Production logs are essential during a crisis. They're often the best (and only) source of real, empirical data on what actually happened. But they're equally valuable in more ordinary circumstances. When developers have routine access to production logs, they can check through them to see how a new feature is behaving. And that's a great way to find out that your feature is not doing what you expected and correct the problem before it's a crisis. When everyone has access to logs, you can react to problems quicker and fix them sooner.

Getting Back to Scaling
The meeting between network and development is all about understanding the full scope of the issues surrounding scaling an ASP.NET application. The actual environment the application operates in directly affects how the code the application runs will work. All the strategies around scaling will impact its behavior in the environment. Applying a specialization strategy such as separating out the SSL-related parts of the application may require changes to the networking environment and possibly changes in the servers themselves.
Even an apparently code-centric change like using caching can impact the environment. That is because when you add data caching to your ASP.NET application, you're decreasing the number of database calls in exchange for an increase in memory usage. The result is that you might increase the number of ASP.NET servers you need, or increase the number of worker thread recycles on the servers, which would trigger events at the network monitoring side.
Scaling your ASP.NET application does impact both development and network personnel—so it's well worth involving both groups in the decision-making process. Often the collaboration will yield original solutions that the teams working independently wouldn't discover. For example, network folks might be aware of existing hardware solutions within the company that can help the developers deliver on performance and scaling requirements. Talking through the details of the application and the environment will reveal these opportunities.

The Cooperative Firefight
Resources for Scaling ASP.NET Apps

  • Developing High-Performance ASP.NET Applications
  • How To: Scale .NET Applications
  • How To: Tune and Scale Performance of Applications That Are Built on the .NET Framework
  • Implementing a Scalable Architecture
  • Maximizing IIS Performance
  • Scaling Strategies for ASP.NET Applications
A crisis in scaling isn't automatically a bad thing—in fact, it's usually a good thing since it's brought on by too many users wanting to run your application, making it proof of the application's value! Now, though, you have to make it work.
Sometimes scaling crises are brought on by another event—perhaps your company is doing a promotion, or you were blogged about, or a social networking site has pointed a tremendous number of people at your Web site all at once. All of a sudden, you're in a firefight to keep the application running. As you might guess, it's best to work through this kind of scenario before it actually happens. And a demonstration of how collaboration can help an organization survive this sort of crisis is a great way to close your meeting of developers and network folks.
The first question in managing the crisis is "How do we detect that the site is dying under load?" All too often the answer is, "we get a call from the CTO." If the first sign that your Web site is in trouble is a call from anyone outside, you have a problem. Some decent metering should inform you of problems before the phone rings. It won't actually stop the phone from ringing, but it will at least give you a head start on how to answer. Saying "Oh really?" to the CTO is not a career-enhancing move.
Next question—who gets called first? Who responds to the event? Often a network person gets the first warning, and that person might be relatively inexperienced so there needs to be a clear escalation plan. Who gets called next? The challenge is doing early, rapid diagnosis to find out what sort of problem you're having. If it's a network outage, that's one thing. But if it's a scaling problem, that's something else entirely. With scaling problems, getting someone from the development team involved early is advisable.
Everyone involved in the firefight needs good information, so effective access rights are key. The goal is to disseminate as much information as possible so that an effective diagnosis can be made, and this should ultimately dictate the scope of the solution.
Often, if the only answer is to write code, the event will be over before the code can be written. Of course the event should be noted in order to help shape development priorities in the future, but it's not reasonable (or smart) to try writing a pile of code and ram it into production without careful testing. Rule #1 of any firefight—don't make the fire any worse.
When it comes to scaling problems, sometimes the solution is to simply wait it out. But that is still a decision that can be made.
At the same time, planning for these contingencies means that different techniques can be rapidly applied. Bringing both network and development resources to bear on a scaling problem often solves it in remarkably good time, perhaps even soon enough to make that promotional event even more of a success.
In conclusion, the lesson learned from the network side is that a high-scaling, well-performing ASP.NET application represents a successful collaboration between the folks who build the application and the ones who deploy and operate it.
The collaboration is inevitable, so it's best to start early with meetings between the teams to share information. The goal of the meetings is to build understanding and agreement on all elements of the application, what it does, how it works, what it depends on, how it behaves under load, and how it copes with problems that arise over time.
When this collaboration is really effective, the result is a more agile company, one that is able to respond quickly when there are problems, and one that is able to communicate clear goals on what the application needs in order to succeed in the future.

Richard Campbell is a Microsoft Regional Director, an MVP in ASP.NET, and the co-host of .NET Rocks, the Internet Audio Talkshow for .NET Developers (go to dotnetrocks.com). He has spent years consulting with companies on the performance and scaling of ASP.NET and is also one of the co-founders of Strangeloop Networks.


From the Editor Live from Tech•Ed
Joshua Hoffman

It's always a little tricky when I sit down to write the Editor's Note every month. Coming up with a topic isn't generally that hard, as there are always plenty of interesting things to talk about. No, the catch is that the Editor's Note needs to be written about six weeks before almost anyone will read it (partly so we have time to get the magazine printed, but mainly so that the copyeditors here have plenty of time to fix all of my mistakes). Unlike blogging, which can be completely reactive to the moment, it's hard to write about "news" in this setting because by the time you read it, it won't be news anymore.
Yet here I am at Tech•Ed 2008, in "sunny" Orlando, and it's really the most important thing going on right now in our techie little world. So while this may be a little dated by the time you read it … well … tough.
Tech•Ed is an exciting event for us for a number of reasons. For one thing, it's one of the only times of the year that we get to see so many of our readers (and even our authors) in person! We have the pleasure of e-mail, IM, and phone calls all year long, but nothing compares to the swell of ideas and interesting conversations that grow out of the congregation of so many people.
Tech•Ed is also a great time to get a look ahead at the coming year. The executives will be here addressing the big-picture strategy, which is certainly informative. We'll hear a lot about new directions in virtualization, the next generation of the Web, revolutionary concepts in systems management, and other things.
However, it's the nitty-gritty stuff that I'm here for. Mark Russinovich, a technical fellow at Microsoft and a contributing editor, will talk about the redefinition of security boundaries. Steve Riley, another contributing editor, will talk about the disappearance of network boundaries altogether—a particularly exciting and innovative concept! And Mark Minasi, Romi Mahajan, Dharmesh Godha, and I will do a panel discussion about the consumerization of IT.
I'll also be speaking about how to get started if you want to write for TechNet Magazine, a presentation I have enjoyed giving for the past few years and one which has inspired quite a few first-time authors. By the time you read this, the slides from my talk will be posted on our blog at blogs.technet.com/tnmag, so feel free to take a look. Who knows? Maybe you'll attend Tech•Ed next year as one of our authors!
Oh, and let's not forget about the Tech•Ed parties. We love the parties, too.
—Joshua Hoffman

Thanks to the following Microsoft technical experts: Sunil Agarwal, Eric Charran, Trina Gorman, Sanjeev Nair, Isaac Roybal, Jez Sadler, Roger Sprague, Stefan Stranger, Kevin Sullivan, Jim Truher, Alan von Weltin, Michael Walsh, and Buck Woody.


Letters Readers Speak Out

SharePoint Content Types
I am looking for information that will give me a detailed view on using content types in Microsoft® SharePoint®. Where can I find this information?
—E.A.

A good source for this information is the Microsoft SharePoint Products and Technologies Team Blog (available at blogs.msdn.com/sharepoint), and Pav Cherny has written an article on standardizing data management with custom content types that you can find at technet.microsoft.com/magazine/cc194408. There is a white paper on managing enterprise metadata that you may find helpful, available at technet.microsoft.com/library/cc262729.
Another resource you might want to explore is Andrew May's blog at blogs.msdn.com/andrew_may. Andrew is a programmer-writer at Microsoft responsible for writing the developer documentation for the enterprise content management features in Windows® SharePoint Services 3.0 and Microsoft Office SharePoint Server 2007. He blogs about content types and is offering posters of technical illustrations that deal with content types and explain the more complicated aspects of enterprise content management in a SharePoint environment.

Taking Security Further
I loved the article "The Great Debate: Security by Obscurity" (June 2008, technet.microsoft.com/magazine/cc510319). However, I was a little disappointed that the whole thing was focused on the changing of the Admin name alone. I mostly land on the side of Roger Grimes and Aaron Margosis regarding the Admin name debate because I do agree it is better than nothing and it has a low degree of creating management headaches. But there are so many meatier challenges—Admin shares was mentioned but not delved into wholly by the group, and that one does have some serious management headaches when it is blocked.
I would also love to see this group debate the pros and cons of attempts to "lower the attack surface." Lowering the attack surface is a slight variant of security by obscurity because most of the stuff people rip out is to shut down anything that broadcasts location or identity. For example, I have many customers ripping out IPv6 in the name of security by obscurity. Service removal is often done just like medieval doctors—close your eyes, reach in, and hope it doesn't kill the patient.
—Shelly B.

We completely agree with you about the limited focus on changing the admin name. Earlier drafts included a number of other examples, but the article would have been 20 pages long if we had not cut it down! So we picked one example to focus on, and the admin account issue seems to be the most often mentioned. Not to worry though—we've already begun brow-beating the authors into writing more on this subject in the near future.

Not Obscure Enough
"The Great Debate: Security by Obscurity" was a good article. Thinking about it, I realized two things. First, I believe that as long as the return on investment (ROI) on obscurity is high enough, then obscurity is a good thing. Second, I realized most IT security has always had an obscurity component.
Take passwords, for example. We have made them more and more obscure over time. In the early days, it was not necessary to have a password on most PCs. Then, as computers became more connected, it was sufficient to have a simple password. As connectivity increased, a password of at least eight characters became important. When that wasn't secure enough, we had to go to mixed characters, then to passphrases, and so on. I suspect we will find that our old obscure protections will continue to become not obscure enough.
—Tom B.

Thanks, Tom, for the very insightful comments. It will be fascinating to see how this trend plays out, especially as security concepts such as multifactor authentication (smart cards plus passwords, for example) become more commonplace.

TechNet Magazine Posters
I would like to get the Microsoft posters that have been included with a few issues of TechNet Magazine for the developers in my company. Can you tell me how to purchase them?
—Helen G.

The three posters available are: Microsoft Exchange Server 2007 Component Architecture, Windows Server® 2008 Active Directory® Feature Components, and Windows Server 2008 Feature Components. You can purchase them by calling 800-444-4881 in the U.S. and Canada, and 785-838-7500 in all other countries.


Toolbox New Products for IT Pros
Greg Steen

Monitor Your Servers
PA Server Monitor Pro
poweradmin.com/servermonitor
You need to keep tabs on all parts of your infrastructure. Knowing something is awry before your end users do is paramount to damage control and communication. (Besides, having the boss breathing down your neck isn't much fun.) One tool to help you keep tabs on your server infrastructure is PA Server Monitor Professional Edition from Power Admin LLC.
Getting up and running with PA Server Monitor is relatively straightforward, especially with the application's built-in Smart Configuration Process, which can discover machines on your LAN and automatically configure a set of monitors for each. If you are not interested in going through the discovery process, you can import a list of machines for which you want to configure monitor checks. And once you've configured a machine, you can export that configuration via the GUI to reuse it as a template for other machines or as a backup in case you ever need to reinstall.
PA Server Monitor offers 18 different monitors you can assign to each of your servers. These monitors include standard checks, such as monitoring disk space, event log errors, remote script execution, file and directory changes, CPU and memory utilization, Internet Control Message Protocol (ping) response, and Windows® service status.
There are several other particularly handy monitors, including TCP port monitoring, which can check basic port status and also send command text and check its response against a predetermined value. Log file monitoring can check a log file for an occurrence of text that you can define as a regular expression or simple text. A directory quota monitor can help you make sure end users don't exceed their limits, and it will notify users via e-mail when they do exceed quotas. You can monitor script execution for any custom tasks you want to run. And finally, you can use file age monitoring to watch your mail queue and ensure it isn't getting backed up.
After you define a monitor for a server, you need to define an action to take if the monitor's threshold is reached. PA Server Monitor provides 15 built-in actions, including e-mail, event log, message boxes, log files, restarting services, Simple Network Paging Protocol (SNPP), and custom scripts for custom actions. When defining actions, you also can define an action escalation chain, which will perform additional actions if the monitor remains in an error state for longer than a specified amount of time. For example, you could set up an action to first restart your Web site automatically. Then, if after another five more minutes the check still fails, an e-mail message is then sent to your handheld device informing you of the site's status.
Over time you likely will want to see statistics on your configured monitors. PA Server Monitor has built-in reporting features to help you get the information to those who need it. Built-in reports include a server summary that shows you stats and the state of a particular server, a general summary that provides a one-liner about each of your monitored machines, and a graphical status map that shows you status indicators for geographically dispersed setups. Reports are generated in HTML so that they can be accessed with a Web browser, which provides a nice touch in your operations center. In addition, you also can generate ad hoc reports with simple graphs and charts, or export to a CSV file for later use. Reports also can be scheduled, allowing you to have the app automatically e-mail your servers' latest stats directly to you on a regular basis.
PA Server Monitor is agentless. One nice benefit is that you can get up and running quickly as long as the service context for the application runs with enough authority to connect to machines on your network to run via Windows Management Instrumentation (WMI) and other remote administrative services, such as remote procedure call (RPC). One of the drawbacks, however, is the elevated account level you need for connecting to the remote machines. Of course, there are other types of server protocol monitors, including HTTP and SMTP, that don't require the elevated permissions.
Another potential drawback is that connecting to machines across a complex and dispersed infrastructure and network topology can be difficult and will be dependent upon domain and network trust. So in the end, whether this agentless approach will work for your environment depends on the layout of your infrastructure and what, exactly, you plan to do with your server monitoring solution.
Price: One-server license for $99 or five-server license for $349 (direct).
Keep tabs on your servers with PA Server Monitor Professional Edition

Clean Up Duplicate Files
dupeGuru
hardcoded.net/dupeguru
If you work with end users or manage shared user disk space, you have undoubtedly noticed that user space gets very cluttered over time. This often is due to a single user saving the same file in multiple places or even many users saving the same file in multiple locations. To help consolidate and clean out that user space, a duplicate finder can be indispensable. One good solution is dupeGuru from Hardcoded Software.
The tool scans your target folder or drive for duplicates, looking at file names and contents. But dupeGuru goes beyond basic string comparison by using a fuzzy matching algorithm to find duplicates that may be missed otherwise. Once your duplicates are marked, you can choose to send them to the recycle bin, delete them forever, move them into a different folder, or have the default application open them up for a final verification. To review the results later, you can export them to XHTML.
In the preferences, you can decide how hard you want the file name and content filter to act when scanning files; this can increase the number of duplicates found, but it also may increase the likelihood of a false-positive match. dupeGuru also has the ability to mark all duplicates that contain a particular word in their file name. This is very useful for finding all those "Copy of" files accidentally created by your users and those empty folders named "New Folder" strewn about your directories.
Versions are available for Windows 2000 and later, and for Mac OS X. If you are looking to reduce duplicates in your music or photos, tailored versions, entitled dupeGuru ME and dupeGuru PE, respectively, are worth a look.
Price: $19.95 (direct).
Clean up duplicate files with dupeGuru

Keep Tabs on Your Environment
pt360 Tool Suite PRO
packettrap.com
It can be very advantageous to have a single interface that lets you view and manage a variety of network monitoring and management tools. Besides saving you a lot of time (and desktop real estate), an integrated suite can make it easy to share data among different monitoring tools while allowing you to survey numerous monitors at once, giving you a more complete view of the state of your infrastructure.
If this sounds good, check out pt360 Tool Suite PRO from PacketTrap Networks. This application offers an integrated UI for the PacketTrap Networks tools, with the addition of centralization and consolidated reporting capabilities. You can create and organize your tool view into a custom dashboard for instant access to your most-used tools and infrastructure status information. The pt360 dashboard also has the ability to pop into full-screen mode. This is great in your operations center, allowing anyone on the team to glance at a screen quickly and see the status of different network and server resources. The application supports multiple dashboards, so you can create various views in your environment according to your needs.
To set up a dashboard, simply select a number of gadgets from a dropdown and click each gadget to configure it. The gadgets are grouped into five categories—Application, Networking, Devices, Availability, and Other—and rely on Simple Network Management Protocol (SNMP) and WMI to monitor your servers, applications, and network devices. The Application gadgets give you hooks into Active Directory®, Microsoft® Exchange Server, and SQL Server® and also provide a browser tool. The Networking gadgets give you insight into device interfaces and provide Top 10 views of average packet loss, highest latency, and interface usage per client. Device gadgets provide performance indicators about CPU and memory usage, available disk space, volume listings, network interface utilization, and more. These are great for keeping tabs on how your servers are performing in real time via their configurable WMI queries. The Availability monitors show you uptime in chart, gauge, text, or list format for your machines. And, finally, Other has a large digital clock.
In addition to the dashboard, the pt360 tools suite gives you networking tools. For example, you get a Cisco Configurator. This lets you archive, modify, upload, download, and compare configuration files from your Cisco equipment. You also get three different ping tools—one that gives you a sortable table view, another that shows you a graphical view, and a third that lets you quickly ping-scan a group of IP addresses. Other types of scanning tools you have at your disposal include a MAC address scanner, a port scan tool, an SNMP scanner, and a WMI scan tool.
The tools suite also can act as a syslog server and send, receive, log, and display syslog messages from and to any device on your network that supports syslog. The display also supports filtering, so you can quickly isolate messages of concern, speeding up the troubleshooting process.
The suite functions as a Trivial File Transfer Protocol (TFTP) server, letting you quickly move files to and from devices around the network. Other tools include trace routing, Whois, and wake-on-LAN capabilities. Of course, since it is a centralized-management solution with connections to your hosts via WMI and SNMP, how the suite stores its credentials for connection is critical. Fortunately, the suite uses a 256-bit Advanced Encryption Standard (AES)-encrypted credential store to which you can add, update, and delete the information needed to monitor your infrastructure.
Price: $999 (direct) for a single license.
Figure 36 Monitor your systems with pt360 Tool Suite PRO

Manage Disk Usage
TreeSize Professional
jam-software.com/treesize/index.shtml
"What is taking up all the disk space on that server!" Is this a familiar cry? Your monitoring application has just sent you an e-mail message notifying you that a low disk space threshold has been hit. OK, so how do you go about cleaning up that machine, you might ask. And how can you quickly judge the best culprit to "Shift-Delete." (That's the quick shortcut for skipping the recycle bin.) One tool that may help you in your quest for the green light in your host monitor is TreeSize Professional from JAM Software.
The application runs on Windows 2000 and later for both 32- and 64-bit versions of Windows, and it gives you an in-depth view of what files are taking up the space on your drives. All you have to do is select the drive to interrogate and the application scans the folder structure automatically. Once that action is complete, you have access to seven different views of the drive. The first is a Chart view, which in turn gives you three different charts to parse: a pie chart, a bar chart, and a relative-size box graph. Each of these charts lets you drill down into the different folders so you can identify the space hogs quickly. You also can customize coloring and style on the charts.
Next is a Detail view that gives you a sortable table view of files, and within this view you can choose from a number of different columns to display. The most notable choices among users include author of Microsoft Office files, the average file size in a folder, the percentage of the parent directory that this file or folder occupies, and hard links.
The Extensions view indicates which types of files are taking up the most space in each directory as you drill down through the folders. The Users view shows the composite view of how much space a user or system account's files are occupying, which is essential in helping you isolate problem users. The File Ages view gives you a graphical view on the percentages of files that have been modified, created, or accessed in the past day, month, year—whatever time frame you prefer—allowing you to identify old files that can be archived or deleted safely. Then there is a Top 100 view, which identifies the 100 biggest files on the drive. And finally, there is the History view, which provides usage patterns on the drive over time.
Each of the seven views can be printed, saved, and copied to the clipboard for later use. Also, TreeSize speeds up common cleanup maintenance tasks via the toolbar by letting you quickly isolate the biggest files, oldest files, temporary files, Internet cache files, and duplicate files with the File Search capability. TreeSize Professional also can scan over the network via UNC shares, but there is a significant performance penalty due to the network overhead involved.
The app also supports a command-line mode. This is great for scheduling and overnight runs. You also can create a Scheduled Task scan, configuring it to scan a path and having it produce a set of reports on a standard schedule. This is useful for monitoring shared user space each night and keeping a close eye on how disk space is being used.
If you are pondering how it is possible that your entire terabyte of user space already has been gobbled up, perhaps TreeSize can help you see how your disk space is being used and start managing it more effectively.
Price: $49.95 (direct) for a single license.
Manage disk usage with TreeSize Professional

Greg Steen is a technology professional, entrepreneur, and enthusiast. He is always on the hunt for new tools to help make operations, QA, and development easier for the IT professional.


SQL Q&A Partitioning, Consistency Checks, and More
Paul S. Randal

Q I accidentally attached a production database to a SQL Server® 2005 server and now I'm trying to attach it to the correct server, which is running SQL Server 2000. I've tried simply detaching the database and attaching to the SQL Server 2000 server, and I've tried doing the same thing with a backup and restore, but neither of these work. Why won't it work on my SQL Server 2000 server? This is the only copy of the database I have.
A The first thing to note here is the importance of having backups. DBAs usually think of needing backups to recover from corruption or some other destructive disaster. But you've hit upon a less obvious disaster—something going wrong during an upgrade, though in this case it was an accidental upgrade. The point remains, however, that you should always have a recent full database backup in case something happens to your database.
An upgrade, whether intentional or accidental, is a one-way operation and it is extremely difficult to reverse its effects. When you upgrade between versions of SQL Server, a series of upgrade steps are performed on the database. Each step usually involves some physical changes to the database, and each step increases the version number of the database.
For example, one of the major changes performed when upgrading a database from SQL Server 2000 to SQL Server 2005 is to change the structure of a database's system catalogs (often called the system tablesor database metadata) that hold various metadata about tables, indexes, column, allocations, and other details regarding the relational and physical structure of the database.
As each of these upgrade steps is performed, the database version number is increased. For example, SQL Server 7.0 databases have version number 515, SQL Server 2000 databases have version number 539, and SQL Server 2005 databases have version number 611 (or 612 if the vardecimal feature is enabled). This allows SQL Server to know the last upgrade step performed on the database.
Releases of SQL Server cannot read databases upgraded to more recent releases of SQL Server (for instance, SQL Server 2000 cannot read a database that's been upgraded to SQL Server 2005). This is because older releases do not have the code needed to interpret the upgraded structures and database layout. And here lies the problem with your database, which has been upgraded to SQL Server 2005: not being able to attach back to SQL Server 2000.
In the absence of a full database backup, your only option is to export all the data from the upgraded database and manually transfer it back to a new SQL Server 2000 database. As long as you haven't used any new features in SQL Server 2005, you can script out the database structure, create the database on SQL Server 2000, and then export/import the data.
To script out the database in SQL Server 2005, you can use the Object Explorer in SQL Server Management Studio (right-click on the database and select Tasks and then the Generate Scripts option). The wizard is self-explanatory and will generate a script to create all the objects, indexes, constraints, triggers, and so on.
Q We've recently redesigned our schema to use table partitioning on the main table—we've been told this helps improve performance. The database is stored on a single 120GB drive and the table is contained in a single filegroup. There's no sliding-window being done, just new partitions being added every week, and all data needs to be available online. Most of our queries process data from within a single week and there are a few that work across the last year. This seems to be easier to do in a single filegroup. Am I correct or is there more to it?
A While it may seem easier to use a single filegroup, you're unlikely to see the benefits of using partitioning that way. The major uses of partitioning are for more efficient database maintenance and for increasing data availability in the event of a disaster, plus you can create a schema that gives performance gains.
The archetypal example of partitioning is for a sales table, with 12 partitions, each representing one month of the last year's sales data. At the end of the month, the oldest partition is switched out (and archived or deleted) and a new partition is switched in. This is the sliding-window scenario to which you referred. The current month's partition is set to read/write and the older months are read-only. Each partition is stored in a separate filegroup. This schema allows all the benefits of partitioning, but is not the most optimal in all circumstances.
My wife, Kimberly, came up with a twist on the schema described above that allows more efficient indexing. Consider splitting the sales table into two tables—one to hold the single read/write partition, and one to hold the 11 read-only partitions, with a partitioned view over both tables.
This allows the read/write table to have a small number of indexes, and the read-only table to have more indexes to support reporting queries. This, in turn, makes Data Manipulation Language (DML) operations on the read/write data significantly more efficient because they do not have to maintain so many non-clustered indexes.
Also, queries on the read/write data don't have to also process the read-only data. Partition elimination in query plans is not perfect in SQL Server 2005, especially in cases with complex predicates, but it has been much improved in SQL Server 2008. For more information on this, see Kimberly's blog post at sqlskills.com/blogs/kimberly/default,month,2007-10.aspx#a71c70243-3d57-4940-9af7-a802b73f2f93.
To show you what I mean, I'll explain some of the features that are enabled by partitioning over multiple filegroups.
Partial Database Availability This is the ability for a database to be online and accessible during a disaster recovery situation as long as the primary filegroup is online. If you have only a single filegroup, then the entire database is down while it is being restored. With the data spread over multiple filegroups, only the damaged filegroups are offline during the restore and your application may be able to continue operating.
Piecemeal Restore This scheme is similar to partial database availability. With a single filegroup, the unit of restoration is either a single page or the whole database. With multiple filegroups, you can restore just a single filegroup—thus allowing partial database availability.
Partitioned Database Maintenance With either of the partitioning schemes discussed above, you can perform per-partition index fragmentation removal, even when all the partitions are on a single filegroup. But with a single filegroup, you lose the ability to do per-filegroup consistency checks, which can drastically cut back on the amount of data that database consistency checking (DBCC) needs to process (and, in turn, the amount of CPU and I/O resources being used).
Simply put, while you can have multiple partitions within the same filegroup, there are numerous benefits to having a 1-1 mapping between partitions and filegroups.
Q We recently had a scare on one of our high-end database servers—a bad memory board was causing corruptions. We discovered these when random data started to appear in our application. We ran DBCC CHECKDB and found all sorts of corruption. Unfortunately, this was also present in our backups so we had to manually prune out the bad data.
Long story short—we replaced the bad hardware and turned on page checksums. We'd like to run regular consistency checks but we don't have a large maintenance window and our 2.4TB database takes a long time to check. What can we do?
A The question of how to perform consistency checks and other maintenance on a VLDB (very large database) is becoming more and more common. Many DBAs give up after finding that DBCC CHECKDB takes longer to run than their maintenance window allows. (In some situations, the database is used 24x7 and there is no valid time to give up the CPU and I/O overhead required by DBCC CHECKDB for an extended period of time.)
Apart from giving up and not running any consistency checks (which I definitely do not recommend) there are four methods you can use. I've personally helped customers use all four of these approaches.
Use the WITH PHYSICAL_ONLY Option of DBCC CHECKDB A regular DBCC CHECKDB runs a large number of logical consistency checks, taking up a lot of CPU (and basically being a CPU-bound operation). Using the WITH PHYSICAL_ONLY option limits the checks to running the very fast DBCC CHECKALLOC allocation bitmap consistency checks, and then reading and auditing every allocated page in the database, forcing any page checksums present on the pages to be tested. This turns DBCC CHECKDB into an I/O-bound operation, with a much shorter run time (in fact, it's sometimes even a magnitude or more faster than a full DBCC CHECKDB, which can translate to pretty significant time savings).
Split Up the Consistency Checking Workload This entails breaking up the tables in the database into equal sized groups (the easiest approach is to do this by number of pages) and then using DBCC CHECKTABLE commands to consistency check all the tables in a single group each night. So, for example, with, say, seven groups being checked one-group-per-day, with a DBCC CHECKALLOC and DBCC CHECKCATALOG once a week, the equivalent of a DBCC CHECKDB can be achieved, albeit spread out over the course of a week.
Use Table Partitioning with Multiple Filegroups The largest tables in the VLDB can be split over several filegroups. A sample consistency- checking schema may be to run a daily DBCC CHECKFILEGROUP on the filegroup holding the read/write partition and also a weekly DBCC CHECKFILEGROUPs on the filegroups holding the read-only partitions. The logic is that the read-only data is comprehensively backed up and not used in day-to-day processing. As a result, it does not need to be consistency checked as often because corruption is not so critical in this data.
Offload the Consistency Checks to Another Server This option involves restoring the regular full database backups onto a different server and running the DBCC CHECKDB on that server. Obviously, this completely offloads the consistency checking workload from the production server. However, the downside is that if a corruption is found, a consistency check must then be run on the production server—but that should be a very infrequent occurrence.
As you can see, there are plenty of options available for DBAs to perform consistency checks on a VLDB, thereby reducing the extra resource load required when running a full DBCC CHECKDB. As I mentioned previously, I've personally helped customers use all four of these approaches. I think you'll find that these options will work for you.
Tip: Using Triggers to Implement Server-Side Logic
In some scenarios, you need to implement server-side logic with triggers. But there are certain pitfalls you must be aware of. Here are some important things you should keep in mind.
  • Triggers are fired by statements, not per row. That said, be sure to place additional logic within your trigger logic to handle the scenario that either multiple or no rows are affected by the statement (triggers are fired per statement, even if no rows are affected). Affected data is held within virtual tables for the Data Manipulation Language (DML) statements. These tables can be joined, allowing you to work with the data.
  • Triggers run synchronously within your transaction. Remember this every time you want to call an external application or access an external resource where you cannot be sure that it will come back in a responsive or reasonable time. For example, if you fire an Update statement against a table and a trigger is fired within that action, the transaction (the implicit of the Update statement) will not finish until all logic of the trigger is completed. If the external application or process returns an error code, SQL Server might cancel the transaction and roll back your transaction (depending on the implemented error handling and error code). So if you need to do something external within a trigger and it's not critical to the transaction (or does not have to be run within the same scope), you should scale it out to another process, picking up data in an asynchronous way. SQL Server 2005 introduced SQL Server Service Broker, which can do such things asynchronously.
  • An error caused by a statement within a trigger is extremely difficult to find. If multiple tables are involved within the transaction, make sure you remember to inspect the triggers in the case of an error and implement proper error handling. If you change the schema within your database, make sure you also keep track of the trigger logic—otherwise a small trigger can have a huge impact on overall performance and stability. Most implications pertaining to schema changes can be checked using Visual Studio® for Database Professionals, doing automatic schema checks while editing the project, and doing static code analysis to check for data type inconsistencies.
—Jens K. Suessmeyer, Database Consultant at Microsoft

Paul S. Randal is the Managing Director of SQLskills.com and a SQL Server MVP. He worked on the SQL Server Storage Engine team at Microsoft from 1999 to 2007. Paul wrote DBCC CHECKDB/repair for SQL Server 2005 and was responsible for the Core Storage Engine during SQL Server 2008 development. An expert on disaster recovery, high availability, and database maintenance, Paul is a regular presenter at conferences. He blogs at SQLskills.com/blogs/paul.


Utility Spotlight Windows Installer CleanUp Utility
Lance Whitney

You try to uninstall or update an application on one of the PCs in your environment, but your command doesn't execute. Instead, an error pops up telling you that the application could not be uninstalled or updated. Sound familiar? If so, you've undoubtedly wondered what causes this glitch.
Software installed using the Microsoft® Windows® Installer (MSI) technology writes certain files and registry settings. If any of those files or settings are missing or corrupted, you may be unable to remove, update, or even reinstall the program. The solution in such cases is to remove all of the Windows Installer information so that the program can be cleanly reinstalled. To do that, Microsoft offers the free Windows Installer CleanUp utility.
The Windows Installer CleanUp utility deletes all files and registry settings associated with a program's Windows Installer configuration. It's important to note that the tool does not remove the program's actual installation files or registry settings. Nor does the tool replace or supersede the Add/Remove Programs command.
Rather, you would resort to this utility if you can't uninstall or update an application through Add/Remove Programs or other methods. By deleting the Windows Installer information, you should be able to reinstall the application. You can then leave the program installed as is, update it, or even remove it.
The Windows Installer CleanUp utility displays a list of all the Windows Installer-based applications on the PC
Some of you may be familiar with the Microsoft command-line program msizap, which removes a program's Windows Installer data. Well, that command is still in use here: Windows Installer CleanUp is simply a friendly front end to msizap, so that you no longer need to wrestle with any cumbersome command-line options.
After you install the utility, you'll find a subdirectory called Windows Installer Clean Up nestled under Program Files. Within this subdirectory are three files: msicuu.exe, msizap.exe, and readme.txt. The msicuu.exe file displays a list of all Windows Installer-based applications on the PC, letting you select, deselect, and remove any of them. This program then passes along your choices to the msizap.exe command, which removes the Windows Installer information for the selected applications.
To run Windows Installer CleanUp, go to the Start Menu | All Programs and find its folder; then click on the Windows Installer Clean Up shortcut. You'll see a list of all the Windows Installer-based applications that are on the PC (as shown in the screenshot). Select the application that you are having trouble with and click the Remove button. That application is then deleted from the Windows Installer database, giving you the opportunity to cleanly reinstall it.
You'll see that after you run the CleanUp tool, the application will no longer appear in Add/Remove Programs. Therefore, if you ultimately want to remove the application, you must reinstall it first. Windows Installer CleanUp works only with software that uses Windows Installer technology, which typically covers installations packaged as MSI files.
Finally, note that you must be logged in as an administrator to run Windows Installer CleanUp. The tool is compatible with all 32-bit and 64-bit versions of Windows. For more information on the utility and to download the software, please point your browser at support.microsoft.com/kb/290301.

Lance Whitney is an IT consultant, trainer, and technical writer. He has spent countless hours tweaking Windows workstations and servers. Originally a journalist, he took a blind leap into the IT world 15 years ago.


Inside SharePoint Troubleshoot Messaging Integration
Pav Cherny

Code download available at: SharePoint2008_08.exe (416 KB)
Contents
General Troubleshooting Approach
SharePoint Messaging Architecture
Obtaining Diagnostic Information
Outbound Message Transfer
Inbound Message Transfer
Directory Management
Conclusion
In a well-designed SharePoint environment, information workers do not need to change their communication habits or work routines to collaborate. Instead of manually checking collaboration sites on an ongoing basis, SharePoint can deliver documents and other information, such as alerts and reminders, directly to their mailboxes.
Users can also participate in workflows by replying to these messages or by sending new items as e-mail messages to document libraries and lists.
However, integrating SharePoint® into a secure messaging environment poses challenges, and it introduces an opportunity for non-SharePoint components to impact SharePoint-based collaboration. The challenge is to isolate and eliminate root causes of problems in the integrated environment efficiently wherever they occur.
In this column, I discuss SharePoint messaging integration based on a proven troubleshooting strategy that delivers fast results. The idea is to locate the trouble spot efficiently and then zoom in with more detailed and targeted troubleshooting steps.
In a large organization, this phased investigation might mean handing over the case to another specialist after an initial analysis, especially if the problem is related to non-SharePoint components. In small and medium organizations, on the other hand, it is not uncommon that a single IT administrator must be able to troubleshoot all involved systems directly, which points even more to the need for using structured approaches.
To keep this column practice-oriented, I use a test environment with Active Directory®, Exchange Server 2007, and WSS 3.0. Step-by-step instructions to build this test environment, as well as the troubleshooting tools discussed in this column, in the companion material, available in the Code Downloads section of the TechNet Magazine Web site.

General Troubleshooting Approach
Perhaps even more so than in other interoperability scenarios, troubleshooting SharePoint messaging integration requires a highly structured approach. There are several factors that complicate this endeavor.
Obviously you have to deal with complex technologies, and while you are struggling with message transfer issues, message-format conversions, security aspects, and directory-management problems, you will find that some SharePoint error messages can be confusing. Internet newsgroups are filled with unresolved discussion threads and suggestions that can lead you down the wrong path—like running all Web applications under the application pool identity for SharePoint Central Administration in order to make messaging integration work.
But there are also positive aspects. SharePoint is very flexible. In fact, it can provide you with detailed troubleshooting information if you know where to look. With a few basic scripts, you can greatly improve your troubleshooting efficiency.
The first troubleshooting objective is to understand the particular issue with which you're dealing. You need to identify where the issue is located and what components it involves. You might also try to reproduce the problem in varying system configurations and study Windows® event logs and text-based log files. But this can be tricky as some issues only occur sporadically. Still, the better you can reproduce an issue, the faster you can identify and eliminate its root cause. Another important objective, often overlooked, is to document all configuration changes and fixes and to make sure they are applied on all relevant systems in the environment, such as on all front-end servers in a Web farm.

SharePoint Messaging Architecture
Whenever you are confronted with an issue related to messaging integration, it is a good idea to get a cup of coffee or tea first and then review the SharePoint messaging integration architecture before engaging in any troubleshooting. Otherwise you might end up fixing the wrong problem or component. For example, an "Access Denied" error that prevents you from creating a contact object in Active Directory is not necessarily an indicator that you need to fix access permissions in Active Directory, as you'll see later.
In any case, it is vital to understand the role of each component involved in messaging integration and how the individual components interact with each other. If you look at Figure 1 you'll see the important components in a SharePoint-Exchange connectivity scenario, and this is covered in detail in subsequent sections.
Figure 1 SharePoint messaging integration architecture

Obtaining Diagnostic Information
One of the most important troubleshooting tools is reliable diagnostic information. The all-time classic is the Windows event log. Among other things, you can control the level of detail SharePoint writes to the Windows event log on Web servers by using SharePoint Central Administration (on the Operations page, under Logging and Reporting, click on Diagnostic Logging). You can specify the least critical event to report to the event log and to the trace log. The trace log (located by default under %CommonProgramFiles%\Microsoft Shared\Web Server Extensions\12\Logs) is very useful if you are looking for low-level information, but can fill very quickly so use this feature sparingly.
Another way to obtain diagnostic information is to enable debugging for the affected Web app by configuring the corresponding web.config file, as outlined in Web Application Config.pdf in the companion material. SharePoint displays detailed trace information directly in the user interface. An advantage of this approach is that you can see the relevant error information without having to parse through megabytes of tracing data. A disadvantage is that you must change the system configuration of your Web application. Don't forget to back up the original web.config file and then revert back to the original configuration when you have finished troubleshooting.
You can also configure the SMTP service to write detailed processing information to the SMTP protocol log (in IIS Manager, select the Enable logging checkbox on the General tab). Make sure you install the ODBC Logging module along with the SMTP Service feature as outlined in SharePoint Messaging Integration.pdf in the companion material, even if you do not plan to use ODBC Logging. Otherwise, the SMTP service will not write the protocol log. By default, the SMTP protocol log is located at %SystemRoot%\System32\LogFiles\SmtpSvc1. It is a text file that enables you to analyze the SMTP communication in detail and determine whether a message has left the SharePoint server.
On the Hub Transport server communicating with the SMTP service, you can also enable protocol logging in the Send Connector and Receive Connector configuration (see the SharePoint Messaging Integration.pdf). You can use a variety of other tools to follow the path messages take in the Exchange Server 2007 environment across Hub Transport servers and Mailbox servers, such as Message Tracking, Queue Viewer, and Pipeline Tracing. If you want detailed information about troubleshooting Exchange Server 2007 mail flow issues, refer to the topic "Transport and Mailflow Issues" at go.microsoft.com/fwlink/?LinkID=120145.
On Active Directory domain controllers, you can gather detailed troubleshooting information by enabling diagnostic logging for Directory Access and other categories by setting corresponding registry entries under HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\NTDS\Diagnostics. A value of 0x0 for a category means that only critical events are logged whereas a value of 0x5 stands for all events, including debug information. Using a logging level greater than 0x3 can degrade server performance and quickly fill the Windows event log. During normal operations, all values should be set to 0x0.
Gathering detailed diagnostic information on SharePoint, Exchange, and Active Directory servers in a troubleshooting situation can help you locate a trouble spot quickly and reliably. Additional troubleshooting tools, such as TCP/IP tools (ping, tracert, telnet, and so on) and Network Monitor, might also come in handy, as most of the communication between these systems is carried out over the computer network. Microsoft® Network Monitor 3.1 is available at go.microsoft.com/fwlink/?LinkId=120143.

Outbound Message Transfer
Resources
  • SharePoint Products and Technologies Web Site
  • Windows SharePoint Services TechCenter
  • Windows SharePoint Services Developer Center
  • Microsoft SharePoint Products and Technologies Team Blog
Equipped with event logs, trace logs, protocol logs, message-tracking logs, and network traces, I am ready to troubleshoot SharePoint messaging integration. Let's start with the easy part: outbound message transfer. SharePoint supports outbound message transfer in a variety of configurations and doesn't require a local SMTP service on the Web servers. However, in a complete messaging integration scenario, it is the recommended configuration to use the local SMTP service. Figure 2 shows the key components in the scenario.
Figure 2 Outbound message transfer
This messaging scenario includes four stages: SharePoint must transfer the message to the SMTP service, the SMTP service must process the message, the Hub Transport server must receive the message, and Exchange Server 2007 must transfer the message to the final destination.
If SharePoint cannot transfer the message to the SMTP service, you will typically receive an error in the user interface, such as that an e-mail notification cannot be sent. The cause may be that the SMTP service is not running. Or perhaps the SMTP protocol log is telling you that the SMTP service rejected the submission attempt with error code 550 (Requested action not taken: mailbox unavailable), which indicates that the issue is located at the SMTP service.
To verify that it is not a SharePoint issue, you can use a basic script (see SMTP.vbs in the companion material) to submit a test message via SMTP with the same sender and recipient information. If it is an SMTP service issue, the script will not succeed. In fact, this script might give you valuable clues about the root cause of the problem, which SharePoint unfortunately does not disclose even with debugging enabled, as illustrated in Figure 3.
Figure 3 Unable to relay messages
By granting the Web server relay permissions in the SMTP service configuration and restarting the SMTP service, you can solve the SharePoint message submission issue. Now the message can reach the SMTP service, and the next important question is whether the SMTP service can route the message to the Hub Transport server.
If the message remains in the Queue folder, the SMTP service is unable to contact the Hub Transport server. That's because either the Microsoft Exchange Transport service is not running or because of a network communication or configuration issue. By using the Telnet client, you can verify quickly and easily whether you can connect to the destination port of a Receive Connector configured for internal communication.
This isn't necessarily TCP port 25. In fact, if the SMTP service uses this port, you might transfer the messages to the default Receive Connector for Hub Transport servers, which is configured to block anonymous message submissions. In this case, you'll see a message file in the Drop folder. (The Windows SharePoint Services Timer service must be stopped; if not, the message disappears after a few seconds.) The message file is a delivery status notification stating that the message was rejected with "Diagnostic-code: smtp;530 5.7.1 Client was not authenticated."
To solve this issue, you should create a dedicated Receive Connector for your SharePoint servers. Because your SharePoint servers are internal systems, choose the Externally Secured authentication option. In this way, you do not need to grant extended rights to the ANONYMOUS LOGON account. Also, consider using IPsec to protect the communication between the SharePoint servers and the Hub Transport server.
If messages leave the SMTP Queue folder and the SMTP protocol logs on the SharePoint server and on the Hub Transport server (check out the %ProgramFiles%\Microsoft\Exchange Server\TransportRoles\Logs\ProtocolLog\SmtpReceive folder) indicate a successful message transfer, you can consider the job done, provided Exchange Server 2007 can route the message to the destination. As mentioned earlier, use mail flow troubleshooting tools included in Exchange Server 2007 to investigate any issues if messages do not reach the intended Exchange recipients. Note that incorrect recipient information, spam filters, and message size restrictions can prevent a final delivery at this stage.

Inbound Message Transfer
In the opposite direction, message transfer is slightly more complicated because potential issues are more subtle. For example, the SharePoint product documentation recommends configuring MX records in DNS for SharePoint e-mail domains, which is a good way to misroute messages in an Exchange organization.
Exchange Server 2007 uses Send Connectors with associated address spaces to transfer messages. It might route your SharePoint messages to Edge Transport servers for transfer over the Internet even though your SharePoint servers are internal. For internal message transfer, use Send Connectors with detailed address spaces instead and specify your SharePoint servers running the SMTP service as smart hosts in the connector configuration (see the SharePoint Messaging Integration.pdf in the companion download). Establishing a well-defined routing topology with dedicated bridgehead servers helps to achieve reliable message transfer from Exchange to SharePoint.
In order to verify that messages arrive at your SharePoint servers, stop the Windows SharePoint Services Timer service. Messages will pile up in the Drop folder because it is the Timer service that processes the messages and places file attachments in document libraries associated with the recipient e-mail addresses, as indicated in Figure 4. If the messages do not arrive in the Drop folder, use the Queue Viewer on your Hub Transport server to see if Exchange routes the messages correctly. Then investigate network communication issues that might prevent the Hub Transport server from communicating with the SMTP service on the SharePoint server.
Figure 4 Inbound message transfer
When you restart the Timer service, you should see the message items disappear from the Drop folder. However, if message attachments do not end up as documents in the destination libraries even though SharePoint indicates successful message processing in the Windows event log, it means that Exchange has sent the messages in Exchange RTF format. To investigate this problem, shut down the Timer service again, send another message with an attachment from your Outlook® client, and then open the message in Notepad after it arrives in the Drop folder. Now search for the string "winmail.dat." If you find it, Exchange Server is sending the messages in the wrong format.
SharePoint requires message attachments to be encoded in MIME or UUENCODE format. Also, SharePoint does not show any winmail.dat-related processing issues in the Windows event log (see Figure 5).The file attachments simply won't appear in the document library. Use Notepad as a troubleshooting tool and eliminate the formatting issue by configuring a Remote Domain definition in Exchange Management Console for the SharePoint e-mail domain. On the Format of original message sent, as attachment to journal report tab, under Exchange rich-text format, select Never use.
Figure 5 Winmail.dat message processing

Directory Management
One of the more useful Timer service events is event 6873, which states that an error occurred while processing an incoming e-mail file because of an unknown alias. This happens if an Exchange user specifies an incorrect SharePoint e-mail address when sending messages, such as a nonexistent document library.
You can mitigate this issue by configuring the Directory Management Service in the Incoming e-mail settings in SharePoint Central Administration to create recipient objects for mail-enabled document libraries in Active Directory. Exchange users can then conveniently select these recipient objects with correct address information from the address book.
Be aware, however, that you are now opening a whole new can of troubleshooting worms. The Directory Management Service fails in a secured system configuration based on the principle of least privilege because this feature has elevated permission requirements on the Web server. What's more, the product documentation (such as technet.microsoft.com/library/cc262947.aspx) somewhat overstates the situation when it suggests that you must grant the Central Administration application pool account the "Create, delete, and manage user accounts" right in the OU you specified for the SharePoint recipient objects in Active Directory.
The Directory Management Service creates contact and group objects, and, therefore, the Central Administration application pool account requires full control over contact and group objects in this OU, but it doesn't require administrative permissions for user accounts. If you set up the Directory Management Service in a Web farm as outlined and try to mail-enable a document library, chances are very good that you will encounter an "Access is denied" error.
The error information points toward Active Directory permission issues, and SharePoint admins are quick to assign full control for Everyone to the SharePoint OU. However, this not only weakens Active Directory security, it also does not solve the problem.
Structured troubleshooting requires that you locate the trouble spot first and then apply targeted configuration changes. If you follow this approach and check the Windows event log on the domain controller and perhaps trace network communication by using Network Monitor, you may find out that SharePoint does not access Active Directory when this problem occurs. So it is unlikely that Active Directory configuration changes will fix the problem. The problem is occurring on the SharePoint server.
Unfortunately it is exceptionally hard to obtain more useful information about this permission issue. SharePoint does not record any additional information in the Windows event log. However, if you enable SharePoint debugging, you can see the call stack (as shown in Figure 6), which suggests that the Directory Management Service uses the CreateContact method of a SharePoint Web service. The SharePoint SDK will tell you that this is the Email Integration Web Service (<WSS_server>:<admin_port>/_vti_bin/SharepointEmailWS.asmx).
Figure 6 Debug output
Server was unable to process request. ---> Access is denied.
   at System.Web.Services.Protocols.SoapHttpClientProtocol.ReadResponse(SoapClientMessage message,      WebResponse response, Stream responseStream, Boolean asyncCall)
   at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke(String methodName, Object[]      parameters)
   at Microsoft.SharePoint.DirectorySoap.SPDirectoryManagementProxy.CreateContact(String Alias,      String FirstName, String LastName, String ForwardingEmail, ContactFlags Flags)
   at Microsoft.SharePoint.SPList.UpdateDirectoryManagementService(String oldAlias, String      newAlias)
   at Microsoft.SharePoint.SPList.Update(Boolean bFromMigration)
   at Microsoft.SharePoint.SPList.Update()
   at Microsoft.SharePoint.ApplicationPages.EmailSettingsPage.SubmitButton_Click(Object sender,      EventArgs args)
   at System.Web.UI.WebControls.Button.OnClick(EventArgs e)
   at System.Web.UI.WebControls.Button.RaisePostBackEvent(String eventArgument)
   at System.Web.UI.Page.RaisePostBackEvent(IPostBackEventHandler sourceControl, String      eventArgument)
   at System.Web.UI.Page.ProcessRequestMain(Boolean includeStagesBeforeAsyncPoint, Boolean      includeStagesAfterAsyncPoint)
Let's take a look at SharepointEmailWS.asmx in a Web browser to see the list of supported operations. You can see the CreateContact method, and clicking on the CreateContact link reveals the SOAP messages you must send to this Web service if you want to create a contact in Active Directory. This is great because now you can create a script-based troubleshooting tool (see SharepointEmailWS.vbs in the companion material) to work outside the SharePoint user interface, making it easier to specify different user accounts during test runs. Figure 7 shows the information returned if you run this script under the Central Administration application pool account (left) and under the SharePoint Administrator account (right).
Figure 7 Two different "Access is denied" errors
The error messages differ! Both messages state that access is denied, but the error on the left indicates a processing issue and the error on the right doesn't. So what is the difference between application pool accounts and the SharePoint Administrator account?
The SharePoint Administrator is a local Administrator on the SharePoint server while application pool accounts are not. Adding the application pool accounts of your Web app to the local Administrators group and restarting the Web server solves the problem, but this is not great news. I consider running application pool accounts with administrative privileges on production Web servers unacceptable.
Why does an application pool account require these elevated permissions? Again, the script can reveal the answer. If you grant all users local Administrator permissions on your Web server for testing purposes, you'll find out that application pool accounts can use the Email Integration Web Service while access remains denied for all other accounts, including SharePoint administrators. This means that the Email Integration Web Service uses the application pool configuration as the basis for access-control decisions.
The application pool configuration is part of the IIS metabase (or applicationHost.config file in IIS 7.0), and access to the metabase is restricted to local Administrators by default. Fortunately it is possible to grant non-administrator accounts access to the metabase using the Metabase Explorer included in the IIS 6.0 Resource Kit tools. On IIS 7.0, it is even easier to grant full control to the applicationHost.config file; simply run the following commands:
CACLS "%SystemDrive%\Windows\System32\inetsrv\config\applicationHost.config"
/G "<application pool account>" :F /E iisreset /noforce
To sum up, in order to create contact objects in Active Directory, the Central Administration application pool requires full control over contact and group objects in the target OU. And the pool accounts of the Web applications require full access to the metabase on the SharePoint servers in the Web farm. Now you are ready to create contact objects using the SharePoint user interface.
But wait, there's more! Creating recipient objects in Active Directory is only half of the directory integration story. The other half is generating Exchange-related recipient attributes and updating server-based address books. For example, if you update the global address list on your Exchange server by using the Exchange Management Shell command Update-GlobalAddressList "Default Global Address List," you can find newly created recipient objects for SharePoint document libraries in the Outlook address book. But sending messages to these recipients results in nondelivery reports because of incorrect address information, as illustrated in Figure 8.
Figure 8 Undeliverable message to a document library
The IMCEAEX acronym stands for Internet Mail Connector Encapsulated Address for Exchange. It indicates an encapsulated non-Exchange recipient address in a format that the old Exchange Internet Mail Connector can handle. But we are dealing with Exchange Server 2007 and native SMTP-based Send Connectors nowadays.
The point is that the SharePoint Email Integration Web Service does not write the address attributes that Exchange Server 2007 requires for message transfer. It only writes the given name, surname, display name, and target address (and, optionally, the ms-ExchRequireAuthToSendTo) attributes. But it doesn't set the mail nickname, or the legacy Exchange DN or proxy addresses, and it doesn't associate the recipient object with an Exchange recipient policy.
To solve this issue, use the Get-Mailbox cmdlet in the Exchange Management Shell to obtain all recipient objects with a target address that points to the SharePoint e-mail domain. And pipe the output to the Set-MailContact cmdlet to activate Exchange recipient policies, as follows:
Get-Contact | where { $_.WindowsEmailAddress –like
  '*sharepoint.contoso.com' } | Set-MailContact
  -EmailAddressPolicyEnabled:$True
Alternatively, use the Exchange Management Console and select the checkbox "Automatically update e-mail addresses based on e-mail address policy" in the contact object properties.

WSS 3.0 and MOSS 2007 support full messaging integration out of the box to enable e-mail based alerts and notifications, workflows, and document submissions. While it is not a trivial task to configure the systems correctly, message integration can increase worker productivity. Specifically, you must make sure that both inbound and outbound message transfer works properly. You should also ensure that directory integration works.
SharePoint error messages are not always intuitive or informative. However, I've shown you ways to get to the bottom of integration issues, locate root causes, and eliminate them reliably without having to jeopardize security on your SharePoint servers.

Pav Cherny is an IT expert and author specializing in Microsoft technologies for collaboration and unified communication. His publications include white papers, product manuals, and books with a focus on IT operations and system administration. Pav is President of Biblioso Corporation, a company that specializes in managed documentation and localization services.


Windows PowerShell A Sneak Peek at Remote Management in Version 2.0
Don Jones

This column is based on a prerelease version of Windows PowerShell. All information herein is subject to change.

Contents
Two Types of Remoting
Synchronous vs. Asynchronous
Reusable Runspaces
Fan-In Remoting
The Killer App in 2.0
Have you had a chance to experiment with the latest Community Technology Preview (CTP) of Windows PowerShell 2.0? The latest version, CTP2, has further refined remote management, and right now is a great time to start becoming familiar with the new capabilities it offers. Before I get started here, you should take a moment to
download it from go.microsoft.com/fwlink/?LinkID=119707.
First, let me clarify a couple of important points. A CTP is pre-beta code that Microsoft provides to let eager users like me get an idea of where Microsoft is headed with the next version of an application. Each CTP milestone or drop (as they call it in the industry) may differ completely from previous drops. This is because the development team gathers feedback, reviews it carefully, and then makes changes to the application based on this user feedback. That methodology brings up an important benefit and an important caveat about your use of the CTP.
Video
Watch Don Jones demonstrate the new remote management capabilities in Windows PowerShell 2.0 CTP2.

The benefit is that when you use the CTP, you're able to give feedback (through the connect.microsoft.com Web site) about the product at a time during the development when the team is able to act on that feedback! If you wait until the beta or, even worse, Release Candidate stage, your feedback is a lot harder to incorporate. During CTP, anything might happen and the team can make vast and sweeping changes, if necessary.
That brings me to the caveat. The CTP is not ready for production. Sure, Windows PowerShell™ 2.0 CTP2 may be one of the most stable pieces of pre-release code you've seen, but keep in mind that the next CTP drop may be a completely different app. So don't start relying on CTP2, because the next version may require you to start all over.
Note that the CTP can't be installed in parallel with Windows PowerShell 1.0. For an ideal setup, the system should also have the Microsoft® .NET Framework 3.5 installed to enable all the available features. Otherwise, some features will be limited.
In addition, since the CTP is very early code, Microsoft has so far placed the most emphasis on the app working on the latest operating systems, meaning Windows Vista® and Windows Server® 2008. Current OS compatibility is no indication of the OS compatibility you can expect for the final released code. Backporting receives attention later in the development cycle.

Two Types of Remoting
In the world of remote management, you usually find two types of remoting: fan-in and fan-out. Fan-in remoting encompasses multiple administrators making secure shell connections to a single server. Windows PowerShell is designed to enable this in a secure, partitioned manner so that, for example, an Exchange Server hosting company can provide its customers with administrative access to their portions of a server. With fan-in remoting, you get secure, remote, interactive access to the copy of Windows PowerShell (version 2.0 only!) installed on a remote server.
Fan-out remoting is when you issue a set of commands to an entire group of remote servers at once. The commands "fan out" from your workstation to the group of servers in parallel. The commands execute on each server, and the results—in the form of Windows PowerShell objects—are returned to your workstation so you can review and work with them. Windows PowerShell supports two core technologies for fan-out remoting—Windows® Management Instrumentation (WMI) and Windows Remote Management (WinRM), which first shipped with Windows Server 2008 and was then updated in the Windows PowerShell 2.0 CTP.

Synchronous vs. Asynchronous
Actually, even Windows PowerShell 1.0 had some basic fan-out capabilities, which were tied to WMI. For example, you could easily create an array of computer names and then retrieve a WMI class from each:
$names = @("server1","server2","server2")
Get-WmiObject Win32_OperatingSystem
    –computer $names
Executing methods—such as rebooting a computer—required a bit more work since version 1.0 offered no bulk way to execute WMI methods. That has changed, however, in the version 2.0 CTP thanks to the Invoke-WmiMethod cmdlet:
$names = @("server1","server2","server2")
Get-WmiObject Win32_OperatingSystem     –computer $names | `
 Invoke-WmiMethod Reboot
There is a problem with this technique, though. It's synchronous, which means each computer is contacted one at a time, and you have to wait for each to finish before you can execute other commands. But the CTP introduces a new concept—background jobs—that allows commands like this to execute in the background. At its simplest, you can have a WMI command run in the background by simply adding the –AsJob parameter:
$names = @("server1","server2","server2")
Get-WmiObject Win32_OperatingSystem     –computer $names -asjob
You can review the resulting job's status by running Get-PSJob, and you can view the final results of the job by running Receive-PSJob. (I'll look at more of the details behind job management in a future column.) However, the Invoke-Command cmdlet provides an even better means of running commands in the background, like so:
$command = { Get-WmiObject     Win32_OperatingSystem }
$names = @("server1","server2","server2")
Invoke-Command –command $command     –computer $names –asjob
This actually pushes the Get-WmiObject command out to each specified computer, and it then executes locally. It typically executes much faster and without having to rely on the WMI remote procedure call (RPC) connections. Instead, Invoke-Command utilizes WinRM, which by default uses port 80 or 443. These ports make it easy to navigate firewalls and are entirely configurable. Invoke-Command also supports additional parameters for alternate credentials and throttling, allowing you to target hundreds of computers but have only a handful running in parallel. This allows you to avoid congestion and excess overhead.

Reusable Runspaces
If you plan to remotely manage a given set of computers more than once, you should consider using runspaces rather than simple lists of computer names. In Windows PowerShell, a runspace is simply an instance of the shell's engine, whether it's running locally on your computer as the shell console window or running in the background on a remote computer. Starting a remote runspace is easy:
$names = @("server1","server2","server2")
New-RunSpace –computer $names
As runspaces also use WinRM, they also use port 80 (or 443, if you specify the –UseSSL parameter) by default. They can also accept alternate credentials, and so on. If you retrieve the resulting runspace objects, you can pass them to Invoke-Command, and Windows PowerShell will push the command out to the computers on which those runspaces exist:
$command = { Get-WmiObject     Win32_OperatingSystem }
$rs = Get-Runspace
Invoke-Command –command $command     –runspace $rs –asjob
The benefit here is that the runspaces remain active for as long as the shell is open, so you can simply reuse them for additional commands.

Fan-In Remoting
Runspaces are also the key to fan-in remoting. For example, Figure 1 shows that I've created a runspace on a remote computer, retrieved a reference to that runspace, and then used the Push-Runspace cmdlet to activate the runspace. At that point, I'm running commands on the remote computer, much like SSH or other remote-shell utilities would allow. Running Pop-Runspace brings back my original, "local" runspace, and the shell prompt helps me keep track of where I am at any moment.
Figure 1 Using runspace to run commands on a remote computer
The exact sequence of commands I ran is as follows:
PS C:\>new-runspace -computer     "WIN-YFZXQMHXAWM"
PS C:\>$server2 = get-runspace -sessionid 2
PS C:\>push-runspace $server2
[win-yfzxqmhxawm]: PS C:\Windows\System32>    pop-runspace
PS C:\>
This technique is called fan-in because multiple admins can open remote interactive runspaces on the same server at the same time—they "fan in" from their individual workstations to the server. A new security model in Windows PowerShell 2.0 allows you to create restricted shells and cmdlets, so each admin can be prevented from making global modifications. Each is limited to his own area of the shell. (These new security techniques require some custom software development in a .NET Framework-targeted language. This is beyond the scope of the Windows PowerShell column, but it is nice to know these capabilities exist.)

The Killer App in 2.0
The Windows PowerShell 2.0 CTP has a stunning list of new features. In my opinion, remoting is the killer app. Every administrator in nearly any environment can benefit from it.
You should familiarize yourself with these features so you can offer your suggestions to the product team. Do you want WinRM default ports to be managed via Group Policy? Should the cmdlets work differently? Are there performance issues? Is WinRM easy enough to configure? You can make an impact by sending suggestions to connect.microsoft.com or share your feedback with MVP Award recipients, including me (to reach me, post your feedback to the forums on ScriptingAnswers.com). So get involved, and help build the killer app for the next generation of Windows PowerShell!
Cmdlet of the Month: Select-Object
Try this out: Get-Service | ConvertTo-HTML | Out-File Services.htm. Now check out the resulting HTML file in your Web browser. There's quite a lot of information, isn't there? If only there was a way to trim it down a bit by selecting the information you are interested in. That is exactly what Select-Object can do for you. For instance, say you just want a list of service names and their current status. You can use this: Get-Service | Select Name,Status | ConvertTo-HTML | Out-File Services.htm.
One thing you need to remember, though, is that Select-Object discards the original object—in this case, services—and produces a custom object (literally a PSCustom object type) that contains only the properties you've specified. Any functionality of the original object is no longer accessible, so you'll likely want to keep Select-Object near the end of your pipeline, working with the original object for as long as you can.

Don Jones is coauthor of Windows PowerShell v2.0: TFM and the trainer behind ScriptingAnswers.com "Special Forces" classroom training (scriptinganswers.com/training.asp). You can reach him at jeepdon@mac.com.


Hey, Scripting Guy! Taking the Express Route
The Microsoft Scripting Guys

In the last few months, one of the Scripting Guys got to experience something new here in the Seattle area. It's a program being tested in various cities throughout the U.S. and it's called electronic tolling. However, in the state of Washington, it's known as Good to Go!
It works something like this: you pay to get an electronic device that attaches to your windshield. You also open an account into which you put a small pile of money. Each time you pass under one of the monitoring devices on the freeway, money is automatically taken from your account. Basically, you're paying to drive on the freeway (so much for the "free" part of freeway—apparently we need to start calling them highways, although we've seen some them under several feet of water recently, which makes them seem not all that high either) without ever having to stop at a toll booth.
Now, in many places where tolls are charged, people who carpool do not have to pay. However, the interesting part about Good to Go! is that it works only in the carpool lane. What's the point of that, you ask? Why would someone go to the trouble of carpooling, then have to pay? Well, they don't. This system only charges people who want to use the carpool lane when they're in their cars by themselves. The obvious reason for wanting to drive in the carpool lane is that there are typically fewer cars in that lane; at times the carpool lane is the only lane where the cars are actually moving.
As we mentioned, this system is experimental right now, though thousands have already signed up for it. There are plenty of people in the Seattle area who are willing to pay money to take an express route to get to wherever it is they're trying to go.
In this month's column, the Scripting Guys have decided to conduct a little experiment of our own. We've decided to show you an express route known as Microsoft® Visual Basic® Express Edition. We'll start off by telling you that one of the coolest things about this route is that it really is free; no tolls will be charged no matter where you use this information, how many people are using it, or what time of day it is. Sounds pretty good, doesn't it? Sure, it may seem like there's a small price to pay in that you have to learn some new things, but you just might decide the benefits are worth it.
Before we go on, we want to mention that we know we're stepping outside our bounds a bit. Typically the Visual Studio® products such as Visual Basic and Visual C#® would be subjects for MSDN® Magazine, not TechNet Magazine. But since MSDN and TechNet seem to be moving more and more toward becoming one big dysfunc—er, happy—family, we're hoping we'll be granted a little leeway here. Also, even though we're talking about working with what has traditionally been a developer tool, we're still talking to Windows® system administrators. Stick with us for a few more minutes and you'll see what we mean.
The first thing you need to do to follow along is install Visual Basic. No, wait. Don't run off saying "I don't have that!" or "I can't afford that!" Microsoft provides an Express Edition of Visual Basic you can download for free from microsoft.com/express/download.
We're not going to go into detail about how to work with Visual Basic Express. There are tutorials in the Help, and the Help itself is actually pretty good. (Hard to believe, isn't it?) What we are going to do is show you how to use Visual Basic Express to solve a particular problem in Windows Vista® (though this example will also work on earlier versions of Windows), and then you can continue to play with it on your own. The problem we're going to solve is this: how do you open a file open dialog box in Windows Vista?
We explained how to open a file open dialog box from a script a few years ago (please see the column "How Can I Show Users a Dialog Box for Selecting Files?" at microsoft.com/technet/scriptcenter/resources/qanda/jan05/hey0128.mspx). That article was published in January 2005, long before Windows Vista came into being. The solution uses the UserAccounts.CommonDialog ActiveX® control to open a file open dialog box, which allows you to browse for a file to open. Unfortunately, this control has been removed from Windows as of Windows Vista, which means this solution won't work on newer systems. (No, no one asked us if this was a good idea.) What this means is that, as far as we know, there is no way to open a file open dialog box using VBScript. So instead of writing a script we're going to use Visual Basic to write an application. One of the things this application will do is open a file open dialog box.
Yes, that's right, the example we're about to show you builds a full application in Visual Basic. That might sound a little scary at first, but you'll see that by using Visual Basic Express, a full application can be as easy to write as a script. This application will consist of a window, or form, that lets you click a button that will open a file open dialog box. You can then select text files from this dialog box, and our application will read from those text files and display a select set of the contents inside the form.
All right, let's get to it. We start by opening Visual Basic Express. The first thing we need to do is to create a new project, which we do by selecting New Project from the File menu. This brings up the New Project dialog box. (Note that if you're using a version of Visual Basic Express earlier than 2008, you'll find a few minor differences from what we present here.) We're going to create a Windows Forms Application, so select the Windows Forms Application template and type in the name you want to give your project. We named ours ReadFiles, as you can see in Figure 1.
Figure 1 Creating a new project in Visual Basic
When the project is first created, it automatically provides a new form to work with. The application we're going to create will present a form to the user that looks like the form in Figure 2. As we mentioned, the form includes a button that, when clicked, will display the file open dialog box, open to the C:\Scripts folder. The user will be able to select one or more text files from anywhere in the file system. The application will then read the first line from each of the selected text files and display the results in the listbox in the form. (If this sounds familiar, you most probably participated in the Beginners Division in the 2008 Scripting Games: microsoft.com/technet/scriptcenter/funzone/games/games08/bevent3.mspx.).
Figure 2 The ReadFiles form that will be presented to users
To create our form, we need to drag controls from the Toolbox onto the project form. By default the Toolbox is open on the left side of the Visual Basic window. (If the Toolbox isn't open, click on the Toolbox button on the left side of the Visual Basic workspace and expand All Windows Forms.) Click Button in the Toolbox, then drag the mouse across the form to create the button the size you want. To add a label to the button, click on the button you just added to the form and type the label you want. (Alternatively, you can edit the Text property in the Properties dialog box, by default on the lower-right side of the Visual Basic workspace.) We labeled our button Browse for Files. Next, click ListBox and again click on the form and drag the mouse to create the listbox. We'll use the listbox to display the lines we read from the text files.
That's all we have to do to create our form. Now comes the fun part: we need to make something happen when we click the button. Start by double-clicking the Button control on the form. This will open a code editor with a new subroutine already created for you, the Button1_Click subroutine:
Private Sub Button1_Click _
  (ByVal sender As System.Object, _
   ByVal e As System.EventArgs) _
  Handles Button1.Click
End Sub
This sub will run each time the button on our form is clicked. As you can see, this sub is empty at the moment; we need to add the code that will tell the app what to do when the button is clicked. And just what is it we want to do? Well, first we want to do the one thing we promised at the beginning of this column that we'd show you: open a file open dialog box. As with almost everything you're going to do with Visual Basic, we'll use the Microsoft .NET Framework for this.
We start by adding a couple of Imports statements up above the sub:
Imports System
Imports System.IO
These statements specify .NET Framework namespaces. We'll be using classes that are in these namespaces, so we import them here so we can simply refer to the classes we want. Note that we said to insert these statements above the sub. Imports statements are global, meaning they can't go inside a subroutine or function; they need to be outside by themselves.
In VBScript, unless you add the Option Explicit statement to the top of your script, you don't need to explicitly declare variables. In Visual Basic, it is automatically added so you need to declare all your variables. We're going to need a variable to hold a reference to the OpenFileDialog object (which just happens to represent the file open dialog box):
Dim dlg As Windows.Forms.OpenFileDialog
Now that we've declared (Dim-ed) a variable of type OpenFileDialog, we can create a new instance of this object, like so:
dlg = New OpenFileDialog()
However, we still haven't displayed the dialog box. First, we need to set some properties:
dlg.Multiselect = True
dlg.InitialDirectory = "c:\scripts"
dlg.Filter = "txt files (*.txt)|*.txt|" & _
  "All files (*.*)|*.*"
The Multiselect property specifies whether the user will be allowed to select more than one file from the file open dialog box. We want them to be able to do this, so we set this property to True. We set the InitialDirectory property to C:\Scripts. If you don't set this property, the dialog box will open to the main computer directory.
And last, but certainly not least, we set the Filter property:
dlg.Filter = "txt files (*.txt)|*.txt|" & _
  "All files (*.*)|*.*"
Here we're filtering which files the dialog box will display to the user. We're going to show them only .txt files. Setting these properties will give us a file open dialog box that looks something like Figure 3.
Figure 3 The Open dialog showing only text files
Now it's finally time to display the dialog box. We do that with the ShowDialog method:
dlg.ShowDialog()
The ShowDialog method returns a value indicating whether the user clicked Open to open the selected files or clicked Cancel to dismiss the dialog box. If they click Cancel, we don't want to do anything. So we put the call to ShowDialog in an If statement to make sure they've clicked Open before we continue:
If dlg.ShowDialog() = _
  Windows.Forms.DialogResult.OK Then
Yes, we know, we checked the OK property rather than the Open property. Behind the scenes, the Open button is really the same as an OK button—it's just that displaying Open to the user makes a lot more sense.
Now we need to know which file or files the user selected from the dialog box. The filenames are contained in the OpenFileDialog object's FileNames property, so we set up a For Each loop to read through the filenames contained in this property:
For Each strName In dlg.FileNames
Oops! We forgot to declare strName:
Dim strName As String
Now we need to open each of these files and read from them. The way we do this in Visual Basic is with a Using block:
Using sr As StreamReader = File.OpenText(strName)
A Using block isn't something that VBScript scripters are used to, since it doesn't actually exist in VBScript. Visual Basic is a little better (OK, in many cases a lot better) at managing system resources than VBScript. The point of a Using statement is to make sure that system resources are freed up when we exit the block. In this case, we're just making sure that we're not hanging on to file contents and large blocks of memory.
We know that this is sounding a little complicated. But really, just look at it like a Set statement. We're simply creating an instance of a new object, in this case a StreamReader object, and assigning it the stream of text returned from the File.OpenText method. Notice that we pass OpenText the name of the file we want to open, the file from our FileNames collection we're currently looping through.
So our StreamReader variable, sr, contains the contents of the file. Now we need another variable:
Dim strLine As String
We'll use this variable to store the first line from the text file we just opened:
strLine = sr.ReadLine()
We read the first line from the file by calling the ReadLine method on our StreamReader object. (Remember that the StreamReader object contains the contents of our file.) We store this line of text in the variable strLine.
Now we simply call the Add method on our ListBox object to add this line to our listbox:
ListBox1.Items.Add(strLine)
For this example, all we want is the first line of the file, so that we can close our StreamReader—we don't want to hang onto the full text of our text file anymore—and close our Using block:
sr.Close()
End Using
After that, we loop around to find the next file that was selected from the file open dialog box until we've read through all the selected files and added all the lines to the listbox. Our completed code looks like Figure 4.
Figure 4 Complete ReadFiles code
Imports System
Imports System.IO

Public Class Form1

  Private Sub Button1_Click _
   (ByVal sender As System.Object, _
    ByVal e As System.EventArgs) _
   Handles Button1.Click
    Dim dlg As Windows.Forms.OpenFileDialog
    Dim strName As String

    dlg = New OpenFileDialog()

    dlg.Multiselect = True
    dlg.InitialDirectory = "c:\scripts"
    dlg.Filter = "txt files (*.txt)|" & _
      "*.txt|All files (*.*)|*.*"

    If dlg.ShowDialog() = _
     Windows.Forms.DialogResult.OK Then
      For Each strName In dlg.FileNames
        Using sr As StreamReader = _
         File.OpenText(strName)
          Dim strLine As String
          strLine = sr.ReadLine()
          ListBox1.Items.Add(strLine)
          sr.Close()
        End Using
      Next
    End If

  End Sub

End Class
See, we told you it wasn't much more difficult to create an application than it is to create a script.
You can test your application by pressing F5 (or pressing the Start Debugging button on the Debugging toolbar). When you click on the button and then select one or more files in the file open dialog box, the results should look something like Figure 5.
Figure 5 The first lines of selected files displayed in the listbox
When your application is running just the way you want it to, select Build ReadFiles (or whatever your project name is) from the Build menu. This will create an .exe file named ReadFiles.exe that you can run just like you'd run any other application, and that you can distribute to your users.
We realize this may not have seemed like an "express" route to using the file open dialog box. But when you consider the fact that you can't do this at all by writing a script as of Windows Vista, well, writing a simple application might be a small price to pay. Just like having to pay the money to be able to actually get somewhere when you're on the freeway: it can be a bit of a hassle, but in the end you get where you want to go.
An added advantage to using Visual Basic for something like this is that now you've started working with a tool that's more powerful than VBScript and that allows you to easily create applications with a nice user interface; that's a huge perk right there. System administrators will find plenty of other uses for Visual Basic too.
And remember, you don't need to relearn how to drive to use the new toll lane; you just need to get used to the fact that the cars ahead of you are actually moving. In the same way, if you know VBScript you don't need to learn a completely new language from scratch to use Visual Basic, you just need to get used to the fact that you can do a lot more.
Dr. Scripto's Scripting Perplexer
The monthly challenge that tests not only your puzzle-solving ability but also your scripting skills.

August 2008: Scripting Alphabet
In this month's puzzle, you need to insert the letters A through Z in the blank spaces to reveal a word that relates in some way to scripting. Each letter (A through Z) should be used only once, and the letters do not appear in alphabetical order. The inserted letter could be the beginning of the word, the end of the word, or anywhere in the middle of the word. For example, in the first line we've inserted the letter C to complete the word SCRIPT:

ANSWER:
Dr. Scripto's Scripting Perplexer
Answer: Scripting Alphabet, August 2008

The Scripting Guys work for—well, are employed by—Microsoft. When not playing/coaching/watching baseball (and various other activities) they run the TechNet Script Center. Check it out at www.scriptingguys.com.


The Desktop Files Windows Deployment Services 101
Wes Miller

Contents
WDS in Windows Server 2003
WDS in Windows Server 2008
Making a Choice
Considerations
Getting Started with WDS
Capture Images
Capturing 101
Image Migration
Unattended Installation
Last month's column reviewed the history of Pre-Boot eXecution Environment (PXE)-based deployment tools from Microsoft. This month, I'll dig into Windows Deployment Services (WDS), specifically discussing the versions that shipped for Windows Server 2003 and the updated version that shipped with Windows Server
2008. After that, I'll get you up and running with WDS.
WDS can best be thought of as the rewritten Remote Installation Services (RIS). Even though they both share some key components, and in fact the version of WDS on Windows Server® 2003 has a legacy mode that is functionally identical to RIS, the actual infrastructure of WDS running in native mode and in 50 percent of mixed mode is brand new.

WDS in Windows Server 2003
When Windows Vista® was released, WDS was offered at the same time. This step was necessary because the Windows Vista setup was completely redesigned using the Windows® Imaging Format (WIM) architecture, making RIS incapable of deploying Windows Vista (and, eventually, Windows Server 2008). WDS was then made available as an out-of-band download for customers running Windows Server 2003 SP1. With SP2, RIS was completely supplanted by WDS.
You should remember that since WDS in Windows Server 2003 has complete backward compatibility, you don't need to move to native mode when installing WDS on your Windows Server 2003 SP2 system(s). You can continue to run in legacy mode—functionally running as an RIS server—or in mixed mode, allowing for either WDS or RIS to be initiated.
Besides being able to deploy Windows Vista and Windows Server 2008, WDS running in native mode or in the native half of mixed mode has several additional features. Rather than utilizing the Windows kernel-mode setup engine and the traditional setup core I discussed last month (technet.microsoft.com/magazine/cc645015), WDS uses Windows PE as its bootstrap preinstallation environment, and it deploys WIM-based images instead of using RISetup or RIPrep-based installs. Doing this makes deployment faster, easier, and more reliable.
Moreover, many of the limitations I mentioned last month, such as the need to use American National Standards Institute (ANSI)-based characters, go away since Windows PE is capable of displaying the same character sets as a full installation of Windows.
In addition, WDS gained a few tools that RIS sorely lacked—a Microsoft® Management Console (MMC) snap-in to assist in management, and a hugely comprehensive command-line configuration tool that enables configuration capabilities far beyond that of RIS.
Since both legacy and mixed modes allow for deploying RISetup- and RIPrep-based images of Windows, you may want to bear one thing in mind. Historically, RIS used a component called Single Instance Storage (SIS) in order to optimize image storage on disk. SIS, which later shipped as a part of Windows Storage Server 2003, utilized a service called Groveler, which runs in the background on RIS (and WDS) servers, looking for identical files. If the size and hash of any files match, SIS saves a copy to its own SIS Common Store folder and then creates a hardlink to each of the original versions it found.
SIS is designed to minimally impact processor utilization while enabling significant storage savings. It was a great concept that was used first by RIS and then broadly on the Windows Server platform. But SIS is no longer used for WDS going forward.
Since WDS on Windows Server 2003 running in native mode—and always on Windows Server 2008—utilizes WIM images for setup, it has its own compression and—more importantly—its own single instance file storage model, and it doesn't utilize SIS at all. The Image Store replaces the storage model used by RIS with a new WIM-based design for WDS. Additional information about SIS is available at go.microsoft.com/fwlink/?LinkId=120302.

WDS in Windows Server 2008
In Windows Server 2008, WDS has become both a superset and a subset of the functionality in Windows Server 2003. It can now deploy images via multicast in standalone mode using Transport Server, a new Trivial File Transfer Protocol (TFTP) server with better performance, support for Extensible Firmware Interface (EFI)-based x64 systems, and enhanced installation metrics reporting. I'll discuss these in next month's column.
You should note that there is one feature of WDS in Windows Server 2003 that has been discontinued in Windows Server 2008—legacy setup support. The ability to deploy RISetup/RIPrep images or use the legacy RIS OS Chooser to start setup is no longer available.

Making a Choice
One of the more frequent questions I've been asked since Windows Server 2008 shipped is, "What version of WDS should I be using?" The answer depends on whether you are deploying pre-Windows Vista versions of Windows and, if so, if you are ready to make the transition from RISetup and/or RIPrep-based images to explicitly using a WIM-based setup.
WDS, even in Windows Server 2008, is capable of deploying Windows all the way back to Windows 2000 if it's in a WIM image—and can do it using multicast. So if you are making the transition from RIS to WDS right now and thinking about Windows Server 2008 in your infrastructure—and you're willing to migrate from RIS-based to WDS-based images—then a migration to WDS in Windows Server 2008 may make sense for you.
You could also migrate from RIS to WDS on your Windows Server 2003 system itself, which essentially happens when you install SP2, and make your shift to WDS-based images. Then you could just upgrade to Windows Server 2008 WDS and go.
That's not the model I'd recommend, though. As with many versions of Windows Server before 2008, Microsoft recommends migrating (versus in-place upgrading) from one version of Windows Server to another.
And as I've mentioned before, considering x64 versions of Windows Server 2008 probably makes a lot more sense in your infrastructure than did x64 versions of Windows Server 2003. If you do make that transition, performing a server-to-server migration of WDS on Windows Server 2003 to Windows Server 2008 will be relatively easy and much more reliable when compared with an in-place upgrade.

Considerations
Here are some points to bear in mind while you are working on your WDS deployment. Note that they are actually not new requirements for WDS—RIS had many of these same considerations—but you'll want to take them into account whether you are building a single WDS server or an entire infrastructure:
Privileges There are two principal items to remember when you are managing a WDS server. You must authorize a WDS server in Active Directory® just as you had to with RIS. To do this, you must be either a domain administrator in the root domain of the forest, an enterprise administrator, or you must have been delegated the appropriate privileges. You also need to be a domain administrator to administer your WDS server, and, of course, you must also be an administrator on the WDS server itself.
Virtualization When running on a Windows Server 2003 or Windows Server 2008 instance virtualized under Microsoft Virtual PC or Virtual Server and communicating with a client PC on the same installation of Virtual PC or Virtual Server, a race condition may occur beginning at PXE boot, leading to a very slow or potentially hung client. It is recommended that you virtualize either clients or servers, but not both—especially on the same host system. Personally, I wouldn't recommend virtualizing a WDS server for production use—this is primarily due to the next point.
Other Services WDS, like RIS, doesn't always play well with other production services. In particular, I've seen issues with RIS/WDS when running on a production system that is also running Microsoft Exchange Server, Systems Management Server (SMS), and System Center Configuration Manager (SCCM).
Generally, I recommend that WDS be the only role on your server if you are using it for production for more than a handful of clients. This isn't unique to WDS—you should carefully consider whether or not you should mix roles of Windows server products and test them before deployment.
Slow- or High-Latency Links As with RIS, WDS isn't meant to be used over a low-bandwidth connection or a high-latency connection such as a satellite link; the PXE handshake may be problematic, and downloading images will likely be painful, if it works at all.

Getting Started with WDS
More on WDS

  • Step-by-Step Guide for WDS in Windows Server 2003
  • Windows Deployment Services Step-by-Step Guide (Windows Server 2008)
  • Deploying and Managing the WDS Update on Windows Server 2003
  • Documentation for Windows Deployment Services in Windows Server 2008
  • Technical Reference: Windows Deployment Services (Windows Server 2008)
To begin, you have to decide whether you are going to use WDS on Windows Server 2003 or Windows Server 2008. If you are using Windows Server 2003, I recommend starting with SP2, which has WDS included natively as an optional component (replacing RIS), instead of using SP1 and the previously available optional download of WDS included in the Windows Automated Installation Kit (WAIK).
For either version of Windows Server, you'll want to build out your server system. You should use a high-quality Network Interface Card (NIC), capable of at least 100Mbps or faster. Do not use NIC teaming, as the PXE infrastructure of WDS may prove problematic with it. Note that if you are using Windows Server 2008, you can install WDS on either Server Core or the full installation of Windows Server 2008.
WDS is installed on Windows Server 2003 as an optional component (as was RIS before it). You can install it during setup of the OS or afterward by running Add/Remove Programs | Add/Remove Windows Components (or running sysocmgr.exe from the command line). WDS is installed in Windows Server 2008 as a Role and can be installed via the initial configuration wizard (click Add roles | Next | and select Windows Deployment Services), via Server Manager (same steps as initial configuration wizard), or from the command line by running ServerManagerCmd –install WDS.
Once you've installed WDS, you need to configure it. Under your Windows Start menu, go to Administrative Tools and select Windows Deployment Services. When the MMC launches, right-click on your server and select "Configure Server" from the menu to launch the configuration wizard.
If you are installing on Windows Server 2003 and upgrading an existing RIS Server, you will want to specify the existing RemoteInstall directory so that the legacy installation can be upgraded to mixed mode. In this situation, you should ensure that the RemoteInstall directory is not located on the System partition, which can impact performance.
Figure 1 shows the WDS console on Windows Server 2008 before the configuration wizard has been completed. Figure 2 shows the Welcome Page for the WDS configuration wizard on Windows Server 2008. Figure 3 shows the PXE configuration step of the WDS configuration wizard. In this step you can specify whether the server should respond to any clients (useful for initial build-out), respond to only known client computers (those that have been pre-staged in Active Directory), or respond to all clients (and whether it should optionally hold all unknown clients so an administrator can choose a further course of action).
Figure 1 WDS installed but not yet configured
Figure 2 The WDS configuration wizard
Figure 3 Choosing a response policy for the PXE server
Figure 4 shows you a fully configured WDS server on Windows Server 2008 (with no images installed yet) with its Properties page opened. If you are using a server that is already a Dynamic Host Configuration Protocol (DHCP) server, be sure to set DHCP Option 60 to PXEClient and set WDS to Do not listen on port 67. Doing so will allow your WDS server to skip a step during the PXE discovery process (as I discussed in last month's column).
Figure 4 A fully configured WDS Server on Windows Server 2008
Once the wizard has completed, you can add new images. Two types of images may be added to a WDS server running in mixed or native mode:
  • Boot Images: these are images containing Windows PE 2.x, generally copied off of the Windows Server 2008 or Windows Vista SP1 installation DVD. Note that you'll want to use the \Sources\Boot.wim file from Windows Server 2008 for your boot images (even if you are deploying Windows Vista or a legacy operating system) if you are using multicast. Boot images generally consist of Windows PE, Windows Setup.exe, and other dependencies.
  • Install Images: these consist of Windows OS images, generally the \Sources\Install.wim file from Windows Vista or Windows Server 2008.
To add a default boot image or default install image, the steps are almost the same. You begin each process by right-clicking the image type and choosing to add an image of that type.
For an install image, you need to specify a name for the image group (given that a default WIM on the Windows DVD contains more than one OS type). Then browse to the appropriate boot.wim or install.wim that you want to add and click Open. For install images, you can then uncheck any volume images (OS images) you do not want to add to the WDS server. Finally, you can PXE-install clients using these default images.

Capture Images
In order to create custom OS images, you need to create a capture image—a Windows PE image that is designed to allow you to capture a Windows system as a WIM image. The easiest way to do this is to use a boot.wim image you installed earlier.
Right-click the desired image and click Create Capture Boot Image. Specify the name and description to use for your image, and the location to save a local copy of the .wim file in case of network connectivity issues when deploying the capture image. Then follow the wizard to completion and click Finish.
Next, right-click the Boot Image folder in the WDS MMC and click Add Boot Image. As before, follow the instructions in the wizard. When you're finished, you are ready to capture up a Windows install image.

Capturing 101
Before you capture an operating system image, you must, of course, install Windows and make sure it includes all your desired applications, customizations, and updates. The next thing you need to do is run Sysprep on your system; you can only capture Windows install images from systems prepared with Sysprep. You must specify a valid local location to save the new image (to avoid potential corruption) and a *.wim file extension.
After you have configured your system and copied the correct version of sysprep.exe to the system, you must follow these steps:
  1. From the command prompt that is on the reference computer, change to the directory where sysprep.exe is located.
  2. Start Sysprep (you can also start by double-clicking sysprep.exe and manually specifying options). On computers running Windows Vista or Windows Server 2008, run the command sysprep /oobe /generalize /reboot. On computers running earlier versions of Windows, run sysprep -mini –reseal –reboot.
  3. When the computer restarts, PXE boot it (the process may vary depending on your client system).
  4. From the boot menu, select your WDS capture image and click Next.
  5. Choose the drive and the name and description for the image.
  6. As mentioned, only systems that have been Sysprepped will be visible. This is by design, and there is no way to bypass it.
  7. Click Browse and then browse to the local folder where you want to store the captured install image. The location you choose can be a mapped network drive.
  8. Select Upload image to WDS server.
  9. Type the name of the WDS server and then click Connect.
  10. From the Image Group list, select the image group you want to store the image in and then click Finish.
This image is now ready to be deployed from your WDS server. There is one thing you RIS veterans out there may have noticed—I didn't capture this Windows image live. WDS requires that the system being captured be offline.
RIPrep, in RIS, was able to capture (in fact, required capture of) an online system. That process was exceedingly fragile. I think you'll find the capture process for WDS relatively easy and certainly more robust.

Image Migration
If you've been using RIS for any time at all, you've got some images laying around. For those images that are RISetup-based (a scripted install), this is the end of the road. You can, however, migrate RIPrep images to WDS WIM-based images. In fact, WDS even allows you to migrate the images via WDSUtil (I'll explain this in more detail next month).
I generally recommend doing this only if you have invested a significant amount of time in an image that you don't want to lose. The process works pretty well, but I've heard of some errors during conversion—which generally means you need to rebuild the image from scratch anyway.

Unattended Installation
Creating an image as I did previously gives you an image that can be installed manually. For a more automated or fully automated install, you need to use two unattended setup files. WDS itself requires an answer file (unattend.xml, stored under the \RemoteInstall\WDSClientUnattend directory on your WDS Server) that automates the WDS setup process. The second file is an actual image unattend file. For Windows Vista or later, this is an unattend.xml file. For earlier Windows images, it is a sysprep.inf file that is stored in a $OEM$ directory structure or under \Unattend in each image's directory. This file is used to automate the actual installation of Windows once WDS has completed.
To create the WDS Unattend file, create an unattend.xml file for WDS using the WAIK. Copy this file to the RemoteInstall directory or a subdirectory. Right-click the install image for which you would like to use the unattend file and click Properties. Under the Client tab, select Enable unattended installation and provide the unattend.xml file specified earlier.
To create the Image unattend file for pre-Windows Vista images, copy the sysprep.inf file to the $OEM$ directory for the image—for example, D:\RemoteInstall\Images\Windows XP\SP2\$OEM$\$1\sysprep\sysprep.inf. For Windows Vista images, specify the image you want to associate, click Properties, and then under the General tab, click Allow the image to install in unattend mode. Then specify the unattend.xml file that is to be used for installation.
We're now halfway there. This column should get you up and running with WDS. Next month, I'll drill into more detail about WDSUtil, Image Store specifics, and multicast. Then I'll wrap up the series with an in-depth look at how and why you might consider building a custom deployment solution on top of WDS. Until then, take a look at the "More on WDS" sidebar that accompanies this column.

Wes Miller is a Senior Technical Product Manager at CoreTrace (www.CoreTrace.com) in Austin, Texas. Previously, he worked at Winternals Software and as a Program Manager at Microsoft. Wes can be reached at technet@getwired.com.


Security Watch Passwords and Credit Cards, Part 2
Jesper M. Johansson

Contents
Pseudo Multifactor Logon Process
The Problem with Browser Add-Ins
The Authenticator Must Never Change
Downgrading to Less Secure Passwords
Bypassing the Pseudo Multifactor Logon
Problems with Compromised Passwords
Some Benefits
Misleading Users with Eye Candy
To Open or Not to Open
Providing Secure Communication
It's About Privacy
Welcome to the second installment of my three-part series on how the IT industry confuses consumers rather than helps them when it comes to security issues. In the July 2008 issue of TechNet Magazine, I discussed inactionable and incorrect security advice as well as confusing logon workflows. (If you haven’t read Part 1 yet, you can find it
at technet.microsoft.com/magazine/cc626076.). I made an argument for how extremely common yet poor security advice and bad logon workflows confuse consumers and undermine their efforts at protecting their personal information.
In this installment, I continue that discussion with more real examples taken from the world of consumer security. The final installment of this series, which will include a "call to arms" for the security professionals of the world, will appear in the next issue of TechNet Magazine.

Pseudo Multifactor Logon Process
In October 2005, the Federal Financial Institutions Examination Council (FFIEC) published its "Authentication in an Internet Banking Environment" guidelines (see www.ffiec.gov/press/pr101205.htm). The timeframe for implementation was a mere 14 months and U.S. financial institutions were soon scrambling to figure out how to meet these new guidelines.
Most institutions failed to meet this timeframe. Many of those institutions that did eventually meet the guidelines met them with measures that serve no other purpose than to meet the guidelines. In other words, the institutions took measures that do not in any way make customers any more secure (they are merely practices in "security theater"). Some of the most interesting examples of unhelpful solutions are the ones that try to create multifactor authentication without actually being multifactor.
For instance, consider the technology that measures typing cadence when a user types the password. Used on a Web site, this solution presents a logon dialog that looks exactly the same as the old logon dialog. This dialog, however, is now an Adobe Flash object.
The Flash object records the user's typing cadence, including characteristics such as how long keys are pressed and how much time there is between key presses. This data is submitted to the Web site along with the password, where it is compared to stored values. As long as the typing cadence data falls within certain variances of the stored data, and the password matches, the logon is accepted. The general idea is that this permits the site to use a pseudo-biometric authentication method, without having to install third-party hardware on the client.
I will refer to this as pseudo multifactor authentication. It is not true multifactor authentication, which measures two or more of the following canonical factors:
  • Something the user has (such as a smart card)
  • Something the user knows (such as a password)
  • Something the user is (such as a fingerprint)
Instead of actually including multiple factors in the authentication process, pseudo multifactor authentication reads multiple parameters off just one single factor—the password. It then uses those additional parameters to make inferences about something the user is.
The technology used for pseudo multifactor authentication suffers from many flaws. Taken together, these flaws render the technology largely ineffective at solving the security problems it purports to solve.

The Problem with Browser Add-Ins
Pseudo multifactor authentication systems rely on browser add-ins to provide the sophisticated client-side processing that they need. All software, of course, has bugs, and some of those bugs cause vulnerabilities. Those vulnerabilities get to be a real problem when the software is difficult to update and the user is unsure whether an update has even been installed.
Browser add-ins are guilty of both of these characteristics—they are difficult to update and don't provide the user with ample info about whether the installed version is up to date. As a result, the user must expose a piece of software to the Internet that might not otherwise be needed.
In some cases, keeping the add-in current will require regular visits to the add-in's Web site for updates. It goes without saying that this is unlikely to happen for most users. Any system that requires an end user to expose additional, unrelated software to the Internet simply to provide security functionality should be taken under significant consideration.

The Authenticator Must Never Change
The "something you are" factor has a trait, perhaps even a shortcoming, that doesn't affect the other two factors. While you can change passwords (something you know) and manufacture new security tokens (something you have), the number of biometric authenticators you can use is fairly limited. With fingerprints, for instance, you get ten, typically for life. And should you lose one of those digits in an accident, you typically cannot replace it.
Now consider how this applies to the pseudo multifactor logon example. The metrics collected as part of another factor cannot be easily replaced or modified. Certainly, when the password changes, the way the user types it will change, but the general way he presses the keys on the keyboard will not change. This is exactly what the technique relies upon. Should these metrics be compromised, it could be possible to synthesize them. At the very least, an attacker can capture these metrics and replay them.

Downgrading to Less Secure Passwords
It is a good idea to use long passwords and to change your passwords often. And, as I discussed in Part 1 of this series, it is also a good practice (contrary to some advice) to record your passwords—in a secure manner, of course. However, these practices are not conducive to typing your password by hand. A long password is more difficult to type, and a recorded password is far more useful when you can copy and paste it. One of the best ways to keep track of a hundred or more passwords is to use a software utility such as Password Safe (at sourceforge.net/projects/passwordsafe). With a tool like this, you can generate fully random passwords, store them in encrypted form, and paste them right into logon dialogs. In fact, with that type of tool, you don't even need to know what your password is.
But here's the rub: you cannot measure typing cadence unless the user types the password. And if you have to type the password into an object that measures typing cadence, this technique for managing passwords begins to break down. Typing a 15-character fully random password correctly is no simple task. And so users will generally opt to simplify their passwords when faced with this type of system. Yet, a 15-character fully random password on its own is far more secure than a shorter password coupled with a pseudo multifactor authentication system.
In effect, pseudo multifactor authentication, implemented exclusively, forces users to use less secure passwords. Unfortunately, as you will see shortly, accommodating users that use proper password management techniques obviates virtually any value offered by the pseudo multifactor authentication systems.

Bypassing the Pseudo Multifactor Logon
Even if a site does implement pseudo multifactor authentication, it must still support a way to bypass the pseudo multifactor system. First, few sites can require all its users to install a "fourth-party" technology to be able to access the site. Some portion of users (certain TechNet Magazine authors, for instance) will always refuse to install this sort of software.
Second, typing cadence can change for many reasons and therefore the sites must accommodate that. For example, say a user sprains her wrist and her typing cadence is temporarily altered. How will she then access the site if the site analyzes her cadence and thinks a different person is trying to log in with the user's credentials? If the injury were permanent, the user could reset the stored cadence value. However, in the case of a temporary injury, the user probably will not want to reset the stored value, assuming her typing cadence will soon return to approximately what it originally was.
Finally, not all users are even capable of using pseudo multifactor authentication systems. For example, a disabled person who interacts with the computer through a speech recognition interface may not be able to fill in the dialog, depending on whether it prevents programmatic entry or not. In that case, an alternate system that accommodates disabled users will likely be required by law.
The simplest and most straight-forward way to accommodate all these scenarios is to also support standard password-based authentication.

Problems with Compromised Passwords
Pseudo multifactor authentication systems are purported to address various problems associated with password-based authentication. However, these systems fail to fully address every significant way passwords can be compromised. There are five primary ways by which systems that rely on password-based authentication can be compromised, where "compromised" means an attacker has obtained and used another user's password:
  • Password guessing
  • Keystroke loggers
  • Phishing attacks
  • Asking the user
  • Breaking into any system that stores the password or a hash of it
Password guessing is not a particularly common attack method for criminals any longer, and it has become greatly overshadowed by keystroke loggers and phishing. Password guessing is also only partially mitigated by pseudo multifactor authentication. A conventional password guessing approach is unlikely to work against a pseudo multifactor authentication system since the attacker must guess the password as well as the typing cadence. While it may be theoretically possible to synthesize the typing metrics, it is rarely necessary to do so because the actual data can usually be captured by a phishing attack or a keystroke logger. In addition, if the system also provides a standard password-only logon interface, the attacker can just use that system for password guessing.
In addition, password guessing attacks are best defeated by strong passwords. If we can teach users to use stronger passwords, possibly with the aid of tools, pseudo multifactor authentication is not needed. (In contrast, the simpler passwords that are typically used with pseudo multifactor authentication systems actually make password guessing a more viable approach.) Thus, any value that may be offered here to mitigate password-guessing is only really adding value if the implementation does not support password-only logons. And, as I pointed out, this is rarely if ever possible. In other words, if users use weaker passwords in the presence of pseudo multifactor authentication systems, password guessing may become a viable attack method again; pseudo multifactor authentication would reduce security, rather than increase it. More research is needed into this question.
Pseudo multifactor authentication also fails to address the problem of keystroke loggers. While I am not aware of any keystroke loggers in common use today that capture typing cadence, there is absolutely no reason why one could not be designed to do this. A keystroke logger is a piece of hardware that sits between the keyboard and the computer or a piece of software that captures keystrokes. In either case, the logger has full access to all the data used by the pseudo multifactor authentication solution.
In fact, the keystroke logger can access even more data since an authentication solution runs in user mode while a keystroke logger sits far beneath that. Without a trusted path between the keyboard and the Web-based object used to capture the password, it is not possible to prevent this type of compromise. If pseudo multifactor authentication solutions become common enough, you can be sure someone will create such a keystroke logger.
Similarly, phishing attacks are not defeated by pseudo multifactor authentication. Instead of using just a fake logon screen, the attacker can use a Flash object to capture the password as well as the typing cadence.
Granted, pseudo multifactor authentication can help in some scenarios where the attacker's technique will only provide the user's password. For example, the easiest way to get a password from a user is to ask the user. Shockingly, this has proven to be an effective technique, whether asking people in person, over the phone, or through phishing messages.
Of course, asking a user for his typing cadence will be far less effective. Likewise, if a corporate password database is compromised, the attacker will likely only gain access to the passwords. But again, these attacks will only be mitigated if the system provides no standard password-only authentication.
I should also point out that if the password database itself has in fact been compromised, the attacker has probably compromised the system in a much deeper way than would be possible with a single, or even many, regular user account passwords. Thus, it is not really valid to attempt to mitigate what an attacker in possession of a password database can do.

Some Benefits
In all fairness, there are a couple problems that pseudo multifactor authentication can help address (assuming that the system provides no standard password-only logon option). For example, it can be used to prevent password sharing. However, this can be a drawback for systems where it is legitimate for multiple users to share accounts, such as joint bank accounts.
In addition, a properly constructed interactive login dialog (not a Web site login) could force a user to go through additional authentication steps if her typing cadence failed to match. This can provide additional security against a compromised account in a highly sensitive environment.

Misleading Users with Eye Candy
One of the best ways to confuse users is to give inaccurate indications of security. The most common is probably the padlock image displayed on a Web page, as shown in Figure 1. This page even goes as far as to display the word "Secure" by the padlock.
Figure 1 An example of padlock icon abuse, a worrisome trend
As you surely know, simply putting a padlock image and the word "Secure" on a page does not make it secure. Yet, the practice is disturbingly common, even among the most reputable, and most targeted, Web sites. The result is that many users are trained to look for these visual safety cues in the body of the Web page instead of looking where these cues actually mean something: in the address bar. (The W3C Web Security Context Wiki has an entry on this problem, available at w3.org/2006/WSC/wiki/PadlockIconMisuse.)
It's unfortunate that there are still so many examples of this misuse. Research has shown that users are unable to identify malicious Web sites even when the certificates are very obvious (see www.usablesecurity.org/papers/jackson.pdf). It comes down to the ability to easily tell fake from real, even when you don't have the real to compare with. This takes skill; misleading security eye candy on Web pages hinders development of that skill as users are drawn to the wrong information.
A particularly disturbing variant of this problem is shown in Figure 2. In this case, the page that displays the information is actually not secure. If you look at the address bar, you see the "http" indicating the protocol. This site uses a very common optimization technique—rather than encrypting the page that contains the logon form, only the form submission is encrypted. The login is secure, just like the page states, as long as you equate "secure" with "encrypted." However—and this is critically important—the user has no way to verify where the credentials are being sent before they are sent! The site does not show a certificate authenticating the site to the user before the form is submitted. It's a game of trust, like falling backwards and assuming the person behind you will catch you. By the time the form is submitted the damage may already be done.
Figure 2 Security eye candy on an insecure page
Secure Sockets Layer (SSL), the protocol that provides the security in HTTPS, serves two important purposes. First, it authenticates the server to the user. Second, it provides an easy mechanism to negotiate a session encryption key that can be used between client and server.
When only the actual form submission is encrypted, the first and most important objective is not achieved. Sites employing this optimization are just using SSL as a way to negotiate keys. Ironically, they could do this simply by employing a standard key negotiation protocol, thereby avoiding the cost and overhead of SSL.
The site shown in Figure 2 is not a rarity. Many sites provide SSL protection for just the form submission, but not the form itself. This particular site, however, demonstrates an even more disturbing trait. If you type https://www.<site>.com (note the secure https indicator) into the browser address bar, the site will redirect you to the non-SSL version of the site! Even if you try to inspect a certificate before sending your credentials to the site, the site refuses to show you the certificate.
Not all sites are that bad, but there are many that are. And two of the largest credit card issues in the United States are among the offenders. In fact, of the three major credit card companies I use, only American Express provides a certificate on the logon form. American Express even redirects HTTP requests to HTTPS. Well done!
One final thought regarding the meaningless security eye candy and the lack of certificates on the logon form. You may be wondering why a site would do this. The reason is simply for economics. Presenting certificates requires encrypting the page, and encrypting the page creates processing overhead. Processing overhead means that more computers are required to service the same load. And more computers cost more money. Unfortunately, when it comes to a choice between protecting customer privacy and increasing profits, many organizations will opt for the increased profits.

To Open or Not to Open
Recently I received an astonishing e-mail message from my health insurance company. Anyone who has used a computer in the past 10 years would know that they should never open unsolicited e-mail attachments. So imagine my surprise when I received the message shown in Figure 3.
Figure 3 E-mail message containing a “secure attachment”
Apparently I had asked a question on the health insurance company's Web site (and I had forgotten about it by the time the suspect message arrived). This is how the company responded. At first I thought it was a clever phishing scheme. As I realized this was a legitimate message, the hairs on the back of my neck started to stand up.
The very first direction is to double-click the attachment to begin decrypting the message. The security community and IT administrators at large have spent the past 10 years trying to teach people NOT to double-click on attachments. And then a company comes along (my health care provider, by the way, is not the only organization using this approach) and says that I must click on the attachment for my security. How should the user act in this situation? What behavior does he learn or unlearn?
Next, I used the preview feature in Microsoft® Office Outlook® 2007 to view it. As you can see in Figure 4, Outlook thought this message might be an attack and warned me not to open it!
Figure 4 Outlook 2007 considers the secure document hostile
It is both ironic and sad that a health insurance company would violate the very basic security checks used in the most popular e-mail client in the world. Considering this company did not even bother to test its new security solution with Outlook, I have to wonder what else this company is doing in the interest of protecting my private information. Or, put more bluntly, what other solutions are being implemented by the company solely to avoid being accused of not adequately protecting customer privacy? This is similar to the security theater performed by the financial institutions. In this case, I think accusation avoidance is the main objective of the solution—not actually protecting customers.
I wanted to see what the attachment really was. It turned out to be an ActiveX® control object. To see the attachment, I had to open it in Internet Explorer® and install the object. I was presented with the reassuring screen shown in Figure 5. As you can see, the designers went to great lengths to make the message look like a regular postal envelope; it even has a stamp that claims the envelope is trusted.
Figure 5 The Secure doc shows an envelope stamped “Trusted”
This type of technology is worrisome for many reasons. First, it reinforces a very bad behavior on the part of the user: opening unsolicited attachments. Second, in the actual message, it gives very bad guidance to configure the system to always open attachments without prompting. Third, in the face of conflicting messages from the computer, where the e-mail says it is trusted, and Outlook says it is not, the message looks very suspicious. Finally, the actual attachment contains meaningless security eye candy to convince the user that it is actually trusted. If the user learns to trust these kinds of messages, the step to trusting malicious messages with a similar look is minimal.

Providing Secure Communication
Admittedly, the issue that this technology is purporting to address is very important. Communicating with customers in a trusted fashion is difficult. However, this particular solution is over-engineered. It will likely cause customer confusion and potentially lead to the opposite end result it aimed to address: the customer's system becoming compromised.
Better-engineered Web sites now use a "message center." In this design, when the company needs to communicate with the customer, it sends an e-mail message that says something to the effect of: "You have a message in the message center. Please go to our Web site, log on, and click the Message Center link to view the message." A company gets bonus points if it uses Secure Multipurpose Internet Mail Extensions (S/MIME) to sign all customer-facing e-mail messages so the user can authenticate the source. There's another bonus if the message does not include any "please click this link" items in the e-mail. The user should type in the company's URL by hand to ensure she is going to the site she expects.
With a message center and a signed message, a company can achieve a trusted path to communicate with the customer. Everything the customer sees during the message workflow is authenticated, and the company does not promote poor security practices.

It's About Privacy
So far, I have spent the first two installments of this series describing how security professionals are actually doing a disservice to users. It is our job to maintain security. But many of the decisions being made and solutions being implemented are confusing users, teaching them to make bad decisions, and giving them a false sense of security. We should not be overwhelming users with these conflicting ideas.
As I pointed out previously, to mainstream users, security is simply about protecting passwords and credit card numbers. They want technology to work and be trustworthy. Unfortunately, they have to make decisions, and it is our job to make sure that they are informed decisions.
In the final installment of this series, I will discuss how some of the most important technologies available to consumers are not living up to expectations. I will also present my call to arms. So please check out the September 2008 Security Watch column.

Jesper M. Johansson is a Software Architect working on security software and is a contributing editor to TechNet Magazine. He holds a Ph.D. in Management Information Systems, has more than 20 years experience in security, and is a Microsoft Most Valuable Professional (MVP) in Enterprise Security. His latest book is the Windows Server 2008 Security Resource Kit.


Sustainable Computing Filtering the Greenwashing
Dave Ohara

Contents
The Need for Numbers
Measuring Energy Usage
Power Factor
Measuring in Production
The Microsoft EnterpriseEngineering Center
Wrapping Up
There are an overwhelming number of products and solutions being marketed as green, as saving energy, as being more efficient. All this marketing hype creates confusion in the market as to what is really eco-friendly. Even after evaluating the specifications on various products, it is difficult, if not impossible, for an IT pro to
determine what equipment should be used when environmental impact is a key concern. When you see all the demonstrations, large energy savings are always highlighted, and this leads you to think that the return on investment (ROI) makes the upgrade easier to justify. After all, the energy savings should reduce the total cost of ownership (TCO).
The move by companies of all types to label seemingly everything as environmental and exploit the current interest in green solutions has led to the concept of "greenwashing," which refers to the over-promising of environmental benefits. So what is the truth of energy savings? This isn't as clear cut as, say, installing new energy-efficient lightbulbs in your home.
As interest in sustainable IT efforts increases and the market for environmentally friendly IT equipment expands, many people and organizations jump to the end result of deploying energy-efficient laptops, desktops, and servers, and using virtualization to reduce energy consumption. Yet few organizations run energy audits to determine the true benefits of what they have purchased.
While the ideal scenario is to actively measure in your production environment, that can also be expensive. If you aren't ready to start measuring in production, you can still move forward by performing your energy audit earlier in the process, doing so in your performance and evaluation labs. (Most companies have some lab or group responsible for testing and evaluating equipment before making a purchase.) There you can add energy performance as one of your test criteria and then take those results into account when making purchasing decisions, rather than relying on the numbers provided by manufacturers. So as you test, create your own device power-consumption database. Then you can ignore the greenwashing and see for yourself what works.
Of course, I should point out that if you want highly accurate numbers for operation under your true load, you'll need to monitor in your production environment. The quicker you begin monitoring your production environment, the better for your organization and your bottom line. This process will be critical to your long-term success.

The Need for Numbers
In an ideal situation, there would be independent tests that evaluate the performance per watt of hardware devices under realistic user loads. But the industry is in the early stages of developing and rolling out tests like these. And those now available, such as the SPEC Power benchmark, are still in their infancy. How well these energy tests will work once they're ready remains to be seen. But as with any test, the manufacturers will learn how to modify their equipment for optimal test results.
Experienced IT pros have learned to question tests conducted by testing labs. And just like car companies make sure they send their best cars for road tests, IT vendors will be sure to send their most efficient equipment configurations for energy tests.
Consider virtualization tests being used to compare a group of over-provisioned servers where no server consolidation has been run to demonstrate the benefits of virtualization. Vendors neglect to mention issues such as potential rebound effects from energy savings projects like virtualization. Essentially, this refers to when energy efficiency decreases costs and demand increases in response, driving up energy use and increasing use﻿. When you see a solution, such as virtualization, marketed as a cure-all for energy savings, you should be suspicious. You really have to see how everything works together and what the ripple effects will be over time.
So how do you discover the right path for your organization? There are plenty of consultants ready to run assessments for a price. But there are some major downsides to this approach. Do you really want to pass along the expertise of being efficient to someone outside your organization? Do you want to rely on a consultant who wants to maintain a long-term contract and create a dependency on their service?
You could turn this into a huge project and get dozens of people involved to determine your environmental sustainability strategy. Avoid the temptation to just pick a spot in your environment and start measuring. While the sentiment is in the right place, the approach doesn't make sense, as this is just random action. You need to find the truth of where your energy consumption lies and where you can make an impact.

Measuring Energy Usage
You need to think in a new way about what is working. IT pros know when a piece of hardware isn't working if, say, a service goes down. But what about when a piece of hardware isn't working properly in terms of power consumption? This probably goes under the radar as most organizations have no data to establish their baseline energy efficiency.
How do you get these numbers? Some people suggest the need for an energy czar. That, however, requires someone who can navigate the company organization and is technical on power issues. A simpler, more approachable method is to add energy-measurement tools to existing functions in performance labs.
There are a range of energy-measuring devices. Unfortunately, at this time there is no perfect device for an IT performance lab. At the low end, you can go with a Watts Up Pro. This is a USB device that provides inline power measurement for 120V. Another device is the Smart-Watt, a networkable power consumption device with a 100-240 voltage and 15-30 amp range. Smart-Watt devices are also available with temperature and humidity sensors to measure environmental factors. Then there are the industrial power analyzing devices, such as the Extech Appliance Tester 380801 and Fluke 345 Power Clamp. You should have no problem finding devices like these to fulfill the needs of your lab tests.

Power Factor
If you're new to power measurement, it's important that you understand the power factor concept. The power factor of an AC electrical system is defined as the ratio of the real power to the apparent power, and is provided in a number between 0 and 1. Real power (watts) is the capacity for the circuit for performing work. Apparent power (VA) is the product of the current and voltage on the circuit.
You may be wondering why this is important. Take a look at Figure 1. The photograph shows an energy-monitoring device from Smart Works being used to compare the energy efficiency of a laptop, a lightbulb, and a capacitor. A lightbulb has a 50-watt load and 50 VA, for a power factor of 1.0. A capacitor with a pure capacitive load draws a 2-watt load and 193 VA, for a power factor of 0.01. The laptop is using 22 watts of power but has a 48 VA reading, for a result of a .47 power factor. The laptop's low power factor is due to an inefficient power supply design and was likely the result of manufacturing cost targets.
Figure 1 Power consumption for a laptop, a lightbulb, and a capacitor
With so much interest in energy efficiency, vendors are now looking to improve the performance of power supplies under expected conditions. If you compare the power factor of existing equipment to that of new equipment, you may find you can gain back some power capacity simply by choosing solutions with more efficient power supplies and higher power factors.

Measuring in Production
The goal of measuring results in the performance lab is to predict performance in the production environment. The problem with measuring energy consumption in production is the cost required to measure all devices. One strategy, however, to reduce the cost of measuring power in production is to measure at the power distribution unit (PDU) and aggregate multiple equipment readings. Figure 2 shows a sample energy reading taken from a PDU with multiple pieces of the same equipment on one unit. Because average power consumption per server can be calculated, this means of measurement provides you with production power-consumption numbers.
Figure 2 Measuring at the power distribution unit
As each piece of equipment is measured, the power information should be integrated into your configuration management database. If this is too difficult, you can create your own database or at least an Excel® spreadsheet that lists devices and their power consumption. As you start to accumulate more devices, you can fill in estimates for others to create an overall power-capacity calculation. Keep in mind that a new energy-efficient server will consume less than half its power load while idle versus at peak. And if you have old servers running obsolete solutions, the old servers will consume significant amounts of power even if they're idle. This is one of the easiest targets for regaining power capacity.
If you want to continue your improvement efforts, the next step is to calculate the power per rack being used in your datacenter. You must also be sure to understand your power and cooling capacities as you alter the environment. Over time, you will start to view your equipment in terms of its power needs and not just the space it requires. Space is an easy concept—it's static and visual. IT pros can readily discuss 1U, 2U, and 4U servers, but start talking about 200-watt servers versus 450-watt servers and you'll find the same IT professionals are not as comfortable with the conversation. This is a new language for IT equipment, and it's evident in today's datacenters. Many have plenty of space for more equipment even though they have reached their max power capacities.

The Microsoft EnterpriseEngineering Center
The Microsoft Enterprise Engineering Center (EEC) recently added power-measurement capabilities to its facility. Located on the Microsoft corporate campus in Redmond, Wash., the EEC (microsoft.com/windowsserver/evaluation/eec/default.mspx) is a state-of-the-art proving ground for the most complex computing environments.
With more than $40 million in hardware and networking equipment (see Figure 3), the EEC can tackle the most complex recreations of real enterprise production environments. The group partners with many leading networking, storage, and client/server solution providers to offer a mix of cutting-edge technologies and legacy platforms. The goal is to ensure each test provides an accurate reflection of the customer's current environment. The latest service being developed in the facility is reporting on energy consumption for solutions to provide a benchmark of performance per watt.
Figure 3 One of many rows in the EEC lab
Over the past few years the EEC has seen many datacenter operators run out of power and cooling for their equipment. State-of-the-art equipment often comes with higher power densities, which put more stress on the facilities. The EEC staff has learned firsthand the costs and time involved in upgrading the power and cooling infrastructure. So to operate more efficiently and be more cost-effective, the EEC has added the ability to measure power per device.
Some of the tools and equipment being used by the EEC were not shipping when this article was written. Because this is a new solution, the EEC will continue to develop better techniques, working with customers, vendors, and Microsoft development teams. The EEC shares the methods it has been developing thus far directly with customers for early feedback, allowing the customers to create their own benchmarks.
And don't assume the results will be the obvious. The EEC has shared some of the interesting results the group has seen when using energy-monitoring features in the EEC performance labs:
  1. 1.Turning off a device doesn't necessarily reduce energy consumption as much as you might expect (see Figure 4). In one case involving server hardware, the EEC discovered a device that actually consumed 100 watts when turned off but still plugged in. This surprised many, and the EEC went over the setup many times. They eventually used an infrared thermometer to measure inlet and outlet temperature and verified that the device did, in fact, consume 100 watts while off.
  2. Software can have a significant impact on power consumption. On identical networking switches, with identical hardware and BIOS configurations, running different networking software displayed a 21 percent difference in power consumption. High-end solutions with more processes and features enabled, like security and monitoring tools, often consume more than their simpler low-end counterparts.
  3. In virtualization scenarios, the EEC has measured power consumption versus I/O utilization and CPU utilization to determine when a given piece of hardware maximizes its performance per watt. The EEC found that a narrow focus on CPU utilization could lead to too many virtual machines loaded on a physical machine, actually decreasing the overall performance per watt.
  4. Higher-density devices, as you might expect, have more power and cooling issues. When deploying higher- density systems, your power and cooling facilities staff should be consulted as early as possible. These devices may be good candidates for their own power-monitoring devices in production if you know the environment will be power constrained.
  5. Dual power supplies can consume considerably more power than a single power supply.
  6. Seemingly identical pieces of hardware with identical configurations can have significantly different power consumption. Observed differences were significant enough to make the EEC staff double-check hardware to ensure they were really configured the same.
  7. The watt ratings on the product plate are not actual consumption numbers, but rated capacity for power supplies.
  8. Maintaining a database of energy consumption tests and results per device and subcomponent is essential for retaining knowledge and comparing data.
  9. Different configurations of equivalent amounts of RAM consume different amounts of energy. Fewer DIMMs typically consume less energy—for example, 4 x 2GB DIMMs versus 8 x 1GB DIMMs. But there have been some cases where fewer DIMMs consumed more energy.
Figure 4 Power-consumption comparison of on versus off

Wrapping Up
By adding the capability to measure power consumption in your performance labs, you can start to accumulate your own database of energy consumption per device with accurate numbers for your real loads. When you deploy those solutions, you should audit the results in a closed-loop feedback to determine the accuracy of your performance lab tests. As the EEC learned by running its own measurement tests, there are some really interesting details you can look for in order to help you filter out the greenwashing and discover the truth.
This method of measuring in performance labs will influence the market overall to begin identifying what truly is most energy efficient. And over time manufacturers will find it necessary to provide more accurate data about their energy-savings claims. Likewise, in time IT professionals will talk about watts per device as commonplace purchasing criteria. This needs to be a common practice, especially for companies that buy large quantities of servers. However, you can lead the charge by starting today to think about power as one of your valuable IT resources.
Keeping Cool in the Datacenter
Datacenter cooling offers a huge potential for reducing energy consumption. It is astounding how much heat can be generated in a datacenter and how much energy is used to keep hardware cooled. But if you want to manage your cooling successfully, fix problems, and develop more efficient cooling solutions, you'll need a temperature-monitoring solution. Consider the solution the Microsoft datacenters use.
Microsoft Research built a temperature sensor network for the datacenters that allows for improved temperature control and also enables evaluation of various cooling improvements. For instance, one Microsoft datacenter was evaluating end-of-aisle air curtains to improve hot and cold air separation. After the curtains were installed, some servers started to send overheat alarms. Naturally, the operation engineers increased the air flow from the cooling system to provide more cool air. To their surprise, however, more servers sent overheat alarms. And all these servers were at the bottom of the rack—the bottom, of course, is usually the coolest area from a raised-floor cooling system.
Using the sensor network, the engineers confirmed that the racks were cooler up higher, with the bottom of the rack the hottest. And they soon figured out that hot air was being drawn from the hot aisle between the bottom of the rack and the flooring—a result of Bernoulli's principle. They easily fixed the overheating by sealing the bottom of the rack and reducing the air flow speed.
This is just the sort of data the Microsoft Enterprise Engineering Center gathers and analyzes when doing performance testing. So the EEC recently notified Microsoft Research that they were ready for a deployment test. Within a day the system was deployed to 10 racks, and the installation took just one hour to complete. The EEC now is able to study and better understand cooling issues and their relationship to hardware performance.
Of course, simply monitoring isn't a solution in itself. The real gain is in your ability to find problem areas that you can fix, make changes, and evaluate various solutions to see if they have the result you are expecting. After all, you don't want to be caught off guard when your new cooling solution unexpectedly causes your racks to overheat.

Dave Ohara has 26 years of experience in technology. He is now working with multiple companies implementing green initiatives.


Field Notes IMAP Saves the World
R'ykandar Korra'ti

Sometimes solutions to new, small problems can solve old, large problems as well. This is one of those cases.
Last winter, our Web administrator bought a MacBook. She had been running Windows® exclusively, but now she'd have both OS X and, in a virtual machine, Windows XP for syncing her Windows Mobile® devices and a few other small tasks.
She needed access to all her mail archives from both operating systems. But she didn't want to run Entourage on the Mac side—that would have been too easy. No, she wanted to run Mail.app, the e-mail client included with OS X. Incidentally, she'd kept everything in her laptop's local store, unmirrored. My job: make this go.
Having archives be readable everywhere presented only the small problem mentioned previously because—and this is the first moral of our story—everything, including Exchange Server, Microsoft® Office Outlook®, and Mail, speaks IMAP4. IMAP4 is a messaging protocol for handling on-server message stores from thin clients. It has fewer features than Exchange users might enjoy, but it works (see How IT Works: IMAP4 in the March-April 2006 issue of TechNet Magazine at technet.microsoft.com/magazine/cc161016).
So, change a few settings to bring IMAP4 over SSL online, copy all of her local folders up onto the server, and ping! She could access mail on all clients and keep her Windows Mobile devices updated. Problem solved.
But coming up with a simple solution to a potentially hairy problem is like getting a new table saw. Suddenly you start looking for more plywood, and before you know it, you've got stage sets built and you're ready to play "Romeo and Juliet."
Our tiny ISP runs Windows, OS X, Mac OS, Linux, and even has one lonely old AmigaOS system hanging around on the fringes looking for a friend. Long ago, we hosted Usenet news and UUCP mail on that Amiga. It created archives stored away in RFC-733/AmigaUUCP format. We also have Microsoft Mail archives (.mmf format) and some similar-vintage UNIX mail archives (.mbox format). They've all just been sitting there, mocking us—a problem we'd consider every so often and then flee, screaming.
This scenario brings us to the second moral of our story, the Danger Hacker motto of lore: Never Erase What You Can't Replace.
We tackled the Microsoft Mail archives first. Outlook hasn't been able to import .mmf files in some time. But as we've upgraded clients over the years, we've kept at least one license of each older product because you just never know. As a result, our license rack forms a timeline of the Microsoft Corporation. Need MS-DOS®? OS/2 1.31? Microsoft Mail? Office 95? Sure! Would you like that in English or in French?
Outlook 2000 imported .mmf, but its IMAP support was strange and occasionally dropped header data. Also, it couldn't talk to IMAP4 and Exchange servers at the same time. This was vexing, but not a showstopper. Import the .mmf files via Outlook 2000, upgrade to our retained license of Outlook 2003, and a few hours of drag and drop later: 15-year-old archives online and viewable anywhere. Similarly for .mbox, throw an mbox-compatible IMAP server onto a Linux machine and put the .mbox-format archives in its working directory. Then use the Outlook IMAP support to connect and copy folders over to the production server of choice.
The Amiga archives, on the other hand, required writing a small bundle of C code to convert the RFC-733 headers into RFC-822 format, ditch the old SysV-style message dividers, and retrieve the strange From: line data from the special hell in which it lived. This is the price for letting problems sit too long. But once done, AmigaUUCP format became .mbox format. And thrown into the same working directory, the Outlook IMAP4 brought the mail right down, good as new. It even packaged up the attachments very nicely for us.
We now have archives back to 1990 online in one unified format, and we're looking around for more archives ... just because we can. Is that healthy?

R'ykandar Korra'ti, postmaster for a small co-op ISP, lives near Seattle with her partner, Anna. Having previously shipped mail products at Microsoft, she's writing technical articles when not working on deeply obscure mandolin song lyrics. Booking agents can reach her at darako@murkworks.net.


Windows Confidential Building on the Past
Raymond Chen

A colleague of mine rescued an Alpha AXP machine from a dusty closet. Upon booting up the system, he discovered that it was running a 64-bit version of Windows® from early 2000. How is that possible?"
I happened to have one of these machines in my office then. In its prime, this machine was a force to be reckoned with. It was about the size of a small refrigerator and generated about as much noise as a vacuum cleaner. It contained four—count 'em, four!—Alpha AXP processors, each running at a mind-boggling 400 MHz. It had one gigabyte of RAM and thirteen gigabytes of hard drive space, striped over a dozen fast SCSI drives. This may sound puny today, but back in the 1990s, these Alpha AXP machines were the envy of the block and made you the popular kid at the lunch table.
In 1999, when Compaq announced it would no longer support Windows on the Alpha AXP, a lot of Alpha AXP systems were suddenly left sitting around Microsoft with no official duties. Some of these machines, however, were pressed into a variety of unofficial duties. I used my Alpha AXP system to index the entire Windows source code. You can imagine how convenient it can be for a programmer to be able to identify all callers of a function in just a few seconds or to locate the source code for a function or dialog box that shows up in a debug trace.
But even for the simple task of indexing the Windows source code, the Alpha AXP was soon overshadowed by x86-class machines. These were cheaper and faster, they offered more hard disk space, and they had more memory. So for this reason, my machine soon joined its forgotten brethren. It was assumed that the end of support for the Alpha AXP was the end of these machines, but then they were given an opportunity to go out in a blaze of glory, breathing their last breaths in service of the next generation.
The 64-bit Windows project was already well under way, and of the 64-bit processors under consideration, only the Alpha AXP was actually available in physical form. The Intel Itanium was still under development and ran only in a simulator, and the AMD64 architecture hadn't even been invented yet. As a result, 64-bit Windows was initially developed on the Alpha AXP.
When Compaq announced that they would no longer support Windows on the Alpha AXP, all these Alpha AXP machines that previously had been used for 32-bit Windows 2000 development and testing got repurposed and began serving a secret life as test machines for a 64-bit operating system that will never ship in that form. The Alpha AXP was merely a proof-of-concept platform.
The Alpha AXP machines served that role well, giving the 64-bit Windows team real hardware to work with instead of having to run the operating system on an Itanium simulator. (You can imagine how slow that was.) Sure, the Alpha AXP wasn't the final target hardware, but it was a big help. And when physical Itanium CPUs began showing up, the niche filled by the Alpha AXPs vanished, and they were once again retired into dusty closets.
My friends who work at the Microsoft Archives called my old, trusty Alpha AXP out of its dusty closet for this photo shoot, posing with one of the products it helped make possible: the 64-bit edition of Microsoft Windows Server® 2003. And yes indeed, that's the original dust.

Raymond Chen's Web site, The Old New Thing, and identically titled book (Addison-Wesley, 2007) deal with Windows history and Win32 programming. He would prefer not to be locked away in a dusty closet.