 MSDN Magazine October 2004

MSDN Magazine October 2004

Bootstrapper: Use the Visual Studio 2005 Bootstrapper to Kick-Start Your Installation

There are plenty of times when you need to get information on running processes, not the least of which is during performance tuning. Using the techniques in this article and special .NET classes you'll see how to get a process' ID, name, priority, number of threads, kernel handle, and memory consumption, as well as its user-mode, kernel-mode, and total elapsed running time and put them to use in a custom app called AssemblyBrowser.

Sean Draine .NET Internals: Examine Running Processes Using Both Managed and Unmanaged Code

There are plenty of times when you need to get information on running processes, not the least of which is during performance tuning. Using the techniques in this article and special .NET classes you'll see how to get a process' ID, name, priority, number of threads, kernel handle, and memory consumption, as well as its user-mode, kernel-mode, and total elapsed running time and put them to use in a custom app called AssemblyBrowser.

Christophe Nasarre ASP.NET 2.0: Speed Up Your Site with the Improved View State in ASP.NET 2.0

View state is a wonderful thing. It allows the ASP.NET developer to maintain state for server-side controls that are not form elements.Used judiciously, it can improve the user experience. But in the wrong hands, it can cause your pages to grind to a halt. The release of ASP.NET 2.0 will include a variety of improvements to view state that will make it easier to use and less likely to slow performance.

Fritz Onion Advanced Serialization: Format Your Way to Success with the .NET Framework Versions 1.1 and 2.0

While .NET offers adequate support for the simple serialization cases, issues arise when you're attempting to support delegates and subscribers, versioning, and class hierarchies. . In addition, the introduction of generics in The .NET Framework breaks new ground in the complexity of the serialization task and the power of serialization tools. This article discusses these issues and related pitfalls, suggests techniques and workarounds that address them, and recommends when and how to best use the new serialization abilities.

Juval Lowy BizTalk 2004: Automate Your Business Processes with Human Workflow Services and Visual Studio

BizTalk Server is an excellent platform for automating B2B and EAI workflows. Process-driven workflows send messages between machines and access data. Until now, such processes were very difficult to automate, but with the introduction of Human Workflow Services in BizTalk Server 2004, this class of workflow automation is a very real possibility. This article explains it all.

Chris Whytock Unit Testing: Mock Objects to the Rescue! Test Your .NET Code with NMock

The problem: unit testing libraries, especially data access components, that have a complex set of dependencies. The solution: providing a dynamic mock implementation of your data access classes. Here the author explains just how to make testing easier and more reliable using NMock.

Mark Seemann Editor's Note: Old, New, Borrowed, Blue

A traditional marriage "rule" is that the wedding should feature something old, something new, something borrowed, and something blue. The reasoning behind this escapes us. Why blue? Why have so many successful marriages (Britney Spears comes to mind) ignored this rule entirely? Can you borrow someone's new, blue thing? How long will this analogy go? In any event, this month we have news from all four of these categories.

New Stuff: Resources for Your Developer Toolbox

MKS Inc. , a provider of tools for porting Unix/Linux applications to Windows®, has just announced the availability of its new 64-bit products. MKS is shipping a beta version of MKS Toolkit for Enterprise Developers 64-bit edition with support for extended-architecture 64-bit platforms such as AMD64 running Windows 64-bit OS.

Nancy Michell Web Q&A: Refreshing Web Pages, Spyware, Group Policy, and More

Edited by Nancy Michell Data Points: Handling Data Concurrency Using ADO.NET, Part 2

Enterprise development has been moving towards a discon-nected model in recent years and ADO. NET development is no exception. While the disconnected model of the ADO. NET DataSet offers great flexibility, that adaptability also means looser control over data updates than you get with a connected data access model.

John Papa Advanced Basics: Building a Progress Bar that Doesn't Progress

In many situations, accurately estimating the length of a certain process (copying a large file, loading data from a server, retrieving files from the Internet) would be both difficult and inefficient.

Duncan Mackenzie .NET Column: P/Invoke Revisited

In the July 2003 installment of the . NET column I covered the basics of Win32® interoperation with the Microsoft® . NET Framework (P/Invoke). Based on reader feedback, this topic is worthy of further coverage, so I have decided to revisit P/Invoke in this column.

Jason Clark Cutting Edge: Binary Serialization of DataSets

The ADO. NET DataSet object plays an essential role in most of today's distributed, multitiered applications. Instances of the DataSet class are used to move data across the tiers and to exchange data with external services.

Dino Esposito Service Station: Securing Web Services with WSE 2.0

Beginning this month, The XML Files will run under the name Service Station. We have made this change so that the column can discuss broader topics such as Web services, service-oriented architecture, and the like.

Aaron Skonnard Security Briefs: Password Minder Internals

In my last column I introduced Password Minder, the tool I use to manage all of my passwords. It generates a long, random password for each site I visit, and makes it possible for me to use the most complex passwords possible, without ever having to see the actual password material or type it in manually.

Keith Brown .NET Matters: ThreadPoolWait and HandleLeakTracker

Stephen Toub C++ Q&A: Create Client Windows, Drag and Drop Between Listboxes

Paul DiLascia { End Bracket }: Microsoft Java Virtual Machine Update

In a settlement agreement reached in January 2001 to resolve a dispute over the Microsoft distribution of its Java implementation, Sun Microsystems and Microsoft agreed to limit the duration of the time Microsoft would use Sun's source code and compatibility test suites to support the Microsoft® Java Virtual Machine (MSJVM).

Brian Keller


Bootstrapper: Use the Visual Studio 2005 Bootstrapper to Kick-Start Your Installation

New information has been added to this article since publication.
Refer to the Editor's Update below.

Bootstrapper

Use the Visual Studio 2005 Bootstrapper to Kick-Start Your Installation

Sean Draine
This article discusses:
  • Rich client application deployment
  • Easy dependency management
  • Deploying redistributable components
  • Deploying the .NET Framework
  • Bootstrapper extensibility
This article uses the following technologies:
Visual Studio, Visual Basic, .NET
Contents
An Integrated Installer for Applications and Prerequisite Components
The Bootstrapper User Experience
Using the Bootstrapper in Visual Studio
Configuring the Bootstrapper
A Closer Look at the Product Manifest
A Closer Look at the Package Manifest
Custom Redistributables in Visual Studio
Generating the Bootstrapper Setup.exe with MSBuild
Conclusion

I
f you have ever written an application intended to run on nonstandardized desktops (or, as some would say, PCs in the wild), you understand that one of the biggest challenges is installing the components required by your application onto the target computers. If any component is missing, your application will most likely fail, typically with a cryptic error message about a missing DLL file that will send your customers straight to your support desk.

As an example, consider a gaming application written in Visual Basic® 2005. The app won't run, of course, unless the .NET Framework 2.0 is installed. In order to present rich and speedy graphics, the application would likely depend on the managed DirectX® libraries. Consider also that the game may use a SQL Server™ 2005 Express Edition database to store game state and history.

To install the game, the user must install the .NET Framework 2.0, DirectX 9.0 for Managed Code, and SQL Server 2005 Express Edition before installing the actual application. This involves a number of complicated steps. First, the user has to check whether each component is already installed on the machine. Then they must manually download and run the installers for each missing component, selecting the appropriate choices in the installation UI. If any reboots are required, they must restart the machine and resume from where they left off. If the components need to be installed in a particular order, there are additional opportunities for things to go wrong. This is a poor user experience, to say the least, and many users would give up.

The Visual Studio® 2005 Bootstrapper solves this problem by allowing you to provide an easy, integrated way to install all of the different prerequisite pieces required by your application.

An Integrated Installer for Applications and Prerequisite Components

The Visual Studio 2005 Bootstrapper lets you provide users with a simple, automated way to detect, download, and install an application and its prerequisites. It serves as a single installer that integrates the separate installers for all the components making up an application, and provides a wealth of features and services.

Minimal initial download size The Bootstrapper is a small executable that weighs in at a mere 222 KB, and takes about one minute to download on a 28.8 Kbps modem. It therefore adds only a small amount of overhead to the installation process, which can save the user from having to download much larger redistributables if he does not need them.

Minimal platform requirements Since the purpose of the Bootstrapper is to easily pre-install application dependencies, it is essential that it have few, if any, dependencies itself. The only hard dependency the Bootstrapper has is the C runtime, which is embedded in the Bootstrapper executable through static linking and therefore does not require a separate installation. The Bootstrapper will load other Windows® DLLs such as msi.dll to perform Windows Installer application code checking, and Wintrust.dll to verify component signatures. These DLLs, however, come pre-installed with Windows 98 and higher.

Platform detection The Bootstrapper can detect whether a component is supported on the target computer's operating system. If the component will not run in the given environment, the Bootstrapper notifies the user and ends the installation before downloading the component. For example, if someone attempts to install the .NET Framework on a machine running Windows 95, the Bootstrapper will alert the user up front, before they have downloaded the 20MB .NET Framework redistributable.

Component detection and download as required A key design goal for the Bootstrapper is to download and install only those components not previously installed. This saves you from having to add deployment servers and subjecting users to lengthy, unnecessary downloads. The Visual Studio 2005 Bootstrapper offers flexible options for detecting whether a component is already installed. Detection can be based on a search through the registry, file system, or the Global Assembly Cache (GAC) (due in Beta 2). It can also be based on the results of a custom executable if more complicated detection logic is required. If a component is detected, the Bootstrapper does not download and install that component.

Handling reboots For some components, a reboot may be required to complete the installation. For example, installing Windows Installer 2.0 will cause a reboot on down-level Windows systems. The Bootstrapper allows graceful handling of reboots because components can specify whether the Bootstrapper should reboot immediately after the component is installed, for example if the installation of subsequent components would otherwise fail. Components can also instruct the Bootstrapper to defer reboots until all components have been installed. This way, the user only has to reboot once, even if more than one of the installed components requires a reboot. When a reboot is required, the Bootstrapper prompts the user to restart the computer and writes a callback to itself to the RunOnce key in the Windows registry. When the computer is restarted, Windows reinvokes the Bootstrapper, which resumes the installation from where it left off.

License agreements Most components include an End-User License Agreement (EULA) that serves to protect the component vendors' intellectual property. The Bootstrapper can be configured to present license agreements for individual components. It can also present a single shared license agreement for multiple components in order to minimize the number of pop-ups the user has to click through.

Install from Web, file shares, and local drives Applications today are distributed through a variety of media, including the Web, network file shares, local drives (CD-ROM, DVD), USB storage devices and Secure Digital (SD) memory cards. The Bootstrapper is designed to handle any of these scenarios. In addition, it can be configured to download prerequisite components from the same location as the app installer, or for purposes of load balancing, to obtain redistributables from a different server.

Extensible to custom redistributables The Bootstrapper is a generic installer that can be configured to bootstrap virtually any redistributable that is packaged as a Windows Installer file or as an executable program. It is programmed through a set of simple XML manifests that specify all the metadata required for the Bootstrapper to manage the installation of the component. The metadata includes the required set of files, platform and component dependencies, detection logic, command-line arguments, return code semantics, the EULA, and more. The Visual Studio build system then writes the metadata into the Bootstrapper executable, resulting in a single setup program that can bootstrap the desired set of prerequisite components for the application.

The Bootstrapper User Experience

The first step for the user is to launch the Visual Studio 2005 Bootstrapper executable, called setup.exe. Setup.exe can be launched from the local file system, a network file share, or a through a hyperlink to a Web address.

Setup.exe first detects whether any of the prerequisites are already installed. If any of the prerequisites are missing, the Bootstrapper shows an install dialog box that lists those components and displays their license agreement. Otherwise, if all of the prerequisites are detected, the Bootstrapper simply launches the application installer. Using the example of the game application described earlier, if the user has already installed DirectX 9.0 for Managed Code but not the .NET Framework 2.0 or SQL Server 2005 Express Edition, the bootstrapper dialog would display only the latter two components in the list of items to be installed.

Once the user accepts the agreement, the download and installation process begins. Otherwise, the installation is canceled and the Bootstrapper displays a progress bar to show how much of the process was completed.

When all of the prerequisites have been installed, the Bootstrapper kicks off the installation process of the application itself. Having launched the application installer, it gets out of the way and leaves it to the application installer to display any UI and perform the necessary installation operations for the application. The Bootstrapper does not dictate which installer technology the app must use; it can use Windows Installer, ClickOnce, or any other technology. In the case of installing a ClickOnce application, the user would see the standard ClickOnce installation dialog box, enabling him to install and run his app.

Using the Bootstrapper in Visual Studio

Visual Studio 2005 deployment tools for ClickOnce and Windows Installer include a UI that enables easy configuration of the Bootstrapper. Figure 1 shows the Prerequisites page of the application, which can be accessed from the property pages for Windows Application, Console Application, and Setup projects.

Figure 1 Visual Studio Prerequisite Dialog
Figure 1 Visual Studio Prerequisite Dialog

The Prerequisites page allows you to select which components your application requires. Visual Studio 2005 will include several frequently used components in the box, including the .NET Framework 2.0, J# Runtime 2.0, SQL Server 2005 Express Edition, Microsoft® Data Access Components (MDAC) 2.8, and Windows Installer 2.0. The Bootstrapper is also extensible, so you can add additional Microsoft and third-party components as needed.

Configuring the Bootstrapper

The Bootstrapper offers a couple of options for where redistributable files can be located. By default, the Bootstrapper assumes that all files, including the redistributables, the Bootstrapper setup.exe itself, and the actual application files, are located together under the same folder. This is ideal for applications that will be installed from local disk (CD-ROM, USB device) or a network share. This option also supports the application publisher who wants to serve up everything from a single deployment server. Alternatively, if the publisher wants to balance the load across different servers, or maintain a central location for prerequisite components that can be shared by multiple applications, the Bootstrapper can be configured to download any prerequisite files from a separate location.

In the case of Web deployment, the user downloads the Bootstrapper setup.exe and then runs it on a local computer, at which point it needs the URL to find the prerequisite files. There is no robust way for an executable to dynamically determine where it was downloaded from, so the origin location is written into the setup.exe file itself. This is important to keep in mind if you are changing deployment servers, because the Visual Studio 2005 Bootstrapper must be reconfigured to download from the new server. Fortunately, this can easily be done through the command-line options available on the Bootstrapper setup.exe. To reconfigure the deployment URL, go to the Windows command prompt and run the setup.exe as follows:

setup.exe –url=http://www.example.com
Here http://www.example.com represents the directory on the new deployment server for the application and its prerequisites.

One of the most powerful features of the Bootstrapper is extensibility. It's a generic installation engine with a flexible configuration system—you can plug into the Bootstrapper just about any installer packaged as a Microsoft Installer (MSI) or executable program. To extend it to a particular component, the developer must author a pair of XML manifests that contain the installation metadata for the component.

The first is the product manifest, which contains any language-neutral metadata for the package. The second is the package manifest, which contains metadata for handling language-specific aspects of the redistributable. A component will have a package manifest for each localized version of that component. The schemas for both of these files are almost identical. The choice of whether to include a given bit of metadata in the product or package manifest depends on how the redistributable has been factored for different languages. The product manifest contains metadata that is common to the different localized versions of the redistributable. The package manifest is used for language-specific metadata, and it typically contains localized error messages.

A Closer Look at the Product Manifest

Figure 2 shows the product manifest for the .NET Framework 2.0. The root tag for the manifest is the <Product> tag, which also specifies the name of the manifest schema. Visual Studio ships a schema file for the Bootstrapper manifests that provides IntelliSense® technology and validation when authoring manifests with the new XML editor.

Figure 2 Product Manifest for the .NET Framework 2.0 (Close Figure 2)
<?xml version="1.0" encoding="utf-8" ?>

<Product
  xmlns="http://schemas.microsoft.com/developer/2004/01/bootstrapper"
  ProductCode="Microsoft.Net.Framework.2.0">

  <!-- Defines list of files to be copied on build -->
  <PackageFiles>
    <PackageFile Name="dotnetfx.exe"/>
    <PackageFile Name="dotnetchk.exe"/>
  </PackageFiles>

  <InstallChecks>
    <ExternalCheck
       Property="DotNetInstalled" PackageFile="dotnetchk.exe" />
    <RegistryCheck Property="IEVersion"
       Key="HKLM\Software\Microsoft\Internet Explorer" Value="Version" />
  </InstallChecks>

  <!-- Defines how to invoke the setup for the .NET Framework redist -->
  <!-- TODO: EstimatedTempSpace, LogFile, updated EstimatedDiskSpace -->
  <Commands Reboot="Defer">
    <Command PackageFile="dotnetfx.exe"
             Arguments=' /q:a /c:"install /q /l"'
             EstimatedInstalledBytes="21000000"
             EstimatedInstallSeconds="300">

      <!-- These checks determine if the package is to be installed -->
      <InstallConditions>
        <!-- indicates the .Net Framework is already installed -->
        <BypassIf
          Property="DotNetInstalled" Compare="ValueNotEqualTo" Value="0"/>

        <!-- Block install if user does not have admin privileges -->
        <FailIf Property="AdminUser"
          Compare="ValueNotEqualTo" Value="1" String="AdminRequired"/>

        <!-- Block install on Windows 95 -->
        <FailIf Property="Version9X" Compare="VersionLessThan"
          Value="4.10" String="InvalidPlatformWin9x"/>

        <!-- Block install on Windows NT 4.0 or less -->
        <FailIf Property="VersionNT" Compare="VersionLessThan"
          Value="5.00" String="InvalidPlatformWinNT"/>

        <!-- Block install if Internet Explorer 5.01 or greater is not
             present -->
        <FailIf Property="IEVersion" Compare="ValueNotExists"
          String="InvalidPlatformIE" />
        <FailIf Property="IEVersion" Compare="VersionLessThan"
          Value="5.01" String="InvalidPlatformIE" />

        <!-- The package should be installed -->
        <InstallIf/>
      </InstallConditions>

      <ExitCodes>
        <ExitCode Value="0" Result="Success"/>
        <ExitCode Value="3010" Result="SuccessReboot"/>
        <ExitCode Value="4096" Result="Fail" String="ImproperUsage"/>
        <ExitCode Value="4097" Result="Fail" String="AdminRequired"/>
        <ExitCode Value="4098" Result="Fail"
          String="WindowsInstallerComponentFailure"/>
        <ExitCode Value="4099" Result="Fail"
          String="WindowsInstallerImproperInstall"/>
        <ExitCode Value="4100" Result="Fail" String="CreateMutexFailure"/>
        <ExitCode Value="4101" Result="Fail"
          String="AnotherInstanceRunning"/>
        <ExitCode Value="4102" Result="Fail"
          String="OpenDatabaseFailure"/>
        <ExitCode Value="4103" Result="Fail"
          String="ReadDatabaseFailure"/>
        <ExitCode Value="4111" Result="Fail"
          String="GetTempDirectoryFailure"/>
        <ExitCode Value="4113" Result="Fail" String="BetaNDPFailure"/>
        <ExitCode Value="4115" Result="Fail"
          String="TempDirectoryTooLong"/>
        <ExitCode Value="4116" Result="Fail"
          String="SourceDirectoryTooLong"/>
        <ExitCode Value="4118" Result="Fail" String="CannotWriteLogFile"/>
        <ExitCode Value="4119" Result="FailReboot"
          String="DarwinServiceHung"/>
        <ExitCode Value="4120" Result="Fail" String="InternalError"/>
        <DefaultExitCode Result="Fail" FormatMessageFromSystem="true"
          String="GeneralFailure" />
      </ExitCodes>

    </Command>
  </Commands>
</Product>

<ProductCode> The <ProductCode> tag specifies a unique ID for the package. This appears in the project file and is used by the build system to determine which components to include in the Bootstrapper. The recommended format for the product code is Publisher.ProductName.Version.

<PackageFiles> The <PackageFiles> section provides a list of all the files in the redistributable. The redistributable can be wrapped up into a single file (as is typical), or it can be a bunch of loose standalone files. The manifest for the .NET Framework specifies just two files. The first is dotnetfx.exe, which is the redistributable file for the Framework. The second is dotnetchk.exe, which is a program the Bootstrapper launches to detect whether the Framework is already installed on the target computer.

<InstallChecks> The <InstallChecks> tag contains all of the detection logic that the Bootstrapper should run prior to downloading and installing the component. The Bootstrapper is capable of a variety of checks that can include searching for files on disk, reading registry keys, searching for MSI product codes, and detecting assemblies in the GAC. If more sophisticated logic is required, the Bootstrapper can launch an external app that returns the detection result to the Bootstrapper through a return code.

In the case of the .NET Framework, the Bootstrapper performs two checks. The first determines if the Framework is already installed by launching dotnetchk.exe and storing the return code value in a property called DotNetInstalled. The second is a simple registry check that obtains the currently installed version of Microsoft Internet Explorer and stores the value in a property called IEVersion. These properties are then used in the <Commands> section, which instructs the Bootstrapper's actions based on the values of these checks.

<Commands> The <Commands> tag specifies all of the actions the Bootstrapper should perform while installing the component. The tag includes a Reboot attribute that specifies how the Bootstrapper should behave if the .NET Framework installer returns a value indicating that a reboot is needed. The attribute has three possible values: Defer, Immediate, and None. Defer prompts the user to reboot after all prerequisites have been installed. Immediate prompts the user to reboot immediately after this component has been installed. None doesn't prompt the user to reboot. The reboot is not critical and can be ignored.

<Command> The <Command> tag dictates a specific action to take during the installation process. Most components will have a single command that simply launches the component installer. In addition to the name of the file to run, the package author can specify command-line arguments, the estimated size of the package (which is used to provide progress UI during download time), and the estimated time the installer needs to run (which is used by the progress UI during install time). For the .NET Framework, the command launches dotnetfx.exe and passes command-line arguments indicating that the installer should be run in silent mode.

<InstallConditions> The <InstallConditions> section specifies the conditions under which the Bootstrapper should install, bypass, or fail the component installation. The <InstallIf> and <FailIf> tags are typically used to express basic requirements that must be met on the target machine in order for the component to be installed successfully. If any of these conditions are not met, the Bootstrapper ends the installation and the user is shown an error message explaining the reason.

In the case of the .NET Framework, there are a number of conditions under which installation would fail, such as:

  • The user does not have administrative privileges on the target machine
  • The machine is running Windows 95 or earlier, which is not supported
  • The machine is running Windows NT 4.0 or earlier, which is not supported
  • The machine is not running Internet Explorer 5.01 or greater, which is a prerequisite for the .NET Framework

The manifest may also contain <ByPassIf> tags, which specify the conditions under which the installation of the component should be silently skipped. The most common reason for skipping a component is that the component or a later compatible version has already been installed. The .NET Framework manifest includes a single bypass condition that skips the installation of the component if it has been previously installed.

<ExitCodes> The section inside <ExitCodes> tells the Bootstrapper how to interpret the various codes that dotnetfx.exe might return after executing. Each <ExitCode> tag specifies a return code value, a success state, and the name of a message to display to the user if necessary.

As shown in the example, a return code of zero from dotnetfx.exe indicates that the .NET Framework was successfully installed and no reboots are required. A value of 3010 indicates a successful install with a required reboot, which could happen if you installed the .NET Framework on a machine that is running Windows 98 without Windows Installer 2.0 installed.

The remaining exit codes are special error codes that dotnetfx.exe returns under various failure conditions (for example, if another instance of this program is already running, the Windows Installer service will be unresponsive). Because dotnetfx.exe wraps an Microsoft Installer file, a number of additional MSI failure codes could also be returned. These situations are all handled by the <DefaultExitCode> tag, which uses system APIs to get the underlying message that will ultimately be displayed to the user. This saves the package author from having to repeat standard Windows Installer error codes and messages and saves lots of extra coding.

A Closer Look at the Package Manifest

Figure 3 shows the English package manifest for the .NET Framework 2.0. The English manifest is somewhat simpler than those of other languages because dotnetfx.exe contains all of the English text, whereas the localized satellites must be installed through a separate Language Pack redistributable called langpack.exe. Just as the product manifest was responsible for downloading and launching dotnetfx.exe, the package manifests are responsible for launching the various language packs. Since we have already reviewed the tags that manage launching installers, we will use the English example here.

Figure 3 Bootstrapper Manifest for the .NET Framework English Language Pack (Close Figure 3)
<?xml version="1.0" encoding="utf-8" ?>

<Package
  xmlns="http://schemas.microsoft.com/developer/2004/01/bootstrapper"
  Name="DisplayName"
  Culture="Culture"
  LicenseAgreement="eula.txt">

  <PackageFiles>
    <PackageFile Name="eula.txt"/>
  </PackageFiles>

  <!-- Defines a localizable string table for error messages-->
  <Strings>
    <String Name="DisplayName">.NET Framework 2.0</String>
    <String Name="Culture">en</String>
    <String Name="AdminRequired">
      You do not have the permissions required to install .NET
      Framework 2.0. Please contact your administrator.
    </String>
    <String Name="InvalidPlatformWin9x">
      Installation of .NET Framework 2.0 is not supported on Windows 95.
      Contact your application vendor.
    </String>
    <String Name="InvalidPlatformWinNT">
      Installation of .NET Framework 2.0 is not supported on Windows NT 4.0.
      Contact your application vendor.
    </String>
    <String Name="InvalidPlatformIE">
      Installation of .NET Framework 2.0 requires Internet Explorer
      5.01 or greater. Contact your application vendor.
    </String>
    <String Name="ImproperUsage">
      Either the syntax used to specify parameters to Dotnetfx.exe
      is incorrect, or the parameters are invalid.
    </String>
    <String Name="WindowsInstallerComponentFailure">
      The Windows Installer installation failed. As a result,
      the installation of .NET Framework 2.0 cannot proceed.
    </String>
    <String Name="WindowsInstallerImproperInstall">
      Due to the improper installation of Windows Installer, the
      installation of .NET Framework 2.0 cannot proceed.
    </String>
    <String Name="CreateMutexFailure">
      A fatal error occurred and the installation of .NET Framework 2.0
      cannot proceed.
    </String>
    <String Name="AnotherInstanceRunning">
      Another instance of setup is already running.
    </String>
    <String Name="OpenDatabaseFailure">Cannot open MSI Database.</String>
    <String Name="ReadDatabaseFailure">
      Cannot read from MSI Database.
    </String>
    <String Name="GetTempDirectoryFailure">
      Cannot get Temporary directory.
    </String>
    <String Name="BetaNDPFailure">
      A beta version of the .NET Framework was detected on the computer.
      You must uninstall any beta versions before attempting to install
      Dotnetfx.exe.
    </String>
    <String Name="TempDirectoryTooLong">
      Temporary directory too long.
    </String>
    <String Name="SourceDirectoryTooLong">
      Source directory too long.
    </String>
    <String Name="CannotWriteLogFile">Cannot write to log.</String>
    <String Name="DarwinServiceHung">
      The Darwin service is hung and requires a reboot in
      order to continue.
    </String>
    <String Name="InternalError">
      An internal error occurred while trying to initialize
      the Darwin Service.
    </String>
    <String Name="GeneralFailure">
      A failure occurred attempting to install .NET Framework 2.0.
    </String>
  </Strings>

</Package>

<Package> The <Package> tag references the Bootstrapper manifest schema, specifies the display name that should appear in the user interface of tools such as Visual Studio, the culture of the package, and a localized license agreement.

<PackageFiles> The <PackageFiles> section provides a list of all the files in the package. In the case of the .NET Framework English version, there is only one additional file in the package, namely, the license agreement. Other languages would include the langpack.exe file in the list.

<Strings> The <Strings> section is a list of localized messages for the user who is installing the component. Each string has a name that can be referenced elsewhere in the product and package manifests. In most cases, these strings will contain the display name for the component, the culture, and a list of error messages to be displayed if a particular failure occurs.

Custom Redistributables in Visual Studio

Adding your own redistributables to Visual Studio requires that you author the product and package manifests, then copy these files along with your redistributable files into a special folder that Visual Studio has reserved for redistributables that plug into the Bootstrapper. This folder is installed by default like so:

C:\Program Files\Microsoft Visual Studio 8\SDK\v2.0\Bootstrapper

To write an installer that copies a redistributable component and its manifests into this folder, you can programmatically discover the Bootstrapper folder location from the registry keys that are shown in the following:

HKCU\Software\Microsoft\GenericBootstrapper\1.0\Path
HKLM\Software\Microsoft\GenericBootstrapper\1.0\Path

Each redistributable component appears in its own subfolder under the packages directory. The product manifest and redistributable files are then placed into this subfolder. Localized versions of the component along with the package manifests are placed in subfolders named after the culture code of that language (EN for English, JP for Japanese, and so on).

Once these files have been copied into the Bootstrapper folder, they automatically appear in the Visual Studio prerequisites dialog box so that the developer can select them by checking a box.

Generating the Bootstrapper Setup.exe with MSBuild

When you have selected a set of components to bootstrap, it is up to Visual Studio and the MSBuild system to configure the Bootstrapper setup.exe to handle those components. Bootstrapper configuration is handled by a special MSBuild task called GenerateBootstrapper. This build task performs a number of steps to produce the setup.exe. First, it obtains a copy of a base setup.exe image and writes the metadata from all of the relevant Bootstrapper manifests directly into the resource header of the setup.exe file. It then writes the URL (if provided) from which the redistributables should be downloaded into the resource header of setup.exe. Last, it copies the Bootstrapper, associated redistributables, and the actual application installer into the output directory.

The GenerateBootstrapper tasks are invoked by the Publish target, which runs whenever an application is published using Visual Studio ClickOnce tools. If Setup projects are used in order to build an MSI file for an application, the Bootstrapper task runs when the MSI file is built. The result is a lightweight setup.exe smart enough to bootstrap the selected components.

Conclusion

The Visual Studio 2005 Bootstrapper is a powerful and flexible new deployment tool that provides an integrated setup experience for an application and its prerequisite components. If you're building applications to run on nonstandardized desktop machines, the Bootstrapper installs all of the components and libraries required by your application, ensuring that at the end of the setup process the app runs smoothly. Especially important is the Visual Studio 2005 Bootstrapper's ability to install the .NET Framework 2.0 onto your customers' machines, thereby removing a major deployment hurdle for Visual Basic .NET and C# client apps.

[ Editor's Update - 9/24/2004: Fortunately, if you are writing applications with Visual Studio .NET 2003 today, and you need to bootstrap the .NET Framework 1.1, you do not have to wait until 2005 for a solution. You can download the Visual Studio .NET 2003 Bootstrapper Plug-In, which integrates with Setup and Deployment projects to configure a setup.exe that can bootstrap both MDAC 2.8 and the .NET Framework 1.1.] .

Sean Draine(seandr@microsoft.com) is the deployment Program Manager on the Microsoft Visual Basic team. Before Microsoft, Sean earned a PhD in Cognitive Psychology and founded Millisecond Software, which produces psychological research software.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


.NET Internals: Examine Running Processes Using Both Managed and Unmanaged Code

.NET Internals

Examine Running Processes Using Both Managed and Unmanaged Code

Christophe Nasarre
This article discusses:
  • Using the classes in the System.Diagnostics namespace
  • Using P/Invoke when necessary
  • Limitations of the ICorPublishxxx interfaces
  • Efficiently utilizing the debugging ICorDebugxxx interfaces
This article uses the following technologies:
C++, C#, COM, .NET
Code download available at: NETProcessBrowser.exe (651KB)
Contents
Enumerating Processes
Process and Other Related Classes
ProcessModule, FileVersionInfo, and ProcessThread Classes
But I Need to Call Unmanaged Functions!
ICorPublish Interfaces Family and Profiling API
Building the AssemblyBrowser Tool
Enter the .NET Debugging API
AppDomain, Assembly, and Module
Deciphering Classes
Conclusion

I
nterested in debugging techniques? You've come to the right place. In this article, you'll discover how to write a utility called AssemblyBrowser, which is similar to the Unix ps tool. AssemblyBrowser allows you to browse managed processes, AppDomains, assemblies, modules, and even the classes loaded by the runtime. You'll see how to get a process' ID, name, priority, number of threads, kernel handle, and memory consumption, as well as its user-mode, kernel-mode, and total elapsed running time. AssemblyBrowser is based on the common language runtime (CLR) debugging services, which supplies a number of services that form the foundation of the tool.

In addition to the specifics of this tool's implementation, I'll also explain the notion of a "responding" process, and show how to extract Win32® Portable Executable (PE) file version information. For processes running on the same machine, you'll see how the Graphical Design Interface (GDI) and USER handle count, and the command line, can be obtained with P/Invoke function calls.

Back in the June 2002 issue of MSDN®Magazine I presented some ways to enumerate processes running on your system along with their loaded DLLs (in "Escape from DLL Hell with Custom Debugging and Instrumentation Tools and Utilities"). If you're developing with managed code, you don't need to use P/Invoke to access functions from psapi.dll or toolhelp32.dll in order to get a list of the running Win32 processes. For pure managed applications and for unmanaged applications hosting the CLR, the .NET Framework offers other ways to fetch information. As you'll discover, it's also easy to get a list of the DLLs loaded at specific addresses within a process, as well as a list of that process' executing threads with their ID, priority, execution state, and user/kernel time.

Enumerating Processes

The System.Diagnostics.Process class allows you to get the list of currently running Win32 processes, regardless of whether they're managed or unmanaged. By calling the static method Process.GetProcesses, you can retrieve an array of Process objects, each of which contains information about a running Win32 process. As an optional parameter to this method, you can specify the name of the machine in whose processes you are interested, and as a result there are three ways to get the list of processes running on the local machine: you can call the parameterless overload of the method, you can specify "." as the machine name, or you can explicitly provide the machine name as the parameter.

To learn the name of your machine, you simply can use the MachineName static property of the System.Environment class. Alternatively, you might be tempted to use the MachineName property of a Process object. Using the Process.GetCurrentProcess static method enables you to easily get access to a Process object wrapping the current application; however, its MachineName property returns ".". This is because each process retrieved via the GetProcessXxx methods has a MachineName property that is either the filter you provided or "." for the parameterless override.

It is interesting to note that the current implementation of the .NET Framework (version 1.1) relies on TOOLHELP32 for Windows 9x systems. Owing to the limitations of this API, it is not possible to access remote machines on these platforms. For Windows NT®, Windows® 2000, and Windows XP, the processes and threads information is fetched using performance counters, even for the local machine. While a slower solution, it allows you to reach remote computers. It is even possible to give the IP address of the remote machine instead of its name. For enumerating modules and DLLs, there is no access provided to those on a remote machine because the Microsoft® .NET Framework relies on some process status API (PSAPI) functions such as EnumProcessModules, GetModuleInformation, GetModuleBaseName, and GetModuleFileNameEx, which were described in detail by Matt Pietrek in his August 1996 Under the Hood column in Microsoft Systems Journal.

If you want to filter the processes by name, you can use another static method provided by the Process class, GetProcessesByName, passing the file name of the application, but without the .exe extension. This is not a regular expression filter, rather a "raw," case-insensitive comparison. So, for example, if you want to retrieve a Process instance for every Windows and Microsoft Internet Explorer process currently running, you'll have to make two calls to GetProcessByName: one specifying "explorer" for the processName parameter and another with "iexplore," instead of using a regular expression which would allow for only one call using the simpler parameter "explore". It isn't too complicated, however, to build your own extended version of GetProcessesByName that allows for filtering based on regular expressions, as shown in Figure 1. Invalid regular expressions, invalid remote machine names, and unsupported remote accesses will all cause exceptions to be thrown.

Figure 1 GetProcessesByName with a Regex (Close Figure 1)
Process [] GetProcessesByName(
    string machine, string filter, RegexOptions options)
{
    ArrayList processList = new ArrayList();

    // Get the current processes
    Process [] runningProcesses = Process.GetProcesses(machine);

    // Find those that match the specified regular expression
    Regex processFilter = new Regex(filter, options);
    foreach(Process current in runningProcesses)
    {
        // Check for a match.
        if (processFilter.IsMatch(current.ProcessName))
        {
            processList.Add(current);
        }
        // Dispose of any we're not keeping
        else current.Dispose();
    }

    // Return the filtered list as an array
    return (Process[])processList.ToArray(typeof(Process));
}

Last, but not least, it is often handy to have a list of processes sorted by name or by ID. The lists you get from static Process methods, however, are not sorted in any particular order. Since these methods return an array, getting the expected sorted process lists should can be achieved by using the Array.Sort static method. For this to work, however, the objects stored in the array must either implement IComparable or you must provide an instance of a class that implements the IComparer interface, which is used to compare two objects. The former condition is not fulfilled because the Process class does not implement IComparable, nor does the .NET Framework offer a class that implements the latter condition. That said, it is easy enough to do it yourself. The following class can be used to sort an array of Process objects by name in alphabetical order:

public class ProcessComparer : IComparer
{
    int IComparer.Compare(object x, object y)
    {
        return ((Process)x).ProcessName.CompareTo(
            ((Process)y).ProcessName);
    }
}
With this class, it is now trivial to retrieve an alphabetically sorted list of all processes whose name includes the string "explore", as shown here:
Process [] processes = GetProcessesByName(
    ".", "explore", RegexOptions.IgnoreCase);
Array.Sort(runningProcesses, new ProcessComparer());
foreach(Process currentProcess in processes)
{
    Console.WriteLine(currentProcess.ProcessName);
}

Process and Other Related Classes

Once you have a reference to a Process object, refer to Figure 2 to see how Process is related to the classes exposed through its various properties. Process.Modules returns an instance of ProcessModuleCollection, a list of loaded modules. Each module is represented by an instance of the ProcessModule class, each associated with its own FileVersionInfo object. Process.Threads returns a ProcessThreadCollection, composed of instances of the ProcessThread class, each representing one thread in the running process.

Figure 2 Processes, Modules, and Threads
Figure 2 Processes, Modules, and Threads

To sum up, Figure 3 presents what you need to do to get the list of processes, their loaded modules, and their constituent threads. In this implementation, the process with an ID of 0 is skipped because this is a "fake" process without any meaningful detail, corresponding to the Idle process in the Task Manager. Before digging into modules and threads, note the wealth of information exposed by the Process class, shown in Figure 4. For more information on the data provided by many of these properties, see Inside Microsoft Windows 2000, Third Edition (Microsoft Press®, 2000), by Mark Russinovich and David Solomon.

Figure 4 Information Exposed by the Process Class (Close Figure 4)

Modules
Type | Property | Description
ProcessModule | MainModule | Module object wrapping the executable file. You should take care that it returns null for the "System" process (its Id is 4).
ProcessModuleCollection | Modules | Array of loaded DLLs. You should take care that it is empty for the "System" process (its Id is 4).
Threads
Type | Property | Description
ProcessThreadCollection | Threads | Array of started threads.
Time
Type | Property | Description
DateTime | StartTime | Moment when the process has been started.
DateTime | ExitTime | Moment when the process died; throws a System.InvalidOperationException if the process is still running: "Process must exit before requested information can be determined."
TimeSpan | PrivilegedProcessorTime | CPU time spent in Kernel mode.
TimeSpan | UserProcessorTime | CPU time spent in User mode.
TimeSpan | TotalProcessorTime | Sum of the two previous properties.
Memory
Type | Property | Description
IntPtr | MaxWorkingSet | Maximum allowable working set size.
IntPtr | MinWorkingSet | Minimum allowable working set size.
int | NonpagedSystemMemorySize | Nonpaged system memory size allocated for the process.
int | PagedMemorySize | Paged memory size.
int | PagedSystemMemorySize | Paged system memory size for the process.
int | PeakPagedMemorySize | Peak paged memory size for the process.
int | PeakVirtualMemorySize | Peak virtual memory size for the process.
int | PeakWorkingSet | Peak working set size for the process.
int | PrivateMemorySize | Private memory size for the process.
int | VirtualMemorySize | Virtual memory size for the process.
int | WorkingSet | Working set size for the process.
Other
Type | Property | Description
int | BasePriority | Base scheduling priority.
int | ExitCode | Exit code; throws a System.InvalidOperationException if the process is still running: "Process must exit before requested information can be determined."
IntPtr | Handle | Handle of the process with PROCESS_ALL_ACCESS rights.
int | HandleCount | Number of allocated kernel objects.
int | Id | Unique ID.
string | MachineName | Matches the machine name used for the enumeration or "." for the parameterless method call.
IntPtr | MainWindowHandle | Handle of the main window.
string | MainWindowTitle | Title of the main window.
bool | PriorityBoostEnabled | Direct call to the Win32 GetProcessPriorityBoost and SetProcessPriorityBoost APIs.
ProcessPriorityClass | PriorityClass | Direct call to the Win32 GetPriorityClass and SetPriorityClass APIs.
string | ProcessName | Name of the executable file.
IntPtr | ProcessorAffinity | Direct call to the Win32 GetProcessAffinityMask and SetProcessAffinityMask APIs.
bool | Responding | Returns true if the main window of the application is responding to messages. This is the same detail the Status column shows in the Applications tab of the Windows Task Manager.
Figure 3 Simplified Source Code for ps.exe (Close Figure 3)
Process[] runningProcesses = Process.GetProcesses(".");

//  sort the list if needed using a ProcessComparer object

foreach(Process currentProcess in runningProcesses)
{
    // skip Idle process
    if (CurrentProcess.Id == 0) continue;

    // dump details of the process
    // ...

    // list the loaded modules for this process
    ProcessModuleCollection loadedModules =
        CurrentProcess.Modules;
    foreach (ProcessModule module in loadedModules)
    {
        // dump module details
        DumpModule(Module);
    }

    // list the running threads
    ProcessThreadCollection RunningThreads =
        CurrentProcess.Threads;
    foreach (ProcessThread thread in runningThreads)
    {
        // dump Thread details
        DumpThread(thread);
    }
}

None of the following members should be used when enumerating existing processes. Rather, they should be used when you create a process yourself:

ProcessStartInfo     StartInfo
StreamReader         StandardError
StreamWriter         StandardInput
StreamReader         StandardOutput

ProcessModule, FileVersionInfo, and ProcessThread Classes

The Modules property of a Process object allows you to list the modules loaded by the application through a strongly typed ProcessModuleCollection. As with most collections, this class exposes typed accessors such as Contains, which in the case of ProcessModuleCollection takes a ProcessModule and returns true if this object is already in the list. Because of the reference semantics of the .NET Framework, if you get a Module from a first ProcessModuleCollection and you need to check whether it is loaded into another ProcessModuleCollection, Contains will return false. That's because this method does not compare the BaseAddress and the ModuleName for the wrapped process; rather, it compares the reference of each ProcessModule object itself. Each module is easily accessed through the bracket array syntax and is wrapped by a ProcessModule object with the properties listed in Figure 5.

Figure 5 ProcessModule Details (Close Figure 5)

Type | Property | Description
IntPtr | BaseAddress | Address where the module is loaded into the process address space (not the one specified at link time).
IntPtr | EntryPointAddress | Address of the entry point (either DllMain or Main).
string | FileName | Full path name of the binary file with its extension.
FileVersionInfo | FileVersionInfo | Wrapper class for the Win32 version embedded into the binary file. Using it, you get access to the same information as displayed by the File Properties dialog box in Windows Explorer.
int | ModuleMemorySize | Space used in memory by the PE file.
string | ModuleName | Name of the binary file with its extension.

Some processes have unfamiliar, non-Win32 file names. For example, "\??\C:\WINDOWS\system32\csrss.exe" is returned for the Win32 subsystem process; "\SystemRoot\System32\smss.exe" is what you get for the Session Manager process. This is not a problem in itself except when you try to access the module's FileVersionInfo. In fact, this ProcessModule property is dynamically computed based on its File name. For the ill-formed cases, you'll end up having to deal with either a System.IO.FileNotFoundException or a System.ArgumentException specifying that there were "Illegal characters in path".

The Process class provides access to its threads through the Threads property—another strongly typed collection represented by ProcessThreadCollection. This class exposes more methods than ProcessModuleCollection, including such methods as Add, Insert, and Remove. I don't see any interest in having these members defined as public, since this threads list should be treated as a read-only snapshot of the process state. Instead, you can use the collection's CopyTo method to get a clean and independent copy of the list for your own purpose.

Now you can retrieve a ProcessThread instance for each thread running in the process. I have a comment to add here to the .NET Framework documentation. Not all of the properties will return a value initially computed when the ProcessThreadCollection is taken, as Figure 6 illustrates. So far, I have not been able to figure out why PriorityLevel gets a different treatment than ThreadState or TotalProcessorTime. This class also provides a ResetIdealProcessor method that permits you to indicate that there is no single ideal processor for this thread.

Figure 6 ProcessThread Details (Close Figure 6)

Type | Property | Computation Time
int | BasePriority | Snapshot
int | CurrentPriority | Snapshot
int | Id | Snapshot
IntPtr | StartAddress | Snapshot
DateTime | StartTime | Snapshot
ThreadState | ThreadState | Snapshot
TimeSpan | PrivilegedProcessorTime | Snapshot
TimeSpan | UserProcessorTime | Snapshot
TimeSpan | TotalProcessorTime | Snapshot
ThreadWaitReason | WaitReason | Snapshot
int | IdealProcessor | Set on demand
bool | PriorityBoostEnabled | Get/set on demand
ThreadPriorityLevel | PriorityLevel | Get/set on demand

Last, but not least, you should take care to read the value of WaitReason if and only if ThreadState is set to the ThreadState.Wait value. Otherwise, be prepared to catch a System.InvalidOperationException exception.

But I Need to Call Unmanaged Functions!

Most of the details provided by the Windows Task Manager are available through the Process, ProcessModule, and ProcessThread classes. If you need a more esoteric piece of information, you can still rely on P/Invoke to call unmanaged Windows APIs directly. Following the example of the .NET Framework, I have built a class called Wrappers whose single purpose is to hide the implementation details needed to access some unmanaged functions.

To call an unmanaged function exported by a DLL, you can define a static extern method signature decorated with the DllImport attribute. This attribute takes the name of the DLL exporting the function as a mandatory parameter. This is a P/Invoke declaration for the CloseHandle function exported by kernel32.dll.

[DllImport("KERNEL32.DLL")] private static extern
int CloseHandle(int hObject);

The major problem when using P/Invoke is not being able to find the name of the function you know so well in Win32 or the function's exporting DLL. Even if you type the wrong name, your code will throw a System.EntryPointNotFoundException with an explanation that reads something like, "Unable to find an entry point named [your function name] in DLL [name you gave to the DllImport file]."

You really begin to scratch your head when you need to "guess" how to define the parameters passed or returned by the function, and what managed types match the Win32 unmanaged types and structures. Adam Nathan has created the site at http://www.pinvoke.net to help developers with this very problem. In addition, .NET Framework assemblies such as system.dll or mscorlib.dll contain some classes whose sole purpose is to provide a polished managed view of P/Invoke calls to well-known unmanaged APIs, as Figure 7 shows. If you double-click on one of these methods, you'll get a managed signature needed for your DllImport-tagged method. (Lutz Roeder's .NET Reflector tool provides the same kind of details in C#, Visual Basic® .NET, or Delphi syntax.) The wrappers.cs file in the code download for this article contains the managed methods that correspond to the APIs explored in my aforementioned June 2002 article, such as GetProcessMemoryInfo, NtQueryInformationProcess, and even ReadProcessMemory which can be used to get the command line that launched a process.

Figure 7 ILDASM and P/Invoke Wrapper Classes System.dll (Close Figure 7)

In addition to the managed classes explored thus far, the .NET Framework offers unmanaged ways to fetch information about running managed applications or unmanaged applications hosting the CLR. I'll now examine a tool based on the CLR debugging services that allows you to browse managed processes, AppDomains, assemblies, modules and even the classes loaded by the runtime.

ICorPublish Interfaces Family and Profiling API

If you are only interested in managed processes or processes hosting the CLR, the classes introduced so far don't give you a way to distinguish between plain unmanaged Win32 applications and the ones you are interested in. Of course, you could cheat and assume that the processes with mscoree.dll or mscorlib.dll loaded in their ProcessModuleCollection are .NET processes, but that won't work correctly, since an application can dynamically load these DLLs to directly call an undocumented function under the control of the CLR.

One simple solution is documented in the debugging API reference guide DebugRef.doc, which is stored in the Tool Developers Guide\docs Framework SDK subfolder of your Visual Studio® .NET installation. A set of COM interfaces, found in the Process Publishing Services, can be used to list .NET processes and their application domains on the local machine. For sample code, my own reusable C++ implementation can be found in ManagedProcess.cpp of this article's code download (for UnmanagedProcesses). Also take a look at Chris Sells's Web site, http://www.sellsbrothers.com/tools, or see "CLR Debugging: Improve Your Understanding of .NET Internals by Building a Debugger for Managed Code," by Mike Pellegrino (MSDN Magazine, November 2002). Mike Woodring provides a really helpful managed wrapper example at ICorPublishProcess, ICorPublishAppDomain Interop Shims.

These interfaces are fine in themselves, but they only provide you access to the .NET processes and their AppDomains. There is no way to get access to their assemblies, modules, or loaded classes through these interfaces. If this is not enough for you, profiling might be the next step. As Anastasios Kasiolas explained in ".NET CLR Profiling Services: Track Your Managed Components to Boost Application Performance" (MSDN Magazine, November 2001), if you build a COM object that implements the ICorProfilerCallback interface, it will be called by the CLR when a variety of events occur in a profiled process. These events include application domain creation or shutdown, assembly loading or unloading, module loading or unloading, and class loading or unloading—exactly the type of information we need!

In fact, there are a few minor issues to overcome in order to use ICorProfilerCallback. First, only one profiler can be alive at a time. Second, your code is implemented as a COM server DLL, not as an executable. In addition to standard COM registration, you have to define two system-wide environment variables to let the .NET runtime know that your COM object is the one to be notified during the execution of all .NET processes. Another solution is to let your profiler run all the time and use a private communication channel to an application dedicated to presenting the profiled .NET processes in a nice user interface.

In addition to the complexity of the implementation and installation of such a solution, you may face performance problems because your profiler is called every time an event you are interested in occurs in each and every process. This final issue is especially relevant for the events related to the loading of classes, an event which can happen frequently.

Building the AssemblyBrowser Tool

In the August 2002 issue, I explained how to use the Win32 debugging API to build a tool that accurately lists the DLLs that are statically or dynamically loaded and unloaded for a debugged process. The major constraint of this tool is that you need to start the target application under a debugger and control its lifetime from birth to death. I want to take a snapshot of already-running processes. Fortunately, the .NET debugging API allows a piece of code to attach to any .NET process, receive the notifications needed about already loaded AppDomains, assemblies, modules, and classes. And to top it off, you can detach from the debuggee without killing it, a capability the Win32 API did not have until Windows XP. I'll explain now how to use the .NET debugging API to build AssemblyBrowser. This tool lists .NET processes, their AppDomains, assemblies, modules, and loaded classes.

AssemblyBrowser is merely a user interface sitting on top of several reusable classes. It's built using a simple object model, as you can see in Figure 8. Each wrapper class defined in ManagedProcesses.h is built following the same model: a set of read-only property-like methods describing the underlying object (Process, AppDomain, Assembly, Module, Classes) and the GetFirst/GetNext enumerator helpers to go deeper into the hierarchy.

Figure 8 AssemblyBrowser Object Model
Figure 8 AssemblyBrowser Object Model

The class root, CManagedProcessList, is responsible for building a snapshot of the running .NET processes, based on the Publishing API already described. The class CManagedProcess is constructed via an ICorPublishProcess interface pointer, from which its PID and full path name are retrieved. Since the IsManaged method always returns true, there is no way through ICorPublishProcess to determine if a .NET process is managed by the CLR or is unmanaged but hosting the CLR. A good rule of thumb would be to take a look at its PE format and try to find out if it references kernel32.dll. If this is the case, chances are that the process is hosting the CLR.

The next task is to determine how you can enumerate the AppDomains created by the process. You could call the EnumAppDomains method from ICorPublishProcess to get a pointer to an ICorPublishAppDomainEnum enumeration interface. With it, you simply call Next until you get the last ICorPublishAppDomain interface pointer. Calling its GetID and GetName methods allow you to take a picture of each AppDomain. For AssemblyBrowser, another solution is used, which I'll cover now.

Enter the .NET Debugging API

First, you need to obtain an ICorDebug interface pointer through CoCreateInstance and call its Initialize method. Before leaving the application, and before releasing this interface, don't forget to call its Terminate method. I wrapped this central interface inside the CManagedDebugger class. Only one instance is available, following a singleton pattern. You simply have to initialize the COM runtime before calling the global function InitEngine, and don't forget to call ShutdownEngine before unloading the COM runtime with CoUninitialize.

Once you have a pointer to an ICorDebug interface, you may be tempted to call its GetProcess member with the process ID as a parameter to retrieve a pointer to the ICorDebugProcess wrapping your process. Unfortunately, this method returns E_INVALIDARG. This is because the debugger is not attached yet, so the application has not been debugged. You face the same problem if you try to enumerate processes with EnumerateProcesses, since it returns S_OK, but the ICorDebugProcessEnum interface pointer you get is useless. Its GetCount method indicates 0 and GetNext returns E_INVALIDARG.

But this is not how ICorDebug works. This interface allows you to either start a new process with CreateProcess or to attach to a running application with DebugActiveProcess. In the November 2002 article I mentioned earlier, Mike Pellegrino explains how to use the former and how to build a script debugger based on ICorDebug. The latter is used in a different way, as I'll demonstrate.

When I started reading the documentation and working with the IDL for ICorDebug, I followed the dead-end path shown in Figure 9. When I called DebugActiveProcess with the ID of the .NET process in which I was interested, I had no problem getting it to return a pointer to an ICorDebugProcess interface. Unfortunately, when I tried to call its EnumerateAppDomains method, its count was 0. This was only the beginning of my problems.

Figure 9 The Wrong Way to Use DebugActiveProcess (Close Figure 9)
ICorDebugProcess* pDebuggee = NULL;
HRESULT hr = pICorDebug->DebugActiveProcess(PID,
    L"We don't debug at the Win32 level" == NULL, &pDebuggee);
if (FAILED(hr) || (pDebuggee == NULL))
{
    TRACE("AttachDetachDebuggee() - "
          "impossible to get process %u...\n", PID);
    return(hr);
}

ICorDebugAppDomainEnum* pAppDomainEnum = NULL;
hr = pDebuggee->EnumerateAppDomains(&pAppDomainEnum);
if (FAILED(hr))
{
    TRACE("AttachDetachDebuggee() - "
          "impossible to enumerate AppDomains %u...\n", PID);
    return(hr);
 }

// try to list AppDomains
ULONG AppDomainCount = 0;
hr = pAppDomainEnum->GetCount(&AppDomainCount);
if (FAILED(hr))
{
    TRACE("AttachDetachDebuggee() - impossible to get "
          "the AppDomains count for process %u...\n", PID);
}

//... list AppDomains

hr = pDebuggee->Detach();
if (FAILED(hr))
{
    TRACE("AttachDetachDebuggee() - "
          "impossible to detach from process %u...\n", PID);
}
pDebuggee->Release();

The call to Detach returned an error and the debuggee was killed because I had neglected a requirement: it is mandatory to first register a callback through ICorDebug::SetManagedHandler. Before you call DebugActiveProcess, you have to register your class which implements the ICorDebugManagedCallback interface. Each of its methods is called by the CLR debugging services when an event occurs within the debuggee. Figure 10 lists the callbacks related to AssemblyBrowser's goals. In DebuggerCallback.cpp, you can uncomment several _TRACE_XXX flags (where XXX is PROCESS, APPDOMAIN, ASSEMBLY, MODULE, CLASS, and THREAD) to allow fine-grained tracing in each handler. This is a good way for a developer to understand when each is called and what arguments it is called with.

Figure 10 Managed Debugging API Workflow (Close Figure 10)

Method | Parameters | Note
CreateProcess | ICorDebugProcess* | Called once just after DebugActiveProcess
CreateAppDomain | ICorDebugProcess* ICorDebugAppDomain* | Called once per AppDomain in the process
LoadAssembly | ICorDebugAppDomain* ICorDebugAssembly* | Called once per Assembly in each AppDomain
LoadModule | ICorDebugAppDomain* ICorDebugModule* | Called once per Module for each Assembly
LoadClass | ICorDebugAppDomain* ICorDebugClass* | Called once per class
CreateThread | ICorDebugAppDomain* ICorDebugThread* | Called once per thread in the process

It is also possible to act as a Win32 debugger and thus to be notified of the corresponding unmanaged debugging events. You would register an implementation of the ICorDebugUnmanagedCallback interface through ICorDebug::SetUnManagedHandler, but we don't need to do that for AssemblyBrowser.

In the Win32 debugging world, once you have started to debug an application, either by launching it or by attaching to a running process, you need to call WaitForDebugEvent to wait for the next event to happen in the debugged app, and ContinueDebugEvent when you have finished handling this particular event. This continues until the debuggee ends or until you decide to leave the game. Your debugging logic stays in the thread waiting for an event to occur in the debuggee.

This process is totally different in the managed world, as Figure 11 illustrates. After you attach to a debuggee, there is no need to call a particular ICorDebug or ICorDebugProcess method in order to receive the events. The ICorDebugManagedCallback methods you have implemented are called from another thread each time an event occurs in the debuggee. Each method receives an ICorDebugController inherited interface pointer as a parameter, such as ICorDebugProcess or ICorDebugAppDomain as context. Once you have finished working on an event, it is your responsibility to call the Continue method of this parameter so the debuggee can keep on running.

Figure 11 Debugger/Debugee Interactions
Figure 11 Debugger/Debugee Interactions

These techniques have one limitation: the CLR does not allow you to attach to a process that's already being debugged. The direct consequence of this limitation is that you can't get information related to one of these processes. But there is another, less obvious consequence. Since Visual Studio .NET is hosting the CLR, it becomes impossible to debug your own code. When a breakpoint that you have added into the code (somewhere before detaching) is triggered, both applications are deadlocked. Visual Studio .NET will never be able to call ContinueDebugEvent to let AssemblyBrowser call Continue.

This is also the reason why you can find the following code in CManagedProcess::AttachDetachDebuggee:

if (::IsDebuggerPresent())
{
   if (wcsstr(m_szPathname, L"devenv.exe") != NULL)
      return(S_OK);
}
Replace "devenv.exe" with the file name of your own debugger if you are not using Visual Studio.

The last architectural issue you need to tackle is how you can synchronize the main thread with your callback so that you can detach from the debuggee when the last interesting event is handled. When you are debugging by attachment, the last event is in fact received by your CreateThread handler. Unfortunately, since more than one thread is always running in any managed process, you can't tell how many times CreateThread will be called. This means you can't detach immediately after the first thread notification. To solve this problem, I created a Win32 Event object named m_hEventNext, which is created before attaching to the debuggee and is signaled in the CreateThread handler. The trick is to call Continue in the handlers of the callback until the first thread is detected, then call Continue in the main thread until ICorDebugProcess::HasQueuedCallbacks returns false, meaning no more events are available in the debuggee. At that point, the main thread can safely detach from the debuggee using the Detach method.

AppDomain, Assembly, and Module

With this architecture in place, the details we are interested in are gathered through the handler methods that were listed in Figure 2 (in the same order). A single CreateProcess is called, and then one CreateAppDomain call is triggered for each AppDomain already created in the process. Each loaded assembly is then signaled by a LoadAssembly call before one LoadModule call per module is issued. To keep track of this stack of calls in the callback class, I store two objects: the CAppDomain object that wraps the current AppDomain in its m_pCurrentAppDomain member, and the CAssembly object that wraps the current assembly in its m_pCurrentAssembly member.

After this set of notifications, your callback LoadClass method is called for each class loaded by the CLR in the process. Finally, your CreateThread method is triggered for each thread detected.

So as not to walk through the source code line by line, let's look at the major implementation details related to each particular handler. In CreateProcess, the m_bHasSeenFirstThread Boolean member is set to false, waiting for CreateThread to reset its value to true. In CreateAppDomain, Attach needs to be called for the given AppDomain, otherwise you will not be notified for its assemblies and modules. Since we are dealing with COM, we need to AddRef this interface, meaning it's necessary to store the corresponding interface pointer in order to call Detach and Release on it once a new AppDomain is detected, or before detaching from the debuggee, as in the case of the last AppDomain.

Sometimes, the ICorDebugAssembly interface pointer given as the parameter to LoadAssembly has the bad habit of returning an empty name through its GetName method. In that case, "<Unknown or dynamic assembly>" is returned as the name of the wrapping CAssembly object. This situation often arises for assemblies dynamically created by Reflection.Emit. According to DebugRef.doc, the GetCodeBase method is not supposed to be implemented. If you try to call it, it returns E_NOTIMPL. This is not important; if you need to know the path corresponding to the assembly, it is returned by GetName.

In the case of LoadModule, you get an ICorDebugModule interface pointer describing each module loaded in the current assembly. Using the GetName method, you can retrieve its full path name. GetBaseAddress returns the address where it has been loaded in the process address space. In the case of a dynamically created AppDomain, for the corresponding modules, IsDynamic returns true and IsInMemory distinguishes between in-memory and on-disk dynamic modules.

Deciphering Classes

If you want to be notified for the loaded classes, you have to explicitly call EnableClassLoadCallbacks on the ICorDebugModule received as a parameter of LoadModule. Otherwise, by default, your LoadClass handler is never called. The CLR debugging services were designed that way for better performance. Since plenty of classes are loaded, your handler will be called hundreds of times, and if you don't need this information, you don't want to pay the performance penalty associated with obtaining it. You should also note that when you are using a particular type, a bunch of other types are implicitly loaded, such as all the types in its inheritance hierarchy as well as the types for any implemented interfaces.

In addition to an ICorDebugAppDomain interface pointer, the ClassLoad method receives an ICorDebugClass interface pointer as a parameter. When the time comes to identify the corresponding type, you can rely on its GetToken method, which returns an mdTypeDef object. For AssemblyBrowser, given a type definition token we need to get its namespace, its outer classes if it's nested, and the type itself. To translate what is returned by GetToken, you need to switch to the unmanaged metadata API detailed in Metadata Unmanaged API.doc as well as by Matt Pietrek in "Avoiding DLL Hell: Introducing Application Metadata in the Microsoft .NET Framework," (MSDN Magazine, October 2000).

The IMetaDataImport interface is the cornerstone for discovering the metadata related to a type. It is usually obtained from the IMetaDataDispenser root interface given an assembly file name. When using the CLR debugging services, you can take a different approach. The GetModule method from ICorDebugClass lets you obtain a pointer to the ICorDebugModule interface wrapping the module defining the type. Call its GetMetaDataInterface method and request the corresponding IMetaDataImport interface. The full name of the type is discovered when you call its GetTypeDefProps method, passing the token representing the type. The token corresponding to its parent is also available through the last parameter of this method. In cases where a type is not defined in the same module, it may be quite a difficult undertaking to get information about it, since GetTypeProps is useless even if it returns S_FALSE as an expected failure.

If you are looking for additional information for the type, such as its namespace, parsing the full name is not enough since you won't be able to tell the difference between a namespace and a nested type. In other words, how do you decrypt foo.bar? Well, foo can be a type or a namespace. The IMetaDataImport interface does not seem to provide any way to get this kind of information directly. If you are feeling energetic, you should take some time to poke around inside the source code of ILDASM, installed with Rotor, a multiplatform shared-source CLI implementation of the CLR, available at Shared Source Common Language Infrastructure 1.0 Release. ILDASM relies on an undocumented IMDInternalImport interface that provides a GetNameOfTypeDef method. GetNameOfTypeDef returns both the namespace and the name of a class, given its token definition. Unfortunately, explaining how to use this interface in the context of the CLR debugging services is far beyond the scope of this article. You will, however, find some hints within the implementation of the CManagedType constructor in ManagedProcesses.cpp.

It is possible to build your own "smart parsing" based on a combination of common sense and another method from the IMetaDataImport interface. The source code listed in Figure 12 shows such a GetClassName method from ManagedProcesses.cpp.

Figure 12 Extracting Namespace and Nested Type from a Type Token (Close Figure 12)
// Given a TypeDef token, return the corresponding type name as well as
// its parent class and namespace.  This method is intended as an
// internal method only and assumes the supplied buffers are of the
// correct size (_MAX_CLASS_NAME+1).
HRESULT CManagedType::GetClassName(
   IMetaDataImport* pIMetaDataImport,
   mdTypeDef tkType,
   wchar_t* szClass,
   wchar_t* szParentClass,
   wchar_t* szNamespace
   )
{
   HRESULT hr = S_OK;

   wchar_t szFullClassName[_MAX_CLASS_NAME+1];
   szFullClassName[_MAX_CLASS_NAME] = L'\0';

   ULONG Length = 0;
   CorTypeAttr tdFlags = tdClass;   // 0 value
   mdToken tkParent = 0;
   hr = pIMetaDataImport->GetTypeDefProps(
         tkType,
         szFullClassName,
         _MAX_CLASS_NAME,
         &Length,
         (DWORD*)&tdFlags,
         &tkParent
         );

   // This could be S_FALSE in some cases.
   ASSERT(SUCCEEDED(hr));
   if (FAILED(hr)) return(hr);

   // The full name is retrieved. If it's a nested type, szFullClassName
   // is the class name only, and I need to do the same for its parent.
   // Otherwise, it contains the namespace plus the class name at the end.
   if (!IsTdNested(tdFlags))
   {
      // Not a nested type:
      // --> szFullClassName = namespace.namespace...namespace.ClassName
      wchar_t* pszNamespace = szFullClassName;
      wchar_t* pszDot = wcsrchr(szFullClassName, L'.');
      if (pszDot == NULL)
      {
         // No namespace, just the class name
         szNamespace[0] = L'\0';
         wcsncpy(szClass, szFullClassName, _MAX_CLASS_NAME);
      }
      else
      {
         // The '.' is just between the namespaces and the class name.
         // The namespace itself starts after the '.'.
         *pszDot = L'\0';
         wchar_t* pszName = pszDot + 1;

         // Extract class and namespace
         wcsncpy(szClass, pszName, _MAX_CLASS_NAME);
         wcsncpy(szNamespace, pszNamespace, _MAX_CLASS_NAME);
      }

      // and no parent class
      szParentClass[0] = L'\0';
   }
   else
   {
      // Get the token for its parent
      hr = pIMetaDataImport->GetNestedClassProps(tkType, &tkParent);
      ASSERT(SUCCEEDED(hr));

      // szFullClassName is the class name and we have
      // to search recursively starting from its parent.
      wchar_t szTmpClass[_MAX_CLASS_NAME+1];
      szTmpClass[_MAX_CLASS_NAME] = L'\0';
      wchar_t szTmpParentClass[_MAX_CLASS_NAME+1];
      szTmpParentClass[_MAX_CLASS_NAME] = L'\0';

      hr = GetClassName(pIMetaDataImport, tkParent,
        szTmpClass, szTmpParentClass, szNamespace);
      ASSERT(SUCCEEDED(hr));

      // build the class name from its parent name
      if (szTmpParentClass[0] != L'\0')
      {
         size_t length = 0;

         wcsncpy(szParentClass, szTmpParentClass, _MAX_CLASS_NAME);
         length = wcslen(szTmpParentClass); // count string just copied

         wcsncat(szParentClass, L".", _MAX_CLASS_NAME - length);
         length++; // count the '.'

         wcsncat(szParentClass, szTmpClass, _MAX_CLASS_NAME - length);
      }
      else wcsncpy(szParentClass, szTmpClass, _MAX_CLASS_NAME);

      // The single class name is returned by GetTypeDefProps()
      wcscpy(szClass, szFullClassName, _MAX_CLASS_NAME);

      // Nothing special to do for namespace as it was already
      // computed during the parent lookup.
   }

   return(hr);
}

The basic idea is quite simple. If a type is not nested, all the words before the last dot are the namespace where it is defined. If it is nested, you need to iterate through the containing types until you reach the innermost type.

In the call to GetTypeDefProps with the type token, the fifth parameter is used to return a flag providing additional information, such as its visibility or whether it is an interface. Calling IsTdNested (defined in CorHdr.h) on this flag returns true if it is a nested type. If it returns false, you know that you only have to deal with namespaces separated by dots within the string returned by GetTypeDefProps. If this is a nested type, you simply have to apply the same idea on the containing type.

With a little help from the GetNestedClassProps method of IMetaDataImport, you can get the token corresponding to the type's enclosing type. You should notice that the name returned by the GetTypeDefProps method is actually the name of the type if the type is nested.

Conclusion

The classes in the System.Diagnostics namespace provide much of the same features as PSAPI and toolhelp. What's more, these classes are gathered into an object model that's not only much easier to understand but also easier to use. Of course, if your needs go beyond what the .NET Framework provides, you can back into the Win32 world using P/Invoke.

The CLR provides several COM interfaces to let you list .NET applications and their content. As you have seen, the ICorDebug set of interfaces opens up the debugging API, leaving you free to use what you need without having to write a real debugger.

Christophe Nasarreis a development manager for Business Objects in France. He has written several low-level tools for Windows since version 3.0 and reviewed many books on Win32, COM, MFC, and .NET. He can be reached at cnasarre@hotmail.com.


ASP.NET 2.0: Speed Up Your Site with the Improved View State in ASP.NET 2.0

ASP.NET 2.0

Speed Up Your Site with the Improved View State in ASP.NET 2.0

Fritz Onion
This article discusses:
  • How view state works
  • ASP.NET 2.0 improvements to view state
  • Using control state to maintain functionality
  • Performance considerations
This article uses the following technologies:
ASP.NET, C#
Code download available at: ViewState.exe (122KB)
Contents
View State Fundamentals
View State Problems
View State Improvements in ASP.NET 2.0
Control State
Declarative Data Sources and View State
Conclusion

I
f you're a seasoned ASP.NET developer, the mere mention of view state probably sends shivers down your spine, as you envision kilobytes of base64-encoded data being sucked through a cocktail straw. Unless you take steps to prevent it, most ASP.NET pages will have a significant amount of supplemental data stored in a hidden field named __VIEWSTATE that in many cases is not even necessary. For fun, surf to your favorite sites built with ASP.NET, view the sources of the pages, and count the number of characters in the hidden __VIEWSTATE fields. I tried it and the sizes ranged from 800 to 7,800 characters.

View state, of course, has an important role in ASP.NET. Used wisely, it can simplify page development and improve user interaction with your site. If left unchecked, it can increase the size of your site's responses dramatically and bring your response times to a crawl over slow connections. The release of ASP.NET 2.0 brings several improvements to the view state mechanism that will make it easier to use without hindering your site's performance. These improvements include a reduction in encoding size, the introduction of control state to separate behavioral state from content, and intelligent integration with data-bound controls.

View State Fundamentals

Before I jump into the improvements in view state in ASP.NET 2.0, a short summary of the purpose and implementation of view state in the current release is in order. View state solves a very specific problem for ASP.NET developers—retaining state for server-side controls that are not form elements. This is important because much of the server-side control model in ASP.NET is built around the assumption that all controls retain their state if the user posts back to the same page. That is, if you modify the contents of any control during the processing of a request, you can count on those modifications still being there on any subsequent POST request back to the same page. As an example of view state in action, try running the page shown in Figure 1.

Figure 1 View State in Action (Close Figure 1)
<%@ Page Language="C#" %>

<script runat="server">
protected override void OnLoad(EventArgs e)
{
  int val = int.Parse(_sum.InnerText);
  _sum.InnerText = (val+1).ToString();
  base.OnLoad(e);
}
</script>

<html>
  <body>
    <form runat="server">
      <h2>ViewState test page</h2>
      <span id="_sum" runat="server">0</span>
      <br />
      <input type="submit" />
    </form>
  </body>
</html>

Each time you press the Submit button, the value in the _sum span is incremented. Because view state retains the previous value across the request, it will start with the last value displayed and show 1, 2, 3, 4, and so on. To see what happens when view state is not enabled, try adding enableviewstate='false' as an attribute on the span element. It will display a value of 1 no matter how many times the page is posted since the previous value of the span is not propagated across requests.

View state completes the control-based programming model in ASP.NET. Without view state, some controls (like textboxes and dropdown lists) would keep some state across POST requests, while others would not, and working with these disparate controls would be a frustrating exercise in remembering special cases. With view state, developers can focus on the programming model and user interaction without having to worry about retaining state. View state can also be hashed or encrypted to prevent a user from tampering with or decoding it. For more information on this topic, see chapter 19 of "Securing Your ASP.NET Application and Web Services," available online.

There is one other important place where view state is used, and that is in controls that issue server-side change events. If the user changes the value in a textbox or switches the selected element in a dropdown list, you can register an event handler and have code execute when the event is raised. These controls compare the current value of the control with its previous value, which is stored implicitly in view state if any process has subscribed to the change event. If you disable view state on a control whose change notification event you are handling, it won't fire correctly since it must always assume that the previous value was the same as the default value in the form.

View State Problems

There are many issues with view state in ASP.NET 1.x, as I indicated earlier. It is on by default, and unless you know to look for it and disable it when it is not needed, it can significantly increase the amount of data rendered by your page. This becomes particularly painful when you use data-bound controls, all of which use view state to save their state across postbacks. As a simple and dramatic example, consider the ASP.NET page shown in Figure 2.

Figure 2 Problems with View State in ASP.NET 1.x (Close Figure 2)
<%@ Page Language="C#" %>
<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="System.Configuration" %>
<%@ Import Namespace="System.Data.SqlClient" %>

<script language="C#" runat="server">
protected override void OnLoad(EventArgs e)
{
  string dsn = ConfigurationSettings.AppSettings["dsnPubs"];
  string sql = "SELECT * FROM Authors";

  using (SqlConnection conn = new SqlConnection(dsn))
  using (SqlCommand cmd = new SqlCommand(sql, conn))
  {
    conn.Open();
    _dg1.DataSource = cmd.ExecuteReader();
    _dg1.DataBind();
  }

  base.OnLoad(e);
}
</script>

<html>
  <body>
    <form id="Form1" runat="server">
      <asp:DataGrid ID="_dg1" Runat="server" />
    </form>
  </body>
</html>

This page has a single DataGrid control that binds to the results of a simple query against the authors table of the pubs database. If you run this page (making any necessary corrections to the connection string) and look at the view state field, you will probably be shocked to find that it has more than 12,000 characters in it. It's even more shocking when you look at the contents of the page and realize that the actual number of characters needed to display the entire contents of the table to the browser is about 1,600. One reason for this explosion is that view state must encode not only the data but also the type of data (metadata); also, base64 encoding generally adds about 33 percent space overhead. Still, this much overhead to simply retain the state of the control across POST requests may seem disproportionate at times and definitely should be avoided if at all possible.

Deciding when to disable view state in your controls is obviously important if you are concerned about the size of your responses. The example just shown is a classic case where view state is being populated but never used, which suggests the primary rule to follow when optimizing view state usage on your site: if you populate the contents of a control every time a page is requested, it is generally safe (and wise) to disable view state for that control.

On the other hand, you may decide to take advantage of the fact that view state is going to retain the state of your controls and only populate control contents on the initial GET requests to a page (just falling through on subsequent POST requests). This saves a round-trip to whatever back-end data source you are using. By modifying my page to leverage this fact, I could change the OnLoad method to check the IsPostBack flag before populating the DataGrid, as shown in Figure 3.

Figure 3 Checking IsPostBack (Close Figure 3)
protected override void OnLoad(EventArgs e)
{
  if (!IsPostBack)
  {
    string dsn = ConfigurationSettings.AppSettings["dsnPubs"];
    string sql = "SELECT * FROM Authors";

    using (SqlConnection conn = new SqlConnection(dsn))
    using (SqlCommand cmd = new SqlCommand(sql, conn))
    {
      conn.Open();
      _dg1.DataSource = cmd.ExecuteReader();
      _dg1.DataBind();
    }
  }
  base.OnLoad(e);
}

There are, of course, other options that may be even better here, such as caching the results of the query on the server and rebinding the control with each request. It is up to you to decide where the state should be stored and what the trade-offs are in your particular architecture and applications.

You may have noticed that I was careful to say that it is "generally" safe to disable view state if you populate the contents of a control each time a page is requested. The exception to this is that some controls also use view state for behaviors as well as plain-state retention. As described earlier, the dropdown list and the textbox controls use view state to store the previous value to properly issue a change notification event on the server. Similarly, the DataGrid class uses view state to issue pagination, edit, and sorting events. Unfortunately, if you want to use features like sorting, paging, or editing in the DataGrid, you cannot disable its view state. This all-or-nothing mentality of server-side controls with respect to view state is one of the most frustrating aspects of the server-side control model in ASP.NET 1.x for developers who are trying to build fast, efficient sites.

View State Improvements in ASP.NET 2.0

Now that I have outlined the problems, it is time to discuss all the improvements in ASP.NET 2.0. The first one is the overall size of view state when it is serialized. In ASP.NET 1.x the serialization of two strings into the view state buffer looks like this:

<p<l<string1;>;l<string2;>>;>;

The serialization format used in ASP.NET 1.x for view state is a tuple format consisting of a hierarchical collection of triplets and pairs serialized using less-than and greater-than characters. The letter preceding the greater-than symbol indicates the type of the object stored (t=triplet, p=pair, i=integer, l=ArrayList, and so on). Each subelement within the less-than and greater-than characters is separated with a semicolon. It's an interesting serialization format, sort of like a compressed XML. If you're concerned with space, however, it is not the most efficient serialization format (only marginally better than XML).

ASP.NET 2.0 changes this serialization format. The serialization of the same two strings into the view state buffer in ASP.NET 2.0 is shown in the following line of code:

[][]string1[]string2
At least that's what it looks like when rendered to a browser. The square boxes are actually nonprintable characters which, if we rewrite using Unicode character references, turn into the code shown here:
&#15;&#5;string1&#5;string2

Instead of using a collection of printable characters to delineate objects in the view state stream ('<', '>', ';', 'l', 'i', 'p' and so on), ASP.NET 2.0 uses a number of nonprintable characters to mark the beginning of an object and to describe what type of object it is.

Using nonprintable characters to delineate the objects stored in view state serves two purposes. First, it improves the efficiency of the lexical analysis during the parsing of the view state string since there is no longer any need to match characters or parse tokens. Second, and more importantly, it reduces the number of characters used to encode the objects in view state. In my simple example of encoding two strings, the first encoding used 16 delineation characters versus 3 in the 2.0 format. This effect quickly compounds to make a significant impact.

For example, if we place a DataGrid on a form and bind it to the authors table in the pubs database, as shown in Figure 3, the size of the view state string in 1.x is 12,648 characters. If we do the same in 2.0, the size of view state is reduced to 6,728 characters, a reduction of nearly half. Figure 4 shows the contrast in space required by view state in ASP.NET 1.x, ASP.NET 2.0, and the number of characters actually rendered to the page for a variety of input sizes. In this case the authors table was augmented with additional rows at 500 row increments up to 2,000.

Figure 4 View State Size Comparison
Figure 4 View State Size Comparison

Without doing anything special, you can look forward to an immediate reduction in the size of your view state with the release of ASP.NET 2.0. This does not mean that you should stop thinking about whether your controls need view state to function properly or not, since it can still contribute significantly to the size of the response, as evidenced in Figure 4. The next improvement in view state, however, can potentially save you much more.

Control State

I mentioned earlier that one of the most frustrating aspects of working with server-side controls in ASP.NET 1.x is the all-or-nothing mentality with respect to view state. Behavioral aspects of controls like pagination in the DataGrid or selection change notifications in textboxes require view state to be enabled to function properly. I'm sure all of you are sufficiently daunted by the prospect of leaving view state enabled in any of your DataGrids at this point. In ASP.NET 2.0, Microsoft addresses this particular problem by partitioning the view state into two separate and distinct categories: view state and control state.

Control state is another type of hidden state reserved exclusively for controls to maintain their core behavioral functionality, whereas view state only contains state to maintain the control's contents (UI). Technically, control state is stored in the same hidden field as view state (being just another leaf node at the end of the view state hierarchy), but if you disable view state on a particular control, or on an entire page, the control state is still propagated. The nice aspect of this implementation is that we can now amend our original principle on optimizing view state for ASP.NET 2.0 to be much stronger: if you populate the contents of a control every time a page is requested, you should disable view state for that control.

As long as control builders properly partition their state into behavioral state (to maintain core functionality) and UI state (to retain contents), this statement should be followed religiously. I would like to tell you to start following this recommendation now with all work done in ASP.NET 2.0, but the DataGrid has not been refactored to keep its behavioral state in control state. Fortunately, the new GridView is the logical successor to the DataGrid and it uses control state properly, as do the new versions of the TextBox and DropDownList controls. Figure 5 shows a list of the controls that currently make use of control state and what properties they store there for your reference. This list will likely change with the final release of ASP.NET 2.0.

Figure 5 Controls That Use Control State (Close Figure 5)

Control | Properties Stored in Control State
CheckBoxList | PreviousItemCount PreviousOffset
ContentPager | CurrentPage PageCount
DetailsView | PageIndex Mode DefaultMode
FormView | PageIndex Mode DefaultMode
GridView | EditIndex SelectedIndex PageIndex SortDirection SortExpression
ListControl (base class for BulletedList, CheckBoxList, DropDownList, ListBox, and RadioButtonList) | SelectedIndices
LoginView | TemplateIndex
MultiView | ActiveViewIndex
Table | CurrentRow ViewMode

If you would like to explore further how the new controls in ASP.NET 2.0 use control state and view state, I have written a utility called the View State Decoder which shows all of the view and control state on a given page. Figure 6 shows a screenshot of this utility in action. You can download the program from the link at the top of this article. Note that this application relies on the character encodings currently used for view state which may change before the final release.

Figure 6 ViewState Decoder (Close Figure 6)

For those of you building controls, the usage model for control state is not quite as convenient as view state. Instead of providing a state-bag with an indexer to insert and remove items, you must override the virtual LoadControlState and SaveControlState methods and manually manage your portion of the object collection that is mapped into control state. The one other thing you must do is call the Page.RegisterRequiresControlState method during initialization so that the page knows to ask for control state at the right time. It is probably a good thing that storing items in control state is more difficult than view state since there is no way for users to disable it (there are also performance advantages to having an opt-in model for control state). Developers should think carefully before storing any state there as it will always be added to the rendering of a page. Figure 7 shows a custom control that stores a string in control state (in this case, its color).

Figure 7 Storing a String in Control State (Close Figure 7)
public class SampleControl : Control
{
  string _color = "black";

  protected override void OnInit(EventArgs e)
  {
    Page.RegisterRequiresControlState(this);
    base.OnInit(e);
  }

  protected override void LoadControlState(
                          object savedState)
  {
    object[] rgState = (object[])savedState;
    base.LoadControlState(rgState[0]);
    _color = (string)rgState[1];
  }

  protected override object SaveControlState()
  {
    object[] rgState = new object[2];
    rgState[0] = base.SaveControlState();
    rgState[1] = _color;
    return rgState;
  }

  •••
}

When you are deciding what should go into control state and what should go into view state, remember that it should be a separation of behavioral state from content or UI state. Things like properties and collections of data should generally stay in view state and not migrate to control state. State that triggers a server-side event is the most typical type to store in control state.

Declarative Data Sources and View State

There is a small problem with the application of our view state optimization principle—you will very often not know whether you are populating the contents of a control on each request or not. The introduction of declarative data sources in ASP.NET 2.0 means that binding data to a control no longer involves explicitly wiring up its Data source property to a DataReader or DataSet and calling DataBind. Instead, you can declaratively place a data source on your page and point your control to that data source, for example using the new GridView class and an associated SqlDataSource bound to the authors table in the pubs database (see Figure 8). If you run this page, you will find that it just works. The GridView and its corresponding data source are smart enough to know when to interact. By the time the page is ready to render, the GridView is full of data from the data source and will properly render the entire authors table to the client.

Figure 8 Pointing to Pubs with a GridView (Close Figure 8)
<%@ Page Language="C#" %>

<html>
<body>
    <form runat="server">
        <asp:GridView Id="_myGridView" Runat="Server"
            DatasourceId="_pubsDs" />
        <asp:SqlDataSource Id="_pubsDs" Runat="server"
            ConnectionString=
                "server=(local);database=pubs;Integrated Security=SSPI"
            SelectCommand="SELECT * FROM Authors" />
        <asp:Button Runat="server" Text="push me" />
    </form>
</body>
</html>

In this simple case, we have not disabled view state on the GridView, so you may think that we are again just wasting view state on storing data that is never used. Fortunately, the ASP.NET 2.0 engine makes the extra effort to do the right thing here, and it will not bother going back to the database when view state is enabled on a control.

Similarly, if you disable view state on the GridView, the data binding to the data source will occur on each request including POST back requests. This functionality is built into the DataBoundControl base class from which the AdRotator, BulletedList, CheckBoxList, DropDownList, ListBox, RadioButtonList, GridView, DetailsView, and FormView controls inherit. All of these controls exhibit this intelligent use of view state when bound to declarative data sources.

Conclusion

The next release of ASP.NET promises many improvements for Web developers, not the least of which is a better chance of sleeping at night without nightmares of base64-encoded data flowing across their Web farms. With a more compact serialization format, a partitioning between behavioral and UI state, and smart interaction with data-bound controls and declarative data sources, those view-state nightmares just might become sweet dreams.

Fritz Onion is a co-founder of Pluralsight, an education and content-creation company, where he focuses on Web development with ASP.NET. Fritz is the author of Essential ASP.NET (Addison Wesley, 2003). Read his blog at http://www.pluralsight.com/fritz.


Advanced Serialization: Format Your Way to Success with the .NET Framework Versions 1.1 and 2.0

New information has been added to this article since publication.
Refer to the Editor's Update below.

Advanced Serialization

Format Your Way to Success with the .NET Framework Versions 1.1 and 2.0

Juval Lowy
This article discusses:
  • Serialization events
  • Secure serialization
  • Serialization and generics
  • The binary formatter and versioning
This article uses the following technologies:
.NET and C#
Code download available at: AdvancedSerialization.exe (245KB)
Contents
Delegates and Serialization
Deserialization Events in the .NET Framework 1.1
Binary Formatter Serialization Events
Applying the Event Attributes
Serialization Events and Class Hierarchies
Using the Deserializing Event
Using the Deserialized Event
Generic Formatters
Serialization and Versioning
Binary Formatter Version Tolerance
Custom Serialization and Versioning
Serialization and Cloning
Constraining Serialization
Manual Base Class Serialization
Securing Serialization
Conclusion

S
erialization is the act of persisting the state of an object, typically to a file or a memory stream. Serialization is crucial when it comes to composing an application out of independent components. If an application contains several components from various vendors, how will it persist its objects into a single file without having the individual components overwrite and destroy the information persisted by other components? By default, objects in .NET are not serializable. But the Microsoft® .NET Framework provides a standard, straightforward way to serialize and deserialize objects. In most cases, all you have to do is add the Serializable attribute and thus grant your consent for serializing the object, as shown here:
[Serializable]
public class MyClass
{
   public string SomeString;
}

The .NET Framework uses reflection during runtime serialization to read the state of the object and write it to the stream. Reflection is also used during deserialization to set the state of the object using values read from the stream. If you want to preclude serialization of individual members, you can use the NonSerialized attribute, like this:

public class MyOtherClass
{...}
[Serializable]
public class MyClass
{
   [NonSerialized]
   MyOtherClass m_Obj;
}

In fact, you must use that attribute when the field's type is not serializable, otherwise you'll get errors when attempting to serialize or deserialize the class or structure that contains it. You can even provide custom serialization by also implementing the ISerializable interface.

While .NET offers adequate support for the simple cases, issues arise when you're attempting to support delegates and subscribers, versioning, and class hierarchies. The .NET Framework 2.0 adds support for serialization events and some degree of versioning tolerance. In addition, the introduction of generics opens new dimensions, both in the complexity of the serialization task and the power now available for serialization tools. Because security must be at the forefront of your development effort, secure serialization is essential. I'll discuss these issues and related pitfalls, suggest techniques and workarounds that address them, and recommend when and how to best use the new serialization abilities.

Delegates and Serialization

All delegates are compiled into serializable classes. This means that when serializing an object that has a delegate member variable, the delegate's internal invocation list is serialized too. This makes serializing delegates very difficult because there are no guarantees that the target objects in the internal list are serializable. Consequently, sometimes the serialization will work and sometimes it will throw a serialization exception. In addition, the object containing the delegate typically does not know or care about the actual state of the delegate. This is even more the case when the delegate is used to manage event subscriptions. The exact number and identity of the subscribers are often transient values that should not persist between application sessions.

As a result, you should mark delegate member variables as nonserializable using the NonSerialized attribute:

[Serializable]
public class MyClass
{
   [NonSerialized]
   EventHandler m_MyEvent;
}
In the case of events, you must also add the field attribute qualifier when applying the NonSerialized attribute so that the attribute is applied to the underlying delegate rather than to the event itself:
[Serializable]
public class MyPublisher
{
   [field:NonSerialized]
   public event EventHandler MyEvent;
}

Deserialization Events in the .NET Framework 1.1

When .NET deserializes an object, it initializes the nonserializable member variables to the default value for that type. It is then up to you to provide code to initialize the variables to their correct value. To that end, the object needs to know when it's being deserialized. To let the object know, you implement the interface IDeserializationCallback:

public interface IDeserializationCallback
{
   void OnDeserialization(object sender);
}

OnDeserialization is called after an object is deserialized, allowing it to perform the required custom initialization steps. The sender parameter is ignored and is always set to null. Figure 1 shows the implementation of IDeserializationCallback. Here, MyClass has a database connection as a member variable of the type SqlConnection, which isn't a serializable type and is therefore marked using the NonSerialized attribute. In its implementation of OnDeserialization, MyClass creates a new connection object, initializes the connection object by providing it with a connection string, and then proceeds to open it.

Figure 1 Using IDeserializationCallback (Close Figure 1)
using System.Runtime.Serialization;

[Serializable]
public class MyClass : IDeserializationCallback
{
   [NonSerialized]
   IDbConnection m_Connection;
   string m_ConnectionString;

   public void OnDeserialization(object sender)
   {
      Debug.Assert(m_Connection == null);
      m_Connection = new SqlConnection();
      m_Connection.ConnectionString = m_ConnectionString;
      m_Connection.Open();
   }
}

Binary Formatter Serialization Events

The .NET Framework 2.0 introduces support for serialization events. Designated methods on your class will be called when serialization and deserialization take place. The .NET Framework 2.0 defines four serialization and deserialization events. The Serializing event is raised just before serialization takes place, and the Serialized event is raised just after serialization. Similarly, the Deserializing event is raised just before deserialization, and the Deserialized event is raised after deserialization. Both classes and structures can take advantage of serialization events. You designate methods as serialization event handlers using method attributes, as shown in Figure 2.

Figure 2 Applying Serialization Event Attributes (Close Figure 2)
[Serializable]
public class MyClass
{
   [OnSerializing]
   void OnSerializing(StreamingContext context)
   {...}
   [OnSerialized]
   void OnSerialized(StreamingContext context)
   {...}
   [OnDeserializing]
   void OnDeserializing(StreamingContext context)
   {...}
   [OnDeserialized]
   void OnDeserialized(StreamingContext context)
   {...}
}

Each serialization event-handling method must have the signature shown in the following line of code:

void <Method Name>(StreamingContext context);
This is required because, internally, delegates with a matching signature are used to invoke the event-handling methods.

Figure 3 Events During Serialization
Figure 3 Events During Serialization

As the attribute names imply, the OnSerializing attribute designates a method to handle the Serializing event, and the OnSerialized attribute designates a method that handles the Serialized event. Similarly, the OnDeserializing attribute designates a method to be used for handling the Deserializing event, and the OnDeserialized attribute designates a method to be used to handle the Deserialized event. Figure 3 is an activity diagram depicting the order in which events are raised during serialization when using the binary formatter.

The serialization mechanism verifies which type of formatter is used. If it is the SOAP formatter (or any custom formatter), then .NET simply performs serialization. However, if the binary formatter is used, .NET first raises the Serializing event, thus invoking the corresponding event handlers (there can be more than one, as you will see shortly). Next, .NET serializes the object, and finally the Serialized event is raised and its event handlers are invoked. The process of deserialization is executed in a similar fashion. Figure 4 is an UML activity diagram depicting the order in which binary deserialization events are raised.

Figure 4 Events During Deserialization
Figure 4 Events During Deserialization

Unlike the serialization events, with deserialization .NET has to accommodate the use of IDeserializationCallback. If the binary format is not used, the formatter performs deserialization and then calls the OnDeserialization method of IDeserializationCallback if the class implements it. When the binary formatter is used, the formatter first raises the Deserializing event, followed by the deserialization itself. If the class implements IDeserializationCallback, then the formatter calls the OnDeserialization method and finally raises the Deserialized event. Note that in order to call the Deserializing event-handling methods, the formatter first has to construct an object, yet it does so without ever calling any of the class's constructors.

Applying the Event Attributes

The .NET Framework 2.0 allows you to apply the same serialization event attributes on multiple methods of the class:

[OnSerializing]
void OnSerializing1(StreamingContext context)
{...}
[OnSerializing]
void OnSerializing2(StreamingContext context)
{...}
This can come in handy when you want to add additional processing to a serialization event without affecting existing event-handling methods. While you can also apply multiple attributes on the same event-handling method, as shown in the following, the usefulness of doing so is questionable:
[OnSerializing]
[OnSerialized]
void OnSerialization(StreamingContext context)
{...}
The method will be called once per attribute, and there is no easy way to detect which event is raised inside the method.

Serialization Events and Class Hierarchies

A significant advantage to using attributes for events as opposed to using interfaces is that the event mechanism is decoupled from the class hierarchy. When using attributes, the event-handling methods are called for each level in a class hierarchy. There is no need to call the base class's event-handling methods and there is no problem if those base methods are private. The events are raised according to the order of the class hierarchy, and the event attributes are not inherited. For example, when serializing an object of the type MySubClass, as shown in the following code, OnSerializing1 is called first, followed by a call to OnSerializing2:

[Serializable]
public class MyBaseClass {
   [OnSerializing]
   void OnSerializing1(StreamingContext context)
   {...}
}
[Serializable]
public class MySubClass : MyBaseClass {
   [OnSerializing]
   void OnSerializing2(StreamingContext context)
   {...}
}

The situation could therefore get messy when virtual methods are involved and the sub class overrides its base class's handling of a serialization event or even calls it. As a result, the serialization infrastructure will throw a SerializationException if any of the event attributes are applied on a virtual method or on an overriding method. Use of the new inheritance qualifier is still allowed in conjunction with serialization events. Since you should not encounter any external party besides the .NET Framework calling a serialization event-handling method, I recommend always designing such methods as private. Similarly, the serialization event attributes throw an exception when applied on a class method that implements an interface method.

Using the Deserializing Event

Since no constructor calls are ever made during deserialization, the Deserializing event-handling method is logically your deserialization constructor. It is designed for performing custom pre-deserialization steps, which typically involve the initialization of nonserializable members. Any value settings performed on the serializable members will be a waste of time because the binary formatter will set those members again during deserialization using values from the serialization stream.

The main difference between the Deserializing event and IDeserializationCallback is that OnDeserialization is called after deserialization is complete, while the Deserializing event is called before deserialization starts. In the Deserializing event-handling method you should place the initialization steps that are independent of the values saved in the serialization stream. In contrast, in OnDeserialization you can take advantage of already deserialized members (such as the database connection string in Figure 1). Other tasks you can perform in the deserialization event-handling method are setting specific environment variables such as thread local storage and signaling global synchronization events.

Using the Deserialized Event

Taking advantage of the Deserialized event makes the use of IDeserializationCallback redundant, as the two are logically equivalent. Both let your class respond to post-deserialization events and initialize nonserializable members while using already deserialized values, as the following code demonstrates:

[Serializable]
public class MyClass {
   [NonSerialized]
   IDbConnection m_Connection;
   string m_ConnectionString;

   [OnDeserialized]
   void OnDeserialized(StreamingContext context) {
      m_Connection = new SqlConnection();
      m_Connection.ConnectionString = m_ConnectionString;
      m_Connection.Open();
   }
}
It performs exactly the same task as in Figure 1, but it relies on the Deserialized event rather than on IDeserializationCallback.

Note that IDeserializationCallback is still useful when you cannot guarantee the use of a BinaryFormatter or the presence of the .NET Framework 2.0 runtime. Let me emphasize that relying on the serialization events mandates that the client always use the BinaryFormatter on the class; the SoapFormatter has no equivalent support for the serialization events.

Generic Formatters

The IFormatter interface is implemented by both the binary and SOAP formatters, as shown here:

public interface IFormatter
{
   object Deserialize(Stream serializationStream);
   void Serialize(Stream serializationStream, object graph);
   /* Other methods */
}
IFormatter's significant methods are Serialize and Deserialize, which perform the actual serialization and deserialization. The problem with IFormatter is that it was defined before generics was available in .NET. With the introduction of generics in the .NET Framework 2.0, you can improve on the available implementations of IFormatter by providing generic and type-safe wrappers around them. You can define the IGenericFormatter interface which provides similar methods to IFormatter, but does so using generic methods, as shown in the following code:
public interface IGenericFormatter
{
   T Deserialize<T>(Stream serializationStream);
   void Serialize<T>(Stream serializationStream,T graph);
}
The use of generic methods is preferable to making the whole interface generic. You could use the same formatter instance, but change the type parameter being serialized or deserialized in every call. Next, implement IGenericFormatter by encapsulating a nongeneric formatter, and delegate the calls to it.

Figure 5 shows the generic class GenericFormatter<F> that implements IGenericFormatter. GenericFormatter<F> is defined using the generic type parameter F, which is constrained to implement IFormatter and provide a default constructor. This enables GenericFormatter<F> to declare a member of the type IFormatter and assign to it a new F object:

IFormatter m_Formatter = new F();
Figure 5 also defines two subclasses of GenericFormatter<F>: GenericBinaryFormatter and GenericSoapFormatter. All they do is provide the binary or the SOAP formatter, respectively, as type parameters to GenericFormatter<F>. You could have defined GenericBinaryFormatter and GenericSoapFormatter with the using statement except that would have had only file scope:
using GenericBinaryFormatter = GenericFormatter<BinaryFormatter>;
using GenericSoapFormatter   = GenericFormatter<SoapFormatter>;
In this case, inheritance is good for strong typing and shorthand across files and assemblies. Figure 6 shows the use of the generic and type-safe formatters.

Figure 6 Using the Generic and Type-Safe Formatters (Close Figure 6)
[Serializable]
public class MyClass
{...}

MyClass obj1 = new MyClass();

IGenericFormatter formatter = new GenericBinaryFormatter();
Stream stream = new FileStream(
    @"C:\obj.bin",FileMode.Create,FileAccess.ReadWrite);

formatter.Serialize(stream,obj1);
stream.Seek(0,SeekOrigin.Begin);

MyClass obj2 = formatter.Deserialize<MyClass>(stream);
stream.Close();
Figure 5 GenericFormatter<F> (Close Figure 5)
public class GenericFormatter<F> : IGenericFormatter where F : IFormatter,new() {
   IFormatter m_Formatter = new F();

   public T Deserialize<T>(Stream serializationStream) {
      return (T)m_Formatter.Deserialize(serializationStream);
   }
   public void Serialize<T>(Stream serializationStream,T graph) {
      m_Formatter.Serialize(serializationStream,graph);
   }
}
public class GenericBinaryFormatter : GenericFormatter<BinaryFormatter> {}
public class GenericSoapFormatter : GenericFormatter<SoapFormatter> {}

Serialization and Versioning

If an application wants to serialize the state of multiple objects of multiple types to the same stream, a simple dump of object state will not do. The formatter must also capture the object's type information. During deserialization, the formatter needs to read the type's metadata and initialize a new object according to the information serialized, populating the corresponding field. The easiest way to capture the type information is to record the type's name and assembly. For each object that is serialized, the formatter persists the state of the object (the values held in its various fields) and the version information and full name of its assembly, including a token of the assembly's public key (if a strong name is used). When the formatter deserializes the object, it loads its assembly and reflects the type's metadata.

By default, the formatters comply with the version-binding and assembly-resolving policies of the common language runtime (CLR). If the serialized type's assembly does not have a strong name, the formatters try to load a private assembly and completely ignore any version incompatibility between the version captured during serialization and the version of the assembly found. If the serialized type's assembly has a strong name, the CLR loader insists on using a compatible assembly. If such an assembly is not found, an exception is thrown.

Both the binary and SOAP formatters provide a way to record only the friendly name of the assembly without any version or public key token, even if the assembly has a strong name. The formatters provide a public property called AssemblyFormat of the enum type FormatterAssemblyStyle:

public enum FormatterAssemblyStyle
{
   Full,
   Simple
}

The default value of AssemblyFormat is FormatterAssemblyStyle.Full. If you set it to FormatterAssemblyStyle.Simple, no version compatibility checks will take place during deserialization:

SoapFormatter formatter = new SoapFormatter();
formatter.AssemblyFormat = FormatterAssemblyStyle.Simple;
However, I strongly discourage you from circumventing serialization version and type verification. At best, a potential incompatibility will result in an exception of type SerializationException. At worst, your application may later crash unexpectedly because the incompatible type required some custom initialization steps.

Binary Formatter Version Tolerance

In the .NET Framework 1.x, there had to be absolute compatibility between the metadata used to serialize a type and the metadata used to deserialize it. This meant that if your application had clients with serialized state of your types, your type members' metadata needed to be immutable or you would break those clients.

In the .NET Framework 2.0, the binary formatter has acquired some version-tolerance capabilities with respect to changes in the type metadata, rather than changes to the assembly version itself. Imagine a class library vendor that provides a serializable component. The various client applications are responsible for managing the serialization media (typically a file). Suppose the vendor changes the component definition by adding a private member variable. Such a change does not necessitate a version change because new client applications can serialize the new component properly. However, the serialization information captured by the old applications is now incompatible and will result in a SerializationException if used in the .NET Framework 1.x. The vendor can, of course, increment the assembly version number, but doing so will prevent the old clients from taking advantage of new functionality. The binary formatter in the .NET Framework 2.0 was redesigned to handle such predicaments.

In the case of removing an unused member variable, the binary formatter will simply ignore the additional information found in the stream. For example, suppose you used the following class definition and serialized it using a binary formatter:

// Version 1.0
[Serializable]
public class MyClass
{
   public int Number1;
   public int Number2;
}
Without changing the assembly version, you can now remove one of the member variables, rebuild, redeploy, and deserialize instances of version 2.0 of MyClass with the serialization information captured using version 1.0 of MyClass:
// Version 2.0
[Serializable]
public class MyClass
{
   public int Number1;
}

The real challenge in type-versioning tolerance is dealing with new members because the old serialization information does not contain any information about them. By default, the binary formatter is not tolerant of the new members, and will throw an exception. The .NET Framework 2.0 addresses this problem by providing the OptionalField field attribute—a simple attribute with a single public member of type int called VersionAdded:

[AttributeUsage(AttributeTargets.Field)]
public sealed class OptionalFieldAttribute : Attribute
{
   public int VersionAdded;
}

Applying the OptionalField attribute has no effect during serialization, and fields marked with it will be serialized into the stream. OptionalField is meant to be applied on new fields of your type as it causes the binary formatter to ignore the new member during the deserialization process:

//Version 1.0
[Serializable]
public class MyClass {
   public int Number1;
}
//Version 2.0
[Serializable]
public class MyClass {
   public int Number1;
   [OptionalField]
   public int Number2;
}

That said, if the new member variable has a good enough default value, such as the application's default directory or user preferences, you can use values provided by the new clients and synthesize a value for the old clients. You will need to provide these values in your handling of the deserializing event. The reason is that if you only set them before deserialization, when the stream does contain serialized values, the deserialization process will override the values previously set.

Consider this class version:

//Version 1.0
[Serializable]
public class MyClass
{
   public int Number1;
}

Suppose you want to add a new class member called Number2, while using the old serialization information. You can provide a handler for the Deserializing event and in it initialize Number2:

[Serializable]
public class MyClass {
   public int Number1;
   [OptionalField]
   public int Number2;

   [OnDeserializing]
   void OnDeserializing(StreamingContext context) {
      Number2 = 123;
   }
}

But what if the values you synthesize are somehow dependent on the version of the class in which they are added? You can store version information in the OptionalField attribute using its VersionAdded member:

[OptionalField(VersionAdded = 1)]
public int Number2;

In the Deserializing event handler you will need to use reflection to read the value of the VersionAdded field and act accordingly, as shown in Figure 7. The example uses the helper method OptionalFieldVersion of the SerializationUtil static helper class. The OptionalFieldVersion method relies on a generic type parameter for the type used, and it accepts the member variable name to reflect, returning the value of the VersionAdded field.

Figure 7 Relying on VersionAdded (Close Figure 7)
[Serializable]
public class MyClass
{
   public int Number1;

   [OptionalField(VersionAdded = 1)]
   public int Number2;

   [OnDeserializing]
   void OnDeserializing(StreamingContext context)
   {
      int versionAdded =
         SerializationUtil.OptionalFieldVersion<MyClass>("Number2");
      if(versionAdded == 1)
         Number2 = 123;
      if(versionAdded == 2)
         Number2 = 456;
   }
}

public static class SerializationUtil
{
   public static int OptionalFieldVersion<T>(string member)
   {
      Type type = typeof(T);
      MemberInfo[] members = type.GetMember(member,BindingFlags.Instance |
                                            BindingFlags.NonPublic|
                                            BindingFlags.Public   |
                                            BindingFlags.DeclaredOnly);
      Debug.Assert(members.Length == 1);
      object[] attributes = members[0].GetCustomAttributes(
          typeof(OptionalFieldAttribute),false);
      Debug.Assert(attributes.Length == 1); //Exactly one is expected

      OptionalFieldAttribute attribute;
      attribute = attributes[0] as OptionalFieldAttribute;
      return attribute.VersionAdded;
   }
}

Custom Serialization and Versioning

Type changes often do involve changes to the version number of the containing assembly. In such cases, you cannot take advantage of the binary formatter type-version tolerance because it is first subject to assembly versioning. In such cases, you need to use custom serialization to deal with version issues between the serialized information and the current class definition. As with binary formatter version tolerance, the component needs to be able to determine the version of its assembly with which the serialized information was saved and then act accordingly.

The SerializationInfo class used in the GetObjectData method of ISerializable provides the AssemblyName property, and you can extract from it the version of the assembly used to serialize the information. For example, suppose version 1.0.0.0 of the serializable class MyClass is distributed to clients:

//Version 1.0.0.0
[Serializable]
public class MyClass
{
   public int Number;
}

The clients persisted this version in the serialized state information. With version 2.0.0.0 of the assembly, the vendor adds the new field NewField to the class definition. Using version checks, the component can decide during deserialization whether to use the serialized value or assign a default value. Figure 8 illustrates this technique. It uses the helper class, SerializationUtil, to extract the version information from the assembly's full name. Note that the version extracted is the version of the assembly in which the original serialized type resided. [ Editor's Update - 3/14/2005: The SerializationInfo.AssemblyName and SerializationInfo.FullTypeName properties have no useful meaning during deserialization. The BinaryFormatter sets these values to match the assembly name and type name of the type being deserialized into. As such, it is a mistake to depend on them for versioning during deserialization.]

Figure 8 Versioning Using Custom Serialization (Close Figure 8)
//Version 2.0.0.0
[Serializable]
public class MyClass : ISerializable
{
   public int Number;
   public int NewField;

   public void GetObjectData(
       SerializationInfo info,StreamingContext context)
   {
      info.AddValue("Number",Number);
      info.AddValue("NewField",NewField);
   }

   protected MyClass(SerializationInfo info,StreamingContext context)
   {
      Number = info.GetInt32("Number");
      Version storedVersion = SerializationUtil.GetVersion(info);
      if(storedVersion.ToString() == "2.0.0.0")
      {
         NewField = info.GetInt32("NewField");
      }
      else
      {
         NewField = 123;//Some default value
      }
   }

   public MyClass()
   {}
}

public static class SerializationUtil
{
   static public Version GetVersion(SerializationInfo info)
   {
      string assemblyName = info.AssemblyName;
      /* AssemblyName is in the form of "MyAssembly, Version=1.2.3.4,
                              Culture=neutral,PublicKeyToken=null" */
      char[] separators = {',','='};
      string[] nameParts = assemblyName.Split(separators);
      return new Version(nameParts[2]);
   }
   //Rest of SerializationUtil
}

Serialization and Cloning

In addition to a FileStream, you can use any other type of Stream, such as a NetworkStream or MemoryStream. You can actually use a MemoryStream to create a deep clone of a serializable object. Figure 9 shows the static Clone method of the SerializationUtil static helper class.

Figure 9 Cloning a Serializable Object (Close Figure 9)
public static class SerializationUtil
{
   static public T Clone<T>(T source)
   {
      Debug.Assert(typeof(T).IsSerializable);

      IGenericFormatter formatter = new GenericBinaryFormatter();
      Stream stream = new MemoryStream();
      formatter.Serialize(stream,source);
      stream.Seek(0,SeekOrigin.Begin);

      T clone = formatter.Deserialize<T>(stream);
      stream.Close();
      return clone;
   }
   //Rest of SerializationUtil
}

The Clone method first verifies that the object passed in is serializable. This is easily accomplished by obtaining the type of the source object. The Type class provides a Boolean property, IsSerializable, which returns true if the type has been marked with the Serializable attribute. The Clone method then uses the GenericBinaryFormatter helper class to serialize and deserialize the object into a memory stream and returns the deserialized object. Using Clone is quite straightforward, as shown in the following code:

[Serializable]
public class MyClass
{...}

MyClass obj1 = new MyClass();
MyClass obj2 = SerializationUtil.Clone(obj1);
You can use SerializationUtil.Clone as an easy way to implement the interface ICloneable:
[Serializable]
public class MyClass : ICloneable
{
   public object Clone()
   {
      return SerializationUtil.Clone(this);
   }
}

Constraining Serialization

A generic class that has generic type parameters as members can still be marked for serialization, like so:

[Serializable]
public class MyClass<T>
{
   T m_T;
}
In such cases, however, the generic class is only serializable if the generic type parameter specified is serializable. Consider this code:
public class SomeClass
{}
MyClass<SomeClass> obj;
Here, obj is not serializable because the type parameter, SomeClass, is not serializable. Consequently, MyClass<T> may or may not be serializable, depending on the generic type parameter used. This may result in a runtime loss of data or system corruption because the using app may not be able to persist the state of the object.

Presently, .NET 2.0 does not provide a mechanism for constraining a generic type parameter to be serializable. There are three workarounds to guarantee deterministic serialization behavior. The first is to mark all member variables of generic type parameter as nonserializable, like so:

[Serializable]
public class MyClass<T>
{
   [NonSerialized]
   T m_T;
}

This, of course, may seriously damage the ability of the generic class MyClass<T> to function properly in all cases where you need to serialize the state of members of a generic type.

The second workaround is to place a constraint on the generic type parameter to implement ISerializable:

[Serializable]
public class MyClass<T> where T : ISerializable
{
   T m_T;
}
This ensures that all instances of MyClass<T>, regardless of the type parameter, are serializable. But it does place a burden on the class developer for implementing custom serialization of all generic type parameters used.

The third and best solution is to perform a single runtime check before any use of the type MyClass<T> and abort the use immediately before any damage can take place. The trick is to place the runtime verification in a C# static constructor. Figure 10 demonstrates this technique.

Figure 10 Runtime Serialization Constraints (Close Figure 10)
[Serializable]
class MyClass<T>
{
   static MyClass()
   {
      SerializationUtil.ConstrainType<T>();
   }
   T m_T;
}

public static class SerializationUtil
{
   public static void ConstrainType<T>()
   {
      Type type = typeof(T);
      bool serializable = type.IsSerializable;
      if(serializable == false)
      {
         string message = "The type " + type.ToString() +
            " is not serializable";
         Exception exception = new SerializationException(message);
         throw exception;
      }
   }
   //Rest of SerializationUtil
}

The C# static constructor is invoked exactly once per type, per app domain, upon the first attempt to instantiate an object of that type. In Figure 10, the static constructor calls the static helper method, ConstrainType, of SerializationUtil. Although ConstrainType does not accept any arguments, it does define a single generic type parameter that the caller needs to supply. ConstrainType then simply verifies that the generic type parameter is serializable by checking the IsSerializable property of its type. If the generic type parameter is not serializable, ConstrainType throws a serialization exception, thus aborting any attempt to use the type.

Performing the constraint verification in the static constructor is a technique applicable to any constraint that you cannot enforce at compile time, yet where you have some programmatic way of determining and enforcing it at run time.

Manual Base Class Serialization

Combining class hierarchies and serialization, whether fully automatic or custom, is straightforward: all classes use only the Serializable attribute or they all do that plus implement ISerializable. The picture isn't so clear when it comes to deriving a serializable class from a class that's not marked with the Serializable attribute, as in this case:

public class MyBaseClass
{}
[Serializable]
public class MySubClass : MyBaseClass
{}

In fact, here a formatter can't serialize objects of type MySubClass at all because it can't serialize their base classes. Trying to serialize an object of type MySubClass results in an exception of type SerializationException. Such a situation may happen when deriving from a class in a third-party assembly where the vendor neglected to mark its class as serializable.

The good news is that you can provide a solution even for such a case. The solution presented next isn't a sure cure. It assumes that none of the base classes require custom serialization steps. The solution merely compensates for an oversight of not marking the base class as serializable.

The workaround is simple: the subclass can implement ISerializable, use reflection to read and serialize the base classes' fields, and use reflection again to set these fields during deserialization. The static SerializationUtil helper class provides the two static methods, SerializeBaseType and DeserializeBaseType, defined as you can see here:

public static class SerializationUtil
{
   public static void SerializeBaseType(object obj,
      SerializationInfo info, StreamingContext context);
   public static void DeserializeBaseType(object obj,
      SerializationInfo info, StreamingContext context);
   //Rest of SerializationUtil
}
All the subclass needs to do is implement ISerializable and use SerializationUtil to serialize and deserialize its base classes, as shown in the following:
public class MyBaseClass {}
[Serializable]
public class MySubClass : MyBaseClass,ISerializable {
   public MySubClass() {}
   public void GetObjectData(SerializationInfo info,
      StreamingContext context) {
      SerializationUtil.SerializeBaseType(this,info,context);
   }
   protected MySubClass(SerializationInfo info,StreamingContext context) {
      SerializationUtil.DeserializeBaseType(this,info,context);
   }
}

If the subclass has no need for custom serialization, and the only reason it implements ISerializable is to serialize its base class, you can use SerializationUtil to serialize the subclass as well. SerializationUtil provides these overloaded versions of SerializeBaseType and DeserializeBaseType:

public static void SerializeBaseType(object obj,bool serializeSelf,
   SerializationInfo info, StreamingContext context);
public static void DeserializeBaseType(object obj,bool deserializeSelf,
   SerializationInfo info, StreamingContext context);

These versions accept a flag instructing them whether to start serialization with the type itself instead of with its base class:

public void GetObjectData(SerializationInfo info,StreamingContext context)
{
   //Serializing this type and its base classes
   SerializationUtil.SerializeBaseType(this,true,info,context);
}
protected MyClass(SerializationInfo info,StreamingContext context)
{
   //Deserializing this type and its base classes
   SerializationUtil.DeserializeBaseType(this,true,info,context);
}

Figure 11 presents the implementation of SerializeBaseType and DeserializeBaseType. When SerializationUtil serializes an object's base class, it needs to serialize all of the base classes leading to that base class as well. You can access the base class type using the BaseType property of Type.

Figure 11 Manual Base Class Serialization (Close Figure 11)
public static class SerializationUtil
{
   public static void SerializeBaseType(object obj,
      SerializationInfo info,StreamingContext context)
   {
      Type baseType = obj.GetType().BaseType;
      SerializeBaseType(obj,baseType,info,context);
   }
   static void SerializeBaseType(object obj,Type type,
      SerializationInfo info, StreamingContext context)
   {
      if(type == typeof(object))
      {
         return;
      }
      BindingFlags flags =
         BindingFlags.Instance|BindingFlags.DeclaredOnly|
         BindingFlags.NonPublic|BindingFlags.Public;
      FieldInfo[] fields = type.GetFields(flags);
      foreach(FieldInfo field in fields)
      {
         if(field.IsNotSerialized)
         {
            continue;
         }
         string fieldName = type.Name + "+" + field.Name;
         info.AddValue(fieldName,field.GetValue(obj));
      }
      SerializeBaseType(obj,type.BaseType,info,context);
   }
   public static void DeserializeBaseType(object obj,
      SerializationInfo info, StreamingContext context)
   {
      Type baseType = obj.GetType().BaseType;
      DeserializeBaseType(obj,baseType,info,context);
   }
   static void DeserializeBaseType(object obj,Type type,
      SerializationInfo info, StreamingContext context)
   {
      if(type == typeof(object))
      {
         return;
      }
      BindingFlags flags =
         BindingFlags.Instance|BindingFlags.DeclaredOnly|
         BindingFlags.NonPublic|BindingFlags.Public;
      FieldInfo[] fields = type.GetFields(flags);
      foreach(FieldInfo field in fields)
      {
         if(field.IsNotSerialized)
         {
            continue;
         }
         string fieldName = type.Name + "+" + field.Name;
         object fieldValue = info.GetValue(fieldName,field.FieldType);
         field.SetValue(obj,fieldValue);
      }
      DeserializeBaseType(obj,type.BaseType,info,context);
   }
   //Rest of SerializationUtil
}

With the Type.GetFields method, you can get all of the fields (private and public) declared by a type as well as any public or protected fields available via its own base classes. This isn't good enough because you need to capture all of the private fields available from all levels of the class hierarchy, including name repetition. The solution is to serialize each level of the class hierarchy separately. SerializeBaseType calls a private helper method also called SerializeBaseType, providing it with the level of the class hierarchy to serialize:

SerializeBaseType(obj,baseType,info,context);
The private SerializeBaseType serializes that level and then calls itself recursively, serializing the next level up the hierarchy:
SerializeBaseType(obj,type.BaseType,info,context);

The recursion stops once it reaches the System.Object level. To serialize a particular level, the private SerializeBaseType calls GetFields with a binding flags mask, which instructs it to return all fields defined by this type only and not its base types (BindingFlags.DeclaredOnly), so that as it visits the next level up, it doesn't serialize fields more than once. It also binds only to instance fields and not to static fields because static fields are never serialized anyway. The private SerializeBaseType then calls GetFields and stores the result in an array of FieldInfo objects:

FieldInfo[] fields = type.GetFields(flags);

The solution needs to deal with a class hierarchy in which some levels actually use the Serializable attribute, such as class A in the following example:

[Serializable]
class A
{}
class B : A
{}
[Serializable]
class C : B,ISerializable
{}
Because class A may have some fields marked with the NonSerialized attribute, the solution needs to check that a field is serializable. This is easy to do using the IsNotSerialized Boolean property of FieldInfo:
foreach(FieldInfo field in fields)
{
   if(field.IsNotSerialized)
   {
      continue;
   }
   //Rest of the iteration loop
}

Since different levels in a hierarchy can declare private fields with the same name, the private SerializeBaseType prefixes a field name with its declaring type, separated by a plus sign:

string fieldName = type.Name + "+" + field.Name;
The value of a field is obtained using the GetValue method of FieldInfo and is then added to the info parameter:
info.AddValue(fieldName,field.GetValue(obj));

Deserialization of the base class (or classes) is similar to serialization and is done recursively as well until it reaches the System.Object level. At each level in the class hierarchy, the private DeserializeBaseType retrieves the collection of the fields for that type. For each field, it creates a name by appending the name of the current level to the name of the field, gets the value from info, and sets the value of the corresponding field using the SetValue method of the FieldInfo class.

Securing Serialization

Imagine a class containing sensitive information that needs to interact with partially trusted clients. If a malicious client could provide its own serialization formatters, it could gain access to the sensitive information or it could deserialize an object with bogus state. To prevent abuse by such serialization clients, a class can demand that its clients have the security permission to provide a serialization formatter during link time, using the SecurityPermission attribute with the dedicated SecurityPermissionFlag.SerializationFormatter flag:

[SecurityPermission(SecurityAction.LinkDemand,
   Flags = SecurityPermissionFlag.SerializationFormatter)]
[Serializable]
public class MyClass
{...}
The link-time demand does not trigger a security stack walk, but rather verifies that the formatter has the demanded permission.

The problem with demanding serialization permission at the class level is that it precludes clients that don't have that permission and don't even need to serialize the class from using the class at all. In such cases, it's better to provide custom serialization and demand the permission only on the deserialization constructor and GetObjectData, as shown in the code listing in Figure 12. You can use the SerializationUtil helper class to automate the implementation of the custom serialization.

Figure 12 Get Permission (Close Figure 12)
[Serializable]
public class MyClass : ISerializable
{
   public MyClass()
   {...}
   [SecurityPermission(SecurityAction.LinkDemand,
      Flags = SecurityPermissionFlag.SerializationFormatter)]
   public void GetObjectData(SerializationInfo info,
      StreamingContext context)
   {...}
   [SecurityPermission(SecurityAction.LinkDemand,
      Flags = SecurityPermissionFlag.SerializationFormatter)]
   protected MyClass(SerializationInfo info,StreamingContext context)
   {...}
}

If all you need are the standard .NET Framework formatters, there is a different solution altogether to the malicious serialization client. Use the StrongNameIdentityPermission attribute to demand at link time that only Microsoft-provided assemblies serialize and deserialize your class, as shown in Figure 13.

Figure 13 Restricting Serialization (Close Figure 13)
public static class PublicKeys
{
   public const string Microsoft = "0024000004 ... 44D5AD293";
}

[Serializable]
public class MyClass : ISerializable
{
   public MyClass()
   {}

   [StrongNameIdentityPermission(SecurityAction.LinkDemand,
      PublicKey = PublicKeys.Microsoft)]
   public void GetObjectData(SerializationInfo info,
      StreamingContext context)
   {...}

   [StrongNameIdentityPermission(SecurityAction.LinkDemand,
      PublicKey = PublicKeys.Microsoft)]
   protected MyClass(SerializationInfo info,StreamingContext context)
   {...}
}

A drawback to this approach is that now the class cannot be serialized using custom formatters by even benign clients with full trust because those clients will not be signed with the Microsoft strong name. Adding another link-time demand for the client's strong name will actually guarantee that no one could use the class, since no assembly could ever be signed with both strong names. To address this, the .NET Framework 2.0 introduces the demand-choice option; when combined with a link-time demand, it instructs the just-in-time (JIT) compiler that the linking client need satisfy only one of the demanded permissions. If you want to allow either Microsoft or clients with the serialization formatter permission to serialize your class, use a link-time demand-choice on both permissions, as shown in Figure 14.

Figure 14 Link-Time Permission Demand-Choice (Close Figure 14)
[StrongNameIdentityPermission(SecurityAction.LinkDemandChoice,
    PublicKey = PublicKeys.Microsoft)]
[SecurityPermission(SecurityAction.LinkDemandChoice,
    Flags = SecurityPermissionFlag.SerializationFormatter)]
public void GetObjectData(SerializationInfo info,
   StreamingContext context)
{...}

[StrongNameIdentityPermission(SecurityAction.LinkDemandChoice,
    PublicKey = PublicKeys.Microsoft)]
[SecurityPermission(SecurityAction.LinkDemandChoice,
    Flags = SecurityPermissionFlag.SerializationFormatter)]
protected MyClass(SerializationInfo info,StreamingContext context)
{...}

Conclusion

On the surface, serialization is a seemingly simple problem to solve, while in fact it poses a series of interdisciplinary challenges, from versioning to generics and security. Although .NET does a great job for most simple serialization cases, you may have to improve on the basic infrastructure for enterprise applications. Luckily, the extensibility of the .NET serialization system makes this possible. The techniques discussed here should put you on the road to more advanced serialization problems. Furthermore, the next generation of serialization (as part of "Indigo, " code name for the communications infrastructure of the next version of Windows®, known as "Longhorn") will offer a compatible programming model that will support many of the programming models and techniques presented in this article.

Juval Lowyis a software architect providing .NET architecture consultation and advanced training. He is also the Microsoft Regional Director for the Silicon Valley. This article contains excerpts from his upcoming book, Programming .NET Components 2nd Edition (O'Reilly). Contact Juval at http://www.idesign.net.


BizTalk 2004: Automate Your Business Processes with Human Workflow Services and Visual Studio

BizTalk 2004

Automate Your Business Processes with Human Workflow Services and Visual Studio

Chris Whytock
This article discusses:
  • The new knowledge worker features in BizTalk Server 2004
  • Human Workflow Services (HWS)
  • Developing and customizing actions
  • Creating and executing a business process in HWS
This article uses the following technologies:
BizTalk Server, Visual Studio, C#
Code download available at: HumanWorkflowServices.exe (1,118KB)
Contents
Workflow Problem: The Interview Process
Create a Process Model
Basic Concepts of Human Workflow Services
Creating the Actions
Developing the Approval Action
Developing the Communicate and Routing Actions
Customizing Actions for Your Environment
Dynamically Creating the Interview Workflow
Creating a Predefined Interview Workflow
Default Parameters
Loop Transitions
Putting It All Together
Next Steps
Conclusion

I
n February of this year, Microsoft released the latest version of its BizTalk® server software, BizTalk Server 2004. BizTalk is the Enterprise Application Integration (EAI) tool that enables businesses to connect their diverse applications and graphically define processes that use the services provided by those applications. This enables a business to automate processes that involve multiple software applications and gain all the benefits that process automation provides, such as consistent processing, audit trails and speed and efficiency improvements.

The latest version of BizTalk adds features such as user-configurable business rule support, Business Activity Monitoring to provide increased insight into processes, and Business Activity Services to help manage interactions with trading partners.

BizTalk Server is an excellent platform for automating B2B and EAI workflows. Process-driven workflows send messages between machines and access data. It's the process, not a person, that decides what happens next. Until now, such processes were very difficult to automate using BizTalk Server. Now, with the introduction of Human Workflow Services (HWS) in BizTalk Server 2004, this class of workflow automation is a very real possibility.

Workflow Problem: The Interview Process

Every company has some type of interview process. This procedure usually requires coordination and communication of information between many people, some of whom are not privy to the overall process. Using HWS to model this process, participants are guided by the system to perform the correct steps in the correct order. This predefined process may be dynamically modified at run time to account for any of the one-off irregularities that typically occur in any human-driven process. Every time the process is run, all information relating to that run of the process will be stored for later retrieval for statistical analysis or legal documentation.

Create a Process Model

Let's explore the steps needed to automate this process. First you need to model your process—break it down into steps and then figure out what type of activities are involved in each step and which users or user types will be involved in each. A typical interview process probably starts with a recruiter choosing a candidate, deciding which job in the company the candidate fits, and assigning the candidate to an assistant. The assistant would then contact the candidate to verify his interest in the position, after which the hiring manager would be notified that a candidate is in the pipeline. Next, the candidate would be screened by a phone interviewer. Following the screening, the assistant would schedule an interview with the candidate and set up a set of interviewers to participate. Feedback from each of the regular interviewers will be collected at some point, and if the candidate passes those interviews, the applicant proceeds to the final interview with the hiring manager. Figure 1 shows a diagram representing the workflow of the interview process.

Figure 1 The Interview Process
Figure 1 The Interview Process

Basic Concepts of Human Workflow Services

As the interview example reveals, business processes are simply an ordering of disparate activities. The two main HWS objects used to create business processes are activity flows, which are individual instances of one business process or another, and actions, which are the disparate activities added to activity flows in order to perform one of the disparate activities in the process. In HWS, to indicate the next step in a process, actions are added to activity flows by human participants. The result is a workflow that represents a human-driven process.

In addition to actions and activity flows, HWS introduces the concept of a task. A task is a message sent from an action to a human participant in the process. The message can be anything, such as a notification that a certain activity has occurred or an instruction telling the person to perform the next step in the process. Once a person receives a task, the person can either respond to the task, add a new action to the activity flow after the task, or both. HWS does not restrict the form a task message takes; it could be e-mail, a Windows® SharePoint® Services task, or even an instant message through MSN® Messenger.

Here's how business processes in HWS work. First, an activity flow is created by starting an action. Second, that action sends out one or more task messages to users of the system. Third, those users can either respond to those task messages or add new actions to the activity flow, after which the business process continues as more and more of these actions are activated in the process over time.

Creating the Actions

Since an activity flow is created by instantiating one or more actions, you must first populate your system with the set of actions required to complete your business process before you can create the activity flow for that process. At first glance, you might think that since the interview process described earlier has a dozen steps, you'll need to write twelve actions. You could certainly do that, but then each process you want to automate would come at a relatively high price.

Actually, a number of the steps involve the same basic activity performed in slightly different ways. For example, the actions of assigning the candidate to an assistant and notifying a phone interviewer that they must conduct an interview are both simply communicating a message to a single user. In fact, there are only three different types of generic actions required to perform the interview process: a Communicate action, which notifies one or more users about something (in Figure 1 those are the steps in purple); an Approval action, which will poll one or more users for a positive or negative answer to a question and collect feedback (red steps in Figure 1); and a Routing action, which will automatically make a branch decision based on what happened in the past (orange steps in Figure 1). Notice that the "Does the candidate fit?" question is not actually a step. This is a human decision which will have to be made by the recruiting assistant after the phone interview feedback since many factors that are not easily quantified (such as how badly the company needs the position filled and the estimation of the phone interviewer) will affect the decision.

Assuming this is the first Human Workflow Services procedure developed for the company, you will have to create each of these particular actions from the beginning. By developing generic rather than specific actions, the more processes you end up automating using Human Workflow Services, the more you'll find that you already have actions defined that will perform most of the activities in the new process.

Developing the Approval Action

The first thing you need to do when developing an action is design the message schemas that the action will use to both send and receive task messages. HWS has defined three extensible schemas for the three types of messages that actions will use: the Activate schema, which defines the message that will start the action; the Task schema, which defines the form of the task messages sent to the user; and the Synchronize schema, which defines how one action talks to the next action in the workflow. Looking at the interview process, you can see that the Approval action will require two different tasks: one task to communicate the issue to the users evaluating the issue and another task to notify a user of the end result of the Approval action.

To create the Approval action through Visual Studio®, you need to create a new BizTalk project using the Human Workflow Project template. Since you need two different task schemas, you need to replicate the original task schema. The easiest way to do this is to make a copy of the file in the file system and then add the copy file as an existing item to your project. This is easier than the manual approach since each HWS schema contains a number of required HWS fields that would need to be replicated.

Now you need to edit the ActionSection of each of the schemas. The ActionSection is the user-defined section of each HWS message type, which is where the designer can add any elements that are required for the action. Since the Approval action needs to send the text of an issue for approval to multiple users, in the activate schema you add four items: a string element for the issue to be evaluated; an unbounded element for the users who will be evaluating the issue; an integer field for the percentage of approvers who must approve the issue; and an element to identify the user who receives the final answer. Your ActionSection will then look like Figure 2. For the task messages, in the first type that communicates the issue, you add an element for the issue text, the user who will approve the issue, an approval field, and a comments field. In the summary task message, you add the issue field, a summary target field, a field for all comments submitted by the approvers, and finally a field for the final approval answer. One important requirement of task messages is that they carry data to and from the user. This is why you have the approval field built into the first task message schema—so that the user can send their approval decision back to the action.

Figure 2 ActionSection
Figure 2 ActionSection

Now you need to add the logic to the action orchestration. Every action created from the HWS action project template contains a scope named ScopeAllActionSpecificLogic. All action-specific logic should be inserted within this scope. The Approval action should next perform three stages: send out all the messages to request approval decisions, wait until it receives completed responses for each message, then compose a summary message and send that out. If you examine the ApprovalAction.odx file, you can see these stages occur in the Send_Approval_Requests scope, the Collect_Responses_Loop while loop, and the Send_Summary_Task scope.

Once the action logic has been developed, three more steps are required to access the action in HWS. First, like all orchestrations, the action must be deployed, enlisted, and started in BizTalk. Next, you will need to register the action with HWS using the HWS admin console. Finally, you must add a constraint to the HWS system to allow your users to activate the action. The final two steps are explained in detail in the BizTalk Server 2004 help files, under the "Enabling an Action for Use in Human Workflow Services" topic, and the first step is covered under the "Managing Orchestrations" topic.

Developing the Communicate and Routing Actions

If you examine the CommunicateAction project in the sample code, you'll see that the messages and action logic of the communicate action is similar to Approval. The code and schemas should be mostly self-explanatory. The routing action, though, is very different. Due to the complex nature of the behavior that the action needs to implement, there would be no easy way to implement the logic using orchestration shapes. The only option left was to write code to implement the action logic and have the orchestration call out into the code to perform its logic. You can see the code in the RouteActionLogic project (specifically the RouteActionLogic.cs file), which at a high level reads a specific value from the sync message received and performs one of the actions encoded in the activate message based on the value. Either method of implementing action logic can be used; however, orchestration shapes are usually much easier for other developers to understand.

Customizing Actions for Your Environment

You may notice that all the actions in the sample code send out rather generic XML messages through their ports. How do these messages make it to the users? The answer is by using the BizTalk pipeline system. Pipelines are designed to shape and transmit the XML messages from an orchestration into meaningful data for the user. This allows for an elegant separation of delivery logic from orchestration logic, making the action orchestration even more reusable. For more detail see the "Processing Messages: Pipelines" topic in the BizTalk documentation. A sample customized pipeline is included in the code download for this article. You can find custom pipelines in both the ApprovalAction and CommunicateAction projects (the .btp files). The bulk of the work these pipelines perform is done by the custom pipeline component created in the SMTPPipelineComponent project. This is a highly configurable component that will process a message, translate the usernames of the targets in Active Directory®, and send out e-mail versions of the message via the .NET SMTP library.

Another benefit of generically designed actions is that by simply writing a new pipeline component, I can change the way I communicate tasks to users from the current AD/e-mail-based notification medium to Windows SharePoint Services tasks or even a Word document on a public share.

Dynamically Creating the Interview Workflow

Now that the actions required for our interview process have been created and added to the system, it is possible to execute the process using what HWS calls ad hoc composition. In HWS, the simplest way to execute a business process is to have the participants in the process dynamically build the process. For the interview process, the recruiter starts a new process and initiates a communication action that directs the assistant to contact the candidate; then the assistant would initiate an approval action to instruct the phone screener to conduct an interview and submit feedback, and so on.

Ad hoc processes are composed in HWS through the HWS Web service. At each stage in the process, the user would call the Web service to retrieve the list of possible actions he or she could initiate, then format the activation message for the desired action and add a new instance of that action to the activity flow. The code for initiating the first step of the interview process this way is shown in the C# sample in Figure 3.

Figure 3 Initiating the Ad Hoc Interview Process (Close Figure 3)
Hws.HwsService hws = new Hws.HwsService();
hws.Credentials = CredentialCache.DefaultCredentials;

Guid interviewProcessId = Guid.Empty;
interviewProcessId = hws.GetNewActivityFlowID();

Hws.Activity[] activities = hws.GetActivityList(interviewProcessId,
  Guid.Empty, Guid.Empty, "", null);

Hws.Activity communicateAction = null;
foreach( Hws.Activity activity in activities )
{
  if( activity.Name == "Communicate" )
    communicateAction = activity;
}

Hws.ActionParameters parameters = hws.GetActionParameters(
 communicateAction.ActionTypeID, null);

XmlDocument activateMsg = new XmlDocument();
activateMsg.Load(new FileStream("CommunicateAction_Activate.xml",
  FileMode.Open));

activateMsg.SelectSingleNode("//ActionSection/MessageBody").InnerText =
  "Please contact Bob Smith (bsmith@hotmail.com) regarding job# 1145";
activateMsg.SelectSingleNode("//ActionSection/Subject").InnerText =
  "Contact Candidate";
activateMsg.SelectSingleNode("//ActionSection/Target").InnerText =
  @"Domain\username";
activateMsg.SelectSingleNode("//ActionSection/ExpectResponse").InnerText
  = "false";

parameters.ParametersDoc = activateMsg.OuterXml;

hws.AddActionToActivityFlow(interviewProcessId, Guid.Empty,
  Guid.Empty, false, parameters, null);

The benefit of ad hoc workflows is that you can get them up and running quickly and easily after developing the requisite actions. This can shorten the development time required for automating a given process, and it can also be used for trying various process configurations. The caveat is that the process is completely defined by the participants. In some processes, this may be a benefit or even a requirement. Certain processes demand a more rigid structure that the participants should not be able to alter. HWS also supports such predefined workflows.

Creating a Predefined Interview Workflow

In HWS, a predefined workflow is referred to as an activity model. Activity models provide a way to structure a workflow prior to execution, and serve up other features such as default parameters and advanced transitions not available in an ad hoc workflow. The basic building blocks of an activity model are steps and transitions, each of which is a class type in the activity model designer API. The API can be found in the Microsoft.BizTalk.Hws.ActivityModelDesignerAPI assembly in the Developer Tools directory of your BizTalk installation. Constructing a workflow can be as simple as creating two-step objects and connecting them with a transition object, as shown in Figure 4. To create more complex workflows, you repeat the same logic, first creating one or more steps, and then connecting those new steps with transitions to steps that are already part of the activity model. This logic can be seen in the creation of the interview activity model, in the AMCreationTool.cs file. There are a few extra features available from the designer API exploited in the creation of the interview activity model, which can enhance a workflow.

Figure 4 Constructing a Workflow (Close Figure 4)
DesignManager dm = new DesignManager();
dm.Connect(System.Environment.MachineName, "BizTalkHwsDb");

ActivityModel interviewProcess;
designManager.CreateActivityModel(out interviewProcess);

Step rootStep;
interviewProcess.AddStep(ActionId, out rootStep);

Step childStep = null;
interviewProcess.AddStep(ActionId, out childStep);

Transition transition;
interviewProcess.AddTransition(
    rootStep.Id, childStep.Id, false, "", "", out transition);

Default Parameters

Generic actions usually require more runtime parameters to fine-tune their functionality to the situation. Requiring the user or client surface to supply all those parameters can be cumbersome, as you saw in the previous ad hoc workflow. To alleviate the burden, HWS allows you to specify default values for some or all of a step's parameters at design time. If you refer to the InterviewProcessCreationApp.cs file in the download, you will find sections of code which set up the default parameters for each step of the process.

// Add a transition from the "Contact Candidate" step to the "Conduct
// Phone Interview" step.  Use the schema of the summary task to make
// sure the next step can only be performed when the summary is sent out.
interviewProcess.AddTransition(
    contactCandidate.Id, phoneInterview.Id, false,
    "http://tempuri.org/CommunicateAction_PrimaryTask", "",
    out newTransition);
When the user asks the HWS Web service for the parameters for this step (via the GetActivityModelParameters method), the ParametersDoc property of the return ActionParameters structure will be pre-populated with an XML string containing the parameter values supplied at design time. The only information this user will need to supply is the ID of the target user and a URL string to substitute for the "{0}" marker in the MessageBody property.

Loop Transitions

In many processes, certain steps may be rerun. In our interview process, a problem may occur when the candidate chooses an interview day that not enough interviewers can commit to. To account for this, the process contains a loop transition from the Schedule Group Interview step back to the Schedule Interview with Candidate step.

Once you have a complete activity model, the final steps are to supply the Name and Description fields of the ActivityModel class instance you've created, and then upload the activity model to your HWS system through the SaveActivityModelToSystem method on the DesignManager class.

Putting It All Together

Now that you have a complete activity model uploaded to your HWS system, it's time to put it to use. The sample code for this article includes a fully functional UI front end for the interview process activity model. It begins with the Interview Process Kickoff Tool, a Windows Forms-based app that initiates an interview process. From then on, all participants in the process interact via the various Web Forms supplied by the InterviewWebApp project. Most of the code in these apps is pretty straightforward. While they are all tailored to one specific action or another, each follows the same basic logic, shown in Figure 5.

Figure 5 Web Form Logic (Close Figure 5)
_HwsService = new Hws.HwsService();
_HwsService.Credentials = CredentialCache.DefaultCredentials;

Task[] tasks = null;

TaskFilter filter = new TaskFilter();
filter.StartTimeEnd = DateTime.MaxValue;
filter.StartTimeStart = DateTime.MinValue;

//Discover tasks assigned to the user
tasks = _HwsService.GetAllTasksForUser(filter, candidateEmail);
Activity[] activities = null;
activities = _HwsService.GetActivityList(tasks[0].ActivityFlowID,
    tasks[0].ActionInstanceID, tasks[0].TaskID, null, null);

XmlDocument taskXml = new XmlDocument();

//Pick an activity and query HWS for params
taskXml.LoadXml(_HwsService.GetTaskMessage(tasks[0].TaskID, null));
Guid _ActivityModelInstanceId = new Guid(taskXml.SelectSingleNode(
    @"//HwsSection/ActivityModelInstanceId").InnerText);
Guid _PartentActionInstanceId = new Guid(taskXml.SelectSingleNode(
    @"//HwsSection/ActionInstanceId").InnerText);

//Populate the parameter list
ActionParameters[] _Parameters = _HwsService.GetActivityModelParameters(
    activities[0].ActivityModelTypeID, activities[0].ActivationBlockID,
    null);
XmlDocument paramsXml = new XmlDocument();
paramsXml.LoadXml(_Parameters[0].ParametersDoc);
paramsXml.SelectSingleNode(
    @"//ActionSection/Issue").InnerText = "My Issue";
paramsXml.SelectSingleNode(
    @"//ActionSection/SummaryTarget").InnerText = "User1";

_Parameters[0].ParametersDoc = paramsXml.InnerXml;

//Activate the Web service
_HwsService.AddActivationBlockToActivityFlow(tasks[0].ActivityFlowId,
    ActivityModelInstanceId, ParentActionInstanceId, tasks[0].TaskID,
    false, _Parameters, null);

The logic at the client site mainly consists of querying the HWS Web service to see what the current user can do and discovering any tasks assigned to the user; picking one of the activities and querying the HWS service for the parameters; populating the parameters that do not provide defaults or overriding those defaults; and finally activating the activity via the Web service. One of the nice features of HWS, as you can see, is that workflows become discoverable. There's no need for a user to keep track of a specific message that told them to do something, since their client UI can ask the HWS system for any information they require.

Next Steps

Now that you have a fully functioning automated interview process, where do you go from here? You really can see the power of the reusable actions created in developing the interview process activity model. Consider if you now wanted to model an expense report approval process in your company, having seen the cost savings that the automated interview process has brought. A basic expense approval process would send the expense report to a manager for approval, the manager would approve, and then send the expense report to accounting for approval. Accounting would then send an e-mail notifying the original submitter of the result and then the funds would appear in his or her paycheck.

While this doesn't resemble the interview process you just created, the steps in the expense process do closely resemble the generic actions you have already created. It would be rather trivial to assemble a new and completely different activity model that is based on the expense report process using the already existing actions. The most difficult part of the process is complete; all you need to do now is compose the activity model and create a UI front end to guide the user. Since the HWS Web service provides a discovery mechanism, you could design a generic UI to automatically handle any new activity models you introduce to your system.

Conclusion

This article has outlined the capabilities of the new Human Workflow Services feature in BizTalk Server 2004. HWS provides a flexible, reusable surface for automating predefined workflows, creating ad hoc workflows, and automatically tracking them at run time. Taking advantage of these new features in BizTalk Server 2004, you will be able to regulate many processes you could not have previously, and will be able to track each process efficiently.

Chris Whytock is a developer for the Microsoft E-Business Servers team. Hailing from Halifax, Canada, he has crossed the continent in his eight years developing software in the financial, media, and business server industries. You can reach Chris at cwhytock@microsoft.com.


Unit Testing: Mock Objects to the Rescue! Test Your .NET Code with NMock

Unit Testing

Mock Objects to the Rescue! Test Your .NET Code with NMock

Mark Seemann
This article discusses:
  • The role of Mock objects
  • How to acquire and use NMock
  • Overview of a shopping sample
  • Design guidelines for testability
This article uses the following technologies:
Testing, .NET, C#
Code download available at: NMock.exe (137KB)
Contents
Understanding the Unit Testing Problem
Mock Objects to the Rescue
The Shopping Sample Library
NMock to the Rescue
Setting Up Results
Setting Expectations
Let the Mock Participate in the Test
Designing for Testability
Conclusion

H
ave you ever considered implementing unit tests for a project but dismissed the idea because the module you wanted to test either had too many dependencies or it was so difficult to isolate the unit itself that the tests started to look like integration tests? Or perhaps setting up and configuring all the external dependencies was impractical or downright impossible to achieve within your time constraints or budgets?

Mock objects can help you overcome these types of obstacles and, as an added benefit, can help enforce good design practices. In this article, I will describe how to use a publicly available dynamic mock object library called NMock, and will provide you with some of the design guidelines that I have found to be of practical use in my own work.

NMock is a great tool, but unfortunately it doesn't currently come with documentation or samples, so it's difficult to figure out what to do with it, or even for what use it is intended. I will try to illustrate the purpose and use of NMock by showing you how to write unit tests for a business logic library which is dependent on a data access layer (DAL).

Understanding the Unit Testing Problem

Before we embark on our journey through the sample code I've prepared for this article, it's necessary to understand the problem: unit testing libraries that have a complex set of dependencies. It is best to design libraries to have a minimal set of dependencies, but sometimes this is not practical, especially when the library has not been designed with testability in mind. When unit testing libraries with a complex set of dependencies, these dependencies may require difficult or time-consuming setup procedures which are detrimental to the test process. Consider the case of a business logic library which uses a data access component to communicate with a database server, as illustrated on the left side of Figure 1.

Figure 1 Mock Testing Concept
Figure 1 Mock Testing Concept

Although the binding between the business logic library and the data access component is interface-based, the business logic library still needs to be able to access properties and methods on the data access component during unit testing, and that's where the trouble begins.

The data access component needs a connection to a database to work. This means that you need to configure the component to access the right database, and you almost certainly need to set up a test instance of the database so that you don't risk corrupting a production database during testing. If you are using SQL Server™ 2000, this may be a fairly manageable task, but what if the data access component provides access to a mainframe-based database? In the latter case, setting up a test instance of the database may be impractical at best. And even when using SQL Server, time may be a prohibitive factor.

However, configuring the DAL for test is not the only problem in this scenario. If your data access component is part of your development project, it may not even exist yet, and even if it does, it most likely will still be under development with a number of bugs, both known and unknown. In this case, you risk possible unit test failure due to bugs in the DAL even if the code under test has no defects. In other words, in such a case the library under test would not be properly isolated from other modules, so at that point the test is actually no longer a unit test.

Mock Objects to the Rescue

Traditionally, unit testing terminology has included the concepts of drivers and stubs. During your career as a software developer, you have probably created quite a few of these drivers and stubs for each module you've needed to test.

A driver is a piece of software that is written with the sole purpose of accessing properties and methods on a library to test the functionality of that library. A stub is a piece of software whose only purpose is to provide the library under test with an implementation of any modules it may need to communicate with to perform its work.

Figure 2 Unit Testing Concepts and Implementation
Figure 2 Unit Testing Concepts and Implementation

As illustrated in Figure 2, this concept translates well into practical terms. With the tools available today, you have a framework for implementing both drivers and stubs in a dynamic and flexible way. As an implementation of a driver, I use NUnit (currently at version 2.2), and for stubs I use NMock (currently at version 1.1) to create dynamic mock objects at run time. If you refer back to the right side of the diagram in Figure 1, a mock object implementing the IMyDataAccess interface has replaced the intended production DAL. NMock can create such a mock object at run time.

It is worth noting that NMock uses reflection to create mock implementations of interfaces at run time, so to use NMock it is necessary to code against interfaces instead of implementations. As this is already a well-regarded best practice, NMock helps to enforce this important design technique.

The Shopping Sample Library

The download for NMock consists of an assembly that can be used right away; no installation or configuration is necessary (see http://www.nmock.org). To illustrate the use of NMock, I'll show you how to write unit tests for a very simplified shopping basket library which uses a DAL. The library uses the DAL by accessing a data access object that implements the IShoppingDataAccess interface, which is shown here:

using System;

namespace NMockExample.Shopping
{
    public interface IShoppingDataAccess
    {
        string GetProductName(int productID);
        int GetUnitPrice(int productID);
        BasketItem[] LoadBasketItems(Guid basketID);
        void SaveBasketItems(Guid basketID, BasketItem[] basketItems);
    }
}

The Basket class illustrated in Figure 3 contains functionality to perform operations such as calculating the subtotal, saving the basket to the data store, and adding a new item to the basket. You may have noticed that the constructor takes as a parameter an object that implements the IShoppingDataAccess interface. The class will use this object to perform data access operations. For more information on this pattern, which is known as "Inversion of Control," see Martin Fowler's article on the subject at Inversion of Control Containers and the Dependency Injection pattern.

Figure 3 Basket Class (Close Figure 3)
using System;
using System.Collections;
using System.Configuration;

namespace NMockExample.Shopping
{
    public class Basket
    {
        private ArrayList basketItems_;
        private Guid basketID_;
        private IShoppingDataAccess dataAccess_;

        public Basket(IShoppingDataAccess dataAccess)
        {
            Initialize(dataAccess);
        }

        public void AddItem(BasketItem item)
        {
            basketItems_.Add(item);
        }

        public void Save()
        {
            dataAccess_.SaveBasketItems(basketID_,
                (BasketItem[])basketItems_.ToArray
                (typeof(BasketItem)));
        }

        public decimal CalculateSubTotal()
        {
            decimal subTotal = 0;
            foreach(BasketItem item in basketItems_)
            {
                subTotal += item.GetPrice();
            }
            return subTotal;
        }

        private void Initialize(IShoppingDataAccess dataAccess)
        {
            dataAccess_ = dataAccess;
            basketItems_ = new ArrayList();
            basketID_ = Guid.NewGuid();
        }
    }
}

The Basket class contains an internal list of BasketItem objects, which are shown in Figure 4. Given a product ID, a BasketItem object uses its internal data access object to provide product information such as unit price and product name.

Figure 4 BasketItem Class (Close Figure 4)
using System;

namespace NMockExample.Shopping
{
    public class BasketItem
    {
        private decimal unitPrice_;
        private int productID_;
        private int quantity_;
        private IShoppingDataAccess dataAccess_;
        private string productName_;

        public BasketItem(int productID,
            int quantity, IShoppingDataAccess dataAccess)
        {
            Initialize(productID, quantity, dataAccess);
        }

        public decimal UnitPrice { get{ return unitPrice_; } }

        public int ProductID
        {
            get { return productID_; }
            set
            {
                productID_ = value;
                unitPrice_ = dataAccess_.GetUnitPrice(productID_);
                productName_ = dataAccess_.GetProductName(productID_);
            }
        }

        public int Quantity
        {
            get { return quantity_; }
            set { quantity_ = value; }
        }

        public string ProductName { get { return productName_; } }

        public decimal GetPrice()
        {
            return unitPrice_ * quantity_;
        }

        private void Initialize(int productID,
            int quantity, IShoppingDataAccess dataAccess)
        {
            dataAccess_ = dataAccess;
            ProductID = productID;
            Quantity = quantity;
        }
    }
}

As you can see, most operations in this Shopping sample library make use of data access methods, which means that I need a way to isolate the Shopping library from the DAL if I want to write meaningful unit tests—tests that truly reflect the correctness of the method under test and that are not affected by one of its dependencies and skewing the results.

NMock to the Rescue

All classes in the Shopping sample library expose a way to externally define the data access implementation they use, so we can use NMock to provide a dynamic mock implementation of IShoppingDataAccess to the Shopping classes.

Let's start by looking at a very simple unit test which demonstrates how to create a mock instance of IShoppingDataAccess using NMock. Before writing any code, it is necessary to add a reference to NMock.dll and add a using statement for the NMock namespace. The following code is a very simple unit test of the Basket class's Save method, though at the moment no assertions are being performed:

DynamicMock dataAccess = new DynamicMock(typeof(IShoppingDataAccess));
Basket b = new Basket((IShoppingDataAccess)dataAccess.MockInstance);
b.Save();

To create a mock object with NMock, instantiate a new DynamicMock object, passing to its constructor the type you want it to implement. The mock object itself doesn't implement the type but is an object which contains information about the behavior of the mock object and can emit an implementation of the specified type. This is exactly what happens in the next line of code, where the Basket constructor is passed a reference to the DynamicMock object's MockInstance property.

DynamicMock uses reflection to emit an implementation of the desired type and expose it through the MockInstance property—in this case an implementation of IShoppingDataAccess. As MockInstance returns an object of type System.Object, it is necessary to cast the returned reference to the desired type.

In this case, the dynamically created implementation of IShoppingDataAccess is passed to the Basket constructor, and when the Save method is called, the Basket object can call the SaveBasketItems method on its internal IShoppingDataAccess member. Nothing was specified in the setup code, so the DynamicMock object doesn't know what to do with the call to SaveBasketItems. In this case, it doesn't matter because the method returns void. As such, this particular unit test only really tests that the Basket constructor and the Save method can be executed without exceptions.

Although this is not the most exciting unit test, it already demonstrates how to dynamically create a mock implementation of an interface. Had I passed a null reference to the Basket constructor instead of the mock object, the code would have thrown an exception when the Save method of the Basket class was called.

Setting Up Results

The default behavior of a mock instance is to return null from every method call that returns a reference type and to return the default value from every method call that returns a value type, so using it in the fashion I just illustrated provides very limited usefulness. It may be adequate for void methods, but falls short when you need to test a method with a return value.

Happily, the DynamicMock class allows me to modify the behavior of its MockInstance property. Before I pass the mock instance to my test target, I can call some methods on the DynamicMock class, which will instruct the mock instance to behave as I need it to.

To illustrate this, I wrote a simple unit test to test the BasketItem constructor. If you examine the code for the constructor in Figure 4, you will notice that during construction, the BasketItem class's ProductID property is being set, which results in two calls to the DAL. Both calls return a value, so if I had repeated my previous action, a NullReferenceException would have been thrown by the call to GetUnitPrice. When dealing with methods with return values, the default behavior of the mock instance isn't sufficient anymore. It is necessary to instruct the mock object about what to do, as shown in Figure 5.

Figure 5 Setting Up Results with NMock (Close Figure 5)
DynamicMock dataAccess = new DynamicMock(typeof(IShoppingDataAccess));

dataAccess.SetupResult("GetUnitPrice", 99, typeof(int));
dataAccess.SetupResult("GetProductName", "The Moon", typeof(int));

BasketItem item = new BasketItem(1, 2,
    (IShoppingDataAccess)dataAccess.MockInstance);

Assert.AreEqual(99, item.UnitPrice);
Assert.AreEqual("The Moon", item.ProductName);
Assert.AreEqual(198, item.GetPrice());

Note that after I've created the DynamicMock object, but before I begin using it in my target library, I use the SetupResult method to modify the behavior of the mock instance. I configure it so that every time it receives a call to the GetUnitPrice method of IShoppingDataAccess, it should return 99, and every time it receives a call to GetProductName, it should return "The Moon." The SetupResult method is defined like this:

void SetupResult(System.String methodName,
    System.Object returnVal, params System.Type[] argTypes)

The first two parameters supply the method name and return value, respectively. The next parameter is an array of types that will be used to identify which method was called. Why is this necessary? Imagine for a moment that there are overloaded versions of the GetUnitPrice method, one taking as a parameter an integer value, and the other taking a System.Guid value. If the parameter list wasn't supplied to the DynamicMock object, it wouldn't know which one of the two GetUnitPrice methods I meant. To uniquely identify the method, I need to supply DynamicMock with enough information to allow it to infer the method's signature.

By calling SetupResult as I do here

SetupResult("GetUnitPrice", 99, typeof(int));
I indicate that when the mock instance receives a call to the GetUnitPrice method that takes a single integer argument, it should return a value of 99. This allows DynamicMock to uniquely identify the method in question.

The last three lines of the unit test are just standard NUnit assertions to test that the newly created BasketItem object contains the expected data, and that it calculates the correct price.

Setting Expectations

The SetupResult method is still a rather blunt instrument. Any call to the configured method is answered with the same return value, no matter what parameters are used in the method call. If I want to test that the Basket class calculates correct subtotals when containing different items, I can't easily use the SetupResult method. If I tried to add two different items to the Basket object, both items would get the same unit price, as the call to GetUnitPrice would return whatever I defined with SetupResult.

To solve this problem, there are other methods in the DynamicMock class I can use, as illustrated in Figure 6. In this test, I add two different BasketItem objects to the Basket object. During object instantiation, each BasketItem object will call the GetUnitPrice and GetProductName methods to retrieve information from the DAL.

Figure 6 Setting Up Expectation with NMock (Close Figure 6)
DynamicMock dataAccess = new DynamicMock(typeof(IShoppingDataAccess));
dataAccess.ExpectAndReturn("GetUnitPrice", 99, 1);
dataAccess.ExpectAndReturn("GetProductName", "The Moon", 1);
dataAccess.ExpectAndReturn("GetUnitPrice", 47, 5);
dataAccess.ExpectAndReturn("GetProductName", "Love", 5);

Basket b = new Basket((IShoppingDataAccess)dataAccess.MockInstance);
b.AddItem(new BasketItem(1, 2,
    (IShoppingDataAccess)dataAccess.MockInstance);
b.AddItem(new BasketItem(5, 1,
    (IShoppingDataAccess)dataAccess.MockInstance);
b.Save();

decimal subTotal = b.CalculateSubTotal();
Assert.AreEqual(245, subTotal);

To configure the mock instance to return appropriate values for different parameter values, I use the ExpectAndReturn method. This method looks similar to the SetupResult method, but has a few important differences. The ExpectAndReturn method configures the mock instance to expect one call to the method, with the supplied parameter values. Instead of using parameter types to identify the target method, the method signature is inferred from the supplied parameter values.

When I use the following parameters

ExpectAndReturn("GetUnitPrice", 99, 1);
the mock instance will return 99 when the GetUnitPrice method is called with a parameter value of 1. If the parameter value is different from 1, a VerifyException is thrown by NMock. Additionally, since the code in Figure 6 defines the next expected call to GetUnitPrice to have a parameter value of 5, a VerifyException will be thrown if the method isn't called with that parameter. Since the mock instance is told to expect a total of two calls to GetUnitPrice, any third call would also result in an exception, even in cases where the parameter is 1 or 5.

Let the Mock Participate in the Test

As you've probably already noticed, the ExpectAndReturn method is a lot less forgiving than the SetupResult method. In my opinion, this is a good thing because it allows you to add a whole new level of control to your tests.

Consider a test where you use SetupResult. Aside from the obvious disadvantage that all calls to the same method return the same value, you don't get much validation out of the mock object. Regardless of whether your target library called a method zero, once, or many times, you wouldn't notice the difference. Conversely, you can use ExpectAndReturn to validate that a method was called the expected number of times, with the expected parameters, and in the expected order. With this approach, a unit test not only has the opportunity to test that a module returns the correct values, but also that it communicates with any external dependencies as designed. In fact, with some modules where most of the implementation is internal, this may very well be the only way you can perform tests in any meaningful way.

Thus, letting the mock object itself participate in the test is a very good idea, so I'll point out a few ways you can make NMock even less forgiving of the unexpected than its default settings allow.

If you take a good look at the unit test code in Figure 6, you may notice something odd: I didn't set an expectation for a call to SaveBasketItems, and even though the Save method is called on the Basket class, the test succeeds. This happens because SaveBasketItems returns void, so NMock can more or less just ignore the call. You can (and should) change this behavior by turning on the Strict option of the DynamicMock class, like this:

dataAccess.Strict = true;
The Strict property defaults to false, but if you change it to true, any call which does not have an expectation set will cause a VerifyException to be thrown. In the case of the unit test in Figure 6, turning on the Strict option will suddenly cause the unit test to fail, as it should. However, in this case the unit test fails because the test code is faulty, not because there is a bug in the test target, so the test code should be corrected.

It may seem obvious to use ExpectAndReturn to set up an expectation for the call to SaveBasketItems, and in fact you can do that by specifying null as the return value. However, for setting up expectations for void methods, the Expect method is a more intuitive alternative. It is used and behaves just like ExpectAndReturn, except that it doesn't take a parameter that defines a return value.

When setting up the expectation for SaveBasketItems, a problem manifests itself. The SaveBasketItems method takes as its first parameter a Guid, which is the ID of the Basket object. If you examine the definition of the Basket class in Figure 3, you will see that this ID is generated internally during object construction, but is not externally accessible (you may argue that this constitutes one of many design flaws of the Basket class, but it illustrates my point). Since I have no way of knowing the value of the basketID parameter in advance, how can I define the expectation? The answer lies in the NMock Constraint classes:

dataAccess.Expect("SaveBasketItems", new IsTypeOf(typeof(Guid)),
    new BasketItem[]{item1, item2});

Instead of a Guid value, I use a new Constraint instance that instructs the mock object that any Guid value is acceptable. For the other parameter (the array of BasketItems), I still indicate the expected value, so although the basketID parameter is allowed to take any Guid value, the basketItems parameter is still as restricted as I can make it. In fact, NMock expectations always work with Constraint classes when setting parameter expectations. The IsEqual constraint is the default, which is why I haven't demonstrated this syntax until now.

The explicit syntax

dataAccess.ExpectAndReturn("GetUnitPrice", 47, new IsEqual(5));
is equivalent to
dataAccess.ExpectAndReturn("GetUnitPrice", 47, 5);
Even when using the Strict option, you are not guaranteed that the mock object validates that all methods are called the expected number of times. In the case where a method is called too many times, NMock throws an exception the first time this happens, but it doesn't automatically discover that a method was called fewer times than expected. Although it has no way of automatically knowing when the unit test is over, you can inform it manually:
dataAccess.Verify();

Calling the Verify method at the end of your unit test tells the mock object that the test is over and that it should verify that all members were accessed the expected number of times. If a member happened to be accessed fewer times than expected, a VerifyException would be thrown.

To summarize the steps thus far, you set the Strict option to true at the beginning, stick with the expectation-setting methods throughout, and call the Verify method at the end of the unit test. With very little effort, you'll have added a whole new level of validation to your unit tests.

Designing for Testability

As I mentioned previously, I have designed my classes so that I can pass a reference to the data access object in the constructor. When using dynamic mocks, the mock is only defined at run time so it is necessary to provide some way for the dependent module to be assigned from outside the consuming module itself. Of course, this does not mean this is the only way. Usually, I overload the constructor so that I have one constructor in which dependencies can be assigned at run time and one in which these parameters are not included. The second type of constructor will usually become the default. For example, the Basket class could be instantiated in the following way:

Basket b = new Basket();

The constructors which explicitly assign dependent objects at run time would typically be used only for tests. The constructor where the assignment is hidden would be used in production code. Why this distinction?

With the decoupled approach used throughout this article, modules communicate with other classes via interfaces. An interface defines methods, properties, indexers, and events, but it doesn't define constructors. This means that when a class needs to instantiate an interface implementation at run time, it has to make assumptions about the constructor.

In my opinion, the most reasonable assumption is to use the default parameterless constructor. Let's imagine that I want to use a discount calculation object with my shopping basket and I have two different types to choose from: a data-based discount class and an algorithm-based class, as illustrated in Figure 7. The data-based class uses a database to look up discounts defined there, whereas the algorithm-based class uses some internal algorithm to calculate discounts. When instantiating an implementation of the IMyDiscount interface, it doesn't make any sense to pass a reference to a data access module. The algorithm-based class doesn't need it, and at design time I don't know which of the two is going to be used. Using a constructor that passes a reference to a data access object would be making an unreasonable assumption about the implementation.

You may think that this is all very good, but then how can you define which interface implementation is going to be used at run time? The answer, of course, is by configuring the application. Consider a simple application configuration file for the Shopping library like the one shown here:

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
   <appSettings>
       <add key="IShoppingDataAccessType"
    value="NMockExample.DataAccess.ShoppingDataAccess, DataAccess" />
   </appSettings>
</configuration>

In this configuration file, I define that for the implementation of the IShoppingDataAccess interface, my application should use the NMockExample.DataAccess.ShoppingDataAccess type, defined in the DataAccess assembly. The default constructor for the Basket class looks like this:

public Basket()
{
    string typeName =
        ConfigurationSettings.AppSettings["IShoppingDataAccessType"];
    Type t = Type.GetType(typeName);
    IShoppingDataAccess dataAccess =
        (IShoppingDataAccess)Activator.CreateInstance(t);
    Initialize(dataAccess);
}

As you can see, that's all there is to it. You get the string defining the type from the configuration file, create the type by calling the static GetType method of the Type class, and create an instance of the type by calling the static CreateInstance method of the Activator class. The CreateInstance method returns an object of type System.Object, so you need to cast it to the type you need to make further use of it.

One problem with this approach is that these three lines of code can't be unit tested with NMock. Either you need to create an empty stub assembly, or you need to test this code using your production implementation of the interface, which has all the problems I described at the beginning of the article. However, this is very simple code that always follows the same pattern: get the string from the configuration file, get the type using the string, and get an instance using the type. Chances are that if you have validated this code to work once, you shouldn't have any more problems with it. Errors might occur within this code, but that should only happen if the string happens to define a type or an assembly which couldn't be found, loaded, instantiated, or cast, or if there is a problem with the configuration file.

Figure 7 Two Discount Classes
Figure 7 Two Discount Classes

If you are still not convinced that this is a good way to define the interface implementation in production code, consider the alternative. In the example from Figure 7, if you don't want to specify the implementation of IMyDiscount in the configuration file, you need to pass an instance of the implementation to the Shopping library at run time. In effect, this only defers the decision to the caller of the Shopping library. It is now the caller's responsibility to figure out which interface implementation to use.

If the decision is made at design time, it means that you end up hardcoding the implementation that should be used, and you lose most of the advantages of an interface-based design.

If the decision is made at run time, it needs to involve code similar to what I just described. As a result, not much has been gained, but you have now moved the decision to a place far from the module where it is needed.

Consider a shopping app that contains a shopping basket, which may need to calculate discounts. Imagine that the Basket class has been extended a bit to implement an IBasket interface and to consume the IMyDiscount interface. The shopping app will instantiate and manipulate an IBasket object, which may or may not be an instance of the Basket class. If you want to use the Basket class, you need to pass it an IMyDiscount object. One implementation of IMyDiscount (the data-based discount module) needs a data access object, so you need to pass this object to the Basket class as well. Now you have not only made the assumption that you are going to use the Basket class, but also that the Basket class is going to use the data-based discount module. This cannot be changed without modifying the code of both the shopping application itself and the Basket class (depending on what you want to change).

Given all of these potential problems, I find the most practical course to be to let any module itself determine and instantiate the interface implementations it consumes.

Conclusion

In real-world software development projects, it is a rare library that doesn't have some kind of external dependency with which it communicates frequently. When unit testing libraries, you need some kind of stub to simulate those external dependencies. NMock provides you with a valuable tool for achieving this with minimal effort. Although writing unit tests still involves work, NMock offers relief from writing, configuring, and deploying your own stub classes over and over again. Defining mock instances and their behaviors at run time also gives you a degree of flexibility and test validation you probably never would have been able to achieve if you wrote your own stubs for each external dependency.

Each time I found myself in a situation in which it seemed like NMock was holding me back, it turned out to be a good time to pause and rethink my design. Each time I discovered that by making my own design better, I could overcome my problem with unit testing and NMock as well. So NMock is not only a great tool for unit testing, it also forces you to make improvements to your own object-oriented code design.

Mark Seemannworks for Microsoft Consulting Services in Copenhagen, Denmark, where he helps Microsoft customers and partners architect, design, and develop enterprise-level applications. Mark can be reached at mseemann@microsoft.com.


Editor's Note: Old, New, Borrowed, Blue

Editor's Note
Old, New, Borrowed, Blue

A traditional marriage "rule" is that the wedding should feature something old, something new, something borrowed, and something blue. The reasoning behind this escapes us. Why blue? Why have so many successful marriages (Britney Spears comes to mind) ignored this rule entirely? Can you borrow someone's new, blue thing? How long will this analogy go? In any event, this month we have news from all four of these categories. Bear with us as we retrofit our theme to the actual content.

Something Old XML is an old friend of ours. For years we have covered XML, in the pages of MIND, MSJ, and MSDN Magazine, as it has moved from a little-used data description language to the default data format of the Internet. Along the way, we've seen its evolution. SOAP became an XML-based format for making cross-machine process calls. Web services, made possible by XML, have become an integral part of today's computing landscape. SQL Server has included native XML capabilities for several years now, and SQL Server 2005 will include even more XML functionality.

When we started to cover XML on a regular basis in The XML Files column, it was still an emerging technology. Aaron Skonnard has tracked its intricacies for years, providing a broad picture of the places where XML can be used and the APIs that help you use it. However, over the years XML has been so widely accepted as part of the basic infrastructure that it's not the most interesting part of the story anymore.

SOA or service-oriented architecture—a broad term that describes systems that are built around loosely coupled software—is the interesting part. Although the term itself can describe just about anything, we normally achieve SOA through a combination of Web services. The data itself is not coupled with the processing it receives, making the architecture of the system much more flexible. As a reliable, universal data standard, XML is an ideal data transport for service-oriented architecture.

As part of our commitment to covering topics of relevance to our readers, Aaron will be writing about SOA in our new Service Station column. Technically it's not a "new" offering; rather, it's the logical extension of his popular XML Files column. So it's new, but it's not. Updated? Spruced up? Refreshed? Slipstreamed? Well, it's one of those exciting adjectives.

Something New Don't forget to sign up for TechNet Magazine! Only a few weeks remain to reserve your copy of the exciting special security issue. If you or a loved one is an IT Professional, make sure to sign up today. You can also visit our new Web site, http://www.microsoft.com/technet/technetmag, or drop by our Usenet presence at microsoft.public.technet.magazine. If you just want to sign up for your free issue, you can do that too. Just surf to http://www.technetmagsubs.com or go through the TechNet Magazine Web site. You'll be glad you did.

Something Borrowed You may have heard that Microsoft will no longer be offering the Java Virtual Machine after December 2007. However, recent news stories have left some confusion as to the future of the Java language within Windows. To clear this up, this month's { End Bracket } column discusses just what you need to know about the future of this language on the Windows platform. If you have any code that uses the Java language, or you are supporting Java-language products, this is important stuff to know.

Something Blue As part of the TechNet Magazine offering, we recently launched a Usenet group for people to come by and ask questions. From where we sit, it's a seemingly simple process to create a newsgroup.

  1. Request a newsgroup, including its name, purpose, and a brief welcome message.
  2. Wait a little bit.
  3. Receive word that the newsgroup, microsoft.public.technet.magazine, has finally been created! Be asked if you want your welcome message to be posted.
  4. Say yes.
  5. Go look at the newsgroup; find that the welcome message has indeed been posted, but that they've used a humorous variation of your actual name. What did they do? You'll have to trip on by the newsgroup yourself to find out. We're sort of blue about the whole thing.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Patrick Baker, Jay Bazuzi, Kawarjit Bedi, Josh Benaloh, Shawn Burke, Pablo Castro, HongMei Ge, Matthew Gibbs, David Guyer, Chris Hollander, Anson Horton, Vishal Kalra, Jonathan Keljo, Ronald Laeremans, Gerrard Lindsay, Ivan Medvedev, Vick Mukherjee, Adam Nathan, Jim Newkirk, Greg Sundberg, Matt Tavis, Dave Templin, Dan Vallejo, Ravinder Vappula, and Joel West.

Active Directory, ActiveX, BizTalk, DirectX, IntelliSense, JScript, Microsoft, MSDN, MSN, Microsoft Press, SharePoint, Visual Basic, Visual C++, Visual C#, Visual J++, Visual J#, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Nancy Michell

Contents
Easily Port Unix Code and Scripts to 64-Bit Windows
Screen Recording Program
Graphical Business Rules Tool
Create Calculation Formulas Dynamically
Generate Flowcharts from Your Source Code
Unit Tests Drive Code Quality
Automate Object Model Documentation
Defend Against Hackers and Malicious Software
The Bookshelf

Easily Port Unix Code and Scripts to 64-Bit Windows

MKS Inc., a provider of tools for porting Unix/Linux applications to Windows®, has just announced the availability of its new 64-bit products. MKS is shipping a beta version of MKS Toolkit for Enterprise Developers 64-bit edition with support for extended-architecture 64-bit platforms such as AMD64 running Windows 64-bit OS.

Enterprise customers and ISVs now have a single solution that will allow them to migrate both 32-bit and 64-bit Unix/Linux applications for deployment on 64-bit Windows platforms. 64-bit Windows is an ideal platform for applications that require large amounts of memory and high performance such as financial transaction analysis, digital content creation, electronic design automation, and advanced scientific and research applications.

MKS Toolkit for Enterprise Developers allows organizations to maintain a single source code baseline and evolve applications as necessary to take advantage of Windows and .NET technologies. As the industry moves toward 64-bit Windows over the next two to three years, MKS Toolkit will help companies quickly and easily migrate legacy applications to Windows.

http://www.mks.com

Screen Recording Program

Blueberry Consultants Ltd. recently announced the release of its new Windows-based program called BB TestAssistant—a helpful aid for software developers, Web designers, and others who need reliable visual feedback. The program streamlines communications within a testing team, helping to eliminate vague and ambiguous feedback.

BB TestAssistant captures screen activity in movie form with little drain on system resources. It uses a custom compression engine to create movies that are easy to e-mail, enabling the entire testing and development team to share feedback. The program aids in beta testing by automatically collecting the computer configuration information to help pinpoint the source of problems.

Additionally, BB TestAssistant can automatically catch Windows application errors. This program enhances team communications by presenting a visual record of what happened so you don't need to write out lengthy descriptions.

BB TestAssistant features a rolling record mode that runs in the background, allowing testers to work as usual, only saving a movie file once a problem has occurred. Also, BB TestAssistant can be configured to create reports automatically, once an error has occurred in an application.

BB TestAssistant is easy to use and requires little training time. This program does not require special integration—BB TestAssistant will work in conjunction with existing beta testing protocols. BB TestAssistant will run under Windows 98, Windows Me, Windows 2000, and Windows XP.

http://www.bbsoftware.co.uk/BBTestAssistant.aspx

Graphical Business Rules Tool

Logic Programming Associates Ltd. recently announced VisiRule, a graphical charting tool that lets you draw, test, and generate business-rule applications and components. VisiRule allows you to design and deliver commercial rule-based systems, expert systems, and knowledge-based systems on the desktop and Internet without having to write any code. You simply draw the decision logic using a point-and-click paradigm and state the conditional logic to use at the relevant branching points.

Now you no longer have to be a sophisticated AI programmer to build decision support systems that may involve complex and interconnected logic. VisiRule lets you rapidly produce accurate charts which you can share and discuss with non- technical colleagues who under- stand the business process at hand. VisiRule runs on all versions of Windows.

http://www.lpa.co.uk

Create Calculation Formulas Dynamically

EasyFormula, recently released by JTech Software & Media, is a software development component used to create dynamic calculation formulas. With EasyFormula, your application can let users create their own formulas and evaluate them at run time. Users can choose how to process and present their data, instead of letting the developer decide.

EasyFormula features include a convenient interface for getting your client application data, seamless integration with any Windows-based application that supports COM, more than 30 built-in functions, support for native C++ classes and ActiveX® controls, and compatibility with Visual Basic®, Visual C++®, and a number of other environments.

With EasyFormula you can separate calculation formulas from source code, keep them in central storage, and evaluate them at run time. You can also change the logic without having to rebuild the system.

http://www.jtech.ca

Generate Flowcharts from Your Source Code

Reliable Union Inc. now offers Code to Chart 2.0, an integrated visual code analyzer and source code flowchart generator. Using Code to Chart, you can quickly get a visual flowchart representation of your source code and achieve synchronization between the flowchart and the code in the editor, easily navigate between function and class definitions, and export the generated flowchart to Visio or a BMP image. You can also save your source code as a RTF file or an HTML file with syntax highlighted.

Code to Chart comes with a classes/functions browser, a syntax highlight code editor, and a flowchart generator. The three components are integrated into one interface and work together. Because Code to Chart 2.0 is specially designed and optimized for C, C++, C#, Java, and PHP, you get especially fast parsing speed.

http://www.reliableunion.com

Unit Tests Drive Code Quality

Are your unit tests effective? Your code may be passing the tests, but that doesn't mean the tests are testing all of your code. It would be safer to know how much of your code is being tested and how critical the code that is not running through the tests is.

Clover.NET, recently released by Cenqua, can help you answer all of these questions for your Microsoft® .NET-based applications. Clover.NET reports are published in HTML format to provide an accessible, easily navigated view of your code. These reports can be shared with the development team or with project management.

Clover can also be used to find "dead code" in an application—code that is no longer used by the app that can be safely removed, helping to prevent code bloat.

Clover.NET is a fully integrated plug-in for Visual Studio® .NET 2003. It provides seamless integration with projects using NAnt and with legacy build systems with command-line interfaces and APIs. It also provides fast, accurate, configurable, detailed coverage reporting of Method, Statement, and Branch coverage. It offers precise control over coverage gathering with source-level filtering.

http://www.cenqua.com/clover.net

Automate Object Model Documentation

ComponentOne recently unveiled Doc-To-Help 7.2 Professional, an upgrade for current Doc-To-Help 7.2 Professional subscribers. Doc-To-Help adds time-saving features that let developers automate object model documentation of their .NET code and features enhanced topic functions and hyperlink usability. It enables users to easily create HTML Help, cross-platform HTML-based Help, JavaHelp, WinHelp, Microsoft Help 2.0, and printed documentation from a single set of Microsoft Word docs.

http://www.ComponentOne.com

Defend Against Hackers and Malicious Software

SoftSphere Technologies has just released Anti-Cracker Shield 1.0, a proactive PC protection software application that explores computers and their contents in order to identify vulnerabilities and weak spots.

Anti-Cracker Shield is a multitier protection system, with several levels of defense—ultra, high, low, and passive defense. The program identifies and instantly blocks any predatory or malicious processes, so it can protect against new and unknown exploits by simply blocking any processes used to infiltrate the system. In addition, the program can do preemptive anti-exploit strikes, such as relocating system libraries, so even if a virus or a trojan is planted, it won't be executed.

One unique feature of Anti-Cracker Shield is that it can protect Windows NT® services without actually turning them off, since they are likely to be used by legitimate applications. The program does not hinder computer performance when an attack happens, leaving the computer fully operational. In addition to the regular version, there is a server version that works with all popular servers and targets server-specific exploits.

With Anti-Cracker Shield, you can specify which particular processes are likely to be used in a hacking attack and thus need to be fully protected. Anti-Cracker Shield protects the entire system, including software applications, network settings, browser, e-mail components, and the operating system itself. When a problem is found, the program informs the user about it, suggesting how to fix it.

http://www.softsphere.com

The Bookshelf

Charles Petzold's book Programming Windows isn't a movie yet, but that doesn't mean you can't mix software development with entertainment. The recently published cyberthriller Stealing the Network: How to Own a Continent does a fine job. It's a fast-paced, hacker-chasing story that also provides lots of information about how to keep your network from being overtaken. The book, by security experts FX, Paul Craig, Joe Grand, Dan Kaminsky, Fyodor, Ryan Russell, Temmingh, Beale (Syngress Publishing, 2004), is the follow-up title to the 2003 bestseller Stealing the Network: How to Own the Box (Syngress Publishing, 2003).

How to Own a Continent operates on a global stage when the network infrastructure of an entire continent is compromised. You may well find yourself having nightmares about the day when your own network goes down.

The book introduces the concept of hacker groups and the damage they can inflict through a concerted, orchestrated string of malicious attacks.

The book also includes a unique appendix, The Making of STC, where the Stealing the Continent mailing list is published. Readers can see the actual e-mail threads where the story was developed, ideas were exchanged, and the project was monitored. Take a look and see if there's anything that strikes a chord.

http://www.syngress.com

Send your New Stuff to newstuff@microsoft.com.

Nancy Michellis a developmental editor at MSDN Magazine where she spends her time manipulating bits of string literals for the good of mankind.


Web Q&amp;A: Refreshing Web Pages, Spyware, Group Policy, and More

New information has been added to this article since publication.
Refer to the Editor's Update below.

Web Q&A
Refreshing Web Pages, Spyware, Group Policy, and More
Edited by Nancy Michell

Q Is there a simple way to refresh a Web page at certain time intervals in ASP.NET?

A You can use the Meta tag to keep the page refreshing. The following tag refreshes the page every 20 seconds:

<META HTTP-EQUIV="Refresh" CONTENT="20">

A programmatic way that is a little more complicated is shown in the following code snippet:

string script = "<script language=JavaScript>\n" +
                "  window.setInterval('postme()',5000);\n" +
                "  function postme()\n" +
                "  {\n" +
                "    document.forms[0].submit();\n" +
                "  }\n" +
                "</script>\n";
RegisterStartupScript("myBlock",script);
An advantage of injecting script from server-side code is that you can use custom logic to set the interval or decide to enable or disable the refresh. Keep in mind, though, that simple is better; you shouldn't do this with scripting unless you have no choice. If you do decide to use scripting but don't need to be able to modify the refresh interval or disable it, you could use the script itself declaratively in the page rather than registering it from the ASP.NET code.

Q What's a good way to prevent spyware or adware from being installed when using Microsoft® Internet Explorer? Are there any upcoming changes to Internet Explorer that would help?

A Windows® XP Service Pack 2 will make it more difficult for you to inadvertently install software. See Changes to Functionality in Microsoft Windows XP Service Pack 2 for details. With Service Pack 2, Internet Explorer provides better user controls and user interfaces to help prevent malicious applications from getting onto your machine. There is also an add-on manager included in Service Pack 2, although the best first line of defense would be to prevent installation in the first place. If you have something really vicious that renames itself and re-registers itself every day, even the add-on manager won't help.

You might want to take a look at Security at home: Fight spyware for more information on spyware. A variety of tools are available from other companies to detect and remove unwanted software from your computer. The aforementioned document recommends Spybot Search & Destroy and Ad-Aware.

There is also a default setting change in Windows XP Service Pack 2 called local machine zone lockdown. Some of the most important aspects of Internet Explorer security are the security zone settings. The current default settings for the My Computer zone (local machine zone) are now regarded as inadequate to protect the local machine. The previous assumption that your local machine is secure has been reviewed and there are new defaults for Windows XP Service Pack 2. This will mitigate one of the main attack vectors for Internet Explorer. See Knowledge Base article 833633 "How to strengthen the security settings for the Local Machine zone in Internet Explorer" for more information.

Q I need to set a Web site as a trusted site for all users in the domain. Is there a way to do this through a group policy object (GPO) or a logon script that edits the registry?

A Yes, you can use a GPO to do this. While you're editing the GPO, expand "Windows Settings" under "User Configuration." Expand "Internet Explorer Maintenance" and select "Security." On the right pane, double-click "Security Zones and Content Ratings." In the "Security Zones and Privacy" group box, select the "Import the current security zones and privacy settings" radio button, and press the "Modify Settings" button. Use the control panel to configure your security zones and privacy settings. Keep in mind that this will not add to the user's current security settings. It will modify their settings to match what you set in the GPO.

Q What does the "Days to keep pages in history" setting in Internet Explorer mean? My understanding is that when you first navigate to a URL, the clock starts ticking and when the "Days to keep pages in history" value is reached, that URL is cleared from the history.

A "Days to keep pages in history" is the number of days something will still show up in the history explorer bar after you have visited the site. If you visit the site again, it actually resets this value. However, there are a few quirks. If you change your clock, the setting may lose accuracy. If you clear your history, the URLs will disappear. The cleanup history code is run when you launch Internet Explorer and the History Manager is instantiated, so if you do not launch Internet Explorer for a long time, items in history may live longer than the numer of days that you have set in the "Days to keep pages in history" setting.

Q Is there a limit on the number of Internet Explorer windows that can be open at one time? If I open about 30, Internet Explorer starts to behave strangely.

A You may be low on desktop heap memory, a setting that you can double on most desktops that allows you to open more Internet Explorer sessions, as shown here:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Session
 Manager\SubSystems]

See Knowledge Base articles 126962 ("Out of Memory Error Message Appears When You Have a Large Number of Programs Running") and 184802 ("PRB: User32.dll or Kernel32.dll Fails to Initialize") for related issues.

Q In ADO.NET, I'm getting an error that says that the timeout period elapsed prior to obtaining a connection from the pool. Can I modify this in the .aspx page or do I need to change something on the SQL side?

A This occurs when the internal connection pool maintained by ADO.NET (on a per-connection-string basis) is full and no connections are being released from it. As an immediate measure, you can modify the SQL Connection string to append the following property to the existing connection string:

Max Pool Size=200
The default is 100, and changing it to 200 will give you relief while you attempt to isolate the issue.

Also, as indicated in Knowledge Base article 310369 ("PRB: No Method in DataReader Closes Its Underlying Connection"), you should make sure to use CommandBehavior.CloseConnection in ExecuteReader method calls when appropriate. The aforementioned document explains that the DataReader object doesn't have a property to get its underlying connection, so unless you have a reference to the connection or have set the CloseConnection behavior, you will not be able to close the underlying connection.

The ADO.NET programming model encourages aggressive opening and closing of connections, so you really shouldn't try to cache the connections in an attempt to amortize connection startup costs; this is done automatically and transparently by the ADO.NET connection pool.

Also be aware that in C#, you should try to use the "using" statement to manage the SqlConnection you are opening so that the Dispose method automatically gets called to close the connection when it goes out of scope.

Q I am creating Microsoft Excel reports and saving them in a particular folder through ASP.NET. But after each report is created, an Excel instance gets stacked up in memory, which I can see in Task Manager. I have the following code in my finally block:

finally
{
  if (oExcel != null)
  {
    oExcel.Quit();
    System.Runtime.InteropServices.Marshal.ReleaseComObject(oExcel);
    oExcel = null;
    GC.Collect();
  }
}

I also tried calling ReleaseComObject in a loop until the count was zero, but that didn't work. What can I do to solve this memory leak problem?

A Make sure that you create explicit objects for each level of the Excel hierarchy—Application, Workbook, Worksheet—and that you then close/quit and destroy each of them in reverse order, before calling the .NET ReleaseCOMObject. You should also read Chris Brumme's thoughts on the ReleaseComObject API.

Q I have ASP and ASP.NET pages creating cookies. Here's what I've been doing: first I create a cookie called "Test" with the value "123" in ASP. Then, I create a cookie called "Test" with the value "456" in ASP.NET. Now the ASP.NET page shows two cookies named "Test" with the two different appropriate values. However, the ASP page shows "Test" with the value "123" only. When I use JavaScript to show the cookies, both of them appear on the ASP page.

I am using Request.Cookies in ASP on the server side and document.cookie in JavaScript. Why don't I see both cookies in ASP on the server side?

A When two cookies with the same name and same path are sent to the server, ASP returns the first cookie with the same name that it has added to the cookies collection. If two cookies with the same name are sent by the client browser, Request.Cookies returns the one with the deeper path structure. For example, if two cookies had the same name but one had a path attribute of /Www/ and the other of /Www/Home/, the client browser would send both cookies to the /Www/Home/ directory, but Request.Cookies would only return the second cookie.

Add the following line to your ASP code and it will show that both cookies are being sent to the server:

Response.Write Request.ServerVariables("HTTP_COOKIE") & "<BR>"

If you want to get both cookies from ASP, you can just parse the cookie collection yourself by accessing the HTTP_COOKIE ServerVariable. You can see an explanation of the cookies collection in the MSDN Library at Request.Cookies Collection.

Q Is there a tool to measure how long a Web or SOAP request takes without adding any code?

A If you have access to the box hosting the Web service, IIS logs are one of the best places to start. You will need the Time Taken property (and you will probably want to enable others to get additional info; most of the log properties, including this one, are not usually turned on by default). Then you can pull the data into Excel to utilize its filtering and sorting capabilities (see Performance and Reliability Monitoring and Publishing Dynamic Data with the Internet Information Server).

If you turn on the Time Taken property for the IIS logs of your Web server, IIS will record the time taken (duration) for your Web resource. It will reflect the time it took from when your Web server receives the request to when the acknowledgment of the last byte came from the client. This will include any time taken calling a downstream Web service because that is included in the execution time of your Web page. But it would not specifically give you the duration of the downstream Web service call separate from the duration of the rest of your Web page. This is all dependent on the fact that the SOAP traffic is over HTTP, of course.

Q We are considering hosting a Windows Forms control in an Internet Explorer Web browser for an intranet Web application. In order to avoid the issue of passing managed events from the Windows Forms control back to unmanaged JScript® code (we may not be able to ask all of our users to make the security changes necessary to allow this), I am thinking of having the JScript code poll the Windows Forms control (maybe four times per second) inquiring, basically, whether any events have fired that the JScript code needs to know about. In rough form, the JScript would look something like this:

window.setInterval(checkWinformEvents, 250);
var bCheckingEvents;
function checkWinformEvents() {
      if(!bCheckingEvents) {
            bCheckingEvents = true;
            var sEvents = objWinform.AnyEventsToReport();
            bCheckingEvents = false;
            if (sEvents) handleWinformEvents(sEvents) ;
      }
}

The managed method, AnyEventsToReport, would then return something like "Double-click on line 14 of List view," and so on. Is it a good idea to check Windows Forms events from JScript?

A Having the JScript code poll the Windows Forms control seems a little weird, but it would give you a workaround for the security restrictions put in place by Internet Explorer. There are a few problems with the code snippet, but nothing that is insurmountable. For instance, where exactly do you make the second call to setInterval? No loop is evident.

The bottom line is that this program is really not robust in the face of exceptions. Plus, there's some uninitialized data in there and you treat a string as a bool, which is a bad idea. Also, why are you doing a re-entrancy check on a method which is not recursive?

You should probably write it like this instead:

window.setInterval(checkWinformEvents, 250) ;
function checkWinformEvents()
{
    try
    {
        handleWinformEvents(winform.AnyEventsToReport());
    }
    finally
    {
      window.setInterval(checkWinformEvents, 250) ;
    }
}
[ Editor's Update - 12/6/2004: setInterval evaluates an expression each time a specified number of milliseconds has elapsed and until the timer is removed with the clearInterval method. As such, it does not need to be called again every time the checkWinformEvents function is invoked.]

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Vineet Batta, Pritam Bhat, Todd Carter, Jim Cheshire, Ben Christenbury, George Chung, Andrew Cussons, Jeff Davis, Allen Dorfman, Ravi Gopinath, Anand Hegde, Mandar Inamdar, Douglas Laudenschlager, Eric Lippert, Dave Massy, Bill Safcik, Johnathan Seal, Krish Shenoy, Jai Singh, John Slyman, Rob Smith, Tommy Unger, Hang Vong, Joe Wilson, and Clement Yip.


Data Points: Handling Data Concurrency Using ADO.NET, Part 2

Data Points
Handling Data Concurrency Using ADO.NET, Part 2
John Papa

Code download available at: DataPoints0410.exe (145KB)
Contents
Planning for Concurrency
Save What You Can!
Using DataViewRowState
Turning the Other Cheek
Conclusion

Enterprise development has been moving towards a discon-nected model in recent years and ADO.NET development is no exception. While the disconnected model of the ADO.NET DataSet offers great flexibility, that adaptability also means looser control over data updates than you get with a connected data access model. One related issue is data concurrency. This month I'll focus on concurrency violations that occur when multiple rows could be saved to the database in a single batch.

When a concurrency violation occurs, the SqlDataAdapter's ContinueUpdateOnError property tells the SqlDataAdapter to either throw an exception or ignore the concurrency violation. I will demonstrate how to show exactly why a concurrency violation occurred by setting the ContinueUpdateOnError property to true and using a series of DataViews bound to ASP.NET DataGrid objects. These DataGrids will use the DataViewRowState enumerators to show what changes have been made to a DataSet.

The SqlDataAdapter also has some events, including RowUpdating and RowUpdated, which can come in handy. To illustrate their use, I will also show how to deal with concurrency by creating a handler for the RowUpdated event.

In last month's Data Points column, I discussed how to deal with concurrency violations and ran through an example of a violation that occurred when only a single row was updated at a time (see Data Points: Handling Data Concurrency Using ADO.NET). In that situation, setting the SqlDataAdapter's ContinueUpdateOnError property to false told the SqlDataAdapter to throw an exception as soon as a concurrency violation was encountered. This technique is ideal when only saving a single row or when you are attempting to save multiple rows and want them all to commit or fail. However, if you want to save the rows that do not encounter concurrency violations and display the rows that do cause concurrency violations, you must use another technique.

Planning for Concurrency

The examples in this column use optimistic concurrency; therefore, the data is only locked during the actual update. This leaves plenty of time for users to read that same data, try to update it, and collide with each other. As such, it is important to implement concurrency violation detection which catches attempts to modify records by a user that have already been modified since the last time the data was retrieved by that user.

Instead of dictating how to handle a concurrency violation, ADO.NET lets the developer decide how to deal with it. ADO.NET throws a data concurrency exception (DBConcurrencyException) when an UPDATE statement affects zero rows. If an UPDATE statement's WHERE clause contains only the primary key(s), it will implement the last-in wins approach, which overwrites whatever values exist in the row. The problem with this technique is that a user's changes can be overwritten and lost without any warning.

By adding another condition to an UPDATE statement's WHERE clause in addition to the primary key(s), a concurrency violation can be caught. There are a few ways to accomplish this, such as adding a timestamp column or a datetime column to the table and using it in the WHERE clause of the UPDATE statement. If you use a SQL Server TIMESTAMP column, it will automatically update itself with a new value every time a value in its row is modified. If you use a DATETIME column, you will have to update its value in every update either explicitly or by another means, such as an AFTER UPDATE trigger, as shown here:

CREATE TRIGGER trEmployees_IU ON Employees AFTER INSERT, UPDATE
AS
    UPDATE Employees
SET LastUpdateDateTime = getdate()
    FROM Employees
INNER JOIN inserted ON Employees.EmployeeID = inserted.EmployeeID

For the examples here, I altered the Employee table by adding an additional versioning column called LastUpdateDateTime and had its value updated by the trigger trEmployee_IU. Using a TIMESTAMP or ROWVERSION column eliminates the need for the trigger, but the DATETIME column solution is easier to demonstrate as it shows an actual date and time to the user. So for the examples here, I will continue to use a DATETIME column, although in a production application I encourage you to consider using a TIMESTAMP column instead.

The next steps are to make sure that UPDATE statements use both the primary key columns(s) and the versioning column and to return the versioning column's value from the UPDATE statement (see Figure 1). The versioning column's value should be returned via the SqlDataAdapter and sent back to the DataSet. If the UPDATE works, this sets the DataSet to the updated versioning column's value, which it will need if it attempts to update the row again. To have the value of the versioning column sent back to the DataSet, a number of key steps must be executed:

  1. The value of the versioning column (LastUpdateDateTime) is set to the current date and time.
  2. The versioning column is returned from the stored procedure via an output parameter.
  3. The SqlDataAdapter maps the output parameter to the LastUpdateDateTime column in the DataSet using the following lines of code:
    oUpdCmd.Parameters.Add(new SqlParameter("@NewLastUpdateDateTime",
    SqlDbType.DateTime, 8, ParameterDirection.Output, false, 0, 0,
    "LastUpdateDateTime", DataRowVersion.Current, null));

oUpdCmd.UpdatedRowSource = UpdateRowSource.OutputParameters;

Figure 1 Using a Versioning Column in an UPDATE (Close Figure 1)
CREATE PROCEDURE prUpdate_Employee
    @EmployeeID INT,          @LastUpdateDateTime DATETIME,
    @FirstName NVARCHAR(10),  @LastName NVARCHAR(20),
    @Title NVARCHAR(30),      @BirthDate DATETIME,
    @HireDate DATETIME,       @Extension NVARCHAR(4),
    @NewLastUpdateDateTime DATETIME OUTPUT
AS
    SET @NewLastUpdateDateTime = getdate()

    UPDATE    Employees
    SET    FirstName = @FirstName,  LastName = @LastName,
           Title = @Title,          BirthDate = @BirthDate,
           HireDate = @HireDate,    Extension = @Extension,
           LastUpdateDateTime = @NewLastUpdateDateTime
    WHERE    EmployeeID = @EmployeeID
      AND    LastUpdateDateTime = @LastUpdateDateTime

    IF @@ROWCOUNT = 0
        SET @NewLastUpdateDateTime = @LastUpdateDateTime

    RETURN

Once you have determined how you will detect data concurrency violations, you need to decide how to deal with them. ADO.NET doesn't automatically deal with concurrency violations, but gives you the flexibility to decide how to do so based on your own situation. You can ignore the violation and just update what you can, you can cancel all changes and give the user the choice of how to react (see Figure 2), or you can give the user no choice and predefine how to react. In the sample application you will see some of the options that can be presented to the user in the event of a data concurrency violation.

Figure 2 Options for Data Concurrency Violations
Figure 2 Options for Data Concurrency Violations

As you saw in Figure 2, one option is to cancel changes and revert back to the state of the data from the last time it was retrieved from the database. A second option is to cancel the changes and reload the data from the database. The difference is that in the latter, the data will reflect any changes that another user might have made since the last time this user got the data. The final option is to save the changes. This option will only save rows that do not cause a concurrency violation. Because I don't want another user's data changes to be overwritten inadvertently, I don't want to provide the user with the last-in wins option.

Save What You Can!

Using this sample application I will demonstrate some ways that you can detect concurrency violations and notify the user. The application attempts to save all rows that were modified. When a row is being updated to the database, if an error occurs the app will note it and continue updating the rest of the rows. In other words, let's say a user modifies five rows and two of those cause errors. These two will not be saved while the other three will be saved properly. So let's start with a scenario in which two users browse to the screen that loads a list of employees that can be edited, as you saw in Figure 2. User 1 changes five rows (EmployeeID's 1 through 5), clicks the Save button, and is notified that all of the rows were saved successfully. Then User 2 changes five rows (EmployeeID's 3 through 7), clicks the Save button, and receives notification that a concurrency violation has occurred. For simplicity, I only change the FirstName field in each row, with User 1 appending a 1 and User 2 appending a 2 to the end of the first name values.

I've also added two radio buttons to the application to show two ways to deal with concurrency. Both techniques have the same net effect to the user, which is to save the rows that it can and to notify the user with as much information as possible about which rows could not be saved due to concurrency violations. Figure 3 shows the information displayed after User 2 clicks the Save button in the scenario I just described.

Figure 3 Different Kinds of Data Errors (Close Figure 3)

Using DataViewRowState

In this example, the rows listed as EmployeeID 3, 4, and 5 were changed by User 1. So when User 2 also saved them, a concurrency violation was encountered. Notice in Figure 3 that there are five ASP.NET DataGrid objects displayed. Each DataGrid shows a different view of the data that errored out because of the concurrency violations. The ShowConcurrency method that is shown in Figure 4 contains the code that gathers the data that will be displayed in each of the DataGrids.

Figure 4 Showing Concurrency Violations (Close Figure 4)
private void ShowConcurrency(dsEmployee.EmployeeDataTable oTblEmp) {
  if (oTblEmp.HasErrors) {
    // Store modified versions of the rows that saved with no problems,
    // then modified versions that had concurrency violations
    dsEmployee oDsModifiedOK = (dsEmployee)oDsAllEmployees.GetChanges();
    dsEmployee oDsModified = new dsEmployee();
    dsEmployee oDsInDB = new dsEmployee();

    // Loop through rows with errors
    foreach (dsEmployee.EmployeeRow oDr in oTblEmp.GetErrors()) {
      // Remove the row from ModifiedOk and add it to Modified, then
      // get matching row from db and add it to the InDB DataSet
      oDsModifiedOK.Employee.Rows.Remove(
        oDsModifiedOK.Employee.Rows.Find(oDr.EmployeeID));
      oDsModified.Employee.ImportRow(oDr);
      Employee oEmployee = new Employee();
      oDsInDB.Employee.ImportRow(oEmployee.GetEmployee(oDr.EmployeeID));
    }
    lblModifiedAll.Visible = lblModifiedOK.Visible =
      lblErroredModifiedCurrent.Visible = lblErroredInDB.Visible =
      lblErroredModifiedOriginal.Visible = true;

    // Bind grdModifiedAll DataGrid to all of the modified rows
    // Bind grdModifiedOK DataGrid to modified rows saved without error
    grdModifiedAll.DataSource = new DataView(oDsAllEmployees.Employee,
      null, null, DataViewRowState.ModifiedCurrent);
    grdModifiedOK.DataSource = new DataView(oDsModifiedOK.Employee,
      null, null, DataViewRowState.ModifiedCurrent);
    grdModifiedAll.DataBind();
    grdModifiedOK.DataBind();

    // Bind grdErroredModifiedOriginal to modified rows saved
    // with errors. Show only original version of row. Same for
    // grdErroredModifiedCurrent but show only current version.
    grdErroredModifiedOriginal.DataSource = new DataView(
      oDsModified.Employee, null, null, DataViewRowState.ModifiedOriginal);
    grdErroredModifiedCurrent.DataSource = new DataView(
      oDsModified.Employee, null, null, DataViewRowState.ModifiedCurrent);
    grdErroredModifiedOriginal.DataBind();
    grdErroredModifiedCurrent.DataBind();

    // Bind grdErroredInDB to rows currently in db that are the
    // counterparts for those with errors
    grdErroredInDB.DataSource = new DataView(
        oDsInDB.Employee, null, null, DataViewRowState.CurrentRows);
    grdErroredInDB.DataBind();
  }
}

The first DataGrid, "All Rows I Tried To Save," shows the five rows and their values that User 2 tried to save to the database. In order to display these rows and the values that are sent to the database, I created a DataView from the main employee DataSet (oDsAllEmployees) and filtered it so that only the current values of modified rows would be in the DataView. I set the DataGrid's Data source property to this new DataView and rebound the DataGrid using the following code:

// Bind the grdModifiedAll DataGrid to all of the modified rows
grdModifiedAll.DataSource = new DataView(oDsAllEmployees.Employee,
    null, null, DataViewRowState.ModifiedCurrent);
grdModifiedAll.DataBind();

The next DataGrid, "Rows that Saved OK," shows all of the rows that User 2 saved successfully, without errors. To show these rows, I created a DataSet called oDsModifiedOK and set it to contain all of the modified rows from the main DataSet using the DataSet's GetChanges method. Then, I looped through the Employee DataTable's rows that contain errors and picked out the EmployeeID for each row that had errors. The DataTable object has a method called GetErrors which returns an array of DataRow objects from the DataTable that contain errors. In this example, the array will contain the three rows that encountered concurrency violations. I then locate the same row for that employee by passing the EmployeeID to the Find method and then remove the row from the oDsModifiedOK DataSet. When the loop completes, only the rows that saved successfully will be left in the oDsModifiedOK DataSet.

The third, fourth, and fifth DataGrids show only the rows that encountered concurrency violations. Each DataGrid shows the same rows, with different values. The third DataGrid, "Rows in Error—Modified Original," shows the three rows and their values as they existed when User 2 last grabbed them from the database.

The fourth DataGrid, "Rows in Error—Modified Current," shows the three rows and their values as they existed after User 2 made changes to them. I created a DataSet called oDsModified and imported the rows that contained errors by looping through the GetErrors array and calling the ImportRow method of the Employee DataTable, as shown in Figure 4. The third DataGrid is bound to a DataView and filtered to show only the rows that match the DataViewRowState.ModifiedOriginal enumerator value.

The fourth DataGrid is bound to a DataView and filtered to show only the rows that match the DataViewRowState.ModifiedCurrent enumerator value. The DataViewRowState filter comes in handy here as it allows me to take a single DataTable and show both its original values and it current values in two different DataGrid objects via two DataView objects.

The final DataGrid, "Rows in Error—In Database," shows the same three rows that encountered concurrency violations as they exist in the database. This DataGrid is key as it shows how the values of the FirstName columns have changed since User 2 last retrieved them from the database. Thus, when User 2 tried to save the values shown in the "Rows in Error—Modified Current" DataGrid, the prUpdate_Employee stored procedure (shown in Figure 1) unsuccessfully tried to match the EmployeeID and LastUpdateDateTime fields between what is shown in the "Rows in Error—Modified Original" and "Rows in Error—In Database" DataGrids. Thus, no rows were updated and concurrency violations were raised for these rows. While I have shown much more information than the user would necessarily want to see in this example, it should help illustrate how concurrency violations occur and the flexibility that ADO.NET offers in dealing with them.

Turning the Other Cheek

In last month's column I demonstrated how to halt and roll back a transaction when any row in the batch update failed. The example here ignores changes to any rows that encounter an error during the update. There are two easy ways to implement this behavior in ADO.NET. One is to set the SqlDataAdapter's ContinueUpdateOnError property to true. This tells the SqlDataAdapter that if an error is encountered, it should not throw an exception. Rather, the SqlDataAdapter will skip the update of the offending DataRow, log the error condition in the offending DataRow's RowError property, and continue updating the rest of the rows in the batch update. A second way to implement this behavior is to use the SqlDataAdapter's RowUpdated event.

The RowUpdating event is raised just before a DataRow is about to be sent to the database to be updated. The RowUpdated event, on the other hand, is raised just after the row was updated in the database. Each of these events can be used to modify the updating behavior before and after it happens, to grab a reference to an updated row, or even to cancel the current update.

The RowUpdated event is less efficient than setting the ContinueUpdateOnError property to true. However, it does expose additional functionality such as exposing the type of action query being performed on the row, whether an error occurred, and the entire row that is being updated. Unless you need to take advantage of the added features, I suggest avoiding the RowUpdated event when managing concurrency.

When setting the ContinueUpdateOnError property to true you respond to errors after the update is finished. The RowUpdated event allows you to respond to errors right when the error is encountered. You can alter the behavior of the update by setting the RowUpdatingEventArgs argument's Status property to one of the settings listed in Figure 5.

Figure 5 Status Settings (Close Figure 5)

Status | Description
Continue | Continue the update as planned
ErrorsOccurred | Abort the update of all rows and throw an exception
SkipAllRemainingRows | Abort the update of all rows but do not throw an exception
SkipCurrentRow | Ignore the current row and continue the update of the remaining rows

In the sample application, if you select the radio button that specifies that the code should use the RowUpdated event handler, it will associate the event handler with the RowUpdated event of the SqlDataAdapter, as you can see here:

oDa.RowUpdated += new SqlRowUpdatedEventHandler(oDa_OnRowUpdated);
The event handler I created (oDa_OnRowUpdated) executes following each individual row that is updated. It checks the status of the update and if it detects that an error occurred, it sets the offending row's RowError property to the error message. Then, it changes the status from UpdateStatus.ErrorsOccurred to UpdateStatus.SkipCurrentRow. This tells the SqlDataAdapter not to throw an exception, but rather to continue trying to update the rest of the rows to the database (thus emulating the behavior of setting ContinueUpdateOnError to true):
private void oDa_OnRowUpdated(object sender, SqlRowUpdatedEventArgs e)
{
    // If an error occurred during the update ...
    if (e.Status == UpdateStatus.ErrorsOccurred)
    {
        // Grab the error message and associate it with the
        // current row that was updated
        e.Row.RowError = e.Errors.Message;
        // ignore error; keep processing updates if there are more
        e.Status = UpdateStatus.SkipCurrentRow;
    }
}

Conclusion

Handling data concurrency issues is especially important when using a disconnected data store like the ADO.NET DataSet object. If an application using the DataSet has the ability to save several rows at once to the database, then it must be prepared to handle the threat of concurrency violations, which can easily occur any time more than one user is accessing the data store.

One way to handle such concurrency violations is to save all rows that can be saved and show the user the rows that could not be saved due to the violation. Another way is to save none of the rows in the batch, basically implementing an atomic transaction. ADO.NET provides a variety of ways to handle concurrency, but choosing the right technique for your app is still up to you.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive.


Advanced Basics: Building a Progress Bar that Doesn't Progress

Advanced Basics
Building a Progress Bar that Doesn't Progress
Duncan Mackenzie

Code download available at: AdvancedBasics0410.exe (151KB)
Contents
What Type of Control Should You Use?
Adding Configuration Properties
Handling Animation
Finally, It is Time to Draw the Control
The Standard Demo Application
Running a Process in the Background
Displaying Updates from Your Background Process
I Just Want It to Stop!
Conclusion

In many situations, accurately estimating the length of a certain process (copying a large file, loading data from a server, retrieving files from the Internet) would be both difficult and inefficient. What you end up with is a process that is going to take long enough to make the user wait, yet you have no easy way to indicate the percentage of the task that has completed. A regular progress bar would be rather meaningless, so you need some form of "Working..." indicator.

Windows® often uses the animated Windows icon in the upper-right corner of Microsoft® Internet Explorer while a page is loading, and you could certainly create and display your own AVI file or animated GIF if you wanted to customize it a bit. Personally, I prefer to use a more flexible approach—a repeating animated progress bar control. By using a control, I can configure the appearance of the animation for different applications, and it is a lot easier to incorporate into a Windows Forms UI.

Figure 1 My Progress Bar
Figure 1 My Progress Bar

In this column, I am going to walk you through the implementation of a custom owner-drawn control—an animated progress bar that will cycle continuously. The particular style of this control was inspired by a recent evening spent staring at the installation UI for Windows XP. By the end of the install, the hypnotic little green squares were burned into my brain, so I just had to try to recreate something similar in my own control. Of course, not everyone likes green squares, so I will try to make the control's appearance as configurable as possible. The result will be a progress bar that displays either squares or circles and exposes properties to control the speed of animation, the size of the graphical elements, and more (see Figure 1). The actual progress indicator will be a single shape (circle or square) drawn a bit larger than the rest, which will appear to move across the control at a steady rate.

What Type of Control Should You Use?

When developing with Windows Forms, I like to classify control development into three categories: user controls, inherited controls, and custom controls. User controls are the traditional (pre-.NET) style of Visual Basic® control, a design surface on which you can combine a variety of other controls. Inherited controls allow you to build on existing controls such as TextBox, writing only the code you need and getting all of the existing functionality for free. Finally, custom controls inherit only from the base class of Control and are perfect for when you want to handle all of the graphics work on your own. Here I'm using a custom control because I'm not going to be hosting any other controls and I'll be doing all of the drawing on my own.

Whenever I'm starting a custom control, I create a Class Library project and then start a control class with the code shown in Figure 2. The class shown inherits from Control, which makes it a Windows Forms control and allows it to be hosted correctly within a Form or other container. The rest of the code just configures the control as owner-drawn, telling the control that my code will be handling all of the drawing, that it should be redrawn whenever it is resized, and that it should use double buffering to reduce flicker. I have added the "DesignerCategory" attribute at the top of the class just for my own ease of use; it makes Visual Studio® open this class into code view by default, instead of into the design view (thanks to Daniel Cazzulino for posting this tip on his blog at http://weblogs.asp.net/cazzu).

Figure 2 Writing a New Custom Control (Close Figure 2)
Imports System.Windows.Forms
Imports System.Drawing

<System.ComponentModel.DesignerCategory("Code")> _
 Public Class NoProgressBar
    Inherits Control

    Public Sub New()

        'I put these at the top of every owner-drawn control...
        Me.SetStyle(ControlStyles.AllPaintingInWmPaint, True)
        Me.SetStyle(ControlStyles.DoubleBuffer, True)
        Me.SetStyle(ControlStyles.SupportsTransparentBackColor, True)
        Me.SetStyle(ControlStyles.UserPaint, True)
        Me.SetStyle(ControlStyles.ResizeRedraw)

    End Sub

End Class

Adding Configuration Properties

From that generic beginning, you are ready to start writing the specific code that your control needs. While the graphics work might seem like the logical next step, I prefer to create all of my supporting Properties, Constants, and Enums at this point. If you leave writing the necessary code for these features until the end, then you will find yourself using placeholders and hardcoded values in your graphic routines, which is just more code that you have to edit. As with any development, this is an iterative process, which means you may need to add or edit one or more properties once you dig into the graphics code. For starters though, let's consider the basic functionality that I want this control to have. I want any developer using it to be able to configure:

  • The shape to be drawn (squares, circles)
  • The size (diameter) of shape to draw
  • The spacing between each shape
  • The background and foreground colors
  • The duration of a complete cycle through the progress bar's items

Given that set of requirements, I am going to create one Enum for the graphic shape, like this

Public Enum ElementStyle As Byte
    Square = 0
    Circle = 1
End Enum
along with properties for ShapeToDraw, ShapeSize, ShapeCount, ShapeSpacing, and CycleSpeed which you can take a look at in the code download for this column, which is available at the link at the top of this article.

The property routines are very simple; they just set and retrieve the value of some private member variables, but each one is marked with the System.ComponentModel.Category attribute to indicate the section under which it should appear in the Visual Studio property window. Note that I don't need to add any properties for the background and foreground colors because the base Control class already provides ForeColor and BackColor.

If you browse through the source for these property routines, you will notice that when a new value is being provided many of these properties call RecalcCountAndInterval and/or force a redraw of the control by calling Me.Invalidate. These calls, along with some basic validation included in these property routines, will be explained a bit later in this column, so for now I will just say that whenever a property change could affect the rest of your control, you will often need to recalculate some layout information or force a redraw of the control.

Handling Animation

To animate the control, which means redrawing all or part of it for every frame or step of the desired animation, I've created an instance of the System.Windows.Forms.Timer class. I create that timer in the constructor of my control, using the WithEvents and Handles keywords together to respond to the Timer class's Tick event, as shown in Figure 3.

Figure 3 Animating a Control with a Timer (Close Figure 3)
Dim WithEvents cycle As Timer
Dim currentActiveItem As Integer = 0

Private Sub cycle_Tick(ByVal sender As Object, _
    ByVal e As System.EventArgs) Handles cycle.Tick
    Dim oldActiveItem As Integer = Me.currentActiveItem

    If Me.currentActiveItem >= m_ShapeCount Then
        Me.currentActiveItem = 1
    Else
        Me.currentActiveItem += 1
    End If

    Me.Invalidate(CalcItemRectangle(oldActiveItem))
    Me.Invalidate(CalcItemRectangle(Me.currentActiveItem))
End Sub

CalcItemRectangle is used to determine the bounds of a single shape, wherever it was drawn on the control, and allows me to invalidate only those areas of the control that need redrawing rather than everything. Reducing the area that is redrawn helps to improve performance and to reduce flicker, although with the built-in double buffering you may not notice a difference either way. The animation and the drawing routines are all dependent on knowing the number of shapes to draw, which is in turn dependent on the shape size, spacing, and the width of the control. Several additional routines are used to keep that information up to date as you can see by taking a look at Figure 4.

Figure 4 Calculating Shape Sizes and Spacing (Close Figure 4)
Protected Overrides Sub OnResize(ByVal e As System.EventArgs)
    RecalcCountAndInterval()
    MyBase.OnResize(e)
End Sub

Private Const MIN_INTERVAL As Integer = 100

Private Sub RecalcCountAndInterval()
    Dim w As Integer = Me.Width
    Dim newShapeCount As Integer
    newShapeCount = Math.Floor((w - m_ShapeSpacing) / _
        (m_ShapeSize + m_ShapeSpacing))
    If newShapeCount <> m_ShapeCount AndAlso newShapeCount > 0 Then
        Dim interval As Integer = Me.m_CycleSpeed \ newShapeCount
        If interval >= MIN_INTERVAL Then
            cycle.Interval = interval
        Else
            cycle.Interval = MIN_INTERVAL
        End If
        m_ShapeCount = newShapeCount
    End If
End Sub

As I mentioned earlier, several of the property routines call RecalcCountAndInterval when they are modified because those values are somehow involved in the calculation. Of course, if you look carefully at the calculations involved in the RecalcCountAndInterval routine, you will see that certain property values will cause an error, specifically any combination of values such that (m_ShapeSize + m_ShapeSpacing) = 0. You could just add a check in the Recalc routine to avoid this issue, but the cleaner solution is to require ShapeSize and ShapeSpacing to be greater than or equal to zero and then check this rule in both the property routines and in RecalcCountAndInterval.

In my property routines, I call Me.Invalidate after calling the RecalcCountAndInterval routine, which forces a redraw of the entire control. It is possible (depending on what has been changed) that only a small area of the control needs to be invalidated, but these property changes should be relatively rare so the code required to determine the affected area wasn't worth the effort.

Finally, It is Time to Draw the Control

As I mentioned earlier, overriding OnPaint and drawing the control might seem like the logical first step, but it is really much easier to work on after you have your configuration options set up. Within the OnPaint routine, you need to draw out every one of the shapes that will appear across the entire control, drawing one shape a bit bigger to indicate that it is the active one. That active shape will appear to be moving across the control, giving the desired appearance of progress (Figure 5 shows the code).

Figure 5 Drawing the Custom Control (Close Figure 5)
Private Const SIZE_INCR As Integer = 2
Protected Overrides Sub OnPaint( _
    ByVal e As System.Windows.Forms.PaintEventArgs)
    Dim g As Graphics = e.Graphics

    ControlPaint.DrawBorder3D(g, Me.ClientRectangle, Me.BorderStyle)
    For i As Integer = 1 To m_ShapeCount
        Dim pos As Point
        Dim x, y As Integer
        pos = CalculateItemPosition(i)
        x = pos.X
        y = pos.Y

        If i = currentActiveItem Then
            DrawShape(g, Me.m_ShapeToDraw, _
                x - SIZE_INCR, y - SIZE_INCR, _
                m_ShapeSize + (SIZE_INCR * 2))
        Else
            DrawShape(g, Me.m_ShapeToDraw, x, y, m_ShapeSize)
        End If
    Next
End Sub

Private Function CalculateItemPosition(ByVal index As Integer) As Point
    Dim pos As Point
    pos.X = (m_ShapeSpacing * index) + (m_ShapeSize * (index - 1))
    pos.Y = (Me.Height \ 2) - (m_ShapeSize \ 2)
    Return pos
End Function
Figure 6 Drawing the Elements
Figure 6 Drawing the Elements

OnPaint is essentially nothing but a loop, but (via the CalculateItemPosition function) it calculates the size and position of the next shape (based on a combination of the ShapeSpacing and ShapeSize properties, as shown in Figure 6) and it then calls the DrawShape method to handle drawing out each individual element with a specific position and size:

Private Sub DrawShape(ByVal g As Graphics, _
    ByVal shape As ElementStyle, ByVal x As Integer, _
    ByVal y As Integer,  ByVal size As Integer)

    Dim shapeBrush As New SolidBrush(Me.ForeColor)
    Select Case shape
       Case ElementStyle.Circle
          g.FillEllipse(shapeBrush, x, y, size, size)

       Case ElementStyle.Square
          g.FillRectangle(shapeBrush, x, y, size, size)
    End Select
End Sub
You should note that the control's ForeColor property is used in order to create the brush, thus making the color of these shapes configurable by the developer.

With the completion of the graphics code, the control is ready for use, although I expect you will want to add to and modify it to suit your own needs (say, for example, you really need it to draw triangles instead of circles or squares).

The Standard Demo Application

Included with the source of this application are two demos, basic and advanced. The first demo is nothing more than a sample of the control in action, complete with an interface for manipulating the properties of the progress bar. The second demo, on the other hand, illustrates how you might use this progress bar in a real program, executing a task on a background thread and putting this control up as the "Working..." indicator. I won't go into great detail on the concepts behind this second demo as Ted Pattison covered it in the May 2004 installment of Basic Instincts, but here is a quick discussion and a walkthrough of the code involved.

Running a Process in the Background

No one likes the "Not Responding" message or, as Mark Boulter calls it (Mark is a Lead Program Manager on the Windows Forms team), the "White Screen of Impatience," but the alternative almost always involves threading. Working with multiple threads is much simpler in the Microsoft .NET Framework than in previous versions of Visual Basic, but that doesn't mean it is easy, so most applications ignore it and the user has no choice but to wait and hope that the application hasn't crashed completely. Using the new progress bar I just created and the simple background process of searching the file system, I will show you a relatively simple implementation of a background thread, then I'll make it more complicated by passing information from that background process to the foreground user interface.

The first step is to encapsulate your background process into a routine that either takes no parameters or accepts parameters through properties because when you start a new thread, you need to give it the address of a "Sub" routine with no parameters. I rarely find a long-running process that requires no information, so I create a class for my process, turn the parameters into properties, and create a "Start" method (see Figure 7).

Figure 7 State Class to Use with Background Thread (Close Figure 7)
Imports System.IO

Public Class FileSystemScan

    Dim m_StartingDirectory As String = "C:\"

    Public Property StartingDirectory() As String
        Get
            Return m_StartingDirectory
        End Get
        Set(ByVal Value As String)
            m_StartingDirectory = Value
        End Set
    End Property

    Public Sub StartScan()
        ScanDirectory(Me.m_StartingDirectory)
    End Sub

    Private Sub ScanDirectory(ByVal dir As String)
        Try
            Dim files As String()
            files = Directory.GetFiles(dir)
            For Each f As String In files
                ' do something with the file path
            Next
        Catch ex As Exception

        End Try

        Try
            Dim directories As String()
            directories = Directory.GetDirectories(dir)
            For Each d As String In directories
                ScanDirectory(d)
            Next
        Catch ex As Exception

        End Try
    End Sub

End Class

Then, in my main application, I create a working form that contains my new progress bar. When it is time to call the long-running process, I pop up the form, create the new thread and execute it, and then wait for that thread to exit (see Figure 8).

Figure 8 Waiting for a Task to Complete (Close Figure 8)
Dim activeThread As Threading.Thread

Private Sub startScan_Click(ByVal sender As System.Object, _
    ByVal e As System.EventArgs) Handles startScan.Click
    Me.Enabled = False
    Dim progress As New ActivityIndicator
    progress.Show()
    Dim myFSS As New FileSystemScan
    myFSS.StartingDirectory = "C:\"
    Dim activeThread As New _
        Threading.Thread(AddressOf myFSS.StartScan)
    activeThread.Start()
    Do While activeThread.IsAlive
        Application.DoEvents()
        Thread.Sleep(50)
    Loop
    progress.Close()
    Me.Enabled = True
End Sub

That is, of course, the simplest case. What if you want to get feedback from the background process, such as the current position in the file system in my example?

Displaying Updates from Your Background Process

Receiving information from the background process is one problem, but the other problem revolves around the use of multiple threads; you can't reliably modify any property of a Form from a different thread than the one the Form is running on (although it can appear to work fine during casual testing). I'll show you what this means to the code in just a moment. First, let's modify the FileSystemScan class to report its progress through a few events. I'll add declarations for two events and then call them during my file-scanning loops, as shown here:

'at the top of the FileSystemScan class
Public Event FileFound(ByVal sender As Object, _
    ByVal e As FileFoundEventArgs)
Public Event ErrorEncountered(ByVal sender As Object, _
    ByVal e As ErrorEventArgs)

I use two custom EventArg-derived classes, one to pass back a message for errors and another to return a file name to indicate progress. Now, back in the calling form, I am going to add a label to the Activity Indicator form (the one with the new progress bar on it) and add a public method to that form to allow the label's text to be set, as shown in the following:

Public Class ActivityIndicator
    Inherits System.Windows.Forms.Form
    ...
    Public Sub SetMessageText(ByVal message As String)
        Me.lblMessage.Text = message
    End Sub
    ...
End Class

Next, I'll rewrite the code in the Start Scan button click event to use a form-level variable (myScanner) that includes event handling and to create and use a delegate pointing at the ActivityIndicator Form to pass along new messages whenever they appear:

Dim WithEvents myScanner As FileSystemScan
Private Delegate Sub MessageHandler(ByVal message As String)
Dim myHandler As MessageHandler

Dim activeThread As Threading.Thread
Private Sub startScan_Click(ByVal sender As System.Object, _
    ByVal e As System.EventArgs) Handles startScan.Click
    ...
    myHandler = New MessageHandler(AddressOf progress.SetMessageText)
    ...
End Sub
With those changes made, I now have a window that displays a nice graphic and shows a continuously changing file path to indicate the current progress.

I Just Want It to Stop!

I need to add a Cancel button to the window and then the sample will be complete (see Figure 9). There are many ways to do this, but the simplest and most generic from my point of view is to interrupt the background thread whenever the Cancel button is pressed. That will cause an exception in the FileSystemScan class, which we can handle by just exiting our long-running process. To accommodate this change, I added a handler for the Cancel button (which calls activeThread.Interrupt, as shown in Figure 10) and error handling to the code in FileSystemScan (see Figure 11).

Figure 11 Handling Thread Interrupts (Close Figure 11)
Public Sub StartScan()
    Try
        ScanDirectory(Me.m_StartingDirectory)
    Catch ex As Threading.ThreadInterruptedException
        'thread interrupted, get out of here!
        Exit Sub
    End Try
End Sub

Private Sub ScanDirectory(ByVal dir As String)
    Try
        •••
    Catch ex As Threading.ThreadInterruptedException
        'thread interrupted, fire it back up the
        'call stack... should go up until it
        'hits "StartScan"
        Throw ex
    Catch ex As Exception
        RaiseEvent ErrorEncountered(Me, _
            New ErrorEventArgs(ex.Message))
    End Try
    •••
End Sub
Figure 10 Starting and Stopping a Background Thread (Close Figure 10)
Dim activeThread As Threading.Thread
Private Sub startScan_Click(ByVal sender As System.Object, _
    ByVal e As System.EventArgs) Handles startScan.Click
    Me.Enabled = False
    Dim progress As New ActivityIndicator
    AddHandler progress.CancelButtonPushed, _
        AddressOf Me.CancelButtonPushed
    progress.Show()
    •••
End Sub

Private Sub CancelButtonPushed(ByVal sender As Object, _
        ByVal e As System.EventArgs)
    Me.activeThread.Interrupt()
End Sub
Figure 9 Adding a Cancel Button
Figure 9 Adding a Cancel Button

While it is centered on only one sample background process, I think that this demo incorporates most of the things I would like to see in an "in progress" indicator for a long-running background process. The sample code included with this column is the culmination of all of these little adventures, and includes the status text area and the Cancel button. Check it out to see the full example running on your own machine.

Conclusion

Sometimes you just need to show that you are doing something, even if you can't say how long it is going to take. One effective way of illustrating this state is to display a cyclic animation. Building your own progress bar is probably the easiest part of this column; understanding how to use it in your own applications is the real trick. I hope that my example helped explain at least some of the issues involved in running a process on a background thread.

Send your questions and comments to basics@microsoft.com.

Duncan Mackenzie is the MSDN Content Strategist for Visual Basic and C# and the author of the Coding 4 Fun column on MSDN online. He can be reached through his blog at http://weblogs.asp.net/duncanma or personal site at http://www.duncanmackenzie.net.


.NET Column: P/Invoke Revisited

.NET
P/Invoke Revisited
Jason Clark

Code download available at: NETColumn0410.exe (121KB)
Contents
Marshaling Structures
Object Lifetime and Pinning
Marshaling Value Types vs. Reference Types
StructLayoutAttribute
Non-Blittable Marshaling
A Word About Complexity

In the July 2003 installment of the .NET column I covered the basics of Win32® interoperation with the Microsoft® .NET Framework (P/Invoke). Based on reader feedback, this topic is worthy of further coverage, so I have decided to revisit P/Invoke in this column. It will build upon the information in the July 2003 issue, so if you are not familiar with basic .NET interop, I suggest reviewing that column before digging into this one.

This month I am going to delve into the interop details of marshaling data structures to native functions. In a future piece, I hope to address calls into native code that calls back into your managed code, and I'll work through an interop case study using Windows® Forms. So let's get started on marshaling structures.

Marshaling Structures

The common language runtime (CLR) is capable of marshaling references to managed memory to native APIs. In order to accomplish this, the runtime has to consider a number of differences between managed memory and native memory. Let's take a look at some of these differences:

  • The field layout of managed reference objects is reorganized by the runtime (by default)
  • Managed reference objects are garbage collected; native memory is not
  • Managed reference objects are moved around in physical memory by the system; native objects are not
  • Certain common native structure layouts, such as inline arrays, are difficult to accomplish using some .NET-compliant languages such as C# and Visual Basic® .NET

These are the major considerations that you and the CLR must account for when moving data between managed and native code. To begin addressing these differences, I'll start with an example. Figure 1 shows an excerpt from the winbase.h header file in the Windows Platform SDK, that contains the declaration of the GetSystemPowerStatus API and the SYSTEM_POWER_STATUS native structure type definition.

Figure 1 Native GetSystemPowerStatus (Close Figure 1)
typedef struct _SYSTEM_POWER_STATUS {
    BYTE ACLineStatus;
    BYTE BatteryFlag;
    BYTE BatteryLifePercent;
    BYTE Reserved1;
    DWORD BatteryLifeTime;
    DWORD BatteryFullLifeTime;
}   SYSTEM_POWER_STATUS, *LPSYSTEM_POWER_STATUS;

BOOL
WINAPI
GetSystemPowerStatus(
    OUT LPSYSTEM_POWER_STATUS lpSystemPowerStatus
    );

Figure 2 shows C# code that makes a call to the native GetSystemPowerStatus API, and outputs information from the resulting data structure. Note that in the C# code it is necessary to redefine the native SYSTEM_POWER_STATUS type definition as SystemPowerStatus, a managed reference type. The type can have any name, but must have a field layout that matches that of the target data structure. It is also possible to marshal managed value types to native code, which I will cover in a moment. But for now, it is important to note that you need to define a managed type that exactly matches the in-memory layout of the native structure that is expected by the API.

Figure 2 GetSystemPowerStatus Call Using Class (Close Figure 2)
class Interop {
  static void OutputBatteryLifeRefType() {
    SystemPowerStatus status = new SystemPowerStatus();
    GetSystemPowerStatusRef(status);
    Console.WriteLine("Battery has {0} percent life remaining.
        State={1}", status._BatteryLifePercent, status._BatteryFlag);
  }

  [DllImport("Kernel32", EntryPoint="GetSystemPowerStatus")]
  static extern Boolean GetSystemPowerStatusRef(SystemPowerStatus sps);

  // In C#, 'class' defaults to automatic field layout
  [StructLayout(LayoutKind.Sequential)]  // Required!
  class SystemPowerStatus {
    public ACLineStatus  _ACLineStatus;
    public BatteryFlag   _BatteryFlag;
    public Byte          _BatteryLifePercent;
    public Byte          _Reserved1;
    public Int32         _BatteryLifeTime;
    public Int32         _BatteryFullLifeTime;
  }

  // Note: Underlying type of byte to match Win32 header
  enum ACLineStatus : byte {
    Offline = 0, Online = 1, Unknown = 255
  }
  enum BatteryFlag  : byte {
    High = 1, Low = 2, Critical = 4, Charging = 8,
    NoSystemBattery = 128, Unknown = 255
  }
}

The SystemPowerStatus class in Figure 2 is attributed with the StructLayoutAttribute class. Any time you use a managed reference type to marshal data to native code, you must apply the StructLayoutAttribute to the type definition and specify a layout kind of Sequential. The LayoutKind.Sequential setting tells the runtime to leave the fields in their defined order, which addresses the first difference in the preceding bulleted list. We will look at the StructLayoutAttribute in more detail in a moment, but first let's dig into the memory ramifications of the code in Figure 2.

Object Lifetime and Pinning

Looking at the call to GetSystemPowerStatusRef in Figure 2, a question arises. What happens if a garbage collection kicks in during the call to the native function? If this happens, won't the object referred to by the status local variable potentially be moved around in memory by the collector? In fact, if status were not referred to by subsequent code, couldn't the collector clean up the object entirely while the native method is executing? The answer to these questions would be yes, were it not for the pinning feature of the CLR.

When the runtime marshaler sees that your code is passing to native code a reference to a managed reference object, it automatically pins the object. What this means is that the object is put in a special state where the garbage collector will neither move the object in memory nor remove the object from memory. Pinned objects hurt the performance of the garbage collector, but they assure that the memory remains intact for the life of the native call; this is critical to the proper functioning of the native code.

When the native function returns, the marshaled object is automatically unpinned. Automatic pinning is very convenient, but it raises another question. What happens if the native function caches the pointer for use later on? When the function returns, won't the collector be free to move the object? The answer is yes, and the solution for your code in such a situation is to manually pin the object using the System.Runtime.InteropServices.GCHandle type.

The following lines of code use the GCHandle type in order to pin and unpin a managed string:

String s = "PinMe";
GCHandle pinHandle = GCHandle.Alloc(s, GCHandleType.Pinned);
try
{
   •••
}
finally
{
   pinHandle.Free();
}

Manually pinning an object is not difficult. The tricky part is knowing when you should. The GetSystemPowerStatus API, for example, does not cache the pointer that is passed in as its only argument. And so, automatic pinning is sufficient for calls to this native API. However, an overlapped call to the native ReadFileEx function would require manual pinning of the second and fourth parameters to the call because the API will continue to utilize the referenced memory after the function returns. You need to know the behavior of the function you are calling well enough to decide whether automatic pinning is sufficient, or whether you will need to manually pin your managed object.

Pinning addresses the differences between managed and native memory in terms of moving memory around and collecting unreferenced objects. Now let's take a look at how the native call in Figure 2 differs if we use a value type rather than a reference type.

Marshaling Value Types vs. Reference Types

The code in Figure 3 functions similarly to the code in Figure 2 in that it makes a call to the native GetSystemPowerStatus function. However, in Figure 3 the SystemPowerStatus type is defined as a managed structure rather than a managed class. Said another way, we are marshaling a value type rather than a reference type.

Figure 3 GetSystemPowerStatus Call Using Struct (Close Figure 3)
class Interop {
  static void OutputBatteryLifeValType() {
    SystemPowerStatus status;
    GetSystemPowerStatusVal(out status);
    Console.WriteLine("Battery has {0} percent life remaining.
        State={1}", status._BatteryLifePercent, status._BatteryFlag);
  }

  [DllImport("Kernel32", EntryPoint="GetSystemPowerStatus")]
  static extern Boolean
      GetSystemPowerStatusVal(out SystemPowerStatus sps);

  // In C#, 'struct' defaults to sequential field layout
  struct SystemPowerStatus {
    public ACLineStatus  _ACLineStatus;
    public BatteryFlag   _BatteryFlag;
    public Byte          _BatteryLifePercent;
    public Byte          _Reserved1;
    public Int32         _BatteryLifeTime;
    public Int32         _BatteryFullLifeTime;
  }
}

When your code defines a variable of a reference type, the variable is either a reference to an object or it is a null reference. In contrast, a variable of a value type is the actual data, and no reference is involved. This difference between the two data types in the managed type system affects marshaling to native code.

When you pass a reference variable to a native function, the runtime marshals the reference into a native pointer to the data structure. However, when you marshal a value to a native function, the runtime pushes a copy of the value object onto the thread's stack. Sometimes this is what you want, but more often a native API will expect a pointer to a structure (as is the case with GetSystemPowerStatus), and so you have to take steps to indicate to the system that a pointer to the variable should be passed, rather than a copy of the variable.

To do this, you can define the parameter to your static extern P/Invoke method as an out or ref parameter. Remember that this tells the runtime to pass a reference to the variable, rather than to the data in the variable on the stack.

Fundamentally, the difference between marshaling classes and structures is one of indirection. When you marshal a managed reference type, you are always passing either a pointer or a pointer to a pointer. When you marshal a managed value type, you are either passing the data wholesale on the stack or you are passing a pointer to the data. Thus, you can marshal value types with either zero or one level of indirection, while reference types can be marshaled using one or two levels of indirection to the native code.

Surprisingly, you often find that marshaling managed value types is preferable to marshaling managed reference types. One reason is that managed structures are somewhat more similar to C structures than are managed classes. The runtime never reorganizes the field layouts of managed structures written in C# or Visual Basic .NET, and the garbage collector never moves unboxed values around in memory. Most structure marshaling can be performed using either value types or reference types. However, until you have a compelling reason to use a class to marshal data to native code, you should generally use structures.

StructLayoutAttribute

You may have noticed that StructLayoutAttribute is absent from the structure definition in Figure 3. This is because the C# compiler emits one automatically for structures. However, you may still need to use StructLayoutAttribute explicitly with your value type definitions for its other features. StructLayoutAttribute allows you to specify four marshaling settings for your managed reference and value types. They are as follows: layout kind, character set, field packing, and size.

The LayoutKind enumerated value of StructLayoutAttribute can be any of three values: Auto, Sequential, and Explicit. We have already addressed Sequential, which is typically the setting you will use and which is the layout emitted by the C# and Visual Basic .NET compilers when compiling value types not explicitly attributed with a StructLayoutAttribute. Reference types not marked with StructLayoutAttribute default to using Auto layout, which is not a valid setting for code that interoperates with native functions. Finally, there is the Explicit value.

When you set LayoutKind to Explicit, you are telling the marshaler that you want to lay out the fields in your type on a field-by-field basis. Using the Explicit setting also requires you to apply FieldOffsetAttribute to each field in your data type, specifying the relative offset (in bytes) of each field. You do not usually need the full flexibility of LayoutKind.Explicit; however, it is useful for marshaling more advanced data structures, such as unions in C, where fields overlap one another.

The CharSet property of StructLayoutAttribute lets you specify whether Char and String fields in your type are marshaled as one-byte ANSI values or two-byte Unicode values. The CharSet enumerated type includes Ansi, Unicode, and Auto settings. The default is Ansi and this is almost never what you want. Instead, when marshaling a structure containing text, you should generally use the Auto setting, which uses ANSI on Windows 95-based systems and Unicode on Windows NT®-based systems. We'll take a look at an example of marshaling a data structure containing text in just a moment.

The Pack and Size properties of StructLayoutAttribute allow you to specify the packing boundary for your fields and the size of your structure, respectively. The default packing size is eight bytes, which is consistent with most of the type definitions in the Win32 SDK. The size of your data type in memory is usually based on the fields in the type definition. But you can use the Size property to manually set the memory footprint of your type. The Marshal.SizeOf method returns the size, in bytes, of the marshaled version of a managed object or type.

Non-Blittable Marshaling

So far we have only looked at marshaling cases where the runtime is able to pass to native code a pointer that points directly to managed memory. Two things make this possible: pinning and a memory layout consistent with what native code expects. Sometimes, however, it is impractical or impossible for your managed memory to be laid out in the same arrangement as a native equivalent data structure.

In these cases, the runtime makes a copy of your managed object using native memory and native layout. A pointer to the copy is then passed to the native call. If the parameter to the extern method is also marked as [Out] using OutAttribute, then the marshaler will also copy the memory back into your managed ob-ject when the call returns. In these cases, you may have to use MarshalAsAttribute to help the marshaler understand what the native memory layout is supposed to look like. Structures and classes that require marshaling in this fashion are called non-blittable types, and they incur substantially more overhead from the runtime due to the memory allocations and copies. Note, though, that there are some non-blittable types where MarshalAs isn't needed. And there are cases where you could use MarshalAs (such as marking an integer with I4), although in reality the setting would be ignored. And so, while the presence of the MarshalAsAttribute is not necessarily an indicator of whether a type is blittable or non-blittable, it is often useful or necessary for marshaling of non-blittable types.

Let's look at an example. The code in Figure 4 is excerpted from header files in the Win32 Platform SDK. It shows the ANSI versions of the GetVersionEx API function and OSVERSIONINFO structure. The OSVERSIONINFO structure contains an inline array of 128 CHAR values. This field is interesting for two reasons. It is not directly expressible in C# as arrays are always reference types. Also, it is a character array and therefore could be a total of 128 bytes or 256 bytes depending on whether ANSI or Unicode is used. Figure 5 contains the C# code that calls this API.

Figure 5 Managed Non-Blittable Call to GetVersionEX (Close Figure 5)
class Interop {
   static void OutputVersionInfo() {
      OSVersionInfo info = new OSVersionInfo();
      GetVersionEx(info);
      Console.WriteLine("Extra version info string = {0}",
         info.CSDVersion);
   }

   [DllImport("Kernel32", CharSet=CharSet.Auto)]
   static extern Boolean GetVersionEx(
      [Out][In]OSVersionInfo, versionInformation);

   [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Auto)]
   class OSVersionInfo {
      public UInt32 OSVersionInfoSize =
         (UInt32) Marshal.SizeOf(typeof(OSVersionInfo));
      public UInt32 MajorVersion      = 0;
      public UInt32 MinorVersion      = 0;
      public UInt32 BuildNumber       = 0;
      public UInt32 PlatformId        = 0;
      // Attribute used to indicate marshalling for String field
      [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 128)]
      public String CSDVersion        = null;
   }
}
Figure 4 Native GetVersionEX (Close Figure 4)
typedef struct _OSVERSIONINFOA {
    DWORD dwOSVersionInfoSize;
    DWORD dwMajorVersion;
    DWORD dwMinorVersion;
    DWORD dwBuildNumber;
    DWORD dwPlatformId;
    CHAR   szCSDVersion[ 128 ];     // Maintenance string for PSS usage
} OSVERSIONINFOA, *POSVERSIONINFOA, *LPOSVERSIONINFOA;

WINBASEAPI
BOOL
WINAPI
GetVersionExA(
    IN OUT LPOSVERSIONINFOA lpVersionInformation
    );

Notice that the OSVersionInfo class definition in Figure 5 has replaced the inline CHAR array in the native structure with a String object reference in the managed object. Notice also that the field is attributed with MarshalAsAttribute, which specifies that the field should be marshaled as a ByValTStr with a size of 128 characters. I could have chosen to marshal this field as a managed Char[] as well, however that would have the same memory layout as String, and so I chose to use the String for the sake of convenience. See Figure 6 for a picture of the memory layout differences between the native structure and the managed representation of the native structure.

Figure 6 Layout Differences in Native and Managed Memory
Figure 6 Layout Differences in Native and Managed Memory

Figure 6 should make it clear why the runtime can't just pass a pointer to the managed object directly to the native API. The native API is expecting a memory layout similar to the block on the left, where the char array is part of the memory block. The native API would simply treat the pointer/reference field in the managed memory block as two or four char values and would continue to clobber memory after the managed object's block. This is why the runtime must make a second copy of the managed structure with a layout equivalent to the native structure.

In brief, when cases arise in which there is no way to directly express the memory layout of the target structure, you can use MarshalAsAttribute to specify alternate memory arrangements for the marshaled version of the data. In all cases like this, the CLR will have to copy the managed fields to a native memory buffer before calling the native API.

In native code, there is no equivalent to managed metadata. This is why you have to carefully define your managed types to match the resulting native memory expected by the target function. However, the native code's lack of knowledge about the passed-in buffer also allows you to make manual adjustments to meet your needs, for example to handle considerations such as performance.

For instance, an alternative implementation of the managed OSVersionInfo class may choose to ignore the string returned in the last field and just leave that field out of the class definition. Of course, in an implementation such as this, the Size property of the StructLayoutAttribute must be used to extend the size of the instance to leave space for the data; otherwise a call to the native function would cause memory corruption.

Similarly, it is possible to define the last field as 128 individual char fields. Doing this, of course, is tedious. But this up-front effort pays off in back-end performance. The sample code for this column, available on the MSDN®Magazine Web site, includes an implementation of OSVersionInfo that takes this approach. Look for OSVersionInfoBlittable in the sources. The OSVersionInfoBlittable structure is both a value type and is blittable. Calls to GetVersionExBlit will be substantially faster than calls to the counterpart function in Figure 5. Meanwhile, you can use a public property to provide friendly String access to the otherwise unfriendly block of Char fields. The get accessor method in the CSDVersion property uses unsafe C# syntax to create a String object using the address of the first Char in the block of Chars. This has the effect of deferring the conversion portion of the marshaling until code accesses the String. It also limits the conversion to the String portion of the data structure.

Also included in the download is a performance test that runs the blittable and non-blittable interop methods a million times while timing them. On my system the blittable call to GetVersionInfo was over 10 times faster than the non-blittable counterpart.

A Word About Complexity

The P/Invoke facilities of the CLR are very complete and should allow for marshaling of many different kinds of data to native code. As the complexity of your data structures increase, however, you will find more use for unsafe code blocks in C#, the MarshalAsAttribute, and other detailed tools in the interop arsenal. Getting interop right can prove tedious as things get complex.

Part of what makes interop complicated using C# is the fact that C# does not know how to natively read C/C++ header files. If it did, it could help out significantly in your P/Invoke work. However, the C++ compiler is capable of building managed code, and can read standard header files directly. You may find that, for interoperation with certain native functions, a helper DLL written using managed C++ saves some trouble, compared to P/Invoking directly from C#.

That's it for this month's column. Browse over to http://www.pinvoke.net, a site created by Adam Nathan to allow developers to find, edit, and add P/Invoke signatures for public consumption. This has helped many people P/Invoke to Win32 and other APIs without having to battle the complexity alone.

Send your questions and comments for Jason to dot-net@microsoft.com.

Jason Clark provides training and consulting for Microsoft and Wintellect and is a former developer on the Windows NT and Windows 2000 Server team. He is the coauthor of Programming Server-side Applications for Microsoft Windows 2000 (Microsoft Press, 2000). You can get in touch with Jason at JClark@Wintellect.com.


Cutting Edge: Binary Serialization of DataSets

Cutting Edge
Binary Serialization of DataSets
Dino Esposito

Contents
The Serialization Problem Defined
How DataSet Serialization Really Works
Avoiding the Problem
Drilling Down a Little Further
DataSet Serialization in ADO.NET 2.0
DataTable Enhancements
Call to Action

The ADO.NET DataSet object plays an essential role in most of today's distributed, multitiered applications. Instances of the DataSet class are used to move data across the tiers and to exchange data with external services. The DataSet has most of the features needed to represent real-world business entities.

First, the DataSet is a disconnected object—you can use it without a physical connection to the data source. In addition, the DataSet provides a rich programming interface. It supports multiple tables of data, accepts relations defined between pairs of tables, and allows you to enumerate its contents. Last but not least, the DataSet is a fully serializable object. It can be serialized in three different ways—to a standard .NET formatter, to an XML writer, and through the XML serializer.

For objects that represent the real-world entities of a multitiered app, serialization is key because the object must be serialized to be moved across tiers. This month, I'll discuss a problem with serialization in ADO.NET 1.x and how serialization in ADO.NET 2.0 will improve upon it. Back in the December 2002 installment of Cutting Edge, I reviewed the serialization capabilities of the DataSet object, which you might want to refer to for a little background.

The Serialization Problem Defined

I have a client with a classic three-tier application in which a number of user interface process components were moving data up and down to business components through a .NET Remoting channel. The runtime environment of .NET Remoting used a custom host application configured to work over binary serialization. The client chose binary serialization, thinking it was the fastest way to move data between processes. But their application was running slowly and experiencing problems moving data. Hence, the client put a sniffer on the network to look at the size of the data being moved back and forth and discovered that too many bytes were being transported; the actual size of the data was grossly exceeding the expected size. So what was going on?

How DataSet Serialization Really Works

Marked with the [Serializable] attribute, the DataSet object implements the ISerializable interface to gain full control over its serialization process. The .NET formatter streamlines, in a default way, objects that don't implement ISerializable and guarantees that data is stored as a SOAP payload if the SoapFormatter is used or as a binary stream if the BinaryFormatter is used. The responsibility of the formatters, however, ends here if the serialized object supports ISerializable. In that case, the formatter passes an empty memory buffer (the SerializationInfo data structure) and waits for the serializee to populate it with data. The formatter's activities are limited to flushing this memory buffer to a binary stream or wrapping it up as a SOAP packet. However, there are no guarantees made about the type of the data added to the SerializationInfo object.

The following pseudocode shows how the DataSet serializes with a .NET formatter. The GetObjectData method is the sole method of the ISerializable interface.

void GetObjectData(SerializationInfo info, StreamingContext context)
{
   info.AddValue("XmlSchema", this.GetXmlSchema());
   this.WriteXml(strWriter, XmlWriteMode.DiffGram);
   info.AddValue("XmlDiffGram", strWriter.ToString());
}

Regardless of the formatter being used, the DataSet always serializes first to XML. What's worse, the DataSet uses a pretty verbose schema—the DiffGram format plus any related schema information. Now take a DataSet with a few thousand records in it and imagine such a large chunk of text traveling over the network with no sort of optimization or compression (even blank spaces aren't removed). That's exactly the problem that I'm sure many of you have been called to solve at one time or another.

Avoiding the Problem

Until ADO.NET 2.0 the prognosis was mixed. There was both good and bad news. The bad news was that this problem is virtually unavoidable if you choose the DataSet as the implementor of your business entities. Alternative choices are raw XML, custom collections, and arrays of custom classes. All of these have pros and cons. All are serializable and enumerable data types. In addition, raw XML and custom types are better at interoperating with components running on platforms such as J2EE than is the DataSet. This assumes you're employing Web services to bridge user interface process components to business entities (see Figure 1). For more information about Web services and DataSets, check out The XML Files column in the April 2003 issue of MSDN®Magazine.

Figure 1 User Interface
Figure 1 User Interface

If you modify your architecture to do without DataSets, you avoid the problem of serialization but lose some usability and simplicity. The DataSet is a great class to code against. Using it you can pass the full DataSet around, and then if needed, you can also easily excerpt only the changed rows in each table in order to minimize bandwidth and CPU usage for a call.

The good news is that there are some workarounds. To see the various options compared, check out the DataSet FAQ. One way to improve the end-to-end transfer speed is to override the DataSet serialization mechanism.

You can create your own DataSet-derived class and implement the members of the ISerializable interface to fulfill your performance and scalability requirements. As an alternative, you can generate a typed DataSet using Visual Studio® .NET and then modify its source code to reimplement the ISerializable interface as needed. A third option is to use a little-known feature of the .NET Framework—serialization surrogates. Jeffrey Richter provides excellent coverage of serialization surrogates in his September 2002 installment of the .NET column. Technically speaking, a serialization surrogate is a class that implements the ISerializationSurrogate interface. The interface consists of two members: GetObjectData and SetObjectData.

By using surrogates you can override the way types serialize themselves. The technique has a couple of interesting practical applications: serialization of unserializable types and deserialization of an object to a different version of its type. Surrogates work in cooperation with .NET runtime formatters to handle serialization and deserialization for a given type. For example, the BinaryFormatter class has a member named SurrogateSelector that installs a chain of surrogates for a variety of types. When instances of any of these types are going to be serialized or deserialized, the process won't go through the object's serialization interface (ISerializable or the reflection-based default algorithm) but rather takes advantage of the surrogate's capabilities using its ISerializationSurrogate interface. Here's an example of how to override the serialization of a DataSet using a custom surrogate DataSetSurrogate:

SurrogateSelector ss = new SurrogateSelector();
DataSetSurrogate dss = new DataSetSurrogate();
ss.AddSurrogate(typeof(DataSet),
    new StreamingContext(StreamingContextStates.All), dss);
formatter.SurrogateSelector = ss;

In the DataSetSurrogate class, you implement the GetObjectData and SetObjectData methods to work around the known performance limitations of the standard DataSet serialization. Next, you add an instance of the DataSet surrogate to a selector class. Finally, the selector class is bound to an instance of the formatter.

Whichever approach you choose to improve the DataSet's serialization performance (a new class or a serialization surrogate), the end goal should be to reduce the amount of information being moved.

The DataSet serializes to an XML DiffGram—a rich XML schema that contains the current snapshot of the data as well as pending errors on table rows and a history of all the changes that occurred on the rows. The history is actually the original value of the row assigned when either the DataSet was created or the last time the AcceptChanges method was invoked.

Since the main problem is the size of the data, compression is one possible answer. The solution outlined in Knowledge Base article 829740 ("Improving DataSet Serialization and Remoting Performance employs a DataSet-like class that is simply marked as Serializable and doesn't implement ISerializable. Other ADO.NET objects (DataTable, DataColumn) are also replaced to take full control of the serialization process. This simple change provides a double advantage: it reduces the amount of data being moved and reduces the pressure on the .NET Remoting framework to serialize and deserialize larger DataSet objects.

Another approach to the problem, that is much more than just a smart way to serialize a DataSet, involves implementing a custom formatter, such as Angelo Scotto's CompactFormatter (available for download at http://www.freewebs.com/compactFormatter/About.html). The CompactFormatter is a generic formatter for both the Microsoft® .NET Framework and the .NET Compact Framework and is capable of producing an even more compact byte stream than the native BinaryFormatter class. In addition, it supports compression, which further reduces the amount of data being moved. More importantly, the CompactFormatter is not DataSet-specific and works with most .NET types.

Drilling Down a Little Further

All .NET distributed systems that make intensive use of disconnected data (as recommended by Microsoft architecture patterns and practices) are sensitive to the size of serialized data. The larger the DataSet, the more CPU cycles, memory, and bandwidth these systems consume. Fortunately, ADO.NET 2.0 provides a great fix. Before I explain it, though, I should make clear that I'm not saying DataSet XML serialization is a bad thing in general. Standard DataSet XML serialization is a "stateful" form of serialization in the sense that it maintains some state information such as pending errors and current changes to rows. It can be customized to a great extent by choosing the favorite combination of nodes and attributes; you can also decide if relations should simply be tracked or rendered by nesting child records under their parents.

What's really interesting is the fact that as long as few records (for example, less than 100) are involved in the serialization, the performance of a standard DataSet and that of an optimized DataSet (custom DataSet, surrogate, compressed, whatever) is negligible. Therefore, I wouldn't spend much time implementing alternative forms of DataSet serialization if my application moves only small chunks of data.

Although performance worsens as the size of the DataSet grows, even for a thousand records the performance is not much of a problem. Note that performance here relates to .NET Remoting end-to-end speed, a parameter that includes the size of the data to transfer and the costs of successive instantiations. As you scale from a thousand to a few thousand records, the performance begins to suffer significantly. Figure 2 shows the difference in .NET Remoting end-to-end speed when XML (standard) and true binary serialization are used (the latter is new to ADO.NET 2.0; more on this in a moment). For the sake of precision, I have to say that the numbers behind the graph have been obtained with a Beta build of ADO.NET 2.0.

Figure 2 Remoting End-to-End Time
Figure 2 Remoting End-to-End Time

Of course the worst scenario is when you move a large DataSet (thousands of rows) with numerous changes. If you're simply forwarding this DataSet to the data access layer (DAL) for applying changes, you can alleviate the issue by using the DataSet's GetChanges method. This method returns a new DataSet that contains only the rows in the various tables that have been modified, added, or deleted.

However, when you're moving a DataSet from tier to tier, or from a business component to an external service that implements a business process, you have no choice but to pass a stateful representation of the data—the whole DataSet with its own set of relations, pending changes, and errors.

Why does the end-to-end speed slow down once you reach a certain threshold? Handling large DataSets poses an additional problem to the .NET Remoting infrastructure aside from the time and space needed to complete the operation. This extra problem has to do with the specific algorithm used to serialize and, more importantly, deserialize DataSets saved as XML DiffGrams. To restore a DataSet from a DiffGram, the binary formatter invokes a protected constructor on the DataSet class (part of the ISerializable implementation). The pseudocode of this DataSet constructor is shown in Figure 3. As you can see, the DataSet's ReadXml method is called to process the DiffGram. Especially for large DataSets, ReadXml works by creating lots of transient, short-lived objects, a few for each row to be processed. This mechanism puts a lot of additional pressure on the .NET Remoting infrastructure and consumes a lot of memory and CPU cycles. Exactly how bad it is will depend on the runtime conditions and hardware equipment of the system. This explains why certain clients experience visible problems when moving only 7MB of data while others won't see problems until they've moved 20MB of data.

Figure 3 Pseudocode for DataSet Serialization (Close Figure 3)
protected DataSet(SerializationInfo info, StreamingContext context)
{
    string schema, diffgram;

    schema = (string) info.GetValue("XmlSchema", typeof(string));
    diffgram = (string) info.GetValue("XmlDiffGram", typeof(string));
    if (schema != null)
         ReadXmlSchema(new XmlTextReader(new StringReader(schema)), true);
    if (diffgram != null)
        ReadXml(new XmlTextReader(new StringReader(schema)),
            XmlReadMode.DiffGram);
}

DataSet Serialization in ADO.NET 2.0

When you upgrade to ADO.NET 2.0, your problems will be solved. In ADO.NET 2.0, the DataSet class provides a new serialization option specifically designed to optimize remoting serialization. As a result, remoting a DataSet uses less memory and bandwidth. The end-to-end latency is greatly improved, as Figure 2 illustrates.

In ADO.NET 2.0, the DataSet and DataTable come with a new property named RemotingFormat defined to be of type SerializationFormat (see Figure 4). By default, the new property is set to SerializationFormat.Xml to preserve backward compatibility. The property affects the behavior of the GetObjectData members on the ISerializable interface and ultimately is a way to control the serialization of the DataSet. The following code snippet represents the pseudocode of the method in ADO.NET 2.0:

void GetObjectData(SerializationInfo info, StreamingContext context)
{
  SerializationFormat fmt = RemotingFormat;
  SerializeDataSet(info, context, fmt);
}
SerializeDataSet is a helper function that fills the SerializationInfo memory buffer with data that represents the DataSet. If the format parameter is SerializationFormat.Binary, the function goes through every object in the DataSet and copies its contents into a serializable structure—mostly ArrayLists and Hashtables. Figure 5 shows the pseudocode. If you compare that to the code in the aforementioned Knowledge Base article 829740, you'll find many similarities. Note, though, that the code in Figure 5 is based on Beta 1 and might change in the future. What shouldn't change, though, is the idea behind the binary serialization. To serialize a DataSet in a true binary fashion, here's what you do:
DataSet ds = GetData();
ds.RemotingFormat = SerializationFormat.Binary;
BinaryFormatter bin = new BinaryFormatter();
bin.Serialize(stream, ds);
Aside from the new RemotingFormat property, there's nothing new in this code. The impact of this new feature on existing code is very minimal.

Figure 5 Pseudocode for DataSet's Binary Serialization (Close Figure 5)
private void SerializeDataSet(
  SerializationInfo info, StreamingContext context,
  SerializationFormat remotingFormat)
{
  info.AddValue("DataSet.RemotingVersion", new Version(2, 0));
  if (remotingFormat != SerializationFormat.Xml)
  {
     int i;
     info.AddValue("DataSet.RemotingFormat", remotingFormat);
     SerializeDataSetProperties(info, context);

     info.AddValue("DataSet.Tables.Count", this.Tables.Count);

     for (i=0; i< Tables.Count; i++)
         Tables[i].SerializeConstraints(info, context, i, true);

     SerializeRelations(info, context);

     for (i=0; i< Tables.Count; i++)
         Tables[i].SerializeExpressionColumns(info, context, i);

     for (int=0; i< Tables.Count; i++)
         Tables[i].SerializeTableData(info, context, i);

     return;
   }

   // 1.x code
}
Figure 4 SerializationFormat Values (Close Figure 4)

Value | Description
Xml | Maintained for backward compatibility, serializes the DataSet object using an XML DiffGram format as in ADO.NET 1.x; this is the default value of the RemotingFormat property
Binary | Instructs the internal serializer to use a true binary format when serializing the DataSet

It is interesting to measure the performance gain that you get from this new feature. Here's a simple technique you can easily reproduce. Fill a DataSet with the results of a query and persist it to a file on disk (see Figure 6). You can wrap the code in Figure 6 in either a Web Form or a Windows Form. Run the sample application and take a look at the size of the files created. Try first with a simple query like this:

SELECT lastname, firstname FROM employees

Figure 6 Testing Performance of Remoting Format (Close Figure 6)
SqlDataAdapter adapter = new SqlDataAdapter(query, connString);
DataSet ds = new DataSet();
adapter.Fill(ds);

BinaryFormatter bin = new BinaryFormatter();

// Save as XML
using(StreamWriter writer1 = new StreamWriter(@"c:\xml.dat"))
{
    bin.Serialize(writer1.BaseStream, ds);
}

// Save as binary
using(StreamWriter writer2 = new StreamWriter(@"c:\bin.dat"))
{
    ds.RemotingFormat = SerializationFormat.Binary;
    bin.Serialize(writer2.BaseStream, ds);
}

If you're familiar with the Northwind database, you know that this query returns only nine records. Quite surprisingly, in this case the XML DiffGram is about half the size of the binary file! Don't worry, there's nothing wrong in the code or in the underlying technology. To see the difference, run a query that will return a few thousand records, like this one:

SELECT * FROM [order details]

Now the binary file is about 10 times smaller than the XML file. With this evidence, look back at the graph in Figure 2. The yellow series shows the performance of the DataSet binary serializer and fortunately increases very slowly as the size of the DataSet grows. The same can't be said for the DataSet XML serializer which reports a sudden upswing and significantly decreased absolute performance as the number of rows exceeds a few thousand.

DataTable Enhancements

In ADO.NET 1.x, the DataTable class suffers from three main limitations. A DataTable can't be used with Web service methods and doesn't provide a direct way to serialize its contents. In addition, while DataTable instances can be passed across a remoting channel, strongly typed DataTable instances cannot be remoted. These limitations have been removed in ADO.NET 2.0. Let's take a look at exactly how this can be accomplished.

Return values and input parameters of Web service methods must be serializable through the XmlSerializer class. This class is responsible for translating .NET types into XML Schema Definition (XSD) types and vice versa. Unlike the runtime formatter classes (such as BinaryFormatter), the XmlSerializer class doesn't handle circular references. In other words, if you try to serialize a DataTable you get an error because the DataTable contains a reference to a DataSet—the DataSet property—and the DataSet, in turn, contains a reference to the DataTable through the Tables collection. For this reason, XmlSerializer raises an exception if you try to pass a DataTable to, or return a DataTable from, a Web service method. DataSet objects have the same circular reference, but unlike the DataTable class, DataSets implement the little-known IXmlSerializable interface.

The XmlSerializer class specifically looks for this interface when serializing an object. If the interface is found, the serializer yields control and waits for the WriteXml/ReadXml methods to terminate the serialization and deserialization process. The IXmlSerializable interface has the following three members:

XmlSchema GetSchema();
void ReadXml(XmlReader reader);
void WriteXml(XmlWriter writer);

In ADO.NET 2.0, the DataTable class fully supports the IXmlSerializable interface so you can finally use DataTables as input parameters or return values in Web service methods.

Note that the GetSchema method exists on IXmlSerializable purely for backward compatibility, and it is safe to choose a trivial implementation of GetSchema that returns null. However, if the class is being used in a Web service method's signature, an XmlSchemaProviderAttribute should be applied to the class to denote a static method on the class that can be used to get a schema. This XSD representation can then be mapped back to the actual object by implementing a SchemaImporterExtension. For more information on XmlSchemaProviderAttribute and SchemaImporterExtension, see New Features for Web Service Developers in Beta 1 of the .NET Framework 2.0.

In addition, the DataTable provides a simple XML I/O channel. A DataTable can be populated from an XML stream using the new ReadXml method and persisted to disk using the new WriteXml method. The methods have the same role and signature as the equivalent methods on the DataSet class. Last, but not least, a DataTable supports the RemotingFormat property and can be serialized in a true binary format.

Note that the list of changes for the DataTable class doesn't end here. For example, the class also features full integration with streaming interfaces and can be loaded from readers (including SQL DataReaders) and its contents can be read through DataReaders. However, these changes don't directly (or necessarily) affect the building of middle tiers of enterprise class applications and in this column I don't have the space to cover them or even the majority of the new ADO.NET 2.0 features.

Call to Action

One of the main ADO.NET 2.0 themes is the emphasis on improved performance and scalability. This encompasses a new, faster index engine to locate table records and the oft-requested binary serialization for DataSets. By adding the same capabilities to the DataTable class, I'd say the team exceeded developers' expectations by a wide margin.

So what's the best thing to do if you have an application that suffers from DataSet serialization performance problems? Once you upgrade to ADO.NET 2.0, fixing the problem is as easy as adding one line of code—the one that sets the RemotingFormat property on the DataSet. The fix is simple to apply and terrifically effective. Especially if you haven't made any progress on alternative routes (like compressors, surrogates, and the like), I suggest you stay and start planning a full migration for when the new platform (or at least its Go-Live license) is available. If you absolutely need a fix today, I suggest that you choose the one that has the least impact on your system. My experience suggests that since most applications use typed DataSets you can just modify the GetObjectData method of your typed DataSets to use surrogates (see the Knowledge Base article 829740) or, more simply, zip the XML DiffGram. The performance improvement is immediate, though not as good as it could be. It's a workaround after all.

ADO.NET 2.0 is evolution, not revolution. It's a fully backward-compatible data access platform in which the RemotingFormat property stands out as the feature that really fixes many existing enterprise applications, allowing Web services to bridge user interface components and business entities.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and the newest Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at http://weblogs.asp.net/despos.


Service Station: Securing Web Services with WSE 2.0

Service Station
Securing Web Services with WSE 2.0
Aaron Skonnard

Code download available at: ServiceStation0410.exe (148KB)
Contents
Requiring UsernameTokens
Authentication and Authorization
Implementing a UsernameTokenManager
Requiring Signatures
Using DerivedKeyTokens
Requiring Encryption
Signing and Encrypting with X.509 Tokens
Automating Security with Policy
Note on the UsernameTokenManager
Where Are We?

Beginning this month, The XML Files will run under the name Service Station. We have made this change so that the column can discuss broader topics such as Web services, service-oriented architecture, and the like.

In the August 2004 issue, I gave you an introductory tour of the various new and improved features in Web Services Enhancements for Microsoft® .NET (WSE) 2.0. This month I'll take you on a tour of the WSE 2.0 security features and show you how to automate security code with declarative policies.

Simply installing WSE 2.0 does not make your ASMX endpoints secure. Configuring your ASMX projects with the WSE SoapExtension makes them capable of processing security elements found in the messages (such as verifying signatures, decrypting elements, and so forth), but these security elements aren't required by the endpoint. You must take action to require such security elements by either writing code or applying a policy. You must also either write code or apply a policy to generate messages that meet the security requirements of a secure endpoint.

Let's start with a simple unsecured ASMX operation:

[WebMethod]
public MortgagePayments CalcMortgage(
  double amount, double interest, double years,
  double annualTax, double annualIns)
{
    MortgageCalc mc = new MortgageCalc();
    return mc.CalculateMortgage(amount, years, interest,
      annualTax, annualIns);
}

As it stands, the CalcMortgage operation is open to a variety of attacks. First, the operation can be invoked by anyone. There's no way to identify the sender and verify that it actually has rights to perform the given operation. Second, there's no way to verify the integrity of incoming messages. Malicious parties can intercept incoming messages, tamper with them, and forward them to the endpoint without being detected. And finally, the messages are available for anyone to read since they're sent in plain text.

I'll show you how to tackle these problems one at a time. First, you need to require the sender to provide a UsernameToken for identification, authentication, and authorization purposes.

Requiring UsernameTokens

Assuming WSE 2.0 is installed and configured properly (note that you must enable the WSE SoapExtension in the service), you can add some code to CalcMortgage to require the presence of a UsernameToken in incoming messages. You can inspect the WS-Security elements found in incoming messages through the SoapContext object. You can look through the SoapContext to determine if a UsernameToken was supplied by the sender.

I've added a new class to the Web service project called WseHelpers, which contains a single method called GetUsernameToken. This method expects the caller to supply it with a SoapContext object that it will inspect for a UsernameToken element, which it returns to the caller when found. The method throws an exception if it doesn't find a UsernameToken (see Figure 1).

Figure 1 Requiring a UsernameToken (Close Figure 1)
public class WseHelpers
{
  public static UsernameToken GetUsernameToken(SoapContext context)
  {
    if (context == null)
      throw new Exception("Only SOAP requests are permitted.");

    // Make sure there's at least one token
    if (context.Security.Tokens.Count == 0)
    {
      throw new SoapException("Missing security token.",
        SoapException.ClientFaultCode);
    }
    else
    {
      // Make sure a UsernameToken is present and return it
      foreach (UsernameToken tok in context.Security.Tokens)
        return tok;
      throw new Exception("UsernameToken not supplied.");
    }
  }
}

You can now use this method to require a UsernameToken in all messages entering CalcMortage by simply calling it at the beginning of the method, as shown here:

[WebMethod]
public MortgagePayments CalcMortgage(
  double amount, double interest, double years,
  double annualTax, double annualIns)
{
  WseHelpers.GetUsernameToken(RequestSoapContext.Current);
  MortgageCalc mc = new MortgageCalc();
  return mc.CalculateMortgage(amount, years, interest,
    annualTax, annualIns);
}

After making this change, you won't be able to invoke the operation without supplying a UsernameToken. To test this, I've provided a sample Windows® Forms application (available in the code download) that is currently configured to invoke this operation without doing anything related to security. Now when you run the client you'll get the "Missing security token" exception.

Now that the CalcMortgage operation requires a UsernameToken, you need to update the client application to provide one. WSE 2.0 provides a UsernameToken class that represents the <wsse:UsernameToken> element. You only have to instantiate a UsernameToken object and add it to the proxy's SoapContext object, as illustrated here:

// retrieve loan values from form
MortgageCalculatorWse mc = new MortgageCalculatorWse();

// instantiate UsernameToken and add to SoapContext
UsernameToken tok = new UsernameToken("mike", "ekim",
  PasswordOption.SendPlainText);
mc.RequestSoapContext.Security.Tokens.Add(tok);

MortgagePayments pay = mc.CalcMortgage(amount, interest,
   year, annualTax, annualIns);
Then, when you invoke any operations through the Web service proxy class, the WSE runtime will automatically add the appropriate security tokens to the SOAP message.

In this case, the password is sent in plain text. You can specify how you want the password to be sent when you instantiate the UsernameToken object. I used the PasswordOptions.SendPlainText option in this example. The other two options allow you to send a hashed password or no password at all. It's obviously not a good idea to send the password in plain text unless you plan to send the SOAP message over a secure channel or to encrypt the UsernameToken element (more on this shortly).

Now when you run the client application supplying a UsernameToken, you won't get the "Missing security token" exception anymore. You must, however, supply the credentials of a valid user account or you'll get the following exception: "The security token could not be authenticated or authorized."

Authentication and Authorization

One of the benefits of sending passwords in plain text is that Windows can automatically authenticate the incoming message by calling the Win32® LogonUser API. If the call fails, WSE 2.0 throws an exception and the request is not allowed through to the ASMX operation. On the other hand, if the call is successful, WSE 2.0 initializes the UsernameToken's Principal property with the authenticated user. The Principal object can be used to authorize access to the operation.

This is accomplished through the IsInRole method, as illustrated in Figure 2. Here we're verifying that the authenticated user is a member of the Broker group on the machine hosting the service. With this code in place, the CalcMortage endpoint knows that only authenticated users belonging to the Broker role will be allowed to access the service.

Figure 2 IslnRole Method (Close Figure 2)
[WebMethod]
public MortgagePayments CalcMortgage(
  double amount, double interest, double years,
  double annualTax, double annualIns)
{
  // retrieve UsernameToken from message
  UsernameToken tok = WseHelpers.GetUsernameToken(
    RequestSoapContext.Current);

  // authorize access based on user roles
  if (!tok.Principal.IsInRole("SKON-COMPAQ\\Broker"))
    throw new Exception("Access denied");

  MortgageCalc mc = new MortgageCalc();
  return mc.CalculateMortgage(amount, years, interest,
    annualTax, annualIns);
}

Although this is certainly a step in the right direction, the endpoint cannot completely rest since it still has the plain text password to deal with. Not addressing this issue could severely compromise the system since someone could sniff the messages and steal the password. Also, since we're not supplying a signature, someone could modify the credentials en route to the endpoint.

Implementing a UsernameTokenManager

There are a few ways to avoid sending the password in plain text, but they all require you to implement a custom token manager. A custom token manager is a class you write and configure on the service to process incoming UsernameToken elements. You can use a UsernameTokenManager to implement a custom authentication scheme where user credentials are stored in a SQL Server™ database (a very common scenario in Web applications today).

You implement a token manager by deriving a new class from UsernameTokenManager and overriding the AuthenticateToken method. Within AuthenticateToken, your job is to look up the password for the supplied user name and return it to the caller, which in this case is the WSE infrastructure. WSE will use the password to authenticate the user by making sure they match (in the plain text case) or by verifying any password-based hashes/signatures. Before returning from AuthenticateToken, you can also create and associate a Principal object with the UsernameToken object that specifies role membership. The Principal object can be used later in the processing pipeline for authorization purposes, as illustrated in the previous section.

Figure 3 shows a sample UsernameTokenManager implementation. For pedagogical reasons, I've hardcoded the passwords for each user, but this is obviously not something you'd want to do in practice. The implementation associates a GenericPrincipal object with each token specifying membership in the Broker role.

Figure 3 Implementing a UsernameTokenManager (Close Figure 3)
public class MyUsernameTokenManager : UsernameTokenManager
{
  protected override string AuthenticateToken(UsernameToken token)
  {
    switch(token.Username)
    {
      case "mike":
        token.Principal = new GenericPrincipal(
          new GenericIdentity(token.Username),
          new string[] {"MACHINE_NAME\\Broker"} );
        return "ekim";
      case "aaron":
        token.Principal = new GenericPrincipal(
          new GenericIdentity(token.Username),
          new string[] {"MACHINE_NAME\\Broker"} );
        return "noraa";
    }
    throw new Exception("Access denied");
  }
}

You have to configure WSE to use your UsernameTokenManager class. You can do this by adding the <securityTokenManager> element to the <microsoft.web.services2> section of the service's Web.config file, as shown here:

<configuration>
    •••
    <microsoft.web.services2>
        <security>
            <securityTokenManager
             type="SecureInvoiceServiceA.MyUsernameTokenManager,
             SecureInvoiceServiceA" xmlns:wsse="http://docs.oasis-
             open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-
             1.0.xsd" qname="wsse:UsernameToken" />
        </security>
    </microsoft.web.services2>

With the UsernameTokenManager class configured in Web.config, the WSE runtime will call into your implementation of AuthenticateToken each time it receives a UsernameToken element. And since you're looking up the password manually and providing it to WSE, you no longer need to have Windows accounts set up on the machine hosting the service. Instead, they can be stored in your location of choice (SQL Server, an XML file, and so on).

The other main benefit of using a UsernameTokenManager is that the sender no longer has to supply the password in plain text. Instead, the sender can send a hashed version of the password. When WSE receives the message and retrieves the password from our UsernameTokenManager, it can use the same algorithm to generate a hash of the value returned by Authenticatetoken and verify that it matches the one sent in the message. When they match, the sender indeed sent the correct password for the user. When they don't match, the sender sent an incorrect password or the token hash has been tampered, which means authentication fails. WSE takes care of all of this for you if you've provided a UsernameTokenManager implementation to supply a correct password.

The following example illustrates how to send a hashed password from the client (again, I've hardcoded the password but only for ease of explanation):

// retrieve loan values from form
MortgageCalculatorWse mc = new MortgageCalculatorWse();
UsernameToken tok = new UsernameToken("mike", "ekim",
  PasswordOption.SendHashed);
mc.RequestSoapContext.Security.Tokens.Add(tok);

This solved the password problem, but you still don't have protection against message tampering. This is what signatures are designed to prevent.

Requiring Signatures

You can require that incoming messages contain signatures by adding more code to inspect the SoapContext object, very much like you did to require UsernameToken elements. In Figure 4 you'll see that I've added a CheckForSignature method to the WseHelpers class shown in Figure 1. This method illustrates how to traverse the SoapContext looking for MessageSignature objects. The presence of a MessageSignature object indicates that at least part of the message was signed. You'd have to extend this method to require signatures for specific elements.

Figure 4 Requiring a Signature (Close Figure 4)
public static void CheckForSignature(SoapContext context)
{
  if (context == null)
    throw new Exception("Only SOAP requests are permitted.");

  bool foundSignature = false;
  foreach (ISecurityElement se in context.Security.Elements)
  {
    if (se is MessageSignature) foundSignature = true;
  }
  if (!foundSignature)
    throw new SoapException("Missing signature",
      SoapException.ClientFaultCode);
}

Adding a call to CheckForSignature at the beginning of CalcMortgage requires signatures on the incoming messages. If you run the client application without adding a signature, you'll get a "Missing signature" exception.

Signing messages from the sending applications is very straightforward with WSE. You can sign messages with a variety of different token types, including UsernameTokens. You sign a message by instantiating a MessageSignature object based on a particular security token and then adding it to the SoapContext, as illustrated in the following lines of code:

MortgageCalculatorWse mc = new MortgageCalculatorWse();
UsernameToken tok = new UsernameToken("aaron", "noraa",
  PasswordOption.SendHashed);
mc.RequestSoapContext.Security.Tokens.Add(tok);
MessageSignature sig = new MessageSignature(tok);
mc.RequestSoapContext.Security.Elements.Add(sig);

By default, WSE signs the body and most of the headers, although you can control which elements you want to sign. When WSE receives this message on the server, it will recalculate the signature value and verify that it matches the one supplied in the message. As such, the message signature can be used as proof that the sender knows the password (known as proof of possession). This makes is possible for the sender to stop sending the password altogether, as illustrated here:

MortgageCalculatorWse mc = new MortgageCalculatorWse();
UsernameToken tok = new UsernameToken("aaron", "noraa",
  PasswordOption.SendNone);
mc.RequestSoapContext.Security.Tokens.Add(tok);
// sign the message with the UsernameToken
MessageSignature sig = new MessageSignature(tok);
mc.RequestSoapContext.Security.Elements.Add(sig);

In this case, WSE will automatically verify the signature and, by so doing, verify the password. Supplying an invalid password causes an exception ("The signature or decryption was invalid") even though the password was not supplied in the message.

You can also tell WSE to sign the response SOAP messages returned by CalcMortgage. One way would be to use the UsernameToken supplied by the sender to sign the response SOAP message, adding the appropriate objects to the current ResponseSoapContext. However, signing the response using the same username token from the request is not commonly used since only the user should use its token to sign a message. A service should not use it to sign anything. Instead, a service should have its own token, normally an X.509 security token (the details of which I'll discuss shortly) to sign the response.

Using DerivedKeyTokens

One of the problems with using a raw UsernameToken to sign messages is that it becomes less secure with each message sent. Signing repeatedly with the same key might expose the system to ciphertext-only attacks that could result in exposing the secret data you're encrypting. To overcome this, by default the WSE runtime adds some randomness to the username token before it is sent so that the keys for different messages are actually different.

WS-SecureConversation also addresses this by defining a new token type called DerivedKeyToken. A DerivedKeyToken is a security token that's derived from another token and then used to sign (or encrypt) the message. A new DerivedKeyToken can be generated from the original shared secret for each message, making it less likely for ciphertext-only attacks to succeed. The following sample illustrates how to modify the sending code to generate a DerivedKeyToken from the original UsernameToken, which it then uses to sign the message:

MortgageCalculatorWse mc = new MortgageCalculatorWse();
UsernameToken tok = new UsernameToken("aaron", "noraa",
  PasswordOption.SendNone);
mc.RequestSoapContext.Security.Tokens.Add(tok);
// derive a key from the UsernameToken and use it to sign
DerivedKeyToken dk = new DerivedKeyToken(tok);
mc.RequestSoapContext.Security.Tokens.Add(dk);
MessageSignature sig = new MessageSignature(dk);
mc.RequestSoapContext.Security.Elements.Add(sig);

Requiring Encryption

Encryption is the last feature you may want the service to require. Encryption prevents eavesdropping and guarantees the privacy of sensitive data. Figure 5 shows how to write a method to check for EncryptedData elements in the SoapContext object. Like CheckSignature, the CheckforEncryption method will return true as long as at least one element was encrypted in the message. You'd have to extend this method if you wanted to require that specific elements are encrypted. Now, if the client application tries to invoke CalcMortgage, with a call to CheckForEncryption in place, it will get the "Encryption required" exception.

Figure 5 Requiring Encryption (Close Figure 5)
public static void CheckForEncryption(SoapContext context)
{
   if (context == null)
      throw new Exception("Only SOAP requests are permitted.");

   if (context.Security.Elements.Count == 0)
      throw new SoapException("Encryption required",
         SoapException.ClientFaultCode);

   bool foundEncryption = false;
   foreach (ISecurityElement se in context.Security.Elements)
   {
      if (se is EncryptedData) foundEncryption = true;
   }
   if (!foundEncryption)
      throw new SoapException("Encryption required",
         SoapException.ClientFaultCode);

}

As with signatures, it is easy to encrypt the messages sent by the client. You can encrypt the message with a raw UsernameToken (like you originally did with signatures) but you should really use a DerivedKeyToken instead. The code in Figure 6 illustrates how to encrypt with a DerivedKeyToken.

Figure 6 Encrypting with DerivedKeyToken (Close Figure 6)
MortgageCalculatorWse mc = new MortgageCalculatorWse();
UsernameToken tok = new UsernameToken("aaron", "noraa",
  PasswordOption.SendNone);
mc.RequestSoapContext.Security.Tokens.Add(tok);

// sign the message
DerivedKeyToken dks = new DerivedKeyToken(tok);
mc.RequestSoapContext.Security.Tokens.Add(dks);
MessageSignature sig = new MessageSignature(dks);
mc.RequestSoapContext.Security.Elements.Add(sig);

// encrypt the message
DerivedKeyToken dke = new DerivedKeyToken(tok);
mc.RequestSoapContext.Security.Tokens.Add(dke);
EncryptedData enc = new EncryptedData(dke);
mc.RequestSoapContext.Security.Elements.Add(enc);

WSE only encrypts the body of the SOAP message by default, but you can specify exactly which elements you'd like to encrypt. You can also modify the WebMethod to encrypt the response messages like you did for signatures.

Signing and Encrypting with X.509 Tokens

The examples I've shown thus far have used UsernameTokens to authenticate, sign, and encrypt SOAP messages. WS-Security allows for a variety of security token types besides UsernameTokens, including X.509 certificates, Kerberos tickets, and custom tokens.

The code in Figure 7 illustrates how to use X.509 tokens to sign and encrypt the message when invoking the CalcMortgage operation. GetX509Token retrieves the certificate from the machine's certificate store. In this example, you're using the client's certificate to sign the message (with the private key) and the server's certificate to encrypt the message (with the public key). When the server receives the message, it will verify the signature and automatically decrypt the message using its private key. WSE 2.0 ships with some sample certificates that you can use to experiment with this type of code (these are the certificates I'm using).

Figure 7 Signing and Encrypting with X.509 Tokens (Close Figure 7)
MortgageCalculatorWse mc = new MortgageCalculatorWse();
UsernameToken tok = new UsernameToken("aaron", "noraa",
  PasswordOption.SendHashed);
mc.RequestSoapContext.Security.Tokens.Add(tok);

// Retrieve client token for signing
X509SecurityToken clientToken = GetX509Token(
  "gBfo0147lM6cKnTbbMSuMVvmFY4=", X509CertificateStore.MyStore);
// Retrieve server token for encrypting
X509SecurityToken serverToken = GetX509Token(
  "bBwPfItvKp3b6TNDq+14qs58VJQ=",
  X509CertificateStore.OtherPeople);

// Must add client token to message for signature processing
mc.RequestSoapContext.Security.Tokens.Add(clientToken);
mc.RequestSoapContext.Security.Elements.Add(
  new MessageSignature(clientToken));

// Encrypt the body
mc.RequestSoapContext.Security.Elements.Add(
  new EncryptedData(serverToken));

... // invoke CalcMortgage

In order to illustrate how to encrypt a specific portion of the message, let's look at how to encrypt a UsernameToken element with an X.509 token. You can do this by adding a new EncryptedData object to the SoapContext, specifying the ID of the element you want to encrypt. The following sample illustrates how to do this for the UsernameToken element:

UsernameToken tok = new UsernameToken("aaron", "noraa",
  PasswordOption.SendPlainText);
...
// Encrypt the body
mc.RequestSoapContext.Security.Elements.Add(
  new EncryptedData(serverToken));

// Encrypt the UsernameToken element
mc.RequestSoapContext.Security.Elements.Add(
  new EncryptedData(serverToken, string.Format("#{0}", tok.Id)));

This scenario allows you to safely use a plain text password in the UsernameToken element since it will be encrypted before crossing the wire. When WSE receives this message on the server, it will decrypt the UsernameToken header before proceeding with the normal authentication process described earlier.

Automating Security with Policy

As you can see, it's not hard to programmatically require security features in a service or to programmatically add security features to messages sent by a client. However, it still requires you to write code that mixes the security infrastructure with business logic. One of the most compelling features of WSE 2.0 is that you can completely automate everything that I've shown here through declarative policies.

Web Services Enhancements 2.0 provides support for WS-Policy, WS-PolicyAssertions, and WS-SecurityPolicy. Policies are used to require security elements when receiving messages and to suggest the need for security elements when sending messages. All you have to do is add an element to your configuration file indicating which policy file to use, as shown in the following lines of code:

<configuration>
  <microsoft.web.services2>
    <policy>
      <cache name="policyCache.config" />
    </policy>

WSE 2.0 provides a wizard integrated with Visual Studio® .NET that makes it easy to generate policies that describe the most common security features. You can launch the wizard from the Policy tab in the WSE Settings Tool. When the wizard appears, select "Secure a service application" and press Next. The next step allows you to specify the security elements that you want to require in the request and response messages. Here you can require signatures and encryption for both the request and response messages.

On the next step you select the type of token the service will require for client authentication. Choose UsernameToken since that's what you've been using throughout this column. Next you specify users or roles that should have access to the operation. In this example, you've been restricting access to users in the Broker role so that's what you'll specify in this step. And finally, you have to specify the server certificate that will be used for request encryption and response signing. You can then finish the wizard and the policy file will be generated and configured in the project's Web.config file (see Figure 8).

Figure 8 WSE Security Settings Tool
Figure 8 WSE Security Settings Tool

In this case, the generated policy states that incoming messages must contain a UsernameToken that is associated with the SKON-COMPAQ\Broker role, a signature based on a DerivedKeyToken (derived from the UsernameToken), and a body encrypted with the server certificate selected in the wizard. This policy will be automatically enforced by WSE. As a result, you can comment out all of the code that you added to the service in order to require the different security elements.

You can also use a policy on the client side to influence how the messages are secured. You just need to run the wizard again, select "Secure a client application," and choose the same settings used for the service. Using a policy on the client makes it possible to automate the code used to add tokens, signatures, and encryption. All you have to do is tell WSE where to find the UsernameToken that it needs to add to the message. You can do this with the PolicyEnforcementSecurityTokenCache, as illustrated here:

MortgageCalculatorWse mc = new MortgageCalculatorWse();
UsernameToken tok = new UsernameToken("mike", "ekim",
  PasswordOption.SendHashed);

// add the UsernameToken to the global cache, it will be
// added to the message automatically by WSE
PolicyEnforcementSecurityTokenCache.GlobalCache.Add(tok);

You can now remove all of the code that you added to the client application throughout this column (except for the last line of code you just saw) since WSE will take care of securing the messages for you. These policies automate all of the security features that were previously implemented manually.

The cool thing about declarative policies is that you can tweak them to control all sorts of things. For example, you can specify that the UsernameToken element needs to be encrypted in the request (when you want to use plain text passwords) by simply adding a reference to the <MessageParts> element. Or you can modify the policy to turn on WS-SecureConversation. This is actually just a checkbox in the very first step of the security wizard. Policy makes it possible to control all of your Web services security needs without any code.

Note on the UsernameTokenManager

When you implement custom authentication using a UsernameTokenManager, your implementation of AuthenticateToken must return the same secret (for example, password) used on the client side to generate the hash/signature, depending on which option you use. This is necessary because WSE needs to recalculate the hash/signature to verify that it's the same. So when the client side uses clear text passwords, UsernameTokenManager must have access to the clear text passwords of all the users in the system. This approach makes security experts cringe because if the machine is compromised, so are all of the passwords. And since passwords are mostly used by humans, and humans typically use the same password everywhere, compromising the machine can have a far-reaching effect. The only way around this is to have the sender and receiver agree on an algorithm for generating a new secret from the original password that it can use instead. Hence, the client would supply the generated secret when instantiating the UsernameToken, and the machine would store the generated password somewhere on the machine. The rest of the process would work the same. This approach doesn't protect the server if it's compromised, but the real passwords are safe.

Where Are We?

WSE 2.0 provides a powerful and flexible security framework that makes it possible to secure incoming and outgoing messages within the context of SOAP. WSE 2.0 accomplishes this by allowing you to attach security tokens that can be used to identify and authorize senders as well as to sign and encrypt (portions of) the messages. You can add these security features to your code manually or through the declarative policy framework that automates everything. WSE 2.0 is a major breakthrough because it's the first available Microsoft Web services framework that makes it possible to secure Web services endpoints without requiring you to become a complete security expert.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a co-founder of Pluralsight, an education and content creation company, where he focuses on XML and Web services technologies. He's a contributing editor to MSDN Magazine and the author of several books, including Essential XML Quick Reference (Addison-Wesley, 2001). Reach him at http://www.pluralsight.com/aaron.


Security Briefs: Password Minder Internals

Security Briefs
Password Minder Internals
Keith Brown

Contents
Stretching and Salting the Master Password
Derived Keys
Password Reminders
Secret Management
Creating Good Passwords
Other Features and Future Direction
Conclusion

In my last column I introduced Password Minder, the tool I use to manage all of my passwords. It generates a long, random password for each site I visit, and makes it possible for me to use the most complex passwords possible, without ever having to see the actual password material or type it in manually. If you didn't have a chance to read that column, you should go do that now, otherwise this column won't make much sense!

This time, I'll take you on a tour of the guts of how Password Minder implements its cryptographic functionality, show a few more features, and ponder future directions. Let's start with a closer look at the cryptography in Password Minder.

Stretching and Salting the Master Password

I keep my main password file on my laptop. Nobody uses the laptop but me, thus it provides real defense against someone recovering my passwords; they'd have to steal my password file before they can even try to crack my passwords! (Of course, it's a lot easier to steal a laptop than a desktop computer, so as an added layer of security I also use the Encrypted File System for the most sensitive folders in my system.) But from time to time I need to roam with my passwords, so I built some extra protection into the latest version of Password Minder.

In early versions, I calculated the master key by simply hashing the master password and taking 128 bits of that hash value as the key. If you consider hashing to be a single step that can be performed in one clock cycle by someone who has specialized hardware, then as I mentioned in my last column, given a 12-character password, an attacker will need about 279 clock cycles to hash all possible keys in this keyspace. On a 3GHz machine, this would take over 14 million years, but given Moore's law and parallel computation, this number doesn't look quite so impressive. And keep in mind that I really didn't start with a 79-bit keyspace given that I didn't pick a truly random master password. However, although Moore's law means computers keep getting faster, humans don't seem to be getting better at remembering longer passwords!

So I fixed Password Minder to use a much stronger mechanism for calculating the master key from the master password. Instead of hashing just once, the password is hashed many times. The algorithm that I use comes from RSA Labs' Public Key Cryptography Standards (PKCS). PKCS#5 is a guideline that specifically describes how to implement password-based cryptography. The guideline recommends a function called the Password Based Key Derivation Function 2 (PBKDF2), which describes a technique for obtaining key material from a password—which is exactly what Password Minder needs to do.

The function works by seeding a cryptographic pseudo-random number generator with the master password and a salt value. With each round, the generator produces output that is XOR'd into the final result. You can choose how many bytes of output you need. In this case, since I use 256-bit keys, I need 32 bytes of output. The generator is based on the HMAC function, which is a message authentication code derived from a hash. You can choose which hash algorithm to use with HMAC, although SHA-256 is probably your best choice today, given the recommendations found in the book Practical Cryptography (Wiley, 2003). Unfortunately, the Microsoft® .NET Framework doesn't implement PBKDF2. There is a class called PasswordDeriveBytes that implements something akin to an older version of the algorithm, and the next release of the Windows® OS, codenamed "Longhorn," appears to have a class that implements the newer algorithm with SHA1. I couldn't wait for Longhorn though, and I prefer to use SHA-256, so I wrote my own implementation based on the .NET Framework SHA256Managed class. This class is shown in Figure 1.

Figure 1 Password-Based Key Derivation Function 2 (Close Figure 1)
using System;
using System.Security.Cryptography;

// implementation of PKCS#5 v2.0
// Password Based Key Derivation Function 2
// http://www.rsasecurity.com/rsalabs/pkcs/pkcs-5/index.html
// For the HMAC function, see RFC 2104
// http://www.ietf.org/rfc/rfc2104.txt
class PBKDF2
{
    // SHA-256 has a 512-bit block size and gives a 256-bit output
    const int BLOCK_SIZE_IN_BYTES = 64;
    const int  HASH_SIZE_IN_BYTES = 32;
    const byte IPAD = 0x36;
    const byte OPAD = 0x5C;

    public static byte[] GetBytes(string password,
        byte[] salt, int iterations, int howManyBytes)
    {
        return GetBytes(
            System.Text.Encoding.UTF8.GetBytes(password),
            salt, iterations, howManyBytes);
    }
    public static byte[] GetBytes(byte[] password,
        byte[] salt, int iterations, int howManyBytes)
    {
        // round up
        uint cBlocks = (uint)((howManyBytes+
            HASH_SIZE_IN_BYTES-1)/HASH_SIZE_IN_BYTES);

        // seed for the pseudo-random fcn: salt + block index
        byte[] saltAndIndex = new byte[salt.Length + 4];
        Array.Copy(salt, 0, saltAndIndex, 0, salt.Length);

        byte[] output = new byte[cBlocks*HASH_SIZE_IN_BYTES];
        int outputOffset = 0;

        SHA256Managed innerHash = new SHA256Managed();
        SHA256Managed outerHash = new SHA256Managed();

        // HMAC says the key must be hashed or padded with zeros
        // so it fits into a single block of the hash in use
        if (password.Length > BLOCK_SIZE_IN_BYTES) {
            password = innerHash.ComputeHash(password);
        }
        byte[] key = new byte[BLOCK_SIZE_IN_BYTES];
        Array.Copy(password, 0, key, 0, password.Length);

        byte[] InnerKey = new byte[BLOCK_SIZE_IN_BYTES];
        byte[] OuterKey = new byte[BLOCK_SIZE_IN_BYTES];
        for (int i = 0; i < BLOCK_SIZE_IN_BYTES; ++i) {
            InnerKey[i] = (byte)(key[i] ^ IPAD);
            OuterKey[i] = (byte)(key[i] ^ OPAD);
        }

        // for each block of desired output
        for (int iBlock = 0; iBlock < cBlocks; ++iBlock) {
            // seed HMAC with salt & block index
            _incrementBigEndianIndex(saltAndIndex, salt.Length);
            byte[] U = saltAndIndex;

            for (int i = 0; i < iterations; ++i) {
                // simple implementation of HMAC-SHA-256
                innerHash.Initialize();
                innerHash.TransformBlock(InnerKey, 0,
                    BLOCK_SIZE_IN_BYTES, InnerKey, 0);
                innerHash.TransformFinalBlock(U, 0, U.Length);

                byte[] temp = innerHash.Hash;

                outerHash.Initialize();
                outerHash.TransformBlock(OuterKey, 0,
                    BLOCK_SIZE_IN_BYTES, OuterKey, 0);
                outerHash.TransformFinalBlock(temp, 0, temp.Length);

                U = outerHash.Hash; // U = result of HMAC

                // xor result into output buffer
                _xorByteArray(U, 0, HASH_SIZE_IN_BYTES,
                    output, outputOffset);
            }
            outputOffset += HASH_SIZE_IN_BYTES;
        }
        byte[] result = new byte[howManyBytes];
        Array.Copy(output, 0, result, 0, howManyBytes);
        return result;
    }
    // treat the four bytes starting at buf[offset]
    // as a big endian integer, and increment it
    static void _incrementBigEndianIndex(byte[] buf, int offset)
    {
        unchecked {
            if (0 == ++buf[offset+3])
            if (0 == ++buf[offset+2])
            if (0 == ++buf[offset+1])
            if (0 == ++buf[offset+0])
                throw new OverflowException();
        }
    }
    static void _xorByteArray(byte[] src,
        int srcOffset, int cb, byte[] dest, int destOffset)
    {
        int end = checked(srcOffset + cb);
        while (srcOffset != end) {
            dest[destOffset] ^= src[srcOffset];
            ++srcOffset;
            ++destOffset;
        }
    }
}

This all sounds really technical, but the idea is to require the solution of a little puzzle in order to turn a password into a master key. Say you require 216 iterations of PBKDF2 in order to calculate your master key. This would effectively stretch your password's strength by 16 bits because the attacker would need to perform 216 steps for each password he wants to try in a brute-force attack. And to ensure that he can't gain an advantage from an economy of scale by attacking a bunch of passwords at the same time, a unique salt value is added to the password before any of this stretching begins. Here's the declaration of my function that computes the PBKDF2 algorithm, showing the inputs it requires:

byte[] GetBytes(byte[] password, byte[] salt,
                int iterations, int howManyBytes);

Password Minder encodes your master password into a byte array and uses a unique salt value that is stored in your password file to initialize the algorithm. It asks for 32 bytes of output, which produces a 256-bit AES master key. The iteration count is entirely up to you.

Figure 2 Change Master Password
Figure 2 Change Master Password

On my laptop, 216 iterations of this algorithm takes about a second. I'm currently using 218 iterations, which takes under five seconds and is about as long as I'll tolerate waiting. But next year when I buy a new laptop, I'll be able to use more iterations. Password Minder accommodates this by asking how much stretching you want whenever you reset your master password via the dialog shown in Figure 2. The slider bar controls the number of iterations you want to use, with the leftmost setting currently at 216 iterations, and each tick to the right increasing the number of iterations by a power of two. The rightmost setting, 225, takes so long I've never even bothered to wait for it to finish. The beauty of this system is that as my hardware gets faster over the years, I can use a larger number of iterations, thus keeping pace with any attacker who manages to get a copy of my password file.

Derived Keys

Password Minder uses a unique key to encrypt each piece of data stored in your password file. These keys are ultimately derived from your master key, which is derived from your master password via a large number of iterations of PBKDF2. In an earlier version of Password Minder, I didn't bother deriving unique keys for each record, until someone noticed that this leaks information. If you have two user names or passwords that are the same, the ciphertext for those items will also be the same. As you'll see, it's trivial to derive unique keys for everything and avoid this problem entirely.

It turns out that this is a common requirement in cryptosystems. For example, to build a secure channel, you need four unique symmetric keys: one for authenticating the data stream sent from the client to the server, another for encrypting that same stream, and two more for doing the same thing for the data stream flowing in the opposite direction. It's easy to generate these keys given a single shared session key: you can just hash the session key along with some unique data for each type of key you need. Here's an example of how this might work:

EKcs = H(SK || "EKcs");
AKcs = H(SK || "AKcs");
EKsc = H(SK || "EKsc");
AKsc = H(SK || "AKsc");

H is a hash function such as SHA-256 and the pipes (||) represent concatenation. Here, SK represents the session key shared by client and server, EK is the encryption key, AK represents the authentication key, cs stands for client to server, and sc stands for server to client.

Password Minder uses PBKDF2 to derive its keys. And it derives these keys in a couple of stages. Stage one derives a unique record key from the master key, and stage two splits that record key into the two keys that are then used to encrypt the user name and password for that record. Here are the formulas for deriving these keys (along with the verification key, which I'll discuss shortly):

RK = PBKDF2(MK, RS);
VK = PBKDF2(MK, "VERIFIER");
UK = PBKDF2(RK, "USERID");
PK = PBKDF2(RK, "PASSWORD");

PBKDF2 is used with SHA-256 (one iteration only). These definitions apply to the formulas: MK stands for the master key (derived from master password), RS represents the salt value for the record, RK is the record key, VK refers to the verifier key, UK is the user name key, and PK is the password key.

You see, whenever you add a new record, Password Minder generates a random salt value for that record. If you look at the XML file where Password Minder stores your encrypted passwords, you'll see this salt value for each and every record:

<record site='amazon.com'
  salt='LonjRW5...ujPwsOtA1w=='
  encryptedUserId='umfrDFlbMaFvjseYnAny1g=='
  encryptedPassword=' iVkOGPXDwVKtvCCEkzI1Cg=='
  lastReset='2003-02-28T00:01:28Z'/>

Password Minder uses this per-record salt value to derive the record key, and then uses a couple of fixed salt values, "USERID" and "PASSWORD", to split that key into the two keys used for encrypting the user name and password, respectively. The overall flow is shown in Figure 3.

Figure 3 Deriving Keys
Figure 3 Deriving Keys

The verification key is used to verify the master password. When you're prompted for your master password, your input is run through the litany of PBKDF2 iterations to derive the master key, and from the master key a verifier is derived (I showed the formula earlier). The verifier is base64-encoded in the password file, and allows Password Minder to verify that you've entered the correct password. If the calculated verifier doesn't match what's in the password file, Password Minder will prompt you to reenter your password. After three failed attempts, Password Minder will shut down.

Because the verifier is produced from the master key via one-way hash operations, an attacker can't use the value of the verifier to deduce the master key, although that alone doesn't stop a brute-force attack. Given that the master key is 256 bits long, brute forcing it against the verifier isn't a useful tactic either.

Password Reminders

I have a few passwords that need to be changed from time to time so that they don't expire, and getting these passwords reset after they expire is a major hassle. The problem is, I don't log into some of these sites enough to notice that my password may be expiring soon. So I included a feature in Password Minder to remind me when a password is about to expire. The program tracks the date of your last password reset for each record, and allows you to specify the validity period for the password, which is typically 45 or 90 days. You can also control how soon Password Minder starts nagging you to reset this password (see Figure 4).

Figure 4 Set Expiration
Figure 4 Set Expiration

Whenever you launch Password Minder, it looks at each record and lets you know if a password needs changing. Once you change your password (which you'll typically do by having Password Minder generate a new random password), the program will record the current date and the reminder cycle will start all over again. This feature has saved me endless headaches!

As an example of how difficult crypto really is, keep in mind that many pseudo-random number generators factor the current time into their sequence generation logic, so exposing the time at which the password was generated is leaking a bit of information to an attacker. A future enhancement to the application might be to not store the exact time the password was reset/updated, but rather to round it to the previous four-hour boundary, for example.

Secret Management

In earlier versions of Password Minder, I was a bit lazy about managing the master key; the main form simply kept the master key in a byte array and passed it to the cryptography helper functions whenever they needed it. The newer version takes advice from the book Writing Secure Code (Microsoft Press®, 2002) and keeps the master key deep down in the helper class that actually needs it. The idea here is that if you must have a secret in your program, you should keep it as close as possible to the code that actually uses that secret. Passing the secret around exposes it more than is necessary. If an attacker is able to somehow compromise a section of your code, it would be best not to have your secret right there on the stack where it is easily obtained.

So I've encapsulated all the cryptography into a class called CryptMaster. This class is responsible for managing all the secrets in Password Minder: everything from collecting your master password to decrypting individual password records and driving the class that inserts those passwords as keystrokes in the input queue. What's interesting is that the resulting code was much cleaner after this refactoring. It seems like a lot of the precautions we take when writing secure code naturally helps us to build more robust and maintainable code as well.

One security precaution that could be added to CryptMaster is the use of pinned byte arrays. It is a good practice to keep keys in pinned byte arrays (as opposed to unpinned arrays) so that you can zero them out when they are not needed anymore. The garbage collector is free to move unpinned arrays during any collection, creating the possibility that passwords and keys will remain in memory for longer than required. Using the new SecureString functionality in the .NET Framework 2.0 can help in this regard. Rather than playing games with pinned byte arrays, Password Minder deals with the problem of secrets in memory by having a very brief process lifetime. This was discussed in my last column.

Creating Good Passwords

Whenever you need to store a new password, Password Minder offers the option to auto generate one for you. The beauty of using Password Minder is that you can generate very large, complicated passwords, and you'll never need to actually type them yourself!

My original implementation of this function always generated 10-character random passwords by using a random sampling of the 26 uppercase and lowercase English letters, in addition to digits and 32 punctuation characters. However, I quickly learned that this simple approach left me out in the cold when I encountered Web sites that restricted the types of passwords they would allow. Some Web sites won't allow any punctuation at all, for example. For the first few months of using Password Minder, I would just manually choose a password for these sites.

But once my friends started complaining about this same problem, I decided to let the user choose the character set and length of the password, and then let Password Minder create a random password given those constraints. Figure 5 shows the resulting dialog. The default password length is 20, which is too big for many sites, and sadly they often won't tell you how long your password can be. Usually they'll only tell you a minimum length. Nobody seems to expect people to use long passwords!

Figure 5 Auto Generate Password
Figure 5 Auto Generate Password

Windows accounts can have very long passwords. Heck, when I generate passwords for a service or IIS application pool, I ask Password Minder to generate a 75-character password. Why not? I'll never have to type or remember it, and using a long password doesn't impact performance. Windows supports passwords up to 127 characters long. Just be careful about creating accounts for users like this because if your users run Windows 9x, they'll be limited to 14-character passwords. Of course, from a theoretical perspective, the strength of a password is only as strong as the master key from which it was produced.

Other Features and Future Direction

Password Minder allows you to import or export passwords. This allows me to share some passwords with my wife for doing online banking and such. It also allows me to export passwords I'll need on the road to a file on a USB memory stick.

Another feature of Password Minder was added to support the few finicky applications that don't like the way Password Minder injects passwords into the input queue. If you ever run into this, just select the entry you want in Password Minder and press Ctrl+C to copy the cleartext password to the clipboard. You can then paste the password wherever you want. This can also be convenient if you find yourself needing passwords over and over. Password Minder clears the clipboard when it shuts down, so to use this feature you'll need to keep Password Minder running until you're finished pasting passwords.

The other day I was having trouble logging into a site, and I got tired of typing my master password over and over and waiting for it to calculate the master key. So I just left Password Minder open and copied and pasted passwords until the problem was solved. Just be careful if you use this feature to remember to close down Password Minder when you're finished.

One last feature bears mentioning. Sometimes you need to type your password twice. To avoid having to restart Password Minder, just press F5 to type the password and keep Password Minder running. When you need to type the password the second time, just click the Password Minder window to bring it to the foreground and press Enter like you would normally. This is useful when you're signing up for a new account and need to confirm your password.

As for future directions, the most interesting idea I've heard so far is to allow Password Minder to act as a Web service client. This would allow you to host your password file on a central Web server so you could use it from several different machines. It's an interesting idea, and when I get some spare cycles I may just add it. The main thing that scares me about this is having my mom log into a machine at some public kiosk and use her password file from there. You should always be wary of what you type into an untrusted machine. The combination of typing your master password on an untrusted keyboard along with having your password file available on a Web share gives me the heebie jeebies. This is not a feature I would enable by default!

A simpler feature that I'm working on right now is tracking usage for each record. By tracking how often a password is used, I can sort the list to bring the most heavily used passwords to the top. When you've got as many passwords as I have, a simple feature like this can make life much easier.

Conclusion

I hope you'll find Password Minder useful, either as a tool or as a sample of how to implement password-based cryptography. If you're doing the latter, I suggest reading the PKCS#5 specification, which you can find at PKCS #5: Password-Based Cryptography Standard. If you're thinking of implementing any sort of cryptography in your system, be sure to read Practical Cryptography by Niels Ferguson and Bruce Schneier to learn just how hard it really is to get this right. Consider hiring a professional cryptographer to look at your design and review your code. Crypto is the one link in an application that everyone assumes is strong. Make sure your assumption is correct!

Oh, and if you don't like Password Minder for one reason or another, find a tool that you do like and use it. Password Safe is another option. Given the large number of credentials we all need to manage these days, you'd be crazy not to find a tool to help you do it well. Just be sure to pick one from a vendor you really trust.

Send your questions or comments for Keith to briefs@microsoft.com.

Keith Brown is a co-founder of Pluralsight, specializing in developing and delivering high-quality training for software developers. Keith's most recent book, The .NET Developer's Guide to Windows Security, is available here. He also keeps a web log and contact information at http://www.pluralsight.com/keith.


.NET Matters: ThreadPoolWait and HandleLeakTracker

.NET Matters
ThreadPoolWait and HandleLeakTracker
Stephen Toub

Code download available at: NETMatters0410.exe (172KB)

Q I have a few tasks that I want to run asynchronously and simultaneously, and I want to wait for them all to finish before continuing. I've seen samples showing how to do this using Thread.Join, but those seem to conflict with the general recommendation to use the ThreadPool whenever possible rather than spinning up new threads. Is there a way to wait for operations queued to the ThreadPool to finish execution?

A As you point out, the classic solution to this problem is to spawn a new thread for each operation, start each thread executing, and then join the main thread to each of the child threads. Such a solution might look something like the following:

ArrayList threads = new ArrayList();
for(int i=0; i<numOperationsToPerform; i++)
{
    Thread t = new Thread(new ThreadStart(DoSomething));
    t.Start();
    threads.Add(t);
}
foreach(Thread t in threads) t.Join();

Join operations block the calling thread until the target thread terminates (or until the specified time elapses if one of the other overloads of Join is used that accepts timeout parameters). Thus, you need to start all of the new threads before joining to any of them; otherwise, if you were to instead execute code like the following, you'd inadvertently be serializing all of the tasks:

for(int i=0; i<numOperationsToPerform; i++)
{
    Thread t = new Thread(new ThreadStart(DoSomething));
    t.Start();
    t.Join();
}

Spinning up new threads for every task may actually be the correct thing to do in your situation. But suppose you code this up and measure it, only to find that the use of new threads is the cause of performance problems (which could be the case if the number of simultaneous tasks is quite large). The .NET ThreadPool was created for situations like this.

A thread pool enables you to use threads more efficiently by providing your application with a set of worker threads that are managed by the system. All you have to do is use the ThreadPool.QueueUserWorkItem method to queue a WaitCallback delegate for execution, and the method referenced by the delegate will execute when a thread from the pool becomes available (or immediately if the thread pool has free threads available). Of course, since QueueUserWorkItem is asynchronous to your method (meaning that it returns immediately and most likely before your method has finished executing), as a user of the ThreadPool you don't know on which thread your method will execute, making it impossible to correctly join to it. Another solution is needed.

What you really need is a way for the main thread to keep track of all of the work items it's spawned and then for each work item to notify the main thread of its completion. A straightforward way to do this is to use an event object, such as System.Threading.ManualResetEvent. ManualResetEvent allows one thread to wait until another thread puts the event into a signaled state. Thus, the main thread can create an event, pass it to the worker thread, and wait for that event to be signaled. The worker thread can then signal the event when it has finished executing, as shown in the first half of Figure 1.

Figure 1 Waiting on ThreadPool Work Items (Close Figure 1)

Waiting on One Work Item
static void DoSomething(object state)
{
    ManualResetEvent mre = (ManualResetEvent)state;
    try
    {
        // ... do work ...
    }
    finally
    {
        mre.Set();
    }
}

static void Main()
{
    ManualResetEvent mre = new ManualResetEvent(false);
    ThreadPool.QueueUserWorkItem(new WaitCallback(DoSomething), mre);
    mre.WaitOne();
    •••
}
Waiting on Multiple Work Items
static void DoSomething(object state)
{
    ManualResetEvent mre = (ManualResetEvent)state;
    try
    {
        // ... do work ...
    }
    finally
    {
        mre.Set();
    }
}

static void Main()
{
    ManualResetEvent mre1 = new ManualResetEvent(false);
    ThreadPool.QueueUserWorkItem(new WaitCallback(DoSomething), mre1);

    ManualResetEvent mre2 = new ManualResetEvent(false);
    ThreadPool.QueueUserWorkItem(new WaitCallback(DoSomething), mre2);

    ManualResetEvent mre3 = new ManualResetEvent(false);
    ThreadPool.QueueUserWorkItem(new WaitCallback(DoSomething), mre3);

    WaitHandle.WaitAll(new WaitHandle[]{mre1, mre2, mre3});
    •••
}

The second half of Figure 1 shows this same approach for supporting waiting on multiple work items. Each work item is given its own ManualResetEvent to signal on completion, and the main thread uses WaitHandle.WaitAll to block until all of the work item events have been set.

I've formalized this approach by writing the class shown in Figure 2. Using this class, you can execute code like the following:

ThreadPoolWait tpw = new ThreadPoolWait();
tpw.QueueUserWorkItem(new WaitCallback(DoSomething1));
tpw.QueueUserWorkItem(new WaitCallback(DoSomething2));
tpw.QueueUserWorkItem(new WaitCallback(DoSomething3));
tpw.WaitOne();

Figure 2 First Attempt at ThreadPoolWait (Close Figure 2)
public class ThreadPoolWait : IDisposable
{
    private ArrayList _list = new ArrayList();

    public void QueueUserWorkItem(WaitCallback callback)
    {
        QueueUserWorkItem(callback, null);
    }

    public void QueueUserWorkItem(WaitCallback callback, object state)
    {
        ThrowIfDisposed();
        ManualResetEvent waitHandle = new ManualResetEvent(false);
        _list.Add(waitHandle);
        QueuedCallback qc =
            new QueuedCallback(callback, state, waitHandle);
        ThreadPool.QueueUserWorkItem(new
            WaitCallback(qc.HandleCallback));
    }

    private class QueuedCallback
    {
        private WaitCallback _callback;
        private object _state;
        private ManualResetEvent _waitHandle;

        public QueuedCallback(WaitCallback callback,
            object state, ManualResetEvent waitHandle)
        {
            _callback = callback;
            _state = state;
            _waitHandle = waitHandle;
        }

        public void HandleCallback(object state)
        {
            try { _callback(_state); }
            finally { _waitHandle.Set(); }
        }
    }

    public bool WaitOne() { return WaitOne(-1, false);  }

    public bool WaitOne(TimeSpan timeout, bool exitContext)
    {
        return WaitOne(timeout.TotalMilliseconds, exitContext);
    }

    public bool WaitOne(int millisecondsTimeout, bool exitContext)
    {
        ThrowIfDisposed();
        WaitHandle [] handles =
            (WaitHandle[])_list.ToArray(typeof(WaitHandle));
        return WaitHandle.WaitAll(
            handles, millisecondsTimeout, exitContext);
    }

    private void ThrowIfDisposed()
    {
        if (_list == null)
            throw new ObjectDisposedException(GetType().Name);
    }

    public void Dispose()
    {
        if (_list != null)
        {
            foreach(ManualResetEvent handle in _list)
            {
                ((IDisposable)handle).Dispose();
            }
            _list.Clear();
        }
    }
}

Under the covers, ThreadPoolWait handles all of the eventing logic necessary for the main thread to block until all of the worker threads have completed execution. When the ThreadPoolWait.QueueUserWorkItem method is called, it creates a new ManualResetEvent and stores it into an internal list of events. It then wraps this event and all of the parameters to the method into a simple state class. This state will be passed to the method invoked on a ThreadPool thread (methods to be executed on a ThreadPool thread must be compatible with the WaitCallback delegate, and thus must take a single object parameter).

ThreadPoolWait.QueueUserWorkItem finishes by delegating to ThreadPool.QueueUserWorkItem, but rather than passing to this method the caller-supplied WaitCallback, instead a new WaitCallback is created that wraps the private QueuedCallback.HandleCallback method (actually, HandleCallback is marked as public, but since its containing class, QueuedCallback, is private to ThreadPoolWait, HandleCallback is limited by the accessibility of its containing class). When executed by the ThreadPool, this method will raise the user-supplied callback with the user-supplied state, and upon completion will raise the ManualResetEvent created in QueueUserWorkItem.

This class works well in some situations, but unfortunately it has a few implementation drawbacks. First, each queued user work item is assigned its own ManualResetEvent. As a result, queueing hundreds of work items means hundreds of ManualResetEvents will be created. Second, and more importantly, WaitHandle.WaitAll has an upper bound to the number of handles on which it can wait; on my machine running Windows® XP, attempting to wait on more than 64 events results in a NotSupportedException being thrown. This means that with the implementation shown in Figure 1, either a maximum of 64 work items can be queued, or a different mechanism than WaitHandle.WaitAll must be used in order to monitor all of the events.

Another viable solution (but one that has similar problems to the approach just described) is to use delegates and a delegate's BeginInvoke method. As opposed to a delegate's Invoke method, which runs the referenced method on the current thread, BeginInvoke queues the method to be run on a ThreadPool thread. The IAsyncResult returned from BeginInvoke exposes a WaitHandle from its AsyncWaitHandle property (most likely a ManualResetEvent, although that is an internal implementation detail on which you shouldn't rely). Just as with the previously discussed approach, WaitHandle.WaitAll can be used to wait on this WaitHandle, which won't be signaled until the queued method has terminated, either successfully or by exception. This still suffers from the WaitHandle.WaitAll limit mentioned earlier.

A better solution that solves both of these problems is shown in Figure 3. Rather than creating one ManualResetEvent per work item, a single ManualResetEvent is used for the entire system. Whether or not the completion of a work item requires that this event be signaled is based on a counter that keeps track of how many work items that have been scheduled still have not finished executing. When a work item is scheduled, the count is incremented using System.Threading.Interlocked.Increment. And when a work item finishes, it decrements the count using Interlocked.Decrement; if the new value of the counter is zero, then the event is signaled.

Figure 3 Better Implementation of ThreadPoolWait (Close Figure 3)
public class ThreadPoolWait : IDisposable
{
    private int _remainingWorkItems = 1;
    private ManualResetEvent _done = new ManualResetEvent(false);

    public void QueueUserWorkItem(WaitCallback callback)
    {
        QueueUserWorkItem(callback, null);
    }

    public void QueueUserWorkItem(WaitCallback callback, object state)
    {
        ThrowIfDisposed();
        QueuedCallback qc = new QueuedCallback();
        qc.Callback = callback;
        qc.State = state;
        Interlocked.Increment(ref _remainingWorkItems);
        ThreadPool.QueueUserWorkItem(
            new WaitCallback(HandleWorkItem), qc);
    }

    public bool WaitOne() { return WaitOne(-1, false); }

    public bool WaitOne(TimeSpan timeout, bool exitContext)
    {
        return WaitOne((int)timeout.TotalMilliseconds, exitContext);
    }

    public bool WaitOne(int millisecondsTimeout, bool exitContext)
    {
        ThrowIfDisposed();
        DoneWorkItem(); // to offset initial value of _remainingWorkItems
        bool rv = _done.WaitOne(millisecondsTimeout, exitContext);
        if (rv)
        {
            _remainingWorkItems = 1;
            _done.Reset();
        }
        else Interlocked.Increment(ref _remainingWorkItems);
        return rv;
    }

    private void HandleWorkItem(object state)
    {
        QueuedCallback qc = (QueuedCallback)state;
        try { qc.Callback(qc.State); }
        finally { DoneWorkItem(); }
    }

    private void DoneWorkItem()
    {
        if (Interlocked.Decrement(ref _remainingWorkItems) == 0)
            _done.Set();
    }

    private class QueuedCallback
    {
        public WaitCallback Callback;
        public object State;
    }

    private void ThrowIfDisposed()
    {
        if (_done == null)
            throw new ObjectDisposedException(GetType().Name);
    }

    public void Dispose()
    {
        if (_done != null)
        {
            ((IDisposable)_done).Dispose();
            _done = null;
        }
    }
}

Notice that the counter's value is initially set to one rather than to zero, which may seem odd. This is done to work around the problem of a work item finishing before all of the work items have been queued; in such a scenario, the event could be signaled before all work items have completed. As a workaround, the counter is initially set to one to ensure that the counter remains above zero until WaitOne is called. WaitOne begins by removing this extra value so that the count accurately represents the remaining work items to be completed. Note, too, that by the time WaitOne is called, all of the work items may have completed and thus the call WaitOne makes to decrement the counter may, in fact, bring the counter to zero. Thus, WaitOne has to perform the same check as the work item, signaling the ManualResetEvent if there is no outstanding work to be done.

Q I have some code that works with IDisposable objects, many of which hold onto unmanaged handles of one sort or another. In my unit tests I want to make sure that I've disposed of any object that requires disposal as soon as possible rather than forcing the finalizer to clean it up. What are some options available to me?

A One common practice used both outside and inside of Microsoft is to take advantage of finalization for notification that an object wasn't cleaned up in a timely manner. As discussed in the .NET Matters column in the May 2004 edition of MSDN®Magazine finalizers should only be implemented in a release version when they're absolutely necessary to ensure that all unmanaged resources are released properly.

However, code can be added to a finalizer that alerts the developer as to when it's run (this should only be done in a debug version unless you have very special circumstances that warrant otherwise). In addition, for classes that implement IDisposable but that don't necessitate a finalizer (for example, a managed class that only holds references to other managed classes that implement IDisposable), a notification finalizer could be conditionally compiled into the class in debug builds.

As an example of classes that were implemented to follow this pattern, take a look at the finalizer for FileStream as coded in the Shared Source Common Language Infrastructure (SSCLI), which is available for download at http://msdn.microsoft.com/net/sscli:

~FileStream()
{
    if (_handleProtector != null) {
        BCLDebug.Correctness(_handleProtector.IsClosed,
            "You didn't close a FileStream & it got finalized.  Name:\""
            + _fileName + "\"");
        Dispose(false);
    }
}

If _handleProtector is not null, then the FileStream was not Disposed, and a debug assert is raised. Of course, calls to BCLDebug.Correctness are only compiled into _DEBUG builds, as dictated by the System.Diagnostics.ConditionalAttribute applied to the Correctness method.

This solution works very well when you're writing the class in question. But what if someone else has written the class and compiled it, and now you're using the assembly they've provided to you? In such a situation, certain types of leaks are easier to discover than others. For example, the base class shown in Figure 4 can be used to help track down native handle leaks using a very simple process: take a snapshot of the handle count before you run the leaky code in question, run the code, take a snapshot afterwards, and then compare the before and after numbers. This approach isn't foolproof and in some situations could result in both false positives and negatives, but in the general case it proves to be a very useful tactic while debugging.

Figure 4 Tracking Handle Leaks (Close Figure 4)
public abstract class HandleLeakTracker : IDisposable
{
    private bool _runGCAtDispose;
    private int _count;

    protected HandleLeakTracker() : this(true, false) {}

    protected HandleLeakTracker(bool runGCNow, bool runGCAtDispose)
    {
        if (runGCNow) RunGC();
        _runGCAtDispose = runGCAtDispose;
        _count = HandleCount;
    }

    private void RunGC()
    {
        GC.Collect();
        GC.WaitForPendingFinalizers();
        GC.Collect();
    }

    void IDisposable.Dispose()
    {
        if (_runGCAtDispose) RunGC();
        int newCount = HandleCount;
        if (_count != newCount) Assert(_count, newCount);
    }

    protected abstract int HandleCount { get; }

    protected virtual void Assert(int oldCount, int newCount)
    {
        Debug.Fail("Possible Handle Leak", "Old Count: " + oldCount +
            Environment.NewLine + "New Count: " + newCount);
    }
}

How do you use the code in Figure 4? The HandleLeakTracker class shown is an abstract base class, so it is assumed you will derive your own specialized class from this in order to track the specific type of resource in which you're interested. For example, if I want to ensure that a specific section of code does not leak a handle opened by the process, I can derive a new class from HandleLeakTracker. This class overrides the HandleCount abstract method and returns the result of calling Process.HandleCount for the current Process:

public class ProcessHandleLeakTracker : HandleLeakTracker {
    protected override int HandleCount {
        get {
            using (Process p = Process.GetCurrentProcess()) {
                return p.HandleCount;
            }
        }
    }
}
Since HandleLeakTracker implements IDisposable, I can surround the leaky code in question with a using block as follows:
using(new ProcessHandleLeakTracker())
{
    StreamWriter w = new StreamWriter(@"C:\test.txt");
    w.WriteLine("Hello, World!");
}

The constructor of HandleLeakTracker stores the current HandleCount. The code in the using block runs, and then the Dispose method (executed from the finally block generated by the C# compiler as part of the using semantics) retrieves the new handle count and performs the comparison. The virtual Assert method uses Debug.Fail to signal any discrepancies.

Note, too, that both the constructor and the Dispose method can call HandleLeakTracker.RunGC depending on the Boolean parameters specified in the constructor. By default, a garbage collection is run before the handle count is initially taken so that a garbage collection that takes place while the code in question is running won't throw off the numbers. Of course, even with this precaution, situations arise which can result in a skewed handle count, such as a multithreaded application in which another thread is creating and destroying handles while the leak detection code is executing, or the garbage collector running arbitrarily during the using block and leading you into a false sense of security. The beauty of using this in your unit tests, though, is that while it won't always point out problems that exist, there's a good chance it'll help you find more than you otherwise would have discovered.

Other variants of this class are possible, such as a HandleLeakTracker-derived class that uses the GetGuiResources function exposed from User32.dll to track Graphical Device Interface (GDI) and User object leaks, shown here:

public class GuiHandleLeakTracker : HandleLeakTracker {
    protected override int HandleCount
    {
        get {
            using (Process p = Process.GetCurrentProcess())
                return GetGuiResources(p.Handle, 0) +
                    GetGuiResources(p.Handle, 1);
        }
    }
    [DllImport("User32")]
    private static extern int GetGuiResources(
        IntPtr hProcess, int uiFlags);
}

The base class could even be modified to support snapshot-capable resources besides handles, or could be integrated with a kernel-mode driver capable of reporting exactly which handles are open, thus allowing the class to determine not only how many handles are erroneously open but also to pinpoint exactly which ones were leaked. As usual, the possibilities are practically endless, and I'd be interested to hear what other solutions you find.

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


C++ Q&amp;A: Create Client Windows, Drag and Drop Between Listboxes

C++ Q&A
Create Client Windows, Drag and Drop Between Listboxes
Paul DiLascia

Code download available at: CQA0410.exe (220KB)

Q How does the system recognize which part of a window is client or non-client when it sends paint messages? And how can I specify the client rectangle when I create a window using ::CreateWindow?

Vipul Solanki

A You don't specify the client area when you create your window, you do it when your window receives a WM_NCCALCSIZE message. Windows® sends this message whenever it wants to know the size of your window's client area. In MFC, you implement an OnNcCalcSize handler. The handler takes two parameters, converted from WPARAM and LPARAM:

void OnNcCalcSize(BOOL bCalcValidRects, NCCALCSIZE_PARAMS* lpncsp);
The flag tells your app whether to "calculate valid rectangles" (more on this later); the NCCALCSIZE_PARAMS struct holds an array of three rectangles, the first of which holds your window's client area. Here's the basic model for implementing OnNcCalcSize:
// got WM_NCCALCSIZE
void CMainFrame::OnNcCalcSize(...)
{
   // do default thing (important!)
   CFrameWnd::OnNcCalcSize(...);
   CRect& rc = (CRect&)lpncsp->rgrc[0];
   // adjust rc; eg, rc.DeflateRect(...);
}

I wrote a little program, NCCalc, that shrinks the standard client area by seven pixels all around and paints this area the standard 3D face color (typically light gray). Figure 1 shows the source highlights. The important functions are OnNcCalcSize, which adjusts the client rectangle, and OnNcPaint, which paints the border. The paint code is straightforward, so I won't bother to explain. See the full source code on the MSDN®Magazine Web site for complete details.

Figure 1 MainFrm (Close Figure 1)
////////////////////////////////////////////////////////////////
// MSDN Magazine — October 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
#include "StdAfx.h"
#include "MainFrm.h"
•••
const int BORDERWIDTH = 7; // width of extra non-client area

//////////////////
// Paint client area: draw "Hello, world" text using tooltip bg color
//
void CMainFrame::OnPaint()
{
   CPaintDC dc(this);
   •••
   CString s = _T("Hello, world");
   dc.DrawText(s, &rc, DT_CENTER|DT_VCENTER|DT_SINGLELINE);
}

//////////////////
// Calculate non-client area:
// Use default non-client area minus BORDERWIDTH on all sides.
//
void CMainFrame::OnNcCalcSize(BOOL bCalcValidRects,
   NCCALCSIZE_PARAMS* lpncsp)
{
   TRACE(_T("CMainFrame:OnNcCalcSize:, bCalcValidRects=%d\n"),
      bCalcValidRects);
   // do default thing (important!)
   CFrameWnd::OnNcCalcSize(bCalcValidRects, lpncsp);
   CRect& rc = (CRect&)lpncsp->rgrc[0];
   rc.DeflateRect(BORDERWIDTH,BORDERWIDTH);
}

/////////////////
// Paint non-client area: draw extra border using 3D face color.
//
void CMainFrame::OnNcPaint()
{
   TRACE(_T("CMainFrame::OnNcPaint\n"));

   CFrameWnd::OnNcPaint();              // do default thing (important!)

   // get client rectangle
   CRect rc;
   GetClientRect(rc);

   // convert to window coordinates
   CRect rcWin;
   GetWindowRect(rcWin);               // window rect (in screen coords)..
   ScreenToClient(rcWin);              // ..convert to client coords...
   rc -= CPoint(rcWin.left, rcWin.top);// ..convert to window coords

   // rc now has client rect in window coords
   CWindowDC dc(this);
   dc.ExcludeClipRect(rc);                   // don't paint in client area
   rc.InflateRect(BORDERWIDTH,BORDERWIDTH);  // expand client rect by
                                             // border width

   // Now draw the rectangle. Since i've clipped the client area, this
   // only draws the border (expanded rectangle minus client rectangle).
   CBrush b;
   b.CreateSolidBrush(GetSysColor(COLOR_3DFACE));
   dc.FillRect(rc, &b);
}

If you override WM_NCCALCSIZE/OnNcCalcSize for a main window, make sure you call the base class (or default window proc) to perform default processing. That way, you'll start with the default client rectangle which you can then adjust. Likewise, you should call the base class/default procedure in your OnNcPaint/WM_NCPAINT handler. Otherwise Windows won't paint the borders, scroll bars, or any other standard non-client elements. If you're implementing your own window class, such as a custom toolbar or palette where you calculate the client rect and paint from scratch, calling the base class/default window proc may not be necessary. Either way, you're responsible for painting the entire non-client area when your window gets WM_NCPAINT.

Some of you may be wondering what the bCalcValidRects and the other rectangles in NCCALCSIZE_PARAMS are used for. If you read the documentation, you'll discover that the semantics of WM_NCCALCSIZE are rather complex. The documentation states that if bCalcValidRects is TRUE, "the application should indicate which part of the client area contains valid information. The system copies the valid information to the specified area within the new client area." In this scenario, "the second [rectangle] contains the coordinates of the window before it was moved or resized. The third [rectangle] contains the coordinates of the window's client area before the window was moved or resized." While all of this seems clear enough, I can't make total sense of it. Nor have I ever seen any app that uses these extra rectangles. Every app I've ever seen ignores bCalcValidRects and simply alters the first rectangle in NCCALCSIZE_PARAMS to set the client area.

I mention this because according to the documentation, if WPARAM/bCalcValidRects is FALSE, then LPARAM does not point to a NCCALCSIZE_PARAMS struct but rather to a single RECT, the client rectangle, whereas MFC casts LPARAM to NCCALCSIZE_PARAMS in all cases. This would seem to be a bug, albeit one that will never disrupt your program as long as you modify only the first rectangle in NCCALCSIZE_PARAMS. I ran some tests to determine that Windows sends WM_NCCALCSIZE with bCalcValidRects=FALSE only once, when the window is first created. Thereafter, Windows sends WM_NCCALCSIZE with bCalcValidRects=TRUE any time the window is resized. You must set the client rectangle in both cases so that your window will display in a proper way.

Alas, I'm afraid in the end I haven't done much to elucidate the purpose of bCalcValidRects. Nor were the Redmondtonians able to shed more light on this mysterious parameter, except to say that it's a performance holdover from the ancient days of Windows 3.1. In these heady times, you'll be fine as long as you make sure to always handle both cases (bCalcValidRects = TRUE and FALSE), and alter only the first rectangle.

Q I'm building a commercial baseball game app. In my UI, I want users to be able to drag items from one listbox to another. Is there a simple way to do this using MFC?

Tom Tippett

A There's nothing built into MFC to handle this, but there are a couple of ways you can do it. COM has its own interfaces for doing general-purpose drag and drop between applications. This requires several interfaces: IDropTarget, IDropSource, and IDataObject, plus a function DoDragDrop that actually does the drag/drop. Using COM for drag and drop requires writing quite a bit of code and may be more than you're up for. Moreover, it may be overkill since the COM stuff is designed to interact with other programs in the most general manner. If all you want to do is drag items from one control to another control within your application, it's simpler and easier to implement your own mechanism from scratch.

I wrote a little class library with a class, CDragDropMgr, that lets you add drag/drop behavior among windows in your app. I also wrote a test program, DDTest, that shows how to use it (see Figure 2). Figure 3 shows it running. DDTest has two listboxes and an edit control. You can drag items from the first listbox to the second, or to the edit control. You can also drag items within the second listbox to rearrange them. DDTest uses CDragDropMgr to do the work. First I'll show you how to use CDragDropMgr; then I'll explain how it works.

Figure 2 DDTest (Close Figure 2)
////////////////////////////////////////////////////////////////
// MSDN Magazine — October 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
#include "stdafx.h"
•••
//////////////////
// Generic MFC application class
//
class CMyApp : public CWinApp {
public:
    CMyApp() { }
    virtual BOOL InitInstance();
} theApp;

//////////////////
// Main dialog with two listboxes and edit control to drag/drop.
//
class CMyDlg : public CDialog {
protected:
    CDragDropMgr m_ddm;      // drag/drop manager
    CListBox m_wndList1;     // first (source) listbox
    CListBox m_wndList2;     // second (source/target) listbox
    CEdit    m_wndEdit1;     // edit control (target)
    •••
    virtual BOOL PreTranslateMessage(MSG* pMsg);
    virtual BOOL OnInitDialog();
    int  GetLBItemUnderPt(CListBox* pListBox, CPoint pt);

public:
    CMyDlg(CWnd* pParent = NULL);    // standard constructor
    afx_msg void OnReset();
    afx_msg LRESULT OnDragEnter(WPARAM wp, LPARAM lp);
    afx_msg LRESULT OnDragDrop(WPARAM wp, LPARAM lp);
    afx_msg LRESULT OnDragAbort(WPARAM wp, LPARAM lp);
    DECLARE_MESSAGE_MAP()
};

BOOL CMyApp::InitInstance()
{
    // InitCommonControls() is required on Windows XP if an application
    // manifest specifies use of ComCtl32.dll version 6 or later to
    // enable visual styles.  Otherwise, any window creation will fail.
    InitCommonControls();
    CWinApp::InitInstance();

    // Run modal dialog
    CMyDlg dlg;
    m_pMainWnd = &dlg;
    dlg.DoModal();

    // return FALSE to exit app
    return FALSE;
}

BEGIN_MESSAGE_MAP(CMyDlg, CDialog)
    ON_COMMAND(IDC_RESET, OnReset)
    ON_REGISTERED_MESSAGE(WM_DD_DRAGENTER, OnDragEnter)
    ON_REGISTERED_MESSAGE(WM_DD_DRAGDROP,    OnDragDrop)
    ON_REGISTERED_MESSAGE(WM_DD_DRAGABORT,    OnDragAbort)
END_MESSAGE_MAP()

//////////////////
// Init dialog: subclass controls and install drag/drop manager
//
BOOL CMyDlg::OnInitDialog()
{
    CDialog::OnInitDialog();
    •••
    OnReset();

    // table describing which child windows are sources and targets
    static DRAGDROPWND MyDragDropWindows[] = {
        { IDC_LIST1, DDW_SOURCE },
        { IDC_LIST2, DDW_SOURCE|DDW_TARGET },
        { IDC_EDIT1, DDW_TARGET },
        { 0, 0 },
    };
    m_ddm.Install(this, MyDragDropWindows);

    return TRUE;  // return TRUE  unless you set the focus to a control
}

//////////////////
// Reset form/dialog contents to initial state. If edit control contains a
// three-letter word like "all," fill listbox #2 also.
//
void CMyDlg::OnReset()
{
    static LPCTSTR MyItems[] = {
        _T("Dopey"),
        _T("Smokey"),
        •••
        _T("Zot"),
        _T("This is a looong item at the end."),
        NULL
    };

    // If contents of edit control are "all", copy all items.
    CString s;
    m_wndEdit1.GetWindowText(s);
    BOOL bAll = s==_T("all");

    m_wndEdit1.SetWindowText(NULL);
    m_wndList1.ResetContent();
    m_wndList2.ResetContent();
    for (int i=0; MyItems[i]; i++) {
        m_wndList1.AddString(MyItems[i]);
        if (bAll)
            m_wndList2.AddString(MyItems[i]);
    }
}

//////////////////
// You must override PreTranslateMessage to pass input to drag/drop
// manager.
//
BOOL CMyDlg::PreTranslateMessage(MSG* pMsg)
{
    return m_ddm.PreTranslateMessage(pMsg) ? TRUE :
        CDialog::PreTranslateMessage(pMsg);
}

//////////////////
// Begin dragging: create new text data from list item under point.
//
LRESULT CMyDlg::OnDragEnter(WPARAM wp, LPARAM lp)
{
    TRACE(_T(_T("CMyDlg::OnDragEnter, wp=%d\n")),wp);
    DRAGDROPINFO& ddi = *(DRAGDROPINFO*)lp;
    CListBox* plb = DYNAMIC_DOWNCAST(CListBox, GetDlgItem((int)wp));
    ASSERT(plb!=NULL);
    int item = GetLBItemUnderPt(plb, ddi.pt);
    if (item>=0) {
        CString text;
        plb->GetText(item, text);
        ddi.data = new CDragDropText(text);
        return TRUE; // do drag/drop
    }
    return FALSE; // no item under mouse: nothing to drag
}

//////////////////
// User let go of the mouse: drop data into child control.
//
LRESULT CMyDlg::OnDragDrop(WPARAM wp, LPARAM lp)
{
    TRACE(_T("CMyDlg::OnDragDrop\n"));
    DRAGDROPINFO& ddi = *(DRAGDROPINFO*)lp;
    LPCTSTR text = (LPCTSTR)ddi.data->OnGetData();

    if (wp==IDC_LIST2) {
        CListBox* plb = DYNAMIC_DOWNCAST(CListBox, GetDlgItem((int)wp));
        ASSERT(plb!=NULL);

        int iNew = GetLBItemUnderPt(plb, ddi.pt);
        int iOld = plb->FindString(0, text);
        if (iOld>=0) {
            plb->DeleteString(iOld);
            if (iOld < iNew)
                iNew—;
        }
        if (iNew>=0)
            plb->InsertString(iNew, text);
        else
            iNew = plb->AddString(text);

        plb->SetCurSel(iNew);

    } else if (wp==IDC_EDIT1) {
        m_wndEdit1.SetWindowText(text);

    } else {
        ASSERT(FALSE);
    }
    return 0;
}

//////////////////
// Drag aborted (for example, user pressed Esc).
//
LRESULT CMyDlg::OnDragAbort(WPARAM wp, LPARAM lp)
{
    TRACE(_T("CMyDlg::OnDragAbort\n"));
    return 0;
}

//////////////////
// Helper to get the listbox item under the mouse.
// This may not be the selected item when dropping.
//
int CMyDlg::GetLBItemUnderPt(CListBox* pListBox, CPoint pt)
{
    BOOL bOutside;
    UINT item = pListBox->ItemFromPoint(pt, bOutside);
    return item>=0 && !bOutside ? item : -1;
}
Figure 3 DDTest in Action
Figure 3 DDTest in Action

To use the drag/drop manager, you first instantiate a CDragDropMgr in your main window or dialog, then initialize it with a table, as shown in the following lines of code:

static DRAGDROPWND MyDragDropWindows[] = {
   { IDC_LIST1, DDW_SOURCE },
   { IDC_LIST2, DDW_SOURCE|DDW_TARGET },
   { IDC_EDIT1, DDW_TARGET },
   { 0, 0 },
};
m_ddm.Install(this, MyDragDropWindows);

Fans of my columns know that one of my top five programming mantras is that one table is worth a thousand lines of code. Tables are more concise, elegant, reliable, and maintainable than long sequences of procedural code. In this case, the table tells the drag/drop manager which child windows can be sources and/or targets for drag/drop. Each table entry holds a child window ID and a combination of flags DDW_SOURCE and DDW_TARGET. In DDTest, the first listbox is a source; the second listbox can be both a source and target, and the edit control is a target only. If you try this at home, don't forget the NULL entry at the end of your table!

Once you've initialized the drag/drop manager with your window table, next override your main window's PreTranslateMessage function to pass messages to the drag/drop manager:

BOOL CMyDlg::PreTranslateMessage(MSG* pMsg)
{
   return m_ddm.PreTranslateMessage(pMsg) ? TRUE :
      CDialog::PreTranslateMessage(pMsg);
}

With this bit of plumbing in place, the drag/drop manager detects when the user tries to drag from one window to another, and notifies your app when it's time to do something. There are four messages/notifications that CDragDropMgr can send: WM_DD_DRAGENTER, WM_DD_DRAGOVER, WM_DD_ DRAGDROP, and WM_DD_DRAGABORT. It's up to you to handle the notifications as they arrive. At a minimum, you must handle WM_DD_DRAGENTER to initiate dragging and WM_DD_DRAGDROP to perform the drop. The other two are optional. WM_DD_DRAGABORT is your chance to clean up when the user cancels; WM_DD_ DRAGOVER lets you do continual processing as the user moves the mouse while dragging. Simple apps like DDTest usually don't need to handle these messages.

When the drag/drop manager sends WM_DD_DRAGENTER, it passes a DRAGDROPINFO struct in LPARAM. Your mission is to set DRAGDROPINFO::data to point to an instance of CDragDropData containing the data you want to drag and then return TRUE. If dragging is not allowed (perhaps the user clicked on a dead area within the listbox), you should return FALSE without setting DRAGDROPINFO::data. CDragDropData is like COM's IDataObject, but much simpler: it holds the data to drop. CDragDropData has three virtual functions: OnGetDragSize to get the bounding rectangle of the drag image, OnDrawData to draw this image, and OnGetData to get the data itself. My drag/drop library provides a class called CDragDropText that implements these functions for text. It holds the text in a CString. OnGetData returns this string, OnGetDragSize calculates the text rectangle, and OnDrawData draws the text:

void CDragDropText::OnDrawData(CDC& dc, CRect& rc)
{
  dc.DrawText(m_text, &rc, DT_LEFT|DT_END_ELLIPSIS);
}

If you want the text, the only function you ever need to call is OnGetData; the drag/drop manager calls OnGetDragSize and OnDrawData internally when it needs to.

How does all of this work in practice? When DDTest receives WM_DD_DRAGENTER, it calls an internal function called GetLBItemUnderPt to figure out which listbox item (if any) lies under the cursor. DDTest then creates a CDragDropText object with this item's text as data and sets the data pointer in DRAGDROPINFO to this object:

// in CMyDlg::OnDragEnter
DRAGDROPINFO& ddi = *(DRAGDROPINFO*)lp;
int item = GetLBItemUnderPt(...);
if (item>=0) {
  CString text = // get item text
  ddi.data = new CDragDropText(text);
  return TRUE;   // allow drag-drop
}
return FALSE; // nothing to drag

CDragDropMgr does the rest. It paints the text as the user drags it around the screen and changes the mouse cursor appropriately depending on whether the cursor is over a drop target or not.

When the user lets go of the mouse, CDragDropMgr sends your app a WM_DD_DRAGDROP message. This is your cue to drop the data. For DDTest, this means setting the text in the edit control if the mouse is over the edit control, or adding the text to the listbox if the mouse is over the listbox. In reality, DDTest is a little more complex because it lets users rearrange the items in the second listbox. DDTest has code to detect whether it needs to add the text or change its position in the listbox. I leave you to ponder these details alone; the basic outline for OnDragDrop is always the same:

// OnDragDrop handler
DRAGDROPINFO& ddi = *(DRAGDROPINFO*)lp;
void* data = ddi.data->OnGetData();
// do something with data
return 0;

So much for text. What if you want to drag some other kind of data? In that case, you'd have to extend my library by deriving from CDragDropData and overriding the three basic functions. For example, to drag an image you'd derive a class CDragDropImage where OnGetData returns a BITMAP or CBitmap, OnGetDragSize returns the dimensions of the bitmap, and OnDrawData calls BltBit or some other function to draw the bitmap.

I've shown you how to use CDragDropMgr, but how does it work? The basic idea is simple. CDragDropMgr::PreTranslateMessage looks for mouse messages sent to one of the drag source windows and sends the appropriate notifications to your main window. CDragDropMgr implements a typical finite state machine with three states: NONE, CAPTURED, and DRAGGING. When the user presses the mouse button, CDragDropMgr enters CAPTURED state. When the user moves the mouse, it goes into DRAGGING state. The details are straightforward.

CDragDropMgr uses PreTranslateMessage instead of subclassing the main window because it needs to intercept mouse messages sent to any one of many possible drag source windows as determined by the drag/drop window table. One of the beauties of MFC is the way it funnels all child window messages through a single virtual PreTranslateMessage method in the main window. This lets CDragDropMgr trap mouse messages sent to any one of many potential drag-source windows in a single function, thus avoiding the need to subclass each window. When CDragDropMgr::PreTranslateMessage sees WM_LBUTTONDOWN, it looks up the window handle (HWND) to see if it's listed as a source in the window table. If it is listed as a source, it initiates dragging; otherwise, it ignores the message.

The mechanics of dragging data are straightforward and a bit boring, so I'll spare you the details. The only thing to highlight is that CDragDropData uses CImageList to paint. If you implement your own drag/drop manager, I encourage you to do the same. CImageList has functions BeginDrag, DragEnter, DragMove, and EndDrag that make short shrift of drawing the bits. They use the proper raster operations to make the image semitransparent, erase it from its previous location, and so on.

These painting details are tedious without CImageList. With it, all CDragDropMgr has to do is draw the drag image once into the image list's bitmap. When the user initiates dragging, CDragDropMgr notifies the main app, which sets DRAGDROPINFO::data to an instance of CDragDropData, as described earlier. The drag/drop manager then calls CDragDropData:: CreateDragImage (see Figure 4), which creates an image list containing the image to draw. CreateDragImage calls the virtual functions CDragDropData::OnGetDragSize to get the size of the drag image and CDragDropData::OnDrawData to draw the data into the image list's bitmap. Once this is done, CDragDropMgr calls the image list functions to draw the image during dragging. For example, each time the user moves the mouse, CDragDropMgr calls CImageList::DragMove. What could be easier? The beauty is that this code is totally generic. To handle new data types, you only have to implement OnGetDragSize and OnDrawData.

Figure 4 DragDrop (Close Figure 4)
////////////////////////////////////////////////////////////////
// MSDN Magazine — October 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.
//
•••
//////////////////
// Create the drag image: create an image list and call virtual draw
// function to draw the data into the image list. Will then use this
// during dragging.
//
CImageList* CDragDropData::CreateDragImage(CWnd* pWnd, CRect& rc)
{
   const COLORREF BGCOLOR = GetSysColor(COLOR_3DLIGHT);

   // create memory dc compatible with source window
   CWindowDC dcWin(pWnd);
   CDC dcMem;
   dcMem.CreateCompatibleDC(&dcWin);

   // use same font as source window
   CFont* pFont = pWnd->GetFont();
   CFont* pOldFont = dcMem.SelectObject(pFont);

   // get size of drag image
   CSize sz = OnGetDragSize(dcMem); // call virtual fn to get size
   rc = CRect(CPoint(0,0), sz);

   // create image list: create bitmap and draw into it
   m_bitmap.CreateCompatibleBitmap(&dcWin, sz.cx, sz.cy);
   CBitmap* pOldBitmap = dcMem.SelectObject(&m_bitmap);
   CBrush brush;
   brush.CreateSolidBrush(GetSysColor(COLOR_HIGHLIGHT));
   dcMem.FillRect(&rc,&brush);
   dcMem.SetBkMode(TRANSPARENT);
   dcMem.SetTextColor(GetSysColor(COLOR_WINDOWTEXT));
   OnDrawData(dcMem, rc); // call virtual fn to draw
   dcMem.SelectObject(pOldFont);
   dcMem.SelectObject(pOldBitmap);

   // create image list and add bitmap to it
   CImageList *pil = new CImageList();
   pil->Create(sz.cx, sz.cy, ILC_COLOR24|ILC_MASK, 0, 1);
   pil->Add(&m_bitmap, BGCOLOR);
   return pil;
}

Figure 2 and Figure 4 show the important highlights from DDTest. As always, you can download the full source. Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.


{ End Bracket }: Microsoft Java Virtual Machine Update

{ End Bracket }
Microsoft Java Virtual Machine Update
Brian Keller

Contents
Step 1—Understand the Dependencies
Step 2—Determine a Migration Strategy

In a settlement agreement reached in January 2001 to resolve a dispute over the Microsoft distribution of its Java implementation, Sun Microsystems and Microsoft agreed to limit the duration of the time Microsoft would use Sun's source code and compatibility test suites to support the Microsoft® Java Virtual Machine (MSJVM). However, as part of the April 2004 settlement agreement, Sun and Microsoft agreed to extend the Microsoft license to use Sun's Java source code and compatibility test suites. This extension allows Microsoft to support the MSJVM and address potential security issues until December 31, 2007.

The MSJVM is being phased out of all current Microsoft products and will not be included in any future products. The MSJVM is obsolete code and will no longer be enhanced or developed. Microsoft will only continue to provide security fixes to help keep customer machines secure.

Sites and applications that currently depend upon the MSJVM will not function correctly when accessed by systems that do not have the MSJVM installed. Microsoft recommends that customers using Microsoft Visual J++® and the Microsoft SDK for Java who have built applications or Web sites that use the MSJVM begin identifying MSJVM dependencies and transition to an alternate solution. For customers who choose to pursue a solution supported by Microsoft, the two migration options are Microsoft Visual J#® .NET and the Microsoft Java Language Conversion Assistant. These products assist developers in migrating existing source code to the .NET Framework.

Step 1—Understand the Dependencies

The first step in eliminating dependencies on the MSJVM is to identify those dependencies. The following are examples of products or applications that may contain dependencies:

  • Internal applications, such as payroll software
  • External applications, such as those sold by ISVs
  • Internal applets, such as those used on a corporate intranet site for employee benefits and the like
  • External applets, such as those accessed by partners, vendors, or customers, or applets that your company relies on which are provided by a third party

Microsoft provides the Diagnostic Tool for the Microsoft VM to help detect such dependencies. To download this diagnostic tool, visit http://www.microsoft.com/java.

Step 2—Determine a Migration Strategy

Once you develop a list of the applications and applets which rely on the MSJVM, the next step is to determine a migration strategy for those components. Microsoft provides several supported tools for migrating MSJVM-based applications, and there are third-party solutions available as well. A summary of some of these options follows.

Visual J# .NET This can be used by developers who are familiar with the Java-language syntax to build applications and services on the .NET Framework. It integrates the Java-language syntax into the Visual Studio® .NET integrated development environment (IDE). Visual J# also supports most of the functionality found in Visual J++ 6.0, including Microsoft extensions such as WFC. J# Browser Controls is a complementary library which provides support for migrating applets.

The Java Language Conversion Assistant JLCA converts Visual J++ 6.0 projects and Java-language files to Visual C#®. By converting these files to Visual C#, you can utilize the existing code base and take advantage of the benefits of the .NET Framework.

Alternate rendering technologies DHTML, ECMAScript, VBScript, JScript®, and others are also viable options for applet migration. These technologies can implement most simple navigational items (for example, menus, rollovers, simple calculators, and chat clients) and are an option for converting low to medium complexity applets dedicated to navigation and user interface tasks.

Third-party Java Runtime Environments These may also be an option. Be aware that these solutions are not supported by Microsoft. Depending on the amount of MSJVM-specific dependencies within the application (for example, WFC, JavaCOM, ms.*), there may be varying degrees of manual conversion that you will be required to perform.

Retiring the application This may be an alternative solution depending on your business requirements.

Detailed information on all these migration strategies can be found in the MSJVM Migration Guide for Developers at http://www.microsoft.com/java.

The MSJVM will reach the end of its life on December 31, 2007. You should be taking steps now to eliminate your dependencies on the MSJVM. By acting now, you can adequately prepare for any migration complexities and can begin taking advantage of fully supported runtime alternatives.