 MSDN Magazine June 2006

MSDN Magazine June 2006

Inside MSBuild: Compile Apps Your Way With Custom Tasks For The Microsoft Build Engine

Learn how you can use MSBuild to customize your builds. Since it ships as a part of the .NET Framework, you don't even need to have Visual Studio installed on your machine.

Sayed Ibrahim Hashimi WCF Essentials: Discover Mighty Instance Management Techniques For Developing WCF Apps

Instance management refers to a set of techniques used by Windows Communication Foundation to bind a set of messages to a service instance. This article introduces the concept and shows you why you need instance management.

Juval Lowy Configure This: Parameterize Your Apps Using XML Configuration In The .NET Framework 2.0

There are a number of ways to configure an application in the .NET Framework 2.0. This article explores the classes of the revamped System.Configuration namespace and explains how to use XML configuration files for your app configuration settings.

Bryan Porter Class To Contract: Enrich Your XML Serialization With Schema Providers In The .NET Framework

The Microsoft .NET Framework 1.x provided minimal options for mapping classes to schemas and serializing objects to XML documents, making this sort of mapping quite a challenge. The .NET Framework 2.0 changes all this with Schema providers and the IXmlSerializable interface.

Keith Pijanowski Share The Load: Report Visual Studio Team System Load Test Results Via A Configurable Web Site

This article discusses a new load test tool in Visual Studio 2005 Team System for performance and stress testing your Web sites, Web services, and other server components. Combined with its handy reporting capabilities, the load test tool provides some powerful options for sharing and managing test results.

Wen Ding Editor's Note: New and Improved Team

We've all been subconsciously conditioned through the years to turn a skeptical eye towards claims of being "new and improved. " After all, the phrase often means that some favorite feature has been turned into a new deficiency.

Toolbox: Generate Office Documents, Monitor Event Logs, and More

Most data-driven Web sites are used as interfaces to collect, process, and summarize information. Reports that summarize the data can be presented to the user in a variety of formats—the most common way is to display the report directly in a Web page.

Scott Mitchell Advanced Basics: Setting Word Document Properties the Office 2007 Way

The last time I wrote this column (March 2006), I shared an application that allows you to update all the Microsoft® Word documents in a folder and its subfolders. Each time the application finds a document in the specified path, it updates the document properties to match those you specified in the application.

Ken Getz CLR Inside Out: Windows Vista Globalization Features

Windows XP and the Microsoft .NET Framework both have APIs that support globalization. Windows Vista™ will further extend globalization support by introducing several new features.

Shawn Steele Data Points: Designing Reports with SQL Server Reporting Services 2005

Many applications require some degree of integration with a reporting tool. A good solution, SQL Server™ Reporting Services 2005, provides Web-based reports and can be integrated into both Windows® Forms and Web-based applications.

John Papa Test Run: Five Ways to Emit Test Results as XML

The use of XML files in software testing has steadily increased over the past few years. Test case data, test harness configuration information, and test result data are now stored as XML. Recently I was writing some .

Dr. James McCaffrey Cutting Edge: A Provider-Based Service for ASP.NET Tracing

When it comes to catching programming errors, the debugger is a developer's best friend. ASP. NET tracing, however, is a nice complement to the debugger and shouldn't be overlooked. It enables your ASP.

Dino Esposito Service Station: WSE 3.0, SOAP Transports, and More

It's that time again. Time to answer some of the questions I get on a regular basis. This month I'll look at service orientation and policy-based compatibility, SOAP's transport-neutral design, and Web Services Enhancements (WSE) 3.0.

Aaron Skonnard Wicked Code: Three Cures for Common Site Map Ailments

Data-driven site navigation is among the niftiest and most useful features in ASP. NET 2. 0. To get it working, all you do is create an XML site map file (or a SQL site map if you're using the MSDN®Magazine SqlSiteMapProvider), add a SiteMapDataSource, and bind a TreeView or Menu to the SiteMapDataSource.

Jeff Prosise Concurrent Affairs: Reader/Writer Locks and the ResourceLock Library

If multiple threads concurrently execute code that writes to or modifies a resource, then obviously the resource must be protected with a thread synchronization lock to ensure that the resource doesn't get corrupted.

Jeffrey Richter .NET Matters: Parameterized ThreadStart, EventWaitHandle, and More

This month Stephen Toub answers readers questions that include: How do I pass data to a new thread? Why can't I convert from "ref string" to "ref object"? And what's the difference between EventWaitHandle, AutoResetEvent and ManualResetEvent?

Stephen Toub Netting C++: The .NET Wrap

This month, we are changing the column name from Pure C++ to Netting C++ to better reflect our focus on Microsoft® . NET programming using C++/CLI, the . NET extensions to Visual C++® that are supported in Visual Studio® 2005.

Stanley B. Lippman C++ at Work: Managed Code in Visual Studio 2005

Many of you are no doubt in the process of upgrading to Visual Studio® 2005, so I thought now would be a good time to relate some of my own experiences with the new compiler. What took me so long? Hey, I'm a retro kind of guy! Better late than never!.

Paul DiLascia { End Bracket }: Singularity

When the C and C++ programming languages were invented, computers were slow, memory was limited, and compilers were simple and memory challenged, so a practical language could be little more than a veneer for assembly language.

James Larus, Galen Hunt, and David Tarditi


Inside MSBuild: Compile Apps Your Way With Custom Tasks For The Microsoft Build Engine

Inside MSBuild

Compile Apps Your Way With Custom Tasks For The Microsoft Build Engine

Sayed Ibrahim Hashimi
This article discusses:
  • Fundamentals of MSBuild
  • Targets, items, transformations, and tasks
  • Extending the build process
  • Implementing custom tasks
This article uses the following technologies:
Visual Studio 2005
Code download available at: MSBuild06.exe (135KB)
Contents
MSBuild Fundamentals
Creating Targets
Items and Transformations
Tasks
Extending the Build Processes
Extending MSBuild
Conclusion

I
n previous versions of Visual Studio, the build process was mostly a black box, and there were few things that you could do to customize your build process. With the release of Visual Studio® 2005 and the Microsoft® .NET Framework 2.0, your managed projects can be built using the Microsoft Build Engine (MSBuild). MSBuild is extensible and allows you to customize each step taken during the course of a build. It uses an XML file (actually the project file, but more on that later) to describe each step, allowing you to easily change and augment how your projects are built.

In this article I will introduce you to MSBuild and show you how to use it to customize your builds. I will demonstrate how to use MSBuild from the command line and show you how you can replicate the same process that is used when your projects are built from the Visual Studio IDE. It's also important to note that MSBuild ships as a part of the .NET Framework, so you don't need Visual Studio installed on a machine in order to have it build your projects. The files supporting MSBuild that are shipped with Visual Studio 2005 only support C#, Visual Basic®, and J# projects; however, MSBuild as a platform can support any language.

MSBuild Fundamentals

The first thing you should know about MSBuild is that your project file is your XML build file. This is important for a few reasons. First, it ensures that the build produced by Visual Studio is repeatable outside of Visual Studio. Second, you get a default build file when you create a Visual Studio project. Let's have a look at the project file for a C# application. For this example I created a new C# Windows®-based application project. When the project is open, right-click on it in the Solution Explorer and select Unload Project. Then right-click on it again and select Edit. Figure 1 shows an excerpt of this project file. From this snippet you can see that various project properties are defined. For example, you can see the Platform and the AssemblyName of the project.

Figure 1 Application Project in C# (Close Figure 1)
<Project DefaultTargets="Build"
         xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <PropertyGroup>
        <Configuration Condition=" '$(Configuration)' == '' ">
            Debug</Configuration>
        <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
        <ProductVersion>8.0.50727</ProductVersion>
        <SchemaVersion>2.0</SchemaVersion>
        <ProjectGuid>{2763BAAA-B9EC-4D3F-8992-11A99E669682}</ProjectGuid>
        <OutputType>WinExe</OutputType>
        <AppDesignerFolder>Properties</AppDesignerFolder>
        <RootNamespace>WindowsApplication1</RootNamespace>
        <AssemblyName>WindowsApplication1</AssemblyName>
    </PropertyGroup>
    ...
</Project>

All of the declarations within the PropertyGroup element are called properties, which are simply name-value pairs. Most Visual Studio configuration settings are stored within properties, which must be contained in a PropertyGroup element. In the project file shown in Figure 1 you'll find this declaration contained within a PropertyGroup element:

<AssemblyName>WindowsApplication1</AssemblyName>

In this example, AssemblyName is the name of the property and WindowsApplication1 is the value of this property. In order to access these values you use the $(PROPERTY_NAME) syntax. For example, for this AssemblyName property you would use $(AssemblyName) to access its value. Later you will see how using properties can make your life a lot easier and how the entire build process is actually tied together with properties.

Elements that will be executed by MSBuild must be contained in a target. When Visual Studio builds your project, it is actually executing an MSBuild target.

A target is simply a container for a set of related tasks that will be executed one after another. Many targets are shipped with MSBuild that pertain to building your projects, such as Compile, Build, Rebuild, and Publish. If you have a look at the WindowsApplication1 project file, you will not find a single target defined in that file. This is because all of the targets are imported from another set of files. I will discuss how this works later.

I'll show you how to execute some of the preexisting targets, then I'll offer a few examples of how to create some new targets. In order to execute MSBuild targets, open the Visual Studio 2005 Command Prompt. To find this tool, click Start | All Programs | Microsoft Visual Studio 2005 | Visual Studio Tools. After opening the command prompt window, navigate to the folder that contains the WindowsApplication1.csproj file. Now, to clean this project, use the following command:

msbuild.exe WindowsApplication1.csproj /t:Clean

In this example you're passing in two parameters: the project file to use and the target to execute. If all went well with cleaning the project, your results should look similar to Figure 2.

Figure 2 Cleaning a Project
Figure 2 Cleaning a Project

You can also build a project by executing the Build target. The command for this is:

msbuild.exe WindowsApplication1.csproj /t:Build

Along with the project name and target, there are several other command-line parameters you can send to Msbuild.exe. For example, you can specify the detail of log output with the /verbosity (/v) parameter or you can specify properties using the /property (/p) parameter. For a complete list of available parameters and their usage, see msdn2.microsoft.com/ms164311.aspx.

By having a look at the core msbuild schema document (located at %FrameworkDir%/MSBuild/Microsoft.Build.Core.xsd), you can see that the target element can have up to five attributes assigned to it—Name, Condition, DependsOnTargets, Inputs, and Outputs. The Name attribute is the reference to the target, as was shown previously. Each MSBuild element can have a Condition associated with it and, if this condition evaluates to false, then the entire element will be ignored. You may have noticed this from the Configuration property declaration shown earlier. Here is the relevant section:

<Configuration Condition=" '$(Configuration)' == '' ">
    Debug</Configuration>

In this declaration, if the Configuration property is empty, it will be assigned a value of Debug. It can be useful to have the same target defined, but different implementations are based on various conditions.

The DependsOnTargets attribute is a semicolon-delimited list of targets that must execute before the declared target. Each of the targets will be examined for execution in the order that they are declared in the DependsOnTargets list. Keep in mind that a target is executed at most one time per MSBuild instance. A target always has an associated build state; the initial state is NotStarted. The other possible build states are InProgress, CompletedSuccessfully, CompletedUnsuccessfully, and Skipped. If a target has a state other than NotStarted, it will not be executed again. Remember this as you design your targets. Typically you'll want completed targets to be skipped, but there are some situations where this is not desirable.

The remaining target attributes are Inputs (files that the target will act upon) and Outputs (files that the target will generate). These lists play a critical role in incremental building, meaning building only the pieces of the application that are out of date.

During the course of a build within Visual Studio, you most likely have seen a message stating that a portion of the build was being skipped. This is incremental building. That portion of the build is being skipped because nothing has changed in that section since the last build. For example, if you have a solution containing many different projects, when you perform a build you'd like to only build projects that have changed, and projects that depend on other projects that have changed. It would be a waste of time to include unaffected projects for each build.

This incremental build concept is supported by the Inputs and Outputs. The idea is that you provide an Inputs list of files that the target will examine. Along with this, you provide an Outputs list of files that your target will create. If Inputs are older than Outputs, then the target will be skipped. As you create targets that perform a lot of work and are time consuming it will be critical to provide correct inputs and outputs so MSBuild can support incrementally building those targets as well.

Creating Targets

Now that you have the basics down, you know almost everything needed to create a simple MSBuild target. The only outstanding issue is tasks. I'll cover this in more depth in another section, but here's a brief description so you can start to build some targets. In MSBuild a task is simply a unit of work. There are many pre-defined tasks that are available for your use, such as copying files, signing files, invoking the C# compiler, and so on.

A task is executed within a target. When a target is being executed, each task contained inside of it will be invoked in the order in which it is declared. For example, take a look at this simple Hello target:

<Target Name="Hello">
    <Message Text="Hello MSBuild" Importance="high"/>
</Target>

This target uses the Message task to send "Hello MSBuild" to the logger, in this case to be printed to the screen. If you place this target into the WindowsApplication1 project file, you can verify that it works correctly. You can achieve this by editing the WindowsApplication1 project from within Visual Studio, and you'll have IntelliSense® to help you.

To inject this target into the project file, I placed it immediately above the closing tag for the Project element. As long as it is an immediate child of the Project element, the location doesn't matter in this case. Verify that this does indeed work by invoking the Hello target on the WindowsApplication1 project file. To do this, execute the following command:

msbuild WindowsApplication1.csproj /t:Hello

You should see a result like this:

Target Hello:
    Hello MSBuild

Items and Transformations

Since builds are heavily reliant on files, it only makes sense to have a construct that makes it easier to work with files. This is the purpose of items. Although items are not limited to being used with files, that certainly is their designed target. An item is a reference to a file (existing or nonexisting) or a group of files. Items can gather sets of related files together and act on them. The item references are declared in the WindowsApplication1 project file like this:

<ItemGroup>
    <Reference Include="System" />
    <Reference Include="System.Data" />
    <Reference Include="System.Deployment" />
    <Reference Include="System.Drawing" />
    <Reference Include="System.Windows.Forms" />
    <Reference Include="System.Xml" />
</ItemGroup>

As you can see, items are contained in an ItemGroup XML element. An item can be decomposed into three parts: an Include list, an Exclude list, and metadata (see Figure 3).

Figure 3 Item Attributes (Close Figure 3)

Attribute | Description
Include | A semicolon-delimited list of files to be included in the item. You can also use wildcards to identify the files.
Exclude | A semicolon-delimited list of files to be excluded from insertion into the item. You can use wildcards on this attribute as well.
Metadata | Data that is associated with the item. There are two kinds of metadata. Well-known metadata is available out of the box and includes such things as FileName and FullPath. Custom metadata is attached to an item in the build file.

The Reference item has the following values associated with it: System, System.Data, System.Deployment, System.Drawing, System.Windows.Forms, and System.Xml. I could have achieved that exact same effect with this more concise declaration:

<Reference Include="System;System.Data;System.Deployment;
    System.Drawing;System.Windows.Forms;System.Xml"/>

The code in Figure 4 indicates that you can use wildcards to initialize items. There are three wildcard elements: ?, *, and **. The ? wildcard replaces a single character with any possible character:

<ItemGroup>
    <MyItem Include="Fi?e.txt"/>
</ItemGroup>

Figure 4 Compile Declaration (Close Figure 4)
<ItemGroup>
    <Compile Include="Form1.cs">
        <SubType>Form</SubType>
    </Compile>
    <Compile Include="Form1.Designer.cs">
        <DependentUpon>Form1.cs</DependentUpon>
    </Compile>
    <Compile Include="Program.cs" />
    <Compile Include="Properties\AssemblyInfo.cs" />
        <EmbeddedResource Include="Properties\Resources.resx">
            <Generator>ResXFileCodeGenerator</Generator>
            <LastGenOutput>Resources.Designer.cs</LastGenOutput>
            <SubType>Designer</SubType>
        </EmbeddedResource>
    <Compile Include="Properties\Resources.Designer.cs">
        <AutoGen>True</AutoGen>
        <DependentUpon>Resources.resx</DependentUpon>
    </Compile>
    <Compile Include="Properties\Settings.Designer.cs">
        <AutoGen>True</AutoGen>
        <DependentUpon>Settings.settings</DependentUpon>
        <DesignTimeSharedInput>True</DesignTimeSharedInput>
    </Compile>
</ItemGroup>

In this declaration MyItem could include files like Fire.txt, File.txt, Fine.txt, and so on, but it would not include files like Fie.txt or FilingCabinet.txt. When MSBuild encounters this item declaration it will examine the current directory (because no path information was provided for the declaration) for files that match the expression. Any file matching this expression is included in the MyItem list.

The * wildcard in a declaration can be replaced with any combination of zero or more characters:

<ItemGroup>
    <MyItem Include="Fi*e.txt"/>
</ItemGroup>

Now MyItem can include files with names that include Fie.txt, File.txt, Fiddle.txt, Fi5e.txt, and so on.

The ** wildcard allows you to tell MSBuild to recursively search all subdirectories, as well as the current directory, for items:

<ItemGroup>
    <MyItem Include="InputFiles\**\Fi*e.txt"/>
</ItemGroup>

When MSBuild encounters this declaration, it will look in the InputFiles folder and each folder under it recursively for files that match the Fi*e.txt pattern. All such files will be included in the MyItem list.

Transformations can seem to be confusing at times, but they are a very important part of MSBuild. An item transformation is a one-to-one mapping between two item lists. For example, if you would like to copy files from one location to another, you will need to use a transformation, and you will use metadata as a part of that transformation.

There are two kinds of metadata: well-known and custom. Well-known metadata is available for all file items. Some examples of well-known metadata are Filename, FullPath, and Extension. These return the file name of the file, the folder path to the file, and the file name extension of the file, respectively. (For a full list of well-known metadata see msdn2.microsoft.com/ms164313.aspx.) Custom metadata is associated with the item directly in the project file.

To illustrate using metadata and transformations, let's copy the files to be compiled to another location. First you need to know how to refer to these files. If you have a look at the WindowsApplication1 project file again, you'll see an item declaration named Compile. This is the set of files that will be sent to the C# compiler. The complete declaration for this item is shown in Figure 4.

Notice that this declaration is quite different from the declaration of References shown previously. This is because the Compile item associates custom metadata to these items. The custom metadata includes the XML elements contained within the items, such as the AutoGen elements, which tell Visual Studio that this is an auto-generated file.

Let's move these files to a folder named SourceCopy in the project directory. The project directory can be determined by using the reserved MSBuildProjectDirectory property. For a list of other reserved properties see msdn2.microsoft.com/ms164309.aspx.

The syntax for a transformation is:

@(ItemName->'TransformationDetails')

ItemName is the name of the item you are transforming, and the TransformationDetails can contain raw text, evaluated properties, and metadata references. The CopyFiles target will copy the files to the location specified:

<Target Name="CopyFiles">
    <!-- If the CopyLocation doesn't exist then create it -->
    <MakeDir Directories="$(CopyLocation)"
        Condition="!Exists('$(CopyLocation)')"/>
    <!-- Now actually copy the files -->
    <Copy SourceFiles="@(Compile->'%(FullPath)')"
        DestinationFiles="@(Compile->
            '$(MSBuildProjectDirectory)
            \$(CopyLocation)\%(RelativeDir)\%(FileName)%(Extension)')"
    />
</Target>

The CopyFiles target first invokes the MakeDir task to create the directory, if it doesn't exist. This is followed by the Copy task that will copy the files to the specified location. (The Copy task is provided by MSBuild. You can get information about built-in MSBuild tasks at msdn2.microsoft.com/7z253716.aspx.) Web Resources

  • MSDN Forum
  • MSBuild Team Blog
  • MSBuild WiKi
  • MSDN Documentation
  • MSDN Reference
  • My Blog

This example passes in two parameters to the Copy task: the SourceFiles and the DestinationFiles. For the SourceFiles the task is extracting the FullPath of the Compile items. The DestinationFiles transformation specification is a little more daunting, so I'll break it down into its components. There are two properties that will be evaluated, $(MSBuildProjectDirectory) and $(CopyLocation). After this there are three pieces of metadata that help piece together the entire path. The metadata used is %(RelativeDir) for path information, %(FileName), and the %(Extension). If you put these together you end up with the path. It is important to note that this process will invoke the Copy task only one time.

Now you need to verify that the task works as intended by executing the CopyFiles target on the WindowsApplication1 project file. From the Visual Studio 2005 Command Prompt, go to the directory containing that project file and execute this command:

msbuild WindowsApplication1 /t:CopyFiles

The result from the execution of this target is shown in Figure 5.

Figure 5 CopyFiles Target Output
Figure 5 CopyFiles Target Output

Let's also take a look at an example of how the Visual Studio project file provides MSBuild functionality right out of the box. Consider this scenario: you have many different projects located on your hard drive and you'd like to clean them all. At first glance you might consider simply deleting all the \bin and \obj directories, but you may inadvertently delete some important files. The ideal solution would be to have each project determine what is needed and what can be thrown out. MSBuild can help you do exactly that. Take a look at the very simple MSBuild project file, CleanAllProjects.proj:

<Project xmlns=http://schemas.microsoft.com/developer/msbuild/2003
    DefaultTargets="CleanAllProjects">
    <!-- Find all projects in or below the current directory -->
    <ItemGroup>
        <Projects Include="**\*.*proj" />
    </ItemGroup>
    <Target Name="CleanAllProjects">
        <MSBuild Projects="@(Projects)" Targets="Clean"
            StopOnFirstFailure="false" ContinueOnError="true">
        </MSBuild>
    </Target>
</Project>

This project has a single item, Projects, which contains all of the project files that are in or under the current directory. Along with this item there is the CleanAllProjects target. When this task is invoked it will call the Clean target on all of the project files contained in the Projects item. This is achieved by using the default MSBuild task, which allows you to create a new instance of MSBuild to process a project file. With this solution you can feel confident that no necessary files will be removed because you are letting the project make the determination. Before Visual Studio 2005 and MSBuild, this type of functionality was not available outside of a custom solution.

Tasks

In MSBuild a task is simply a unit of work that is to be performed. Each task is independent of every other task and is not natively aware of any other task. Examples of tasks are those that copy files, compile C# files, create directories, and so on. A task can have a set of parameters associated with it and can have parameters that it returns. These parameters are the means that a task communicates with the MSBuild project file that is invoking it.

MSBuild ships with several useful tasks. For a complete list, see the Microsoft.Common.tasks file located in the %WinDir%\Microsoft.NET\Framework\v2.0.50727 directory. Some frequently used tasks are summarized in Figure 6. For more specific information about how to use these and other tasks shipped with MSBuild see the MSDN® MSBuild Task Reference.

Figure 6 MSBuild Tasks (Close Figure 6)

Task | Description
CreateItem | Creates items. Since items are evaluated at the beginning of the build, if you need files generated by your build included in an item then you must use this task. If the item that is being created already exists then it will be appended.
CreateProperty | Creates a property. Properties are evaluated at the beginning of the build (except those created by CreateProperty). If the property that is being created exists, it will be overwritten.
Copy | Copies files from one location to another.
Delete | Deletes files.
Error | Notifies MSBuild that an error has occurred. Usually this is used with a Condition attribute that means something went wrong.
Exec | Invokes an executable file.
Message | Passes a message to the MSBuild logger.
MSBuild | Invokes MSBuild on another project file. You can also use the Exec task to start Msbuild.exe, but the MSBuild task has advantages such as getting the output of executed targets.

Previously you saw an example of how to use the Message task to output a message to the console logger:

<Target Name="Hello">
    <Message Text="Hello MSBuild" Importance="high"/>
</Target>

This task invocation passes two pieces of information to the Message task: Text and Importance. As you use tasks you'll quickly notice that each task has its own inputs and outputs. For example the Message task has an input named Text, but the Copy file does not. You can determine the correct inputs and outputs from the task documentation or the source code. The Message task, for example, does not create any outputs.

To demonstrate using a task output, I will use the CreateItem task. For the CreateItem task you need to specify the list of files to be included in the item created. The name of this input is Include. This is also the output that you will use to retrieve the newly generated item. In the WindowsApplication1 sample, I have added the following towards the end of the project file:

<Target Name="PrintOutputFiles" DependsOnTargets="Build">
    <!-- First create an Item that contains these files.
     $(OutputPath) is the property that contains the
     locations where built files are placed. -->
    <CreateItem Include="$(OutputPath)\**\*">
        <Output TaskParameter="Include" ItemName="OutputFiles" />
    </CreateItem>

    <!-- Now send these values to the logger -->
    <Message Text="@(OutputFiles->'%(Filename)%(Extension)')"
     Importance="high"/>
</Target>

As you can see, the CreateItems task is invoked to create the Output Files item, which is then used in the Message element to print the values to the console. The OutputFiles item is available immediately after the CreateItem task completes, and it is available to all other targets as well, not just the one currently executing.

Extending the Build Processes

The entire build process used by MSBuild, and therefore by Visual Studio, is defined by your project file. When creating projects by using Visual Studio, a set of files is imported into your project file. Files are imported using the Import element. For C# projects, the Microsoft.CSharp.targets file is imported. This file contains specific information about how to build a C# project, such as defining the step to call the C# compiler (CSC). It then imports the Microsoft.Common.targets file. This file is responsible for defining the generic steps that are used during the course of all managed builds. Much of the build process is defined in this file.

From the Microsoft.Common.targets file, let's look at how the build target is defined:

<PropertyGroup>
    <BuildDependsOn>
        BeforeBuild;
        CoreBuild;
        AfterBuild
    </BuildDependsOn>
</PropertyGroup>
<Target
    Name="Build"
    Condition=" '$(_InvalidConfigurationWarning)' != 'true' "
    DependsOnTargets="$(BuildDependsOn)"
Outputs="$(TargetPath)"
/>

This is the target that will be invoked by default when you build your project using Visual Studio. You can see that this target itself does nothing. It relies completely on its dependent targets to do all of the work. This is specified using the DependsOnTargets list. As I mentioned previously, this is the list of targets that must be executed before the target gets invoked. In this case, this list is defined in the property BuildDependsOn. This makes it easier for you to customize the necessary steps to execute the target.

To show you just how this works, let's inject a step into the middle of a build process. Do this by modifying the CoreBuildDependsOn list (see Figure 7). This is the list of targets that the CoreBuild target uses. CoreBuildDependsOn is redefined in the WindowsApplication1 project file. Along with this redefinition, you place the target needed for your particular project.

Figure 7 CoreBuildDependsOn (Close Figure 7)
<PropertyGroup>
    <CoreBuildDependsOn>
        BuildOnlySettings;
        PrepareForBuild;
        PreBuildEvent;
        UnmanagedUnregistration;
        ResolveReferences;
        PrepareResources;
        ResolveKeySource;
        Compile;
        MyCustomStep;
        GenerateSerializationAssemblies;
        CreateSatelliteAssemblies;
        GenerateManifests;
        GetTargetPath;
        PrepareForRun;
        UnmanagedRegistration;
        IncrementalClean;
        PostBuildEvent
    </CoreBuildDependsOn>
</PropertyGroup>
<Target Name="MyCustomStep">
    <Message Text="Inside of MyCustomStep target" Importance="high"/>
</Target>

The MyCustomStep target is placed into the dependency list after the following statement in the WindowsApplication1 project:

<Import Project="$(MSBuildBinPath)\Microsoft.CSharp.targets" />

The BuildDependsOn property must be inserted after the Import statement because the Microsoft.Common.targets also defines this property. When properties are defined multiple times, the last definition is the one that is used. So if you placed this definition above the Import statement, it would simply be overridden.

The MyCustomStep passes a message to the MSBuild logger. You can verify that you've successfully injected the step by building the project. At the Visual Studio 2005 Command Prompt execute the following command:

msbuild WindowsApplication1.csproj /t:Build

The results from this are shown in Figure 8.

Figure 8 Build Output for MyCustomStep
Figure 8 Build Output for MyCustomStep

There are other ways to inject your custom steps into the build and many targets are created for this sole purpose. For instance, the BeforeBuild and AfterBuild targets are empty by default and are designed to be overridden. If you need a quick and easy way to perform steps, you can override one of these targets and it will be called at the appropriate time. But there is a drawback to using this technique. If there is more than one place where the same target is overridden, only one of the overrides will be preserved. If you need to add steps before or after a target that has a DependsOn property, then the best option is to append to that list. For example, instead of overriding the BeforeBuild target, you can define a target and prepend it to the BuildDependsOn property:

<PropertyGroup>
    <BuildDependsOn>
        CustomBeforeBuild;
        $(BuildDependsOn);
    </BuildDependsOn>
</PropertyGroup>
<Target Name="CustomBeforeBuild">
    <!-- Your steps here -->
</Target>

This redefinition uses the existing value for the BuildDependsOn property, accessed through $(BuildDependsOn), and simply adds to it. If this was performed multiple times then all of the modifications would exist and no target execution would be lost.

Extending MSBuild

MSBuild has two main ways to be extended: custom tasks and custom loggers. I will show how to write a custom task later in this article. An MSBuild logger, on the other hand, is the object responsible for taking the build event notifications and processing them. MSBuild ships with two loggers by default, the console logger and a file logger. You can use the Msbuild.exe command-line parameters to declare which logger to use. I don't have enough space here to delve into custom loggers but you can find information about them at msdn2.microsoft.com/ms171470.aspx.

If you look at the Microsoft.Common.tasks file, you will see many tasks declared. You'll find a Copy task and a Delete task, but no Move task. Typically during a build you will perform copy operations and even delete files; however, you will not be required to move many files. Yet there are certain times when you will want this functionality, and it would be nice if there was a task to take care of this.

When creating a task, keep in mind that as with a good subroutine, you want to make it reusable. Because of this, a task should complete small steps. You don't want a task that is responsible for doing too many things. Also, you need to consider how you want this task to communicate with other components. This communication is facilitated through the task's inputs and outputs. Input and output types that are supported include strings and all other built-in types, plus Microsoft.Build.Framework.ITaskItems and arrays of these objects. ITaskItem is the underlying interface for the items used by MSBuild. So in your project file when you create an item you're actually creating one or more ITaskItems. If your task contains ITaskItems then you can access their metadata in a similar way as you would in the project file.

Every MSBuild task must implement the Microsoft.Build.Framework.ITask interface. ITask has two properties (BuildEngine and HostObject) and one method (Execute). The Execute method will be called to carry out the activity of the task at the appropriate time. This is a pretty simple interface and is easily implemented.

When you write your own task, it is best to actually extend the abstract Microsoft.Build.Utilities.Task class. This class will take care of implementing the two properties, and it will help with a few other things. By doing this you can focus on writing the logic behind your task, and take the MSBuild aspect for granted.

The Move task will just extend the Task class.The Move task contains four properties and the Execute method. The class diagram for the Move class is shown in Figure 9. From the four properties, three are designed as possible inputs: SourceFiles, DestinationFiles, and DestinationFolder. The remaining MovedFiles property is an MSBuild output. This will contain a list of the files that were moved. Let's have a look at how the output, MovedFiles, is declared:

private ITaskItem[] movedFiles;

[Output]
public ITaskItem [] MovedFiles
{
    get { return this.movedFiles; }
}

Figure 9 Move Class Diagram
Figure 9 Move Class Diagram

This property returns an array of ITaskItem objects, and you know that it is available as an output because it has the Output attribute (Microsoft.Build.Framework.OutputAttribute) attached to it. Every output must have this attribute, and it also must have a public get accessor. The input properties are shown in Figure 10.

Figure 10 Move Input Properties (Close Figure 10)
private ITaskItem[] sourceFiles;
private ITaskItem[] destFiles;
private ITaskItem destFolder;

[Required]
public ITaskItem [] SourceFiles
{
    get { return this.sourceFiles; }
    set { this.sourceFiles = value; }
}

public ITaskItem [] DestinationFiles
{
    get { return this.destFiles; }
    set { this.destFiles = value; }
}

public ITaskItem DestinationFolder
{
    get { return this.destFolder; }
    set { this.destFolder = value; }
}

The three input properties look like standard properties, with one exception. The SourceFiles property has the Required attribute (Microsoft.Build.Framework.RequiredAttribute). By using this attribute, the task will not be executed unless that input parameter has been given a value. The other input properties are optional. Since these are ITaskItems, you can access their metadata using the GetMetadata and SetMetadata methods. The Execute method simply looks at what properties have been set, does some error checking, and then moves the files one by one (creating the destination directories as necessary), as shown in Figure 11.

Figure 11 The Execute Method in Move Task (Close Figure 11)
public override bool Execute()
{
    Log.LogMessageFromText("Starting move", MessageImportance.Normal);
    bool allSucceeded = true;

    //Now we need to actually implement the move
    if (this.SourceFiles == null || this.SourceFiles.Length <= 0)
    {
        //if nothing to move just leave quietly
        this.DestinationFiles = new ITaskItem[0];
        Log.LogMessageFromText("Nothing to move",
            MessageImportance.Normal);
        return true;
    }

    if (this.DestinationFiles == null && this.DestinationFolder == null)
    {
        Log.LogError("Unable to determine destination for files");
        return false;
    }

    if (this.DestinationFiles != null && this.DestinationFolder != null)
    {
        Log.LogError("Both DestinationFiles & DestinationFolder " +
            "were specified, only one can be defined");
        return false;
    }

    if (this.DestinationFiles != null &&
       (this.DestinationFiles.Length != this.SourceFiles.Length) )
    {
        //# of items in source & dest don't match up
        Log.LogError("SourceFiles and DestinationFiles differ in length");
        return false;
    }

    if (this.DestinationFiles == null)
    {
        //populate from DestinationFolder
        this.DestinationFiles = new ITaskItem[this.SourceFiles.Length];
        for (int i = 0; i < this.SourceFiles.Length; i++)
        {
            string destFile;
            try
           {
               destFile = Path.Combine(
                   this.DestinationFolder.ItemSpec,Path.GetFileName(
                       this.SourceFiles[i].ItemSpec));
           }
           catch(Exception ex)
           {
               Log.LogError("Unable to move files; " + ex.Message,null);
               this.DestinationFiles = new ITaskItem[0];
               return false;
           }
           this.DestinationFiles[i] = new TaskItem(destFile);
           this.SourceFiles[i].CopyMetadataTo(this.DestinationFiles[i]);
       }
   }

   this.movedFiles = new ITaskItem[this.SourceFiles.Length];
   //now we can go through and move all the files
   for (int i = 0; i < SourceFiles.Length; i++)
   {
       string sourcePath = this.SourceFiles[i].ItemSpec;
       string destPath = this.DestinationFiles[i].ItemSpec;
       try
       {
           string message = string.Format(
               "Moving file {0} to {1}", sourcePath, destPath);
           Log.LogMessageFromText(message, MessageImportance.Normal);
           FileInfo destFile = new FileInfo(destPath);
           DirectoryInfo parentDir = destFile.Directory;
           if (!parentDir.Exists) parentDir.Create();
           File.Move(sourcePath, destPath);
           this.movedFiles[i] = new TaskItem(destPath);
       }
       catch (Exception ex)
       {
           Log.LogError("Unable to move file: " + sourcePath + " to " +
               destPath + "\n" + ex.Message);
           allSucceeded = false;
       }
   }

   return allSucceeded;
}

At this point I have a completed task and can use it in an MSBuild project. I created a blank project file named MoveFiles.proj for this purpose. To use this task I have to build the assembly that contains it. My assembly is named Tasks.dll. After building the assembly, I place it in a known directory. In this case I'm placing the assembly in a directory named SharedTasks. When you want to use a custom task in your projects, you have to tell MSBuild what task you intend to use and where it is located. You do this with the UsingTask element. In the case of my example project, the declaration looks like this:

<PropertyGroup>
    <!-- Location of the shared tasks directory -->
    <SharedTasksDir>..\..\SharedTasks</SharedTasksDir>
    <SourceFolder>SourceFiles</SourceFolder>
    <DestFolder>DestFolder</DestFolder>
</PropertyGroup>

<UsingTask AssemblyFile="$(SharedTasksDir)\Tasks.dll" TaskName="Move"/>

The UsingTask declaration states where to find the needed assembly (AssemblyFile) and the name of the task (TaskName). Instead of the AssemblyFile attribute, you can use the AssemblyName attribute. AssemblyFile causes MSBuild to use Assembly.LoadFrom to load the target assembly, whereas using AssemblyName causes MSBuild to use Assembly.Load. See the UsingTask documentation for details about the difference.

To demonstrate this task in action I have created a target, MoveFiles:

<Target Name="MoveFiles">
    <Move SourceFiles=
            "@(SourceFiles->'%(FullPath)')"
        DestinationFiles=
            "@(SourceFiles->'$(DestFolder)\
             %(Filename)%(Extension)')">
        <Output TaskParameter="MovedFiles"
            ItemName="FilesMoved"/>
    </Move>
    <!-- Print out the files so we can see
         where they moved to -->
    <Message Text=
        "Moved files:%0d%0a
         @(FilesMoved->'%(FullPath)','%0d%0a  ')"/>
</Target>

In this target I call the Move task just like I would any other task. I specify the input properties as parameters and capture the output in the Output element. Now I can call this target and verify that this task does indeed work (see Figure 12).

Figure 12 MoveFiles Target Output
Figure 12 MoveFiles Target Output

Included in the source code for this article, available for download from the MSDN Magazine Web site, you'll find three other custom tasks: GetDate, GetRegKey, and TempFile. The GetDate task populates a property with the current date and time, based on a specified format. The GetRegKey task retrieves a registry value. The TempFile task creates a temporary file. Take a look at these tasks and their corresponding example project files for more insight on how to create custom tasks.

Conclusion

MSBuild is designed to work behind the scenes, and as a result many developers don't even realize that it exists. After having read this article you should feel confident about customizing your build process to suit your particular needs. I've given a broad overview of MSBuild here, but like many technologies, it cannot be completely covered in a single article. The best way to learn about MSBuild is to start using it.

This is not all there is to know about writing custom tasks, but from the information provided here you should be able to write many tasks that will make your work easier. From this point forward you can write your own custom tasks to take care of various build steps that you need executed. Some other issues that I did not discuss but are worth investigating include tasks executing in their own application domains and assigning metadata to items inside of a task.

There are many different resources available on the Web that you can turn to when you are having difficulties. A list of some of these resources is provided in the "Web Resources" sidebar. The MSBuild forum is a great place to go when you need a specific question answered.

Sayed Ibrahim Hashimi has a computer engineering degree from the University of Florida. He is a developer and architect in Jacksonville, Florida. He is an expert in the financial, education, and collection industries. His primary focus is working with .NET.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


WCF Essentials: Discover Mighty Instance Management Techniques For Developing WCF Apps

WCF Essentials

Discover Mighty Instance Management Techniques For Developing WCF Apps

Juval Lowy
This article discusses:
  • Per-call services
  • Per-session services
  • Shareable services
  • Instance deactivation
This article uses the following technologies:
Windows Communication Foundation
Code download available at: InstanceManagement2006_06.exe (230KB)
Contents
Per-Call Services
Per-Session Services
Shareable Services
Duplicating a Proxy
Sharing an Instance
Singleton Services
Demarcating Operations
Instance Deactivation
Throttling
Configuring Throttling
Reading Throttled Values
Conclusion

I
nstance management refers to a set of techniques used by Windows® Communication Foundation to bind a set of messages to a service instance. It's necessary because applications widely differ in their needs for scalability, performance, throughput, transactions, and queued calls, and there simply isn't a one-size-fits-all solution to address these varied demands. Understanding instance management is essential to developing scalable and consistent service-oriented applications. This article provides the rationale for various instance management modes, offers guidelines for when and how to use them, and discusses some related topics such as behaviors, contexts, demarcating operations, and instance deactivation.

By and large, the service instance mode is strictly a service-side implementation detail that should not manifest itself on the client side. To support that and a few other local service-side aspects, Windows Communication Foundation defines the notion of behaviors. A behavior is a local attribute of a service that does not affect its communication patterns.

The ServiceBehaviorAttribute configures service behaviors—a behavior that affects all endpoints of the service and is applied directly on the service implementation class. As shown in Figure 1, the attribute defines the InstanceContextMode property of the enum type InstanceContextMode whose value controls which instance mode is used for the service.

Figure 1 InstanceContextMode (Close Figure 1)

public enum InstanceContextMode
{
    PerCall, PerSession, Shareable, Single
}

[AttributeUsage(AttributeTargets.Class)]
public sealed class ServiceBehaviorAttribute : Attribute, ...
{
    public InstanceContextMode InstanceContextMode {get;set;}
    ... // More members
}

Per-Call Services

Per-call services are the Windows Communication Foundation default instantiation mode. When the service type is configured for per-call activation, a service instance, a common language runtime (CLR) object, exists only while a client call is in progress. Every client request gets a new dedicated service instance. Figure 2 illustrates how this single-call activation works.

Figure 2 Per-Call Instantiation
Figure 2 Per-Call Instantiation

  1. The client calls the proxy and the proxy forwards the call to the service.
  2. Windows Communication Foundation creates a service instance and calls the method on it.
  3. After the method call returns, if the object implements IDisposable, then Windows Communication Foundation calls IDisposable.Dispose on it.

In the classic client-server programming model, using languages such as C++ or C#, every client gets its own dedicated server object. The fundamental problem with this approach is that it doesn't scale well. The server object may hold expensive or scarce resources such as database connections, communication ports, or files. Imagine an application that has to serve many clients. Typically, these clients create the objects they need when the client application starts and dispose of them when the client application shuts down. What impedes scalability with the client-server model is that the client applications can hold onto objects for long periods of time, while actually using the objects for only a fraction of that time. If you allocate an object for each client, you will tie up such crucial or limited resources for long periods of time and you will eventually run out of resources.

A better activation model is to allocate an object for a client only while a call is in progress from the client to the service. That way, you have to create and maintain only as many objects in memory as there are concurrent calls, not as many objects as there are outstanding clients. Between calls, the client holds a reference on a proxy that doesn't have an actual object at the end of the wire. The obvious benefit is that you can now dispose of the expensive resources the service instance occupies long before the client disposes of the proxy. By that same token, acquiring the resources is postponed until they are actually needed by a client. The second benefit is that forcing the service instance to reallocate or connect to its resources on every call caters very well to transactional resources and transactional programming because it eases the task of enforcing consistency with the instance state.

To configure a service type as a per-call service, you can apply the ServiceBehavior attribute with the InstanceContextMode property set to InstanceContextMode.PerCall:

[ServiceContract]
interface IMyContract {...}

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
class MyService : IMyContract {...}

Because InstanceContextMode.PerCall is the default value for the property, there is no need to actually apply the attribute. Consequently, the following two definitions are equivalent:

class MyService : IMyContract {...}

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
class MyService : IMyContract {...}

Although in theory you can use per-call instance activation mode on any service, in practice, you need to design the service and its contracts to support per-call activation mode from the ground up. The main problem is that the client doesn't know it's getting a new instance each time.

If a per-call service must be state-aware, meaning that state from one call must persist to a future call, the implementation itself must proactively manage the state, giving the client the illusion of a continuous session. An instance of a per-call service is created just before every method call and destroyed immediately after each call. Therefore, at the beginning of each call, the object should initialize its state from values saved in some storage, and at the end of the call it should return its state to the storage. Such storage is typically either a database or the file system, but it can also be volatile storage like static variables. However, not all of the object's state can be saved as-is. For example, if the state contains a database connection, the object must reacquire the connection at construction or at the beginning of every call and dispose of the connection at the end of the call or in its implementation of IDisposable.Dispose.

Using per-call instance mode has one important implication for operation design: every operation must include a parameter to identify the service instance whose state needs to be retrieved. Examples for such parameters are the account number for bank account service, the order number for services processing orders, and so on. Figure 3 shows a template for implementing a per-call service.

Figure 3 Implementing a Per-Call Service (Close Figure 3)

[DataContract]
class Param {...}

[ServiceContract]
interface IMyContract
{
    [OperationContract]
    void MyMethod(Param objectIdentifier);
}

class MyPerCallService : IMyContract, IDisposable
{
    public void MyMethod(Param objectIdentifier)
    {
        GetState(objectIdentifier);
        DoWork();
        SaveState(objectIdentifier);
    }
    void GetState(Param objectIdentifier) {...}
    void DoWork() {...}
    void SaveState(Param objectIdentifier) {...}
    public void Dispose() {...}
}

The MyMethod operation accepts a parameter of type Param (a pseudo-type invented for this example) that identifies the instance, as shown in the following:

public void MyMethod(Param objectIdentifier);

The instance then uses the identifier to retrieve its state and to save the state back at the end of the method call. Any piece of state that is common to all clients can be allocated at the constructor and disposed of in Dispose.

Also, the per-call activation mode works best when the amount of work to be done in each method call is small and there are no more activities to complete in the background once a method returns. For this reason, you should not spin off background threads or dispatch asynchronous calls back into the instance because the object will be discarded once the method returns.

Per-call services clearly offer a trade-off in performance (the overhead of reconstructing the instance state on each method call) with scalability (holding onto the state and the resources it ties in). There are no hard-and-fast rules as to when and to what extent you should trade some performance for a lot of scalability. You may need to profile your system and ultimately design some services to use per-call activation and redesign some not to use it.

Per-Session Services

Windows Communication Foundation can maintain a private session between a client and a particular service instance. When the client creates a new proxy to a service configured as session-aware, the client gets a new dedicated service instance that is independent of all other instances of the same service. That instance will remain in service usually until the client no longer needs it. Each private session uniquely binds a proxy to a particular service instance. Note that the client session has one service instance per proxy. If the client creates another proxy to the same or a different endpoint, that second proxy will be associated with a new instance and session.

Because the service instance remains in memory throughout the session, it can maintain state in memory, and the programming model is very much like that of the classic client-server model. Consequently, it also suffers from the same scalability and transaction issues as the classic client-server model. A service configured for private sessions cannot typically support more than a few dozen (or perhaps up to a few hundred) outstanding clients due to the cost associated with each such dedicated service instance.

Supporting a session has two facets: contractual and behavioral. The contractual facet is required across the service boundary because the client-side Windows Communication Foundation runtime needs to know it should use a session. The ServiceContract attribute offers the Boolean property Session:

[AttributeUsage(AttributeTargets.Interface|AttributeTargets.Class,
    Inherited=false)]
public sealed class ServiceContractAttribute : Attribute
{
    public bool Session {get;set;}
    ... // More members
}

Session defaults to false. To support sessions, you need to set Session to true at the contract level:

[ServiceContract(Session = true)]
interface IMyContract {...}

To complete the configuration, you need to instruct Windows Communication Foundation to keep the service instance alive throughout the session and to direct the client messages to it. This local behavior facet is achieved by setting the InstanceContextMode property of the ServiceBehavior attribute to InstanceContextMode.PerSession, as shown in the following:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)]
class MyService : IMyContract {...}

The session typically terminates when the client closes the proxy, which notifies the service that the session has ended. If the service supports IDisposable, then the Dispose method will be called. Figure 4 shows a contract and service configured to use a private session and their client. As you can see from the output, the client got a dedicated instance.

Figure 4 Per-Session Service and Client (Close Figure 4)

Service Code
[ServiceContract(Session = true)]
interface IMyContract
{
    [OperationContract]
    void MyMethod();
}

[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerSession)]
class MyService : IMyContract,IDisposable
{
    int m_Counter = 0;

    MyService()
    {
        Trace.WriteLine("MyService.MyService()");
    }

    public void MyMethod()
    {
        m_Counter++;
        Trace.WriteLine("Counter = " + m_Counter);
    }

    public void Dispose()
    {
        Trace.WriteLine("MyService.Dispose()");
    }
}
Client Code
MyContractProxy proxy = new MyContractProxy();
proxy.MyMethod();
proxy.MyMethod();
proxy.Close();
Output
MyService.MyService()
Counter = 1
Counter = 2
MyService.Dispose()

The per-session instance mode provides for a session at the application level between the client and a service instance. This session is only as reliable as the underlying channel session. Consequently, a service that implements a session-aware contract requires that all the endpoints that expose the contract use bindings that support reliable transport session. This constraint is verified at service load time, yielding InvalidOperationException if there is a mismatch. Make sure to use a binding that supports reliability and that you explicitly enable it at both the client and the service, either programmatically or administratively. One exception to this rule is the named pipes binding. This binding is considered a reliable transport and has no need for the reliable messaging protocol. Besides, all calls will be on the same machine anyway.

Just as reliable transport session is optional, so is ordered delivery of messages, and Windows Communication Foundation will provide for a session even when ordered delivery is disabled. Typically, a client that interacts with a session-aware service expects that all messages are delivered in the order they are sent. As such, ordered delivery is enabled by default when reliable transport session is enabled, so no additional setting is required.

Typically, the session will end once the client closes the proxy. However, in case the client fails to terminate gracefully or encounters a communication problem, each session also has an idle time timeout that defaults to 10 minutes—the session will automatically terminate after 10 minutes of inactivity from the client, even if the client still intends to use the session. Once the session has terminated due to the idle timeout, if the client tries to use its proxy, the client will get a CommunicationObjectFaultedException.

Both the client and the service can configure a different timeout by setting a different value in the binding. The bindings that support reliable transport-level session provide the ReliableSession property with the InactivityTimeout property used for configuring the idle timeout. For example, the following shows the code that is required to programmatically configure an idle timeout of 25 minutes for the TCP binding:

NetTcpBinding tcpSessionBinding = new NetTcpBinding();

tcpSessionBinding.ReliableSession.Enabled = true;

tcpSessionBinding.ReliableSession.InactivityTimeout =
    TimeSpan.FromMinutes(25);

Here is the equivalent configuration setting using a config file:

<netTcpBinding>
    <binding name="TCPSession">
        <reliableSession enabled="true" inactivityTimeout="00:25:00"/>
    </binding>
</netTcpBinding>

If both the client and the service configure timeouts, then the shorter timeout prevails.

Shareable Services

Windows Communication Foundation does not allow you to pass object references across service boundaries. Objects are technology-specific entities, and sharing objects goes against the grain of service-oriented, technology-neutral interactions. However, sometimes one client may want to share the current state of its session with another client. The solution is a shareable service. A shareable service behaves much like a per-session service, with one important additional aspect: the instance has a unique ID, and when a client establishes a session with a shareable service, the client can pass a logical reference to that instance to another client. The second client will establish an independent session but will share the same instance. Also, each of these sessions may use different inactivity timeouts, and expire independently of any other session.

You configure a shareable service by setting the InstanceContextMode property to InstanceContextMode.Sharable:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Sharable)]
class MyService : IMyContract {...}

There are two ways to take advantage of a shareable service and both are variations on the same technique. The first is to simply duplicate a proxy in the same app domain and the second is to pass an endpoint address over Windows Communication Foundation to another client.

Passing around an instance reference presents an interesting challenge to Windows Communication Foundation: what if the first client creates the instance, passes a reference to it to a second client, and closes its proxy before the second client has had the chance to create its own proxy using the reference? To deal with that, Windows Communication Foundation employs a linger timeout that defaults to one minute. Windows Communication Foundation keeps track of the sessions wired up to the same instance. Closing a proxy by any one of the clients does not dispose of the service instance. The instance is disposed of when the last session ends and the linger timeout has expired. Windows Communication Foundation also allows for configuring the linger timeout. In the service host config file, add a behaviorConfiguration property to the service element, pointing to a behavior section. In that section, set the instanceContextIdleTimeout property to the desired linger timeout:

<services>
    <service type="MySharableService"
     behaviorConfiguration="ShortLingerBehaviour">
    ...
    </service>
</services>
<behaviors>
    <behavior name="ShortLingerBehaviour"
     instanceContextIdleTimeout="00:05:00" />
</behaviors>

If the linger timeout is not good enough for your application, then the participating clients need to coordinate among themselves when it is permissible to close the proxies.

Duplicating a Proxy

Duplicating a proxy is handy when two clients in the same app domain want to share a service instance. If the two clients share a proxy reference, then the clients need to coordinate which is responsible for closing the proxy and when. This need for coordination will introduce an unwanted degree of coupling between the two clients. The solution is to have the two clients use separate proxies and yet point to the same instance. Because the two proxies will have independent sessions, each client can close its proxy without affecting the other.

To duplicate a proxy, create a proxy to the service as usual and then explicitly resolve the service instance by creating an endpoint address reference. This contains the address of the service and some additional addressing headers information required to connect to the existing instance. To obtain an endpoint reference, use the ResolveInstance method of the IClientChannel interface:

public interface IClientChannel : ...
{
    EndpointAddress ResolveInstance();
    ... // More members
}

You obtain IClientChannel through the InnerChannel property of the ClientBase<T> proxy base class.

Calling ResolveInstance requires an exchange with the service to both verify it is configured for sharing and to obtain the instance ID. If the service is not configured for sharing, ResolveInstance throws a CommunicationException. Once you have the endpoint address, you create a new proxy with it. You can use the proxy constructor that takes the endpoint section name from the config file and the address, in which case the address in the config file will be ignored. Alternatively, you can use the proxy constructor that takes a binding and address objects. You can also set the endpoint address explicitly post-construction through the Endpoint property of the proxy. Regardless of which method you choose to create the proxy, make sure the endpoint address's transport matches that of the binding used. Now you have two distinct proxies that share the same service instance, albeit in separate sessions. Figure 5 demonstrates this technique.

Figure 5 Duplicating a Proxy (Close Figure 5)

MyContractProxy proxy1 = new MyContractProxy();
proxy1.MyMethod();
IClientChannel channel = proxy1.InnerChannel;

//For simplicity, just grab binding from first proxy
Binding binding = proxy1.Endpoint.Binding;
EndpointAddress address = channel.ResolveInstance();

MyContractProxy proxy2 = new MyContractProxy(binding,address);
proxy2.MyMethod();

proxy1.Close();
proxy2.Close();

Using the same definitions as in Figure 4 except with the service configured for InstanceContextMode.Sharable, Figure 5 yields the same output as Figure 4.

Sharing an Instance

If the service is configured as shareable, Windows Communication Foundation lets you share an instance of that service across service boundaries, meaning you can pass a reference to a shared instance from one client to another over Windows Communication Foundation. The receiving side needs to expose a contract with an operation that takes EndpointAddress as a parameter. This sort of interaction is illustrated in Figure 6. In this figure, the client has a session with an instance of the shareable Service A:

  1. The client obtains an endpoint address from Service A.
  2. The client then passes the address over Windows Communication Foundation to an instance of Service B.
  3. Service B constructs a proxy using the endpoint address and calls the instance of Service A.

Figure 6 Passing a Reference
Figure 6 Passing a Reference

The B instance can also pass the endpoint address to another internal client, or pass it to another service, and so on.

It is worth noting that while the original client and the A instance need to have a session, and while the B instance (or any client that uses the address) and the A instance need to have a session, the client and the B instance do not need to have a session at all.

As I just mentioned, the receiving end must implement some kind of a contract that can accept an EndpointAddress as a parameter, as the following example shows:

[ServiceContract]
interface ISomeContract
{
    [OperationContract]
    void PassReference(EndpointAddress10 serviceAddress);
}

EndpointAddress is not serializable. However, you can safely work around this by using EndpointAddress10, which corresponds to the wire-format of WS-Addressing 1.0.

Assume for a moment that the service implementing ISomeContract is called MyOtherService and is defined as follows:

class MyOtherService : ISomeContract
{
   public void PassReference(EndpointAddress10 serviceAddress)
   {
      MyContractProxy proxy = new MyContractProxy();
      proxy.Endpoint.Address = serviceAddress.ToEndpointAddress();
      proxy.MyMethod();
      proxy.Close();
   }
}

Using the same definitions as in Figure 4, but with the service configured for InstanceContextMode.Sharable, the following code yields the same output:

MyContractProxy proxy1 = new MyContractProxy();
proxy1.MyMethod();

IClientChannel channel = proxy1.InnerChannel;
EndpointAddress serviceAddress = channel.ResolveInstance();

SomeContractProxy proxy2 = new SomeContractProxy();
proxy2.PassReference(EndpointAddress10.FromEndpointAddress
    (serviceAddress));

proxy1.Close();
proxy2.Close();

I find that a shareable service instance is a cumbersome instance management technique that probably harbors more harm than good. It does offer a singular benefit for some specific sharing scenarios, but it raises complicated issues with regard to session management and potential lifecycle coupling if the linger timeout is inadequate for your application. Try to avoid this technique and find a design solution that does not require you to pass a service reference or share a service instance's state.

Singleton Services

The singleton service is the ultimate shareable service. When a service is configured as a singleton, all clients get connected to the same single well-known instance independently of each other, regardless of which endpoint of the service they connect to. The singleton service lives forever, and is only disposed of once the host shuts down. The singleton is created exactly once when the host is created.

Using a singleton does not require clients to maintain a session with the singleton instance, or to use a binding that supports transport-level session. If the contract that the client consumes has a session, by closing the proxy the client will only terminate the session, not the singleton instance. In addition, the session will never expire. If the singleton service supports contracts without a session, those contracts will not be per-call—they too will be connected to the same instance. With a singleton service, you can't call ResolveInstance. The reason is obvious: by its very nature the singleton is shared, and each client should simply create its own proxies to it.

You configure a singleton service by setting the InstanceContextMode property to InstanceContextMode.Single:

[ServiceBehavior(InstanceContextMode = InstanceContextMode.Single)]
class MySingleton : ...
{...}

Figure 7 demonstrates a singleton service with two contracts, one that requires a session and one that does not require one. As you can see from the client call, the calls on the two endpoints are routed to the same instance, and closing the proxies does not result in terminating the singleton.

Figure 7 Singleton Service and Client (Close Figure 7)

Service Code
[ServiceContract(Session=true)]
interface IMyContract
{
    [OperationContract]
    void MyMethod();
}

[ServiceContract]
interface IMyOtherContract
{
    [OperationContract]
    void MyOtherMethod();
}

[ServiceBehavior(InstanceContextMode=InstanceContextMode.Single)]
class MySingleton : IMyContract, IMyOtherContract, IDisposable
{
    int m_Counter = 0;
    public MySingleton()
    {
        Trace.WriteLine("MyService.MyService()");
    }
    public void MyMethod()
    {
        m_Counter++;
        Trace.WriteLine("Counter = " + m_Counter);
    }
    public void MyOtherMethod()
    {
        m_Counter++;
        Trace.WriteLine("Counter = " + m_Counter);
    }
    public void Dispose()
    {
        Trace.WriteLine("MyService.Dispose()");
    }
}
Client Code
MyContractProxy proxy1 = new MyContractProxy();
proxy1.MyMethod();
proxy1.Close();

MyOtherContractProxy proxy2 = new MyOtherContractProxy();
proxy2.MyOtherMethod();
proxy2.Close();
Output
MyService.MyService()
Counter = 1
Counter = 2

Sometimes you may not want to create and initialize the singleton using just the default constructor. Perhaps initializing the scarce resource the singleton manages takes too much time and you would not want to penalize the first client. Perhaps initializing that state requires some custom steps or specific knowledge not available to the clients (or the clients should not be bothered). To support such scenarios, Window Communication Foundation allows you to create the singleton instance directly using normal CLR instantiation beforehand, initialize the instance, and then open the host with that instance in mind as the singleton service. The ServiceHost class offers a dedicated constructor that accepts an object:

public class ServiceHost : ServiceHostBase,...
{
    public ServiceHost(object singletonInstance,
        params Uri[] baseAddresses);
    public virtual object SingletonInstance { get; }
    ... // More members
}

Note that the object must be configured as a singleton. For example, consider the code in Figure 8. The class MySingleton will be first initialized and then hosted as a singleton. The client making the first call will be connected to the already initialized instance.

Figure 8 Initializing and Hosting a Singleton (Close Figure 8)

Service Code

[ServiceContract]
interface IMyContract
{
    [OperationContract]
    void MyMethod();
}
[ServiceBehavior(InstanceContextMode=InstanceContextMode.Single)]
class MySingleton : IMyContract
{
    int m_Counter = 0;

    //Accesses m_Counter
    public int Counter {get;set;}

    public void MyMethod()
    {
        m_Counter++;
        Trace.WriteLine("Counter = " + Counter);
    }
}
Host Code

MySingleton singleton = new MySingleton();
singleton.Counter = 42;
ServiceHost host = new ServiceHost(singleton);
host.Open();
Client Code

MyContractProxy proxy = new MyContractProxy();
proxy.MyMethod();
proxy.Close();
Output

Counter = 43

If you do initialize and host a singleton this way, you may also want to access it directly on the host side. Windows Communication Foundation enables downstream objects to reach back into the singleton directly using the SingletonInstance property of ServiceHost. Any party on the call chain leading down from an operation call on the singleton can always access the host through the operation context's read-only Host property:

public sealed class OperationContext : ...
{
    public ServiceHostBase Host {get;}
    ... // More members
}

Once you have the singleton reference you can interact with it directly, like shown here:

ServiceHost host = OperationContext.Current.Host as ServiceHost;
Debug.Assert(host != null);
MySingleton singleton = host.SingletonInstance as MySingleton;
Debug.Assert(singleton != null);
singleton.Counter = 388;

If no singleton instance is provided to the host, SingletonInstance returns null.

Having a singleton implies the singleton has some valuable state that you want to share across multiple clients. The problem is that when multiple clients connect to the singleton, they may all do so concurrently on multiple worker threads. The singleton must synchronize access to its state to avoid state corruption. This in turn means that only one client at a time can access the singleton. This may degrade responsiveness and availability to the point that the singleton is unusable as the system grows.

In general, use a singleton object if it maps well to a natural singleton in the application domain. A natural singleton is a resource that is by its very nature single and unique. Examples of natural singletons are a global logbook that all services should log their activities to, a single communication port, and a single mechanical motor. Avoid using a singleton if there is even the slightest chance that the business logic will allow more than one such service in the future, such as adding another motor or a second communication port. The reason is clear: if your clients all depend on being implicitly connected to the well-known instance and more than one service instance is available, the clients would suddenly need to have a way to bind to the correct instance. This can have severe implications on the application's programming model.

Demarcating Operations

Sometimes when dealing with session contracts there is an implied order to operation invocations. Some operations cannot be called first while other operations must be called last. For example, consider this contract used to manage customer orders:

[ServiceContract(Session = true)]
interface IOrderManager
{
    [OperationContract]
    void SetCustomerId(int customerId);
    [OperationContract]
    void AddItem(int itemId);
    [OperationContract]
    decimal GetTotal();
    [OperationContract]
    bool ProcessOrders();
}

The contract has the following constraints: the client must first provide the customer ID against which items are added; then the total is calculated. When the order processing is complete, the session is terminated.

Windows Communication Foundation allows contract designers to designate contract operations as operations that can or cannot start or terminate the session using the IsInitiating and IsTerminating properties of the OperationContract attribute:

[AttributeUsage(AttributeTargets.Method)]
public sealed class OperationContractAttribute : Attribute
{
    public bool IsInitiating {get;set;}
    public bool IsTerminating {get;set;}
    ... // More members
}

Using these properties may demarcate the boundary of the session, so I call this technique demarcating operations. Both a session-aware service and a singleton can implement a contract that uses demarcating operations to manage their client sessions.

The default values of these properties are IsInitiating set to true and IsTerminating set to false. Consequently these two definitions are equivalent, as shown here:

[ServiceContract(Session = true)]
interface IMyContract
{
    [OperationContract]
    void MyMethod()
}

[ServiceContract(Session = true)]
interface IMyContract
{
    [OperationContract(IsInitiating = true, IsTerminating = false)]
    void MyMethod()
}

By default, operations do not demarcate the session boundary—they can be called first, last, or between any other operation in the session. Using non-default values enables you to dictate that a method is not called first, or that it is called last, or both, to enforce the interaction constraints (see Figure 9).

Figure 9 Specifying Demarcating Operations (Close Figure 9)

Service Code

[ServiceContract(Session = true)]
interface IOrderManager
{
    [OperationContract]
    void SetCustomerId(int customerId);

    [OperationContract(IsInitiating = false)]
    void AddItem(int itemId);

    [OperationContract(IsInitiating = false)]
    decimal GetTotal();

    [OperationContract(IsInitiating = false, IsTerminating = true)]
    bool ProcessOrders();
}
Client Code

OrderManagerProxy proxy = new OrderManagerProxy();
proxy.SetCustomerId(123);
proxy.AddItem(4);
proxy.AddItem(5);
proxy.AddItem(6);
proxy.ProcessOrders();
proxy.Close();

When IsInitiating is set to true (the default), it means the operation will start a new session if it is the first method called by the client, but that it will be part of the ongoing session if another operation is called first. When IsInitiating is set to false, it means the operation can never be called as the first operation by the client in a new session, and the method can only be part of an ongoing session.

When IsTerminating is set to false (the default), the session continues after the operation returns. When IsTerminating is set to true, the session terminates once the method returns, and the client will not be able to issue additional calls on the proxy. Note that the client must still close the proxy because the operation does not dispose of the service instance—it simply rejects subsequent calls.

Instance Deactivation

The session-aware service instance management technique as described so far connects a client (or clients) to a service instance. The real picture is more complex. Each service instance is hosted in a context, as shown in Figure 10.

Figure 10 Contexts and Instances
Figure 10 Contexts and Instances

Sessions actually correlate the client messages not to the instance, but to the context that hosts it. When the session starts, the host creates a new context. When the session ends, the context is terminated. By default, the lifeline of the context is the same as that of the instance it hosts. However, for optimization purposes, Windows Communication Foundation provides the service designer with the option of separating the two lifelines and deactivating the instance separately from its context. In fact, Windows Communication Foundation also allows the situation of a context that has no instance at all. I call this instance management technique context deactivation. The usual way for controlling context deactivation is through the ReleaseInstanceMode property of the OperationBehavior attribute:

public enum ReleaseInstanceMode
{
    None, BeforeCall, AfterCall, BeforeAndAfterCall
}
[AttributeUsage(AttributeTargets.Method)]
public sealed class OperationBehaviorAttribute : Attribute,...
{
    public ReleaseInstanceMode ReleaseInstanceMode {get;set;}
    ... // More members
}

The various values of ReleaseInstanceMode specify when to release the instance in relation to the method call: before, after, before and after, or not at all. When releasing the instance, if the service supports IDisposable, the Dispose method is called.

You typically apply instance deactivation only on some, but not all, of the service methods, or with different values on different methods. If you were to apply it uniformly, you would end up with a per-call-like service, so you might as well have configured it that way. If relying on instance deactivation assumes a certain call order, you can try and enforce that order using demarcating operations.

The default value for the ReleaseInstanceMode property is ReleaseInstanceMode.None. ReleaseInstanceMode.None means that the instance lifeline is not affected by the call, as shown in Figure 11.

Figure 11 Comparing Instance Deactivation Modes
Figure 11 Comparing Instance Deactivation Modes

When a method is configured with ReleaseInstanceMode.BeforeCall, if there is already an instance in the session, Windows Communication Foundation will deactivate it, create a new instance in its place, and let that new instance service the call (as shown in Figure 11). While the client blocks, the incoming thread handles the process of deactivating the instance and calling dispose before the call occurs. This ensures that the deactivation is indeed done before the call and not concurrently. ReleaseInstanceMode.BeforeCall is designed to optimize methods like Open, which acquire some valuable resources and yet want to release the previously allocated resources. Instead of acquiring the resource when the session starts, you wait on the Open method, and then both release the previously allocated resources and allocate new ones. After Open is called you are ready to start calling other methods on the instance. These methods are typically configured with ReleaseInstanceMode.None.

When a method is configured with ReleaseInstanceMode.AfterCall, Windows Communication Foundation deactivates the instance after the call (see Figure 11). This is designed to optimize methods (such as Close) that clean up valuable resources the instance holds, without waiting for the session to terminate. ReleaseInstanceMode.AfterCall is typically applied on methods called after methods that have been configured with ReleaseInstanceMode.None.

When a method is configured with ReleaseInstanceMode.BeforeAndAfterCall it has the combined effect , as its name implies, of ReleaseInstanceMode.BeforeCall and ReleaseInstanceMode.AfterCall. If the context has an instance before the call is made, then Windows Communication Foundation deactivates that instance just before the call, creates a new instance to service the call, and deactivates the new instance after the call, as shown in Figure 11.

ReleaseInstanceMode.BeforeAndAfterCall may look superfluous at first glance, but it actually complements the other values. It is designed to be applied to methods called after methods marked with ReleaseInstanceMode.BeforeCall/None or before methods marked with ReleaseInstanceMode.AfterCall/None. Consider a situation where the session-aware service wants to benefit from state-aware behavior (like a per-call service), while holding onto resources just when needed to optimize resource allocation and security lookup. If ReleaseInstanceMode.BeforeCall was the only available option, there would be a period of time after the calls when the resources would still be allocated to the object but not in use. A similar situation would occur if ReleaseInstanceMode.AfterCall was the only available option, because there would be a period of time before the call when the resource would be wasted.

Instead of making a design-time decision on which methods to use to deactivate the instance, you can make a run-time decision to deactivate the instance after the method returns. You do that by calling the ReleaseServiceInstance method on the instance context. You obtain the instance context through the InstanceContext property of the operation context:

public sealed class InstanceContext : CommunicationObject,...
{
    public void ReleaseServiceInstance();
    ... // More members
}

public sealed class OperationContext : ...
{
    public InstanceContext InstanceContext {get;}
   ... // More members
}

Figure 12 demonstrates this technique.

Figure 12 Using ReleaseServiceInstance (Close Figure 12)

[ServiceContract(Session = true)]
interface IMyContract
{
    [OperationContract]
    void MyMethod();
}

[ServiceBehavior(InstanceContextMode=InstanceContextMode.PerSession)]
class MyService : IMyContract,IDisposable
{
    public void MyMethod()
    {
        //Do some work then
        OperationContext.Current.InstanceContext.
            ReleaseServiceInstance();
    }

    public void Dispose() {...}
}

Calling ReleaseServiceInstance has a similar effect as using ReleaseInstanceMode.AfterCall. When used in a method decorated with ReleaseInstanceMode.BeforeCall, it has a similar effect as using ReleaseInstanceMode.BeforeAndAfterCall.

Instance deactivation is an optimization technique, and like all such techniques it should be avoided in general cases. Consider using instance deactivation only after failing to meet both your performance and scalability goals and when careful examination and profiling has proven beyond a doubt that using it will improve the situation. If scalability and throughput are your concern, choose the simplicity of the per-call instancing mode, and avoid instance deactivation.

Throttling

While not a direct instance management technique, throttling enables you to restrain client connections and the load they place on your service. Throttling allows you to avoid maxing-out your service and the underlying resources it allocates and uses.

The default throttling setting is unlimited. When engaged, if the throttling settings you configured are exceeded, Windows Communication Foundation automatically places the pending callers in a queue and serves them out of the queue in order. Throttling is done per service type—that is, it affects all instances of the service and all its endpoints. This is done by associating the throttle with every channel dispatcher the service uses.

Windows Communication Foundation allows you to control the service consumption parameters shown in Figure 13. With a per-session service, Max Instances is both the total number of concurrently active instances and the number of concurrent sessions. With a shared-session service, Max Instances is just the total number of concurrently active instances (since each shared instance can have multiple sessions). With a per-call service, the number of instances is actually the same as the number of concurrent calls. Consequently, the maximum number of instances with a per-call service is the minimum of Max Instances and Max Concurrent Calls. Max Instances is ignored with a singleton service since it can only have a single instance anyway.

Figure 13 ServiceThrottle Parameters (Close Figure 13)

Parameter | Description
Max Connections | The overall number of outstanding clients connected to the service.
Max Concurrent Calls | The total number of calls currently in progress across all service instances.
Max Instances | The total number of contexts concurrently alive. How instances map to contexts is a product of the instance context management mode.

Configuring Throttling

Throttling is typically configured in the config file. This enables you to throttle the same service code differently over time or across deployment sites. The host can also programmatically configure throttling based on some run-time decisions.

Figure 14 shows how to configure throttling in the host config file. Using the behaviorConfiguration tag you add to your service a custom behavior that sets throttled values.

Figure 14 Administrative Throttling (Close Figure 14)

<system.serviceModel>
    <services>
        <service type = "MyService"
                 behaviorConfiguration = "ThrottledBehavior">
        ...
        </service>
    </services>
    <behaviors>
        <behavior name="ThrottledBehavior">
            <throttling
               maxConcurrentCalls = "12"
               maxConnections = "34"
               maxInstances = "56"
            />
        </behavior>
    </behaviors>
</system.serviceModel>

The host process can programmatically throttle the service based on some run-time parameters. You can only do so before the host is opened. Although the host can override the throttling behavior found in the config file by removing the configuration and adding its own, you typically should provide a programmatic throttling behavior only when there is no throttling behavior in the config file.

The ServiceHostBase class offers the Description property of the type ServiceDescription:

public class ServiceHostBase : ...
{
    public virtual ServiceDescription Description {get;}
    ... // More members
}

As its name implies, the service description is the description of the service with all its aspects and behaviors. ServiceDescription contains a property called Behaviors of the type KeyedByTypeCollection<I> with IServiceBehavior as the generic parameter:

public class KeyedByTypeCollection<I> : KeyedCollection<Type,I>
{
    public T Find<T>();
    public T Remove<T>();
    ... // More members
}

public class ServiceDescription
{
    public KeyedByTypeCollection<IServiceBehavior> Behaviors {get;}
}

IServiceBehavior is the interface that all behavior classes and attributes implement. KeyedByTypeCollection<I> offers the generic method Find<T>, which returns the requested behavior if it is in the collection and null otherwise. A given behavior type can only be found once in the collection at most.

Figure 15 shows how to set the throttled behavior programmatically. First the hosting code verifies that no service throttling behavior is provided in the config file. This is done by calling the Find<T> method of KeyedByTypeCollection<I> using ServiceThrottlingBehavior as the type parameter. ServiceThrottlingBehavior is defined in the System.ServiceModel.Design namespace:

public class ServiceThrottlingBehavior : IServiceBehavior
{
    public int MaxConcurrentCalls {get;set;}
    public int MaxConnections {get;set;}
    public int MaxInstances {get;set;}
    ... // More members
}

Figure 15 Programmatic Throttling (Close Figure 15)

ServiceHost host = new ServiceHost(typeof(MyService));

ServiceThrottlingBehavior throttle;
throttle = host.Description.Behaviors.Find<ServiceThrottlingBehavior>();

if(throttle == null)
{
    throttle = new ServiceThrottlingBehavior();
    throttle.MaxConcurrentCalls = 12;
    throttle.MaxConnections  = 34;
    throttle.MaxInstances   = 56;
    host.Description.Behaviors.Add(throttle);
}

host.Open();

If the returned throttle is null, the hosting code creates a new ServiceThrottlingBehavior, sets its values, and adds it to the behaviors in the service description.

Reading Throttled Values

The throttled values can be read at run time by service developers for diagnostics and analysis purposes. The service instance can access at run time its throttled properties from its dispatcher. First, obtain a reference to the host from the operation context. The host base class ServiceHostBase offers the read-only ChannelDispatchers property—a strongly typed collection of ChannelDispatcherBase objects. Each item in the collection is of the type ChannelDispatcher. ChannelDispatcher offers the property ServiceThrottle which contains the configured throttled values (see Figure 16).

Figure 16 Reading the Throttled Values (Close Figure 16)

class MyService : ...
{
   public void MyMethod() //Contract operation
   {
      ChannelDispatcher dispatcher = OperationContext.Current.Host.
          ChannelDispatchers[0] as ChannelDispatcher;

      ServiceThrottle serviceThrottle = dispatcher.ServiceThrottle;

      Trace.WriteLine("MaxConcurrentCalls = " +
          serviceThrottle.MaxConcurrentCalls);
      Trace.WriteLine("MaxConnections = " +
          serviceThrottle.MaxConnections);
      Trace.WriteLine("MaxInstances = " + serviceThrottle.MaxInstances);
   }
}

Note that the service can only read the throttled values, and has no way of affecting them. If the service tries to set throttled values it will get an InvalidOperationException.

Conclusion

While every application is unique when it comes to scalability, performance, and throughput, Windows Communication Foundation does offer canonical instance management techniques that are applicable across the range of applications, thus enabling a wide variety of scenarios and programming models. Understanding Windows Communication Foundation instance management and choosing the right activation mode is critical.Yet applying each mode is surprisingly easy to accomplish. In addition, you can augment the straightforward modes with demarcating operations and instance deactivation.

Juval Lowy is a software architect with IDesign providing Windows Communication Foundation training and architecture consulting. Juval is currently working on a comprehensive Windows Communication Foundation book. He is also the Microsoft Regional Director for the Silicon Valley. Contact Juval at www.idesign.net.


Configure This: Parameterize Your Apps Using XML Configuration In The .NET Framework 2.0

Configure This

Parameterize Your Apps Using XML Configuration In The .NET Framework 2.0

Bryan Porter
This article discusses:
  • Configuration with the .NET Framework 1.x
  • Configuration with the .NET Framework 2.0
  • Implementing custom configuration sections, collections, elements, and groups
  • Creating custom edit controls
This article uses the following technologies:
Visual Basic, .NET Framework 2.0
Code download available at: Configuration2006_06.exe (187KB)
Contents
The Basics
Collections
Putting It to Use
Creating the Edit Controls
Final Details
Conclusion

A
pplications are complicated. Even the simplest software invariably contains numerous settings that have to store some sort of value in order to function at even a basic level. Database-driven applications usually have the luxury of storing the lion's share of their configuration information in a database of some sort, but there are always settings and options whose values can't be stored there (such as the location of the database itself) or are inconsequential to system reliability but absolutely necessary to preserve the user experience.

I've done my fair share of work on projects as a "clean-up" man, where the functional specifications had all been met, but there was a growing dissatisfaction on the part of the end users. Management couldn't understand what the problem was—the application did all that had been requested! Why weren't the users satisfied? After doing some end-user interviews, I discovered that most of the complaints concerned all the little ancillary things—like windows not opening at the same location on the screen each time, or button and menu shortcuts being inconsistent. All these little things the original engineers hadn't thought about gave rise to unhappy users who felt that the app was broken. In fact, it wasn't broken at all; it was just poorly configured.

Configuration in the Microsoft® .NET Framework 1.x was a great boon for developers. The structured XML configuration file was easy to work with from a developmental standpoint, and it freed developers using .NET from having to store configuration information in the Windows® Registry. Using the XML configuration file, xcopy deployment was finally a reasonable option for many applications. That's not to say that the configuration system in the .NET Framework 1.x was without its own set of problems. Lack of an API for editing the configuration file and the unstructured nature of custom configuration sections all gave rise to inconsistencies in application configuration and management.

Thankfully, those idiosyncrasies have been addressed in the .NET Framework 2.0. A new set of classes now give you read/write access to your application's configuration file and allow you to house structured, strongly typed and validated configuration data. For any developer who spent much time with .NET Framework 1.x configuration files and IConfigurationSectionHandler, this is a welcome change.

In this article, I'll discuss different ways to configure an application in the .NET Framework 2.0, exploring the classes of the newly revamped System.Configuration namespace. Along the way I'll show you how, with just a little bit of planning, you can construct a dialog box to automatically manage the configuration settings of your application.

The Basics

Deciding what configuration information to store, and how to store it, is a big step in determining the best way to architect configuration for your application. Take the case of storing the majority of user-specific configuration data in the database of a database-driven application. This can be a good decision for some usage scenarios, since any change users might make to configuration values will follow them around to any machines on which they run that application. The flip side, of course, is that you've now got a ton of extra information in your database that isn't particularly pertinent to the operation of your application.

Alternatively, you can use the XML configuration file. Visual Studio® 2005 introduces support for simplifying this process, complete with an auto-generated object model to make accessing simple configuration values a snap. If you double-click on the My Project folder in Solutions Explorer (Properties for a C# project), you'll open the properties document for your project. Select the Settings tab and choose User as the configuration scope for any settings you want to add. Once you've added User-scoped settings to your project, you can access these settings through the My.Settings property (Properties.Settings.Default in C#) at run time. After creating a few different settings, open the app.config file in Visual Studio, and you'll see something similar to Figure 1.

Figure 1 App.config User Settings (Close Figure 1)
<configuration>
  <configSections>
    <sectionGroup name="userSettings"
        type="System.Configuration.UserSettingsGroup, System,
              Version=2.0.0.0, Culture=neutral,
              PublicKeyToken=b77a5c561934e089">
      <section name="MSDNSampleSettings.My.MySettings"
        type="System.Configuration.ClientSettingsSection, System,
             Version=2.0.0.0, Culture=neutral,
             PublicKeyToken=b77a5c561934e089"
        allowExeDefinition="MachineToLocalUser"
        requirePermission="false" />
    </sectionGroup>
  </configSections>
  ...
  <userSettings>
    <MSDNSampleSettings.My.MySettings>
      <setting name="Setting" serializeAs="String">
        <value>SomeDefaultValue</value>
      </setting>
    </MSDNSampleSettings.My.MySettings>
  </userSettings>
</configuration>

The exact XML will be different depending on what settings you've specified, but the general structure will be the same. The great thing about the settings designer in Visual Studio 2005 is that all the features of the configuration system that it takes advantage of to produce the configuration file can be used by the developer to do the same. This allows you to define configuration elements structured just like the .NET Framework assemblies. In fact, if you open the machine.config file for your workstation, you'll see a very similar structure.

The Visual Studio designer uses implementations of ConfigurationSection and ConfigurationSectionGroup to perform this magic. While the built-in Visual Studio designer can handle many of your configuration needs, eventually you may run into a situation where your configuration data is a little more complicated than a simple name-value combination. It's in these situations that the new .NET Framework configuration model really shines.

A configuration section, like the MSDNSampleSettings.My.MySettings section in Figure 1, is created by defining a class that inherits from ConfigurationSection, defined in the System.Configuration assembly. (New for the .NET Framework 2.0, the System.Configuration namespace now spans both the System and System.Configuration assemblies.) The attributes and their corresponding values for a given configuration section element are mapped to the properties of a ConfigurationSection implementation through the ConfigurationProperty attribute. The code in Figure 2 shows a sample implementation of a custom configuration section.

Figure 2 ConfigurationSection (Close Figure 2)
Imports System.Configuration
Imports System.ComponentModel

Public Class CustomConfigurationSection
    Inherits System.Configuration.ConfigurationSection

    Public Sub New()
        MyBase.New()
    End Sub

    <ConfigurationProperty("testString", _
     IsRequired:=True, IsKey:=False, _
     DefaultValue:="AStringMoreThanTenLong"), _
     StringValidator(InvalidCharacters:="xyz", _
         MaxLength:=50, MinLength:=10)> _
    Public Property TestString() As String
        Get
            Return CType(Me("testString"), String)
        End Get
        Set(ByVal value As String)
            Me("testString") = value
        End Set
    End Property

    <ConfigurationProperty("testInt", IsRequired:=True, IsKey:=False, _
     DefaultValue:=10), _
     IntegerValidator(ExcludeRange:=False, _
     MaxValue:=500, MinValue:=10)>
    Public Property TestNumber() As Integer
        Get
            Return CType(Me("testInt"), Integer)
        End Get
        Set(ByVal value As Integer)
            Me("testInt") = value
        End Set
    End Property

    <ConfigurationProperty("testBool", IsRequired:=True, IsKey:=False, _
     DefaultValue:=False)> _
    Public Property TestBool() As Boolean
        Get
            Return CType(Me("testBool"), Boolean)
        End Get
        Set(ByVal value As Boolean)
            Me("testBool") = value
        End Set
    End Property
End Class

Pay special notice to the application of the ConfigurationProperty attribute as shown in the following:

<ConfigurationProperty("testString", IsRequired:=True, IsKey:=False, _
    DefaultValue:="AStringMoreThanTenLong"), _
    StringValidator(InvalidCharacters:="xyz", _
        MaxLength:=50, MinLength:=10)> _
    Public Property TestString() As String
The ConfigurationProperty attribute takes a number of different arguments in its constructor, and the one you should be most interested in is the first argument, which defines the corresponding attribute name for this custom implementation's corresponding XML node. The name property maps your class's property to the property name in an internal property collection provided by the base class. My class simply takes whatever value is passed and inserts it into this internal property collection. The .NET Framework maps class property metadata to corresponding XML attribute properties in the app.config file. Additionally, the implementation here uses a ConfigurationValidator to specify that the value of this property cannot contain characters x, y, or z, must be no longer than 50 characters, and must be at least 10 characters long. I'll talk a bit more about ConfigurationValidators later on.

An example of an XML element that corresponds to this custom configuration section implementation in Figure 2 is:

<sampleCustomConfig testString="testtesttesttest" testInt="250"
 testBool="false" />
If you were to type the code from Figure 2 into your Visual Studio project and add the XML fragment just shown to your app.config, you'd quickly find your application has no access to the defined element. Even though you've constructed a perfectly valid ConfigurationSection implementation, you haven't yet instructed the .NET Framework configuration system to use this new configuration information. Because of this, the .NET Framework has no idea what the sampleCustomConfig element means.

For the .NET Framework configuration system to make use of your custom configuration implementation, you must first define it in the <configSections> element of your app.config file. For instance:

<configSections>
    <section name="sampleCustomConfig"
        type="MSDN.Configuration.Demo.CustomConfigurationSection,
              MSDN.Configuration.Demo" />
</configSections>
This tells the .NET Framework configuration system that while parsing your app.config file, whenever it encounters an XML node named sampleCustomConfig, it should map the attributes and values of that element according to the rules specified by the property attributes on the CustomConfigurationSection class.

Referring back to the code in Figure 2, notice that some of the properties I've defined and marked with a ConfigurationProperty attribute have a *Validator attribute applied as well. These attributes allow you to validate the data that gets populated in the properties of your class. A number of configuration validators ship with the .NET Framework, such as ones for validating based on regular expressions, strings, integers, and more. For a full list, see the MSDN® documentation on ConfigurationValidatorBase, ConfigurationValidatorAttribute, and the types that derive from each.

While the mapping and validation might seem straightforward, there is a lot more happening behind the scenes. Exactly how does the .NET Framework convert the string representation of your configuration value to the correct type for storage in your ConfigurationSection derived class? Additionally, is this conversion mechanism only available for standard .NET types or could you use your own types? For instance, take the TestNumber property used in the sample. This property gets or sets an Integer value, but the XML config file stores that integer as plain text, between quotes. How is this conversion performed automatically?

There are actually two answers to these questions: IConvertible and TypeConverter. IConvertible defines a number of methods for converting a type between the different standard .NET types, and is implemented by the standard .NET value types (Int32, String, and so on). Alternatively, an implementation of a custom TypeConverter for a particular class would allow you to convert a string representation of a value to an entirely custom value.

Suppose I wanted to return a type from one of my configuration section's property classes that had no TypeConverter defined for converting itself to a string representation. At run time, I'd get an exception since the .NET Framework wouldn't know how to do the conversion. In that instance, I'd have to define a TypeConverter. Defining TypeConverters is beyond the scope of this article, but for more information, see the MSDN documentation regarding the IConvertible interface, the TypeDescriptor class, and the TypeConverter class and its associated attribute.

Now, configuration sections standing on their own are plenty useful, but for organizational purposes you might want to group them together. Enter ConfigurationSectionGroup—this class allows you to define a grouping tag to keep your configuration elements ordered logically.

To specify a grouping tag to be used by your configuration section, modify the <configSections> entry in your app.config file as shown in the following snippet:

<configSections>
    <sectionGroup name="sampleSectionGroup">
        <section name="sampleCustomConfig"
         type="MSDN.Configuration.Demo.CustomConfigurationSection,
         MSDN.Configuration.Demo" />
    </sectionGroup>
</configSections>
Then, modify your app.config file to nest the <sampleCustomConfig> element defined earlier to appear inside a matching set of <sampleSectionGroup> tags:
<sampleSectionGroup>
    <sampleCustomConfig testString="testtesttesttest"
     testInt="250"
     testBool="false" />
</sectionGroup>
You can leave off the type specification for a section group in your <configSections> tag, and the .NET Framework will use the default implementation of ConfigurationSectionGroup (which is suitable for most purposes). But I recommend that instead of relying on this default behavior, you go ahead and define a custom ConfigurationSectionGroup-derived class, even if you don't provide any additional functionality in it. By so doing, you can easily augment the base functionality of the ConfigurationSectionGroup class in the future and provide additional details as to the contents of your ConfigurationSectionGroup (something I will take advantage of shortly).

An implementation of a custom configuration section group is exceedingly trivial. Simply define a class that inherits from ConfigurationSectionGroup and whose constructor calls down to the base class constructor. Once that's done, the associated entry in the <configSections> tag looks like this:

<configSections>
    <sectionGroup name="sampleSectionGroup"
     type="MSDN.Configuration.Demo.CustomConfigurationSectionGroup,
     MSDN.Configuration.Demo">
        ...
    </sectionGroup>
</configSections>

Collections

There is one more facet of the configuration system I've yet to discuss: collections. Frequently, you'll encounter configuration sections that use <add>, <remove> or <clear/> elements in the XML element body. Internally, these elements are actually populating a ConfigurationElementCollection property on the ConfigurationSection. Defining one is relatively easy, using skills I've already discussed. Simply derive two new classes, one each from ConfigurationElement and ConfigurationElementCollection. Your ConfigurationElement-derived class will become the class that's instantiated for each <add> element in your XML file. The ConfigurationElementCollection class has two key methods you'll need to override: CreateNewElement and GetElementKey.

When you are defining your properties in the ConfigurationElement-derived class, it's important to specify at least one property as that element's key. Without this specification, the configuration system won't be able to sort the elements in your ConfigurationElementCollection-derived class. Figure 3 contains the code detailing a sample implementation of ConfigurationElement and ConfigurationElementCollection.

Figure 3 Defining a Collection (Close Figure 3)

CustomConfigurationElement.vb
Imports System.Configuration
Imports System.ComponentModel

Public Class CustomConfigurationElement
    Inherits System.Configuration.ConfigurationElement

    <ConfigurationProperty("testString", DefaultValue:="", IsKey:=True, _
     IsRequired:=True) > _
    Public Property TestElementString() As String
        Get
            Return Me("testString").ToString()
        End Get
        Set(ByVal value As String)
            Me("testString") = value
        End Set
    End Property
End Class
CustomConfigurationElementCollection.vb
Imports System.Configuration

Public Class CustomConfigurationElementCollection
    Inherits System.Configuration.ConfigurationElementCollection

    Protected Overloads Overrides Function CreateNewElement() _
            As ConfigurationElement
        Dim configurationElement As New CustomConfigurationElement()
        Return configurationElement
    End Function

    Protected Overrides Function GetElementKey( _
            ByVal element As ConfigurationElement) As Object
        If (TypeOf (element) Is CustomConfigurationElement) Then
            Dim configElement As CustomConfigurationElement = _
                CType(element, CustomConfigurationElement)
            Return configElement.TestElementString
        End If

        Throw New ArgumentException( _
            "The specified element is not of the correct type.")
    End Function
End Class

To add this collection to the previously defined CustomConfigurationSection class, add the following to the code shown in Figure 2:

<ConfigurationProperty("namedCollection", IsDefaultCollection:=False, _
 IsRequired:=False)> _
Public ReadOnly Property ElementCollection() As _
        CustomConfigurationElementCollection
    Get
        Return CType(Me("namedCollection"), _
            CustomConfigurationElementCollection)
    End Get
End Property
Alternatively, you can replace the namedCollection property name with empty quotes and set IsDefaultCollection equal to true; this has the effect of establishing a default collection for the ConfigurationSection, allowing you to write XML similar to the following snippet:
<sampleSectionGroup>
    <sampleCustomConfig testString="testtesttesttest" testInt="250">
        <add testString="someString" />
        <add testString="someOtherString" />
    </sampleCustomConfig>
</sampleSectionGroup>
To get a better understanding of the inheritance relationship of these custom configuration classes and their .NET Framework base classes, take a look at Figure 4. Notice that ConfigurationSection and ConfigurationElementCollection share a common base class, ConfigurationElement. You can take advantage of this inheritance hierarchy when constructing ConfigurationElementCollection and ConfigurationElement editor controls later on.

Figure 4 Custom Configuration Classes and .NET Framework Base Classes
Figure 4 Custom Configuration Classes and .NET Framework Base Classes

I haven't yet discussed how to actually access the configuration information stored using our new, structured ConfigurationSection derived classes. Information stored in the configuration file is accessed through the Configuration class, which is returned by the ConfigurationManager class and its OpenExeConfiguration method. The OpenExeConfiguration method takes a value from the ConfigurationUserLevel enumeration as an argument, to indicate which of the configuration files you actually want to open: the configuration file that applies to all users, the configuration file only applying to roaming users for the current user, or the configuration that applies to the current user. For more information on the OpenExeConfiguration method, and the ConfigurationManager class in general, see the MSDN documentation.

Once you've obtained a reference to the current application's Configuration object, you can access any sections not nested inside ConfigurationSectionGroup through the Configuration object's Sections property. Additionally, there is a SectionGroups property on the Configuration class that returns a collection of the defined ConfigurationSectionGroups. You can access both of these by either name or ordinal:

Dim mySection as CustomConfigurationSection = _
    CType(config.Sections("sampleCustomConfig"), _
          CustomConfigurationSection)
...
Dim mySectionGroup as CustomConfigurationSectionGroup = _
    CType(config.SectionGroups("sampleSectionGroup"), _
          CustomConfigurationSectionGroup)
One thing you are sure to notice when poking around the Configuration object for your application is that there are a lot of sections and section groups returned by the object that aren't in your app.config file. These values are actually contained in a parent configuration file. For the examples shown here, your configuration file inherits numerous default values from the system's machine.config file.

Putting It to Use

Having a well-structured configuration system is all well and good, but do you really want your users wading through XML in order to change a setting in your application? Absolutely not! While you could certainly hardcode a dialog that could manipulate your custom ConfigurationSection-derived classes, this would be tedious and potentially error-prone. Instead, I'll show you how to build a Windows Forms dialog, similar to the one shown in Figure 5, that can handle modifying the configuration for you.

Figure 5 Editing the Custom Configuration
Figure 5 Editing the Custom Configuration

Start out by creating a new Windows Forms form and place a SplitContainer on it. In the left-hand pane of the split container, drop a TreeView control for a normal, Explorer-like set up. Below the SplitContainer, add two buttons, OK and Cancel, and drop a Checkbox control aligned with the buttons on the left-hand side of the form. The TreeView control will show the "outline" of your configuration file, and you can use the TreeNode's tag property to house the individual ConfigurationSection, ConfigurationSectionGroup, and ConfigurationElementCollections your dialog will be set up to edit. The Checkbox control will be used to enable and disable the display of system-defined configuration sections.

There are five methods that need closer attention: RefreshTree, GetSectionNodes, GetSectionGroupNodes, ShouldIncludeSection, and ShouldIncludeSectionGroup (see Figure 6). GetSectionNodes and GetSectionGroupNodes recurse through the hierarchy of ConfigurationSections and ConfigurationSectionGroups accessible through the Configuration class. Since a ConfigurationSectionGroup can hold nested ConfigurationSectionGroups, you have to handle any recursive scenarios that might arise. The GetSectionNodes and GetSectionGroupNodes methods both return an array of TreeNodes for easy insertion into the TreeView.

Figure 6 Building the Tree Structure of the Configuration File (Close Figure 6)
Private Sub RefreshTree()
    treeSections.Nodes.Clear()
    Dim systemNode As New TreeNode("System Configuration")

    systemNode.Nodes.AddRange(GetSectionNodes())
    treeSections.Nodes.AddRange(GetSectionGroupNodes(Nothing))

    If (systemNode.Nodes.Count > 0) Then
        treeSections.Nodes.Add(systemNode)
    End If
End Sub

Private Function GetSectionNodes(ByRef config As _
        System.Configuration.Configuration) As TreeNode()
    Dim nodes As New List(Of TreeNode)

    For Each section As ConfigurationSection In config.Sections
        If (ShouldIncludeSection(section)) Then
            Dim sectionNode As New TreeNode(GetSectionName(section))
            sectionNode.Tag = section

            BuildSectionNode(section, sectionNode)

            nodes.Add(sectionNode)
        End If
    Next

    Return nodes.ToArray()
End Function

Private Function GetSectionGroupNodes(ByRef group _
    As ConfigurationSectionGroup) As TreeNode()
    If (group Is Nothing) Then
        Dim nodes As New List(Of TreeNode)

        For Each childGroup As ConfigurationSectionGroup _
            In config.SectionGroups

            If (ShouldIncludeSectionGroup(childGroup)) Then
                nodes.AddRange(GetSectionGroupNodes(childGroup))
            End If
        Next

        Return nodes.ToArray()
    Else
        Dim node As New TreeNode(GetSectionGroupName(group))

        For Each childGroup As ConfigurationSectionGroup _
            In group.SectionGroups

            If (ShouldIncludeSectionGroup(childGroup)) Then
                node.Nodes.AddRange(GetSectionGroupNodes(childGroup))
            End If
        Next

        For Each section As ConfigurationSection In group.Sections
            If (ShouldIncludeSection(section)) Then
                Dim childNode As New TreeNode(GetSectionName(section))
                childNode.Tag = section

                BuildSectionNode(section, childNode)

                node.Nodes.Add(childNode)
            End If
        Next

        Return New TreeNode() {node}
    End If
End Function

Private Function ShouldIncludeSection(ByRef section As _
        ConfigurationSection) As Boolean
    Dim sectionType As Type = section.GetType()
    If (hideSystemConfig.Checked And _
        sectionType.Assembly.FullName.StartsWith(«System»)) Then
        Return False
    End If

    Return True
End Function

Private Function ShouldIncludeSectionGroup(ByRef sectionGroup As _
    ConfigurationSectionGroup) As Boolean
    Dim groupType As Type = sectionGroup.GetType()

    If (hideSystemConfig.Checked _
        And groupType.Assembly.FullName.StartsWith(«System») _
            And Not groupType.FullName.Contains(_
            «.ConfigurationSectionGroup»)) Then
        Return False
    End If

    Return True
End Function

The RefreshTree method simply starts the whole process by first ensuring that the TreeView is cleared and constructing a parent node to handle ConfigurationSections that aren't parented by a ConfigurationSectionGroup. The ShouldIncludeSection and ShouldIncludeSectionGroup contain some filtering logic to separate out ConfigurationSections and ConfigurationSectionGroups that are defined in machine.config. Use these in combination with the checkbox control to optionally display or hide system-defined configuration elements.

Since sections can appear outside of section groups and section groups can be nested, you need two different methods to enumerate sections and section groups. When GetSectionGroupNodes is called the first time (from RefreshTree), Nothing (null in C#) is passed as the argument. The method checks this argument, and if it's Nothing, you enumerate through the root section groups returned by the Configuration objects SectionGroups property.

You then call GetSectionGroupNodes again for each of the ConfigurationSectionGroups returned. Since successive calls to GetSectionGroupNodes pass a reference to a ConfigurationSectionGroup object (instead of Nothing) as the method argument, a second code path is executed that enumerates the ConfigurationSections and ConfigurationSectionGroups of the passed ConfigurationSectionGroup object. A TreeNode is constructed for each ConfigurationSectionGroup and ConfigurationSection reference found inside the passed ConfigurationSectionGroup object, and each tree node has its corresponding ConfigurationSectionGroup or ConfigurationSection reference stored in the nodes Tag property. This way, when you wire up events to handle TreeNode selection, you immediately have access to the selected nodes associated ConfigurationSection or ConfigurationSectionGroup.

Filtering out sections defined by the framework is as simple as calling RefreshTree from within the CheckedChanged event on the forms checkbox. For production applications, you'll probably want to filter out system-defined configuration sections at all times, but during development it might be handy to enable that functionality, if only to see what else you've got defined in your app.config file.

One problem with enumerating the different sections and section groups in this manner is one of usability: the names of the different sections or section groups make sense in the context of your XML configuration file, but most users won't understand what they mean. You'll need to provide a way to add descriptive information to the ConfigurationSection and ConfigurationSectionGroup-derived classes. Luckily, the .NET Framework has done much of the work already.

The Description attribute class resides in the System.ComponentModel namespace and is useful for attaching simple descriptive strings to class elements. By modifying your custom configuration section and applying the Description attribute, you can provide a descriptive name for your configuration sections and groups that are a little more user friendly than the values provided by SectionName property of the ConfigurationSections or the SectionGroupName property of the ConfigurationSectionGroups. The resulting class declaration now looks like this:

<Description("Custom Configuration Section")> _
Public Class CustomConfigurationSection
    Inherits System.Configuration.ConfigurationSection
At run time, you can now read the metadata of the class definition to provide a superior user experience. This same problem occurs with the properties defined in the custom configuration section, but by applying the Description attribute to those properties, you can present a nicely descriptive user interface to the end user. What's more, the PropertyInformation objects (describing the configuration properties handled by your ConfigurationSection-derived class) retrieved from a configuration section's ElementInformation.Properties collection, will populate their Description property with whatever text you specify in the Description attribute applied to a particular property in the class definition. For instance, you can access the properties description attribute that has been defined in this manner:
<ConfigurationProperty("testString", IsRequired:=True, _
    IsKey:=False, DefaultValue:="AStringMoreThanTenLong"), _
    StringValidator(InvalidCharacters:="xyz", MaxLength:=50, _
    MinLength:=10), _
 Description("A Test String")> _
Public Property TestString() As String
To access the properties, you'd use code as simple as this (assuming mySection is a reference to your ConfigurationSection):
mySection.ElementInformation.Properties("testProp").Description
The rest of the configuration system doesn't display PropertyInformation's penchant for Description attributes, so I wrote up a few different private helper methods to grab the Description attribute for sections or section groups. Figure 7 contains the code for three methods, GetSectionName, GetSectionGroupName, and GetFriendlyName. GetSectionName and GetSectionGroupName call GetFriendlyName, passing in a reference to the Type object that defines the ConfigurationSection or ConfigurationSectionGroup. GetFriendlyName then inspects the passed Type object for any Description attributes that might be applied to it and returns the string contained in the first Description attribute it encounters, or returns Nothing if it found none.

Figure 7 Getting Friendly Names (Close Figure 7)
Private Function GetSectionName(_
    ByRef section As ConfigurationSection) As String

    Dim sectionName As String = Nothing

    sectionName = GetFriendlyName(section.GetType())

    If (sectionName IsNot Nothing) Then
        Return sectionName
    Else
        Return section.SectionInformation.SectionName
    End If
End Function

Private Function GetSectionGroupName(ByRef group As _
        ConfigurationSectionGroup) As String

    Dim groupName As String = Nothing

    groupName = GetFriendlyName(group.GetType())

    If (groupName IsNot Nothing) Then
        Return groupName
    Else
        Return group.SectionGroupName
    End If
End Function

Private Function GetFriendlyName(ByRef classType As Type) As String
    Dim attributes As System.ComponentModel.DescriptionAttribute()
    attributes = classType.GetCustomAttributes( _
        GetType(System.ComponentModel.DescriptionAttribute), True)

    If (attributes.Length > 0) Then
        Return attributes(0).Description
    End If

    Return Nothing
End Function

If GetFriendlyName returned Nothing, GetSectionName and GetSectionGroupName both return the values found in the SectionName or SectionGroupName, respectively. Otherwise, GetSectionName and GetSectionGroupName just return the value returned by GetFriendlyName. I use these when I'm populating the TreeView with the configuration outline.

But back to the task at hand, for each ConfigurationSection found in the configuration object, you call a private method named BuildSectionNode with the ConfigurationSection you're currently working with and the ConfigurationSection's corresponding TreeNode as arguments. BuildSectionNode inspects the configuration properties of the ConfigurationSection, looking for any properties that return a ConfigurationElementCollection-derived type. If one is found, it creates another node beneath the passed node with either the name/description of the collection property or Values (it needs some sort of name) if the property returns a default collection. You then set this new child node's Tag property to the ConfigurationElementCollection reference.

Now that you've parsed the configuration objects and populated the tree, its time to create edit controls.

Creating the Edit Controls

In creating the edit controls, I had two major goals: to provide validation by respecting the ConfigurationValidator attributes applied to the exposed properties of the ConfigurationElements and ConfigurationSections, and to provide collection editing support. The first item is handled by a custom control derived from Panel, the second by a UserControl.

Editing an item is fairly straightforward. The general idea is to take a ConfigurationElement, enumerate the properties, and for each property create a label and corresponding textbox, storing the PropertyInformation reference that a given textbox is meant to edit in that textbox's Tag property. Then, wire up to the Enter, Leave, TextChanged, KeyUp, and Validating events of the textbox. During the validation event, manually fire the associated property validator. If it fails, raise an error on the textbox with a standard .NET error provider, setting the error text to be the message contained in the exception thrown by the property validator. The Enter, Leave, TextChanged and KeyUp events handle rolling back an edit made if the user presses the escape key while editing.

Laying out the different labels and textboxes before the release of the .NET Framework 2.0 was a bit of a challenge. Thankfully, the .NET Framework 2.0 introduces a number of new layout containers that ease this burden. For the purposes of the ConfigurationElement editor, I decided on the TableLayoutPanel, which lets you arrange controls in rows and columns, similar to HTML tables, and is most handy for making sure labels are appropriately aligned with the textboxes. Figure 8 shows the code used to set up this control.

Figure 8 Setting Up the Element Editor (Close Figure 8)
Private Sub ProcessElement(ByRef element As ConfigurationElement, _
        ByRef panel As TableLayoutPanel)

    Dim propertyLabel As Label = Nothing
    Dim propertyValue As TextBox = Nothing

    For Each propInfo As PropertyInformation In _
        element.ElementInformation.Properties

        If (propInfo.Type.IsSubclassOf( _
            GetType(ConfigurationElementCollection))) Then

            Continue For
        ElseIf (propInfo.Type.IsSubclassOf( _
            GetType(ConfigurationElement))) Then

            ProcessElement(CType(propInfo.Value, _
                ConfigurationElement), panel)
            Return
        End If

        propertyLabel = New Label()
        propertyValue = New TextBox()

        If (propInfo.Description <> String.Empty) Then
            propertyLabel.Text = propInfo.Description
        Else
            propertyLabel.Text = propInfo.Name
        End If

        propertyLabel.AutoSize = True
        propertyLabel.TextAlign = Drawing.ContentAlignment.MiddleLeft

        propertyValue.Anchor = AnchorStyles.Left Or _
            AnchorStyles.Right Or _
            AnchorStyles.Top

        If (propInfo.Value IsNot Nothing) Then
            propertyValue.Text = propInfo.Value.ToString()
        End If

        propertyValue.Tag = propInfo

        If (propInfo.IsLocked) Then
            propertyValue.ReadOnly = True
        End If

        _validator.SetIconPadding(propertyValue, -20)

        AddHandler propertyValue.Enter, New EventHandler(AddressOf  _
            propertyValue_Enter)
        AddHandler propertyValue.KeyUp, New KeyEventHandler(AddressOf  _
            propertyValue_KeyUp)
        AddHandler propertyValue.TextChanged, _
            New EventHandler(AddressOf  _
            propertyValue_TextChanged)
        AddHandler propertyValue.Validating, _
            New CancelEventHandler(AddressOf _
            propertyValue_Validating)

        panel.Controls.Add(propertyLabel, 0, _currentRow)
        panel.Controls.Add(propertyValue, 1, _currentRow)

        Dim width As Integer = propertyLabel.Width
        propertyLabel.AutoSize = False

        propertyLabel.Width = width
        propertyLabel.Height = propertyValue.Height

        propertyLabel = Nothing
        propertyValue = Nothing

        _currentRow = _currentRow + 1
    Next

    _currentRow = 0
End Sub

Validating the input values for a given property can be tricky, since you must first take an input value and convert it into a type supported by the PropertyInformation object's declared ConfigurationValidator—just passing a string instance won't do. Luckily, the PropertyInformation class exposes a Converter property that returns a TypeConverter for the type that the PropertyInformation object can hold. Unfortunately, you can't be certain that every PropertyInformation object you run across in the ConfigurationElement editor will know what TypeConverter to use; some types don't specify one, in which case you'll have to assume that the correct type to store the value in is just a string.

Another problem is one of type identity. You may have a Type object describing what type the PropertyInformation object can hold, but there may be no way of actually doing a cast at run time to the declared Type. Since you're doing this all in a late-bound fashion, you have to box the resulting value inside an Object and pass that to your validator's Validate method.

The validator for a given PropertyInformation object is returned by the aptly named Validator property. ConfigurationValidators expose two public methods that are relevant to this project: CanValidate and Validate. The CanValidate method takes a System.Type as its argument and returns a Boolean indicating whether it knows how to validate that type of object. The Validate method takes an Object as its argument and, unlike CanValidate, doesn't return a Boolean indicating the success or failure of the validation, but rather throws an exception should any problem in validating the supplied object arise. This behavior enables the Validate method to provide much greater detail as to why it can't validate the passed value—the value could be of the wrong type, for instance, or the passed value could be Nothing. A simple Boolean flag wouldn't really help in those types of situations, and it wouldn't allow you to provide descriptive feedback to your users.

Since a validator throws an exception, you have to wrap your call to Validate in a Try/Catch block and catch any exception that's thrown. This is actually quite handy, since you'll just be taking the contents of the Message property on the thrown exception, and using that as the error message to display to the end users with an ErrorProvider.

Now that you've got an ElementEditor control all set up to edit ConfigurationElements, its time to move on to editing collections, which is a bit trickier. Since you don't know what type of object a ConfigurationElementCollection can hold, only that it can hold an object derived from ConfigurationElement, how do you create a new element for a particular implementation of ConfigurationElementCollection?

The answer, as it turns out, is making a call to the CreateNewElement method on ConfiguationElementCollection. CreateNewElement is overridden by the implementers of a given ConfigurationElementCollection, and by convention should return a reference to a new ConfigurationElement object of the appropriate type. CreateNewElement is declared as Protected, meaning it's only available to the class in which it is defined and to any base classes. In order to call this method, you must use reflection to get a reference to the MethodInfo object that describes the method, then call the Invoke method on your MethodInfo reference to invoke the method on your ConfigurationElementCollection reference. At this point, you just pass the returned ConfigurationElement object to an instance of ElementEditor for editing purposes.

That done, you now have a new ConfigurationElement instance all set up and initialized that needs to be added to ConfigurationElementCollection. But how? Again, reflection is the answer.

ConfigurationElementCollection exposes a number of protected members that you'll need to access. In particular, you need the BaseAdd method for adding new items, the BaseIndexOf method for finding the ordinal of an item housed in the collection, and the BaseRemoveAt method for removing an item from the collection at a specific index. Since the control could potentially perform numerous add and remove operations on its target collection, I decided to initialize MethodInfo objects for these methods in the constructor of the control.

Invoking the various protected methods through the MethodInfo objects is straightforward. The MethodInfo class defines an Invoke method that takes two arguments: a reference to the ConfigurationElementCollection object that you want to invoke the method against and an Object array that holds the various arguments this method might accept. For instance, the call to CreateNewElement looks something like this:

Dim configElement As ConfigurationElement = CType( _
    createElement.Invoke(colInstance, Nothing), _
    ConfigurationElement)
In this case colInstance is the ConfigurationElementCollection instance and createElement is the MethodInfo instance for CreateNewElement. Since the MethodInfo Invoke method returns an Object—again, using reflection, the reflected MethodInfo can't be strongly typed—you must first cast this value to the desired type, ConfigurationElement.

It's a similar story for BaseAdd, BaseIndexOf, and BaseRemoveAt. BaseAdd and BaseRemoveAt return no value, so don't pay any attention to the returned value from the Invoke method. BaseIndexOf, however, does return a value, this time an Integer. A call to that method looks something like this:

Dim elementIndex As Integer = CInt( _
    getElementIndex.Invoke(colIntance, new Object(){myElement}))
myElement is the ConfigurationElement you're looking for, and getElementIndex is the MethodInfo instance for BaseIndexOf.

With the index of a specific element, you can then make a call to, for instance, BaseRemoveAt, which would remove the element at the specified index:

removeAtIndex.Invoke(colInstance, new Object(){elementIndex})
It's important to remember that all these hoops you're jumping through to call these methods are due to the fact that the methods themselves are declared Protected, and are thus inaccessible to your code without using reflection. If you had inherited from ConfigurationElementCollection and wanted to call these methods from within your derived class, you'd just call them directly.

As I said before, the collection editor control is a user control; this is primarily to simplify the construction of the control, since it's much easier to achieve the desired layout using the Windows Forms designer (especially when dealing with a composite control like the one I've constructed). The control itself consists of a ListView control and three buttons: Add, Edit, and Remove. The ListView control is populated by the items contained in the collection you're editing, and the ListView columns all correspond to the PropertyInformation objects found on the contained ConfigurationElements. Additionally, the Edit and Remove buttons should only be enabled when an item from the collection displayed in the ListView is selected. The finished control, hosted in the ConfigurationDialog, is shown in Figure 9.

Figure 9 Editing a ConfigurationElementCollection
Figure 9 Editing a ConfigurationElementCollection

One challenge in constructing this control was determining the headers to display. Since the ConfigurationElementCollection doesn't contain any info regarding the different PropertyInformation objects returned by the ConfigurationElements it contains, just grab one of the ConfigurationElements in the collection to enumerate the PropertyInformation objects. What if the ConfigurationElementCollection is empty? Make a call to CreateNewElement (through the MethodInfo object created in the control's constructor), enumerate the names (or descriptions) of the PropertyInformation objects of the returned ConfigurationElement, and use those names as the ListView headers (see Figure 10).

Figure 10 Determining List View Headers (Close Figure 10)
Private Sub ProcessElementCollection(ByRef elementCollection As _
        ConfigurationElementCollection, ByRef list As ListView)

    Dim propInfos() As PropertyInformation

    list.Clear()

    If (elementCollection.Count > 0) Then
        Dim elements(elementCollection.Count - 1) As ConfigurationElement
        elementCollection.CopyTo(elements, 0)

        ReDim propInfos(elements(0).ElementInformation.Properties.Count _
            - 1)

        elements(0).ElementInformation.Properties.CopyTo(propInfos, 0)
    Else
        Dim element As ConfigurationElement
        element = CType(_createNewElement.Invoke(_
            elementCollection, Nothing), ConfigurationElement)

        ReDim propInfos(element.ElementInformation.Properties.Count - 1)

        element.ElementInformation.Properties.CopyTo(propInfos, 0)
    End If

    For Each propInfo As PropertyInformation In propInfos
        Dim header As New ColumnHeader()
        Dim headerText As String

        If (propInfo.Description = String.Empty) Then
            headerText = propInfo.Name
        Else
            headerText = propInfo.Description
        End If

        header.Text = headerText
        list.Columns.Add(header)
    Next

    For Each subElement As ConfigurationElement In _
        CType(elementCollection, ConfigurationElementCollection)

        Dim subItem As ListViewItem = Nothing

        For Each propInfo As PropertyInformation In propInfos
            If (subItem Is Nothing) Then
                subItem = New ListViewItem( _
                    subElement.ElementInformation.Properties(_
                    propInfo.Name).Value.ToString())
            Else

                If (propInfo.Value IsNot Nothing) Then
                    subItem.SubItems.Add(_
                        subElement.ElementInformation.Properties( _
                        propInfo.Name).Value.ToString())
                Else
                    subItem.SubItems.Add("")
                End If
            End If
        Next

        If (subItem IsNot Nothing) Then
            subItem.Tag = subElement
            list.Items.Add(subItem)
        End If
    Next

    Dim headerWidth As Integer = list.Width / list.Columns.Count
    For Each header As ColumnHeader In list.Columns
        header.Width = headerWidth
    Next
End Sub

Removing a ConfigurationElement from a ConfigurationElementCollection by using the control is simple. But in order to edit a newly added ConfigurationElement, or edit an existing one, you need another dialog to pop up. To handle this, construct a new dialog that hosts a single instance of the previously created EditControl, which you'll use to pop up a separate window for editing a ConfigurationElement inside a ConfigurationElementCollection. You'll display this dialog from the event handler of the Add and Edit buttons click event.

Final Details

With the two edit controls constructed—one for individual ConfigurationSections or ConfigurationElements, and one for ConfigurationElementCollections—just drop them on the form created earlier, set their visible properties to false, and add some code to the AfterSelect event of the TreeView control to load each control with the selected ConfigurationSection or ConfigurationElementCollection and make the control visible. Figure 11 details all this action.

Figure 11 Displaying the Correct Control (Close Figure 11)
Private Sub treeSections_AfterSelect(ByVal sender As System.Object, _
    ByVal e As System.Windows.Forms.TreeViewEventArgs) _
    Handles treeSections.AfterSelect
    If (e.Node Is Nothing Or e.Node.Tag Is Nothing) Then
        Return
    End If

    If ((TypeOf (e.Node.Tag) Is ConfigurationSection Or _
        TypeOf (e.Node.Tag) Is ConfigurationElement) And Not _
        (TypeOf (e.Node.Tag) Is ConfigurationElementCollection)) Then

        _elementEditor.Visible = True
        _elementCollectionEditor.Visible = False

        _elementEditor.Element = e.Node.Tag
    ElseIf (TypeOf (e.Node.Tag) Is PropertyInformation) Then
        Dim propInfo As PropertyInformation = _
            CType(e.Node.Tag, PropertyInformation)

        If (TypeOf (propInfo.Value) Is _
            ConfigurationElementCollection) Then

            _elementEditor.Visible = False
            _elementCollectionEditor.Visible = True
            _elementCollectionEditor.ElementCollection = _
                CType(propInfo.Value, _
                ConfigurationElementCollection)
        End If
    End If
End Sub

But what about editing existing items and newly created items within a ConfigurationElementCollection? You need a separate place to host an ElementEditor control for these purposes. After debating a few different ways to achieve this, I decided on creating a separate dialog for editing new or existing items in a ConfigurationElementCollection. This dialog is launched whenever a user presses the Add or Edit buttons on the collection editor.

Saving the edited configuration is as simple as making a call to the Save method on the Configuration object stored in the main dialog. Adding this call to the main dialog's OK button takes care of that, and writes out all changes to the app.config file.

The example is relatively full-featured and should work well for most production environments. There are a few places where some great extensions could be worked out. For instance, instead of using TextBoxes to perform the PropertyInformation value editing, you could use a system similar to the .NET Framework PropertyGrid to provide smart type editing. Providing support for statically locking a given set of configuration properties (database connection strings, for example) would allow the dialog to display values that a user might need to read off to support personnel, but shouldn't change themselves.

Additionally, the dialog currently only supports editing collections nested one level beneath a ConfigurationSection; it's possible to construct a configuration system whereby each ConfigurationElement contained in a ConfigurationElementCollection also exposes a ConfigurationElementCollection. Adding the necessary support for a nested collection would also improve the utility of the dialog.

Conclusion

The .NET Framework provides great support for application configuration. This can be a real time-saver when it comes to changing an application to work for a user, as opposed to getting that user to work around your application. Simple things like default window positions and which toolbars are displayed can make a big difference in the acceptance of an application.

I've given you a whirlwind tour of the configuration system in the .NET Framework; there is certainly a lot more to cover in this robust system. I hope you've learned something in the process and that I've piqued your interest enough to delve deeper into application configuration under the .NET Framework 2.0.

Bryan Porter is the Chief Software Architect for Software Plus, where he oversees the various software engineering teams that drive Software Plus's business process management software. Additionally, he drove the design and architecture of Monocle, Software Plus's Software Asset Management solution for distributed organizations.


Class To Contract: Enrich Your XML Serialization With Schema Providers In The .NET Framework

Class To Contract

Enrich Your XML Serialization With Schema Providers In The .NET Framework

Keith Pijanowski
This article discusses:
  • The .NET Framework attribution model
  • Converting between object instances and XML
  • Converting between schemas and classes
  • Customizing XML serialization and deserialization
This article uses the following technologies:
.NET Framework 2.0, XML
Code download available at: SchemaProviders2006_06.exe (191KB)
Contents
The .NET Attribution Model
Objects and XML Documents
Schema Providers
Custom Serialization and Deserialization
Web Services
Conclusion

M
apping classes to schemas and serializing objects to XML documents leads to interoperability. Consider the relationship that exists among classes, schemas, objects, and XML documents (as illustrated in Figure 1). Classes are constructs that describe a software system's data in a proprietary way, as opposed to schemas which allow data to be described using a proven standard. By using a standard, schemas allow for interoperability between systems, platforms, and organizations. When working with classes, you might not consider that the data being described may some day be visible to another system if a Web service is implemented within your system. By mapping classes to schemas, data that was originally meant for internal system use can be set free, traveling from system to system in the form of SOAP messages.

Figure 1 Mapping Schemas and Objects
Figure 1 Mapping Schemas and Objects

Unfortunately, the Microsoft® .NET Framework 1.x provides only minimal options for mapping classes to schemas and serializing objects to XML documents. As a result, when architects design service-oriented architectures with object-oriented technologies, the task of performing this sort of mapping offers quite a challenge. The .NET Framework 2.0, however, addresses the limitations with the introduction of Schema providers and the IXmlSerializable interface. These new tools put the mapping of class to schema and XML serialization/deserialization in the hands of developers and architects.

In this article, I'll show you how to use these tools and explain why they are important. In order to better understand the issues involved in mapping classes to schemas and objects to XML documents, I will review the current capabilities found in the .NET Framework 1.1, focusing on its architectural shortcomings. I'll then discuss Schema providers, elaborating on how they allow developers to programmatically create the XML Schema that will be used to represent a class. And then I will take a look at the added support for XML serialization and deserialization in the .NET Framework 2.0.

The .NET Attribution Model

Using the capabilities of the .NET Framework 1.1, it is possible to let the Framework itself handle most of the work needed to map classes to schemas and objects to XML documents. Both mapping and serialization are accomplished by using the attributes and utilities available in the System.Xml.Serialization namespace. This technique, which I will refer to as the .NET attribution model, provides a declarative programming model that is easy to use. However, it places limits on what can be put in both the schema and the document.

Consider this simple Product class:

namespace SimpleExample
{
   public class Product
   {
       public int ProductID;
       public string Name;
       public decimal ListPrice;
   }
}
If this needs to be used as the return type of a Web service or the type of a Web service input parameter, then there would have to be a way to map the .NET Framework type system into the XML Schema type system. The .NET Framework will do this mapping for you automatically if you let it. For example, say the Product class example is packaged into an assembly named BusinessEntities.dll and the Product class is in the SimpleExample namespace. If this is the case, then the schema shown in Figure 2 will be generated if I issue the following command from the Visual Studio command prompt:
xsd BusinessEntities.dll /type:SimpleExample.Product
Note that the xsd.exe command-line tool does not have an output parameter to allow for the specification of an output file. When you execute this command, the generated schema is placed in a file named schema0.xsd, located in the directory in which the command was executed.

Figure 2 Product Class as an XML Schema (Close Figure 2)
<?xml version="1.0" encoding="utf-8"?>
<xs:schema elementFormDefault="qualified"
    xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="Product" nillable="true" type="Product" />

    <xs:complexType name="Product">
        <xs:sequence>
            <xs:element minOccurs="1" maxOccurs="1"
             name="ProductID" type="xs:int" />
            <xs:element minOccurs="0" maxOccurs="1"
             name=»Name» type=»xs:string» />
            <xs:element minOccurs="1" maxOccurs="1"
             name="ListPrice" type=»xs:decimal» />
        </xs:sequence>
    </xs:complexType>

</xs:schema>

It is not a good idea to let the .NET Framework generate a schema from a class like the one shown in Figure 2. The problem with this schema is that no target namespace has been defined. The namespaces in XML Schemas serve the same purpose as namespaces in an object-oriented language—they help to ensure unique names for custom types. A schema's target namespace is the namespace in which all complex types defined within that schema will be placed.

Another issue with the schema in Figure 2 is that the root element declaration has the same name as its type: Product. This would be like declaring a variable with the same name as the variable's type, as shown here:

Product Product = new Product();
This would make your code difficult to understand. Likewise, it makes the schema difficult to comprehend.

Fortunately, the creators of the .NET Framework have provided a few tools in the form of .NET attributes to allow you to correct these problems. The following class is an improved version of the one shown previously:

[XmlType("product_type",Namespace =
    "http://Attribution.Example.org/Product.xsd")]
[XmlRoot(Namespace = "http://Attribution.Example.org/Product.xsd"
    ElementName = "product_root", IsNullable = false)]
public class Product
{
    [XmlElement("product_id")]
    public int ProductID;
    [XmlElement("name")]
    public string Name;
    [XmlElement("list_price")]
    public decimal ListPrice;
}
It uses the XmlType attribute to specify a target namespace for the complex type that will be used to represent the Product class in the resulting schema. This attribute also allows the developer to assign a type name that is different from the class name. In this example the target namespace is assigned a value of "http://Attribution.Example.org/Product.xsd" by using the Namespace property of the XmlType attribute, and the type name is assigned a value of product_type by using the first parameter of this attribute.

The XmlRoot attribute is used to assign a name to the element declaration that will represent the document root. In the new version of the Product class, this attribute is used to assign product_root as the name of the element that will represent the root in the XML document being described by the schema. This attribute also allows the developer to assign the root element to a namespace. This will typically be the target namespace specified through the XmlType attribute.

Finally, note the use of the XmlElement attribute. This attribute is used to explicitly state that the resulting schema should use element normal form for all the fields. This attribute also allows the developer to specify each field's element name (as opposed to each element being given the same name as the field itself). The schema generated for the new Product class is shown in Figure 3.

Figure 3 Attributed Product Class as an XML Schema (Close Figure 3)
<?xml version="1.0" encoding="utf-8"?>
<xs:schema
  targetNamespace=http://AttributionExample.xsd/Product.xsd
  xmlns:tns="http://AttributionExample.xsd/Product.xsd"
  elementFormDefault="qualified"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="product_root" type="tns:product_type" />

    <xs:complexType name="product_type">
       <xs:sequence>
           <xs:element minOccurs="1" maxOccurs="1"
               name=»product_id» type=»xs:int» />
           <xs:element minOccurs=»0» maxOccurs=»1»
               name=»name» type=»xs:string» />
           <xs:element minOccurs=»1» maxOccurs=»1»
               name=»list_price» type=»xs:decimal» />
        </xs:sequence>
    </xs:complexType>

</xs:schema>

Objects and XML Documents

XML serialization (and deserialization for that matter) is relatively painless with the .NET Framework 1.1. There are some disadvantages, though. Let's first look at the simplicity of performing such tasks in the .NET Framework 1.1.

The following code snippet instantiates the Product class, fills it with data, and serializes it:

Product p = new Product();
p.ProductID = 1234;
p.ListPrice = 500.00M;
p.Name = "Mountain Bike";

XmlSerializer ser = new XmlSerializer(p.GetType());

StringWriter s = new StringWriter();
ser.Serialize(s, p);

txtXML.Text = s.ToString();
(Note that this code is part of the download that accompanies this article.) The serialization code creates the XML document that is shown in the following snippet:
<?xml version="1.0" encoding="utf-16"?>
<product_root
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xmlns:xsd="http://www.w3.org/2001/XMLSchema"
 xmlns="http://AttributionExample.xsd/Product.xsd">
    <product_id>1234</product_id>
    <name>Mountain Bike</name>
    <list_price>500.00</list_price>
    <dealer_price>450.00</dealer_price>
</product_root>
Notice that to serialize an object to XML, there is no need for code that is specific to the object's type. The XmlSerializer class uses the information specified through the attributes discussed in the previous section to create an XML document that contains the information found in the object. Specifically, notice that the name of the root element and the names of the elements that represent the object's public fields conform to the names specified through attribution.

Since the .NET Framework generates both the schema and the XML document, you can safely assume that the document can be validated against the schema in Figure 3.

Deserializing an XML document into an object is just as easy as serializing an object to an XML document. The following code snippet will deserialize the previous XML document and produce a Product object:

StringReader sr = new StringReader(txtXML.Text);
Product p = new Product();
XmlSerializer ser = new XmlSerializer(p.GetType());
p = (Product) ser.Deserialize(sr);
From this example it is easy to see that the big advantage to using the .NET attribution model is ease of implementation. By utilizing just three attributes—XmlType, XmlRoot, and XmlElement—a class can be accurately mapped to a schema and the .NET Framework will take care of serialization and deserialization.

What you won't glean from this example, however, are the limitations placed on architects and developers as a result of these easy to use utilities. There are a few subtle annoyances with the attribution model. First, you cannot take full advantage of the power of XML Schemas. It would be nice, for example, to limit the name element to a specified number of characters. This capability is provided by the XML Schema standard, but there is no way to express this restriction using the .NET attribution model. The XML Schema standard also provides options such as pattern matching, numeric ranges, and enumerations that can be used for restricting values of simple data types. These restrictions are either not supported by the .NET attribution model or they are cumbersome to implement using attributes.

Another minor issue with .NET attribution is that you have no way to execute business logic during serialization and deserialization. If a complex object graph is being recreated during deserialization, you may want to temporarily turn off all validation logic during deserialization and then fire a validating event once you know that all objects are fully hydrated.

By far the biggest issue with the .NET attribution model is the fact that the data contract that the outside world sees is tightly coupled with the object-oriented accessibility options of the .NET Framework. Specifically, only read/write public properties and fields can be attributed using the attributes found in the System.Xml.Serialization namespace. When using the .NET attribution model, you cannot expose a private member variable (or any other non-public member) as part of your data contract. Additionally, read-only properties and fields are not supported with this model.

Schema Providers

Schema providers are a new tool for designing the data contracts of Web services. They provide complete control and flexibility for mapping class to schema. A schema provider, which is a static function specified by the XmlSchemaProvider attribute (a class-level attribute), is responsible for setting up a schema for the class in which it resides. Specifying a schema provider causes the default class-to-schema mapping to be overridden and places control directly in the hands of the developer. When a schema provider is specified, tools like xsd.exe will call into this static function when creating schemas or when looking for complex type definitions that will be used to represent the class to the outside world.

Specifying a custom schema necessitates custom serialization and deserialization. Therefore, the .NET Framework 2.0 now provides support for the IXmlSerializable interface.

With the flexibility offered by schema providers, you don't have to rely on .NET attribution. You can design the schema. And then, once you are satisfied with the design of the schema, you can then implement your class.

The schema in Figure 4 represents the redesigned data contract for the Product class. This schema is similar to the schema discussed earlier but has been given a few improvements that demonstrate the flexibility of schema providers. First, the name element is limited to 25 characters. Second, there is now an additional element declaration, dealer_price, to represent the dealer price.

Figure 4 Revised Product Schema (Close Figure 4)
<?xml version="1.0" encoding="utf-8"?>
<xs:schema xmlns:tns="http://SchemaProvider.Example.org/Product.xsd"
    elementFormDefault="qualified"
    targetNamespace=http://SchemaProvider.Example.org/Product.xsd
    xmlns:xs="http://www.w3.org/2001/XMLSchema">

    <xs:element name="product_root" type="tns:product_type" />

    <xs:complexType name="product_type">
        <xs:sequence>
            <xs:element name="product_id" type="xs:int" />
            <xs:element name="name" type="tns:NameType" />
            <xs:element name="list_price" type="xs:decimal" />
            <xs:element name="dealer_price" type="xs:decimal" />
        </xs:sequence>
    </xs:complexType>

    <xs:simpleType name="NameType">
        <xs:restriction base="xs:string">
            <xs:maxLength value="25" />
        </xs:restriction>
    </xs:simpleType>

</xs:schema>

Some improvements are also needed in the design of the Product class. ProductID is going to become a read-only property that is mapped to the product_id element. A new field, dealerPrice, is mapped to the new dealer_price element, and it is a private member variable that is not accessible to other objects within the system.About the Code Download

The code download for this article is a simple Windows Forms test harness built with Visual Studio 2005. It implements the classes described in this article. This test harness is designed for educational purposes and can be used to view the schemas and XML documents that are generated from classes and objects, respectively. Specifically this test harness can be used to do the following:

  • View the schema that is generated by a schema provider.
  • View the XML document generated by IXmlSerializable.WriteXml.
  • Test serialization and deserialization to make sure that what is serialized by IXmlSerializable.WriteXml can be utilized by IXmlSerializeable.ReadXml to correctly recreate an object from an XmlDocument.
  • Validate the XML generated by IXmlSerializable.WriteXml against the schema that is produced by the schema provider.

This test harness has three groups of command buttons used to compare the .NET attribution model to the new capabilities of the .NET Framework 2.0 (schema providers and the IXmlSerializable interface). The first group of buttons makes use of an attributed version of the Product class described in this article. The second group of command buttons uses a Product class with a schema provider and an implementation of the IXmlSerializable interface. The third group of buttons is an implementation of using schema providers with an object graph or objects that have other objects as their properties. (This was not covered in the article, so check out this implementation if you plan to use schema providers to describe a complex class hierarchy.)

When it comes to ensuring the quality of your classes, the unit testing capabilities of Visual Studio Team Developer and Visual Studio Team Test offer a better option than this test harness. Unit tests can easily be generated to create schemas, serialize objects to XML, deserialize XML into objects, and validate XML against schemas. Such unit tests can be tied to check-in policy and can be set up to automatically run against the builds created with Team Foundation Server.

Figure A Schema Provider Test Harness
Figure A Schema Provider Test Harness

The code in Figure 5 shows a schema provider that builds the schema shown in Figure 4. This code shows the portion of our Product class that declares and implements a schema provider. The schema provider is specified by using the XmlSchemaProvider attribute, which merely specifies the name of the function that should be called when schema information for this class is needed. In this case, the name of the function is GetSchemaFile. As noted earlier, this function must be static because tools like xsd.exe do not want to instantiate the class in order to use the schema provider.

Figure 5 Schema Provider for the Product Class (Close Figure 5)
[XmlRoot(ElementName="product_root", DataType="product_type",
    Namespace="http://SchemaProvider.Example.org/Product.xsd",
    IsNullable = false)]
[XmlSchemaProviderAttribute("GetSchemaFile")]
public class Product : IXmlSerializable
{
    public static XmlSchemaComplexType GetSchemaFile(
        System.Xml.Schema.XmlSchemaSet xs)
    {
        string xsdFile = Directory.GetCurrentDirectory() +
            "\\Product.xsd";
        XmlSerializer schemaSerializer =
            new XmlSerializer(typeof(XmlSchema));
        XmlSchema schema =
            (XmlSchema)schemaSerializer.Deserialize(
                XmlReader.Create(xsdFile));
        xs.Add(schema);

        // target namespace
        string tns = „http://SchemaProvider.Example.org/Product.xsd";
        XmlQualifiedName name =
            new XmlQualifiedName(„product_type", tns);
        XmlSchemaComplexType productType =
            (XmlSchemaComplexType) schema.SchemaTypes[name];

        return productType;
    }

    ...
}

The signature of this static function is interesting and deserves a few comments. Notice that it is expecting to be passed an XmlSchemaSet object and not just an XmlSchema object. With this design, you are free to build up as many schemas as you need, set them into this XmlSchemaSet object, and have them reference each other through namespaces in any fashion. Having a collection of schemas is useful when your class contains properties that are not simple data types, but typed as other classes that have schema providers of their own. For the sake of simplicity, the example I will describe builds up a single schema that is structured similarly to the schema that was created using the .NET attribution model, but which takes advantage of schema capabilities not exposed by the .NET attribution model.

Finally, note that the schema provider in Figure 5 returns an XmlSchemaComplexType object to inform the .NET Framework which complex type represents the class. This is necessary when the schema or schemas contained in the XmlSchemaSet object contain more than one complex type definition.

The GetSchemaFile function illustrates the simplest way to implement a schema provider. This provider reads an external schema file, Product.xsd, which is located in the same directory as the assembly, and streams it into an XmlSchema object. The schema object is then added to the XmlSchemaSet parameter that was passed into the GetSchemaFile function.

Using an external schema file allows you to develop your schema with the editor of your choice. As long as you have solid processes for source code control, source code versioning, and assembly deployment, you will not have a problem.

It is also possible to build a schema programmatically using the classes found in the System.Xml.Schema namespace. This technique allows for the creation of a schema provider that is not dependent on any external resources. However, the classes in this namespace can be tricky to use and you will find yourself writing a lot of code if you have a lot of fields and properties to map to a schema. Nonetheless, if you are interested in investigating this technique, you'll find a purely programmatic schema provider that produces the schema of Figure 4 in the code download for this article, which is available on the MSDN Magazine Web site.

Custom Serialization and Deserialization

Specifying a custom schema implies that custom code is required to correctly produce an XML document that corresponds to the schema coming out of the schema provider. The Product class of Figure 5 implements the IXmlSerializable interface. This interface allows the .NET Framework to call into your object for custom serialization as well as custom deserialization.

The IXmlSerializable interface has three functions: GetSchema, WriteXml, and ReadXml. The GetSchema function should not be used for producing schema information; a schema provider is a better option. For the purposes of backward compatibility, it was decided not to deliver a new version of this interface with the .NET Framework 2.0. The implementation of this function should throw an error in the event that it is erroneously called. The following code snippet is an example of how to throw an error out of the GetSchema function:

XmlSchema IXmlSerializable.GetSchema()
{
    throw new System.NotImplementedException(
        "The method or operation is not implemented.");
}
If you look closely at the signature of IXmlSerializable.GetSchema, you'll see why a schema provider is a better option. IXmlSerializable.GetSchema does not accept an XmlSchemaSet object in case you need to build up a collection of schemas to represent your class. Also, this function does not return an XmlSchemaComplexType object to inform the .NET Framework which complex type represents your class in the case that your schema contains more than one complex type definition.

The following code shows a valid implementation of the IXmlSerializable.WriteXml method for the Product class:

void IXmlSerializable.WriteXml(System.Xml.XmlWriter writer)
{
    string ns = "http://SchemaProvider.Example.org/Product.xsd";
    writer.WriteElementString("product_id", ns, ProductID.ToString());
    writer.WriteElementString("name", ns, Name.ToString());
    writer.WriteElementString("list_price", ns, ListPrice.ToString());
    writer.WriteElementString("dealer_price", ns,
        dealerPrice.ToString());
}
And here is the XML document that will be produced when a product object is serialized to XML:
<?xml version="1.0" encoding="utf-16"?>
<product_root xmlns="http://SchemaProvider.Example.org/Product.xsd">
    <product_id>1111</product_id>
    <name>Mountain Bike</name>
    <list_price>500.00</list_price>
    <dealer_price>450.00</dealer_price>
</product_root>
The root element of the document will be created for you according to what is specified in the XmlRoot attribute. Hence, you do not have to write this element out yourself using WriteElementStart and WriteElementEnd.

Notice that there are no limitations to what can be serialized. Since this function resides within the Product class, you have access to all the object's resources and any entity may be written to the XmlWriter stream, including read-only members and non-public (private, protected, internal, or internal protected) members.

This function can also contain any business logic that needs to execute during serialization. For example, you can place auditing logic in this function in order to make note of sensitive information leaving the system.

Deserialization is accomplished with the IXmlSerializable.ReadXml method, as shown here:

void IXmlSerializable.ReadXml(System.Xml.XmlReader reader)
{
   string ns = "http://SchemaProvider.Example.org/Product.xsd";
   reader.ReadStartElement();

   ProductID = reader.ReadElementContentAsInt("product_id", ns);
   Name = reader.ReadElementContentAsString("name", ns);
   ListPrice list = reader.ReadElementContentAsDecimal("list_price", ns);
   dealerPrice = reader.ReadElementContentAsDecimal("dealer_price", ns);

   reader.ReadEndElement();
}
There are a few caveats to be aware of when implementing this function. First, this code is responsible for reading the root element. This is in contrast to the serialization process during which the .NET Framework writes out the root element automatically. Second, if any elements being read in are qualified with a namespace, this code must pass in the namespace to the appropriate function of the XmlReader. Failure to specify the correct namespace for namespace qualified elements results in a run-time error because the element reading functions will not be able to find the correct element to read within the XML stream.

The control and flexibility of schema providers and the IXmlSerializable interface come with great responsibility. There is no guarantee that the XML created within IXmlSerializable.WriteXml will validate against the schema generated by the class's schema provider. Furthermore, if you are not careful, the XML written by IXmlSerializable.WriteXml may not be recognizable by IXmlSerializable.ReadXml. It is always a good idea to implement a test harness where you can test your schema provider and your implementation of IXmlSerializable to ensure that data is correctly serialized and deserialized, and that the XML you serialize out of your object can be validated against the schema provided by the schema provider. The code download for this article contains a test harness for the Product class I've described in this article. (See the sidebar "About the Code Download" for details.) It also contains code that can be easily modified for any class.

Web Services

Thus far a good amount of time has been spent on the basics of schema providers. Understanding schema providers is essential to understanding the fundamentals of describing data in two different type systems (CLR and XML Schema), as well as understanding the mechanics of moving data from one type system to the other. This section moves beyond the basics to discuss how you can use schema providers in real systems that need to be designed and implemented for interoperability. In other words, I'll show you how to use schema providers to build interoperable Web services.

Consider the Web service shown in Figure 6. This Web service contains a single WebMethod that accepts an integer and returns an instance of the Product class that is equipped with a schema provider. Figure 7 shows the Web Services Description Language (WSDL) for this Web service. The types element is the foundation of a WSDL document. It contains element declarations for all request messages and for all response messages. If additional type definitions are needed to build up a request message or a response message, then both of these type definitions are also included in the types element.

Figure 7 WSDL for Product Web Service (Close Figure 7)
<?xml version="1.0" encoding="utf-8"?>
<wsdl:definitions ... Namespace declarations omitted for brevity ... >

  <wsdl:types>
    <s:schema elementFormDefault="qualified"
      targetNamespace="http://YourCompany.com/MyService">
      <s:import namespace=
        "http://SchemaProvider.Example.org/Product.xsd" />
      <s:element name="GetProductByID">
        <s:complexType>
          <s:sequence>
            <s:element minOccurs="1" maxOccurs="1" name="id"
                type="s:int" />
          </s:sequence>
        </s:complexType>
      </s:element>

      <s:element name="GetProductByIDResponse">
        <s:complexType>
          <s:sequence>
            <s:element minOccurs="0" maxOccurs="1"
                name="GetProductByIDResult" type="s1:product_type" />
          </s:sequence>
        </s:complexType>
      </s:element>

    </s:schema>

    <s:schema xmlns:tns="http://SchemaProvider.Example.org/Product.xsd"
      xmlns:xs="http://www.w3.org/2001/XMLSchema"
     elementFormDefault="qualified" targetNamespace=
         "http://SchemaProvider.Example.org/Product.xsd">
      <xs:element name="product_root" type="tns:product_type" />
      <xs:complexType name="product_type">
        <xs:sequence>
          <xs:element name="product_id" type="xs:int" />
          <xs:element name="name" type="tns:NameType" />
          <xs:element name="list_price" type="xs:decimal" />
          <xs:element name="dealer_price" type="xs:decimal" />
        </xs:sequence>
      </xs:complexType>
      <xs:simpleType name="NameType">
        <xs:restriction base="xs:string">
          <xs:maxLength value="25" />
        </xs:restriction>
      </xs:simpleType>
    </s:schema>

  </wsdl:types>

  <wsdl:message name="GetProductByIDSoapIn">
    <wsdl:part name="parameters" element="tns:GetProductByID" />
  </wsdl:message>
  <wsdl:message name="GetProductByIDSoapOut">
    <wsdl:part name="parameters" element="tns:GetProductByIDResponse" />
  </wsdl:message>

  <wsdl:portType name="ServiceSoap">
    <wsdl:operation name="GetProductByID">
      <wsdl:input message="tns:GetProductByIDSoapIn" />
      <wsdl:output message="tns:GetProductByIDSoapOut" />
    </wsdl:operation>
  </wsdl:portType>

  <wsdl:binding name="ServiceSoap" type="tns:ServiceSoap">
    <soap:binding transport="http://schemas.xmlsoap.org/soap/http" />
    <wsdl:operation name="GetProductByID">
      <soap:operation style="document" soapAction=
        "http://YourCompany.com/MyService/GetProductByID" />
      <wsdl:input>
        <soap:body use="literal" />
      </wsdl:input>
      <wsdl:output>
        <soap:body use="literal" />
      </wsdl:output>
    </wsdl:operation>
  </wsdl:binding>

  <wsdl:service name="Service">
    <wsdl:port name="ServiceSoap" binding="tns:ServiceSoap">
      <soap:address location=
        "http://localhost:9266/WebServiceTest/Service.asmx" />
    </wsdl:port>
  </wsdl:service>

</wsdl:definitions>
Figure 6 Product Web Service (Close Figure 6)
using System;
using System.Web;
using System.Web.Services;
using System.Web.Services.Protocols;

[WebService(Namespace = "http://example.com/MyService")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
public class Service : System.Web.Services.WebService
{
    [WebMethod]
    public SchemaProviderExample.Product GetProductByID(int id)
    {
        // sample data
        decimal list = 500.00M;
        decimal dealer = 450.00M;
        string name = "Mountain Bike";

        return new SchemaProviderExample.Product(id, name, list, dealer);
    }

}

Let's walk through of the WSDL document so you can get a concrete example of these concepts. You can see three distinct entries in the types element of the WSDL. The first entry is an element declaration for an element named GetProductByID. This describes the request message for the WebMethod in Figure 6. The GetProductByID element declaration merely states that this message is composed of exactly one integer. The next element declaration is GetProductByIDResponse, which describes the response message of the same method. This states that the response is an instance of the product_type complex type. The final entry in the types element of the WSDL is all the schema information that is returned from the schema provider implemented by the Product class. The complex type named product_type, which is used by the response message, is defined in this final entry.

When this WSDL was generated by wsdl.exe, the schema provider of the Product class was used to define the data contract of the Product class, since an instance of the Product class is the return value of the GetProductByID Web method. When this data contract was placed in the WSDL, it became a part of the message contract for the Web service. This ability to control a message contract with a schema provider represents the value of schema providers.

At run time, the Product class's implementation of IXmlSerializable is used for serialization and deserialization of SOAP messages. In the previous example, the relevant portions of the SOAP response message are created by the Product class's implementation of IXmlSerializable.WriteXml. If an instance of the Product class is needed by a request message (in other words, the Product class is the type of an input parameter) then the implementation of IXmlSerializable.ReadXml is used to deserialize the relevant portions of the SOAP request into an instance of a Product object.

Schema providers are useful when developing Web services in a contract-first fashion. If an application architect is given the WSDL in Figure 7 and asked to build a Web service that conforms to that contract, then the product_type type definition can be copied directly from the WSDL and used to implement a schema provider for an existing Product class or a new Product class. The bottom line is that whether you need to control a message contract with a class or build a class based on a message contract, schema providers give you the flexibility needed to get the job done.

Conclusion

By using schema providers along with the IXmlSerizable interface, you gain flexibility and control over the data contract and the serialization/deserialization process. Mapping non-public class members to a class's data contract is now possible and schema providers allow all the capabilities of XML Schemas to be used within the data contract of a class. Additionally, classes that are used as input parameters and return values of Web services can be equipped with schema providers in order to provide greater control over the Web service's message contract. Finally, the IXmlSerializable interface is now supported and provides access to all of an object's members. This interface can also be used to invoke custom business logic during the serialization/deserialization process.

Implementation is not as easy as with the .NET attribution model. The .NET attribution model is still supported in the .NET Framework 2.0, and you will need to choose the correct tool for each design scenario that requires interoperability.

Keith Pijanowski is a .NET Architect Evangelist for Microsoft Corporation specializing in the .NET Framework. Keith helps Microsoft customers apply new technologies to business problems. Keith is also a frequent speaker at Microsoft events and community events in the New Jersey and New York area.


Share The Load: Report Visual Studio Team System Load Test Results Via A Configurable Web Site

Share the load

Report Visual Studio Team System Load Test Results Via A Configurable Web Site

Wen Ding
This article discusses:
  • Using the Visual Studio load test tool
  • Load test reporting Web site
  • Architecture of the test reporting site
  • Using and extending the test reporting site
This article uses the following technologies:
Visual Studio 2005, Team System, SQL Server 2005
Code download available at: LoadReporting2006_06.exe (332KB)
Contents
Using the Load Test Tool
The Load Test Reporting Site
Report Site Architecture
Load Test Run Management
SQL Server Reporting Pages
Going Further
Conclusion

V
isual Studio 2005 Team System includes a newly designed load test tool. Formerly code-named "Ocracoke," this new tool provides many exciting features you can use for performance and stress testing your Web sites, Web services, and other server components. And, combined with its handy reporting capabilities, the load test tool provides some powerful options for sharing and managing test results.

The load test tool executes performance test runs and can be configured to log data into a central SQL database. After each test run, you can view collected test data, such as performance counter values, test error messages, and the test execution load, and then analyze the test data from within the Visual Studio® 2005 UI.

I find it particularly helpful to create performance test reports in a summary view, showing key performance results like latency numbers, test load, server resource utilization, and any errors encountered during execution. These performance reports can be viewed with a Web browser from other machines that might not have Visual Studio installed. Because the load test tool can store test data in a SQL Server™ database, users can connect to the database directly and dig into the test data repository to create a variety of customized performance test reports. In this article, I'll introduce one approach to creating a customized performance reporting Web site from test data generated by the load test tool.

The reporting site I discuss in this article (shown in Figure 1) provides two main features. The first is an ability to show reports of performance test runs that can be viewed from any Web browser. The reports can also be exported to Microsoft Excel® or Office Web Components-based HTML documents for e-mail delivery. For each test run, the site displays several useful reports including latency numbers for both test cases and transactions; summary and detailed views of performance counter logs that show the amount of server resources consumed (such as CPU and memory usage); test load information including the number of requests per second; and error messages.

The second feature provides very handy management functions. The number of performance test runs grows over time, especially when your team has been performance testing multiple projects. This could result in hundreds of different performance test runs being stored in the database, making it difficult to search for information or clean up any obsolete test runs within Visual Studio. This sample Web site lets you easily filter test runs by, say, scenario name or date range, and then remove any unneeded test run data from the database. Other useful management functions include editing project and scenario names and adding analysis comments.

Using the Load Test Tool

When you develop a high-traffic Web site, a Web service, or some other server components, it's important to know how much load your site or service can handle. A performance test can help determine whether the product meets your performance goals. You can use a load test tool like the one in Visual Studio 2005 Team System to simulate multiple users simultaneously sending requests to the service. Although I won't go through all the exciting new features of that tool, I will describe some basic features that will help you understand how the test reporting site works. You can find more information on working with the Visual Studio 2005 load test tool at msdn2.microsoft.com/ms182561.aspx.

The load test tool is fully integrated into Visual Studio 2005 Team System, so you do not need to search for and launch it separately. The Load Test Wizard lets you easily create a load test project. To start a performance test run you create unit test cases covering your desired performance test scenarios, and then create a load test project. At this point you can choose any unit test cases that you want to run performance tests against, and add them to the load test project. The Load Test Wizard guides you through all the necessary steps including creating multiple performance test scenarios with different run settings. After these tasks, you can run a scenario for performance testing.

There are two test case types that can be added to a load test project: Web page tests and unit tests, including any functional code or Web service calls. To create a Web page load test, you use the Web Page Recorder to capture the navigation and user input performed in the Web browser. The recorded Web requests, post data, query strings, and cookies can then be replicated by the load test tool. The Load Test Wizard will guide you through all the replay settings: how many simultaneous users, the mix of browser types, the run duration, thinking time between Web requests, the mix of different network bandwidths, what types of performance counters to collect, and so on.

For load tests on unit test cases, such as Web service calls or calls to server components, you can create a load test project and add unit cases to it. The same Load Test Wizard guides you through settings that are similar to those mentioned previously. You can choose different stress patterns, including consistent load, step-up load, and so on. Load tests can be done on a single test client machine, or initiated from a set of remote machines (called a "rig"). A controller machine is used to control the test run on the rig, collect all the performance test data from the remote clients, and place all the data into the results database.

The load test tool is able to collect various types of test data from remote servers and log this data into the database. For performance test reporting, it is important to know what kinds of test data are collected after the load test run. There are four types of information you will typically put in the reports:

  • Latency for each type of request
  • Throughput information, or how much load the server received (for example, requests per second)
  • Server-side resource utilization (such as CPU usage, memory consumption, and so on)
  • Test run configurations (such as test duration, number of simultaneous users, and so on)

There are other useful types of information that can be logged in the results database. These include any error messages encountered and SQL Server trace information that can help you find which query or stored procedure is the slowest during the test run.

The Load Test Reporting Site

My load test reporting Web site is designed to provide a number of useful capabilities that can help you run tests and create reports. First of all, it has a hierarchical structure that makes it easy to navigate between test runs. The site organizes test runs by date range—an approach that makes it easier to find and manage tests. It allows for easy modification of important information like project and scenario name, test description, and analysis comments. And my reporting Web site offers a variety of test reports, including a summary report, a detailed performance counter report, and a Web page performance test report. (Note that my sample site uses Scott Mitchell's RoundedCorners Web control, which you can read more about at aspnet.4guysfromrolla.com/articles/072804-1.aspx.)

On the left side of the home page in Figure 1, a tree view navigation control lists all the load-tested projects in the database. When you click on a specific project name in the tree view, all the scenarios and test runs for this project are displayed in the right panel. The relationship between a project name (the LoadTestName field in the database tables), scenario names, and RunIds is like a branching tree-structure. One project can contain multiple scenarios. And for one test scenario, multiple test runs can be executed. Under the tree view control, there is a dropdown list control that is used to filter the number of test runs displayed. To limit the number of test runs displayed on the page, you can specify dates to display only test runs that happened on those days.

Figure 1 Load Test Reporting Site
Figure 1 Load Test Reporting Site

On the right side of the home page, the Results column provides three links for each run: Summary (which displays a test summary report, as shown in Figure 2), LoadChart (which displays a user load chart), and PageResults (which shows results for a Web request test). The Edit links allow you to modify the project name, scenario name, test run description, and analysis comments for a specific run, and the Delete links let you remove all the logged test data for a specific run from the database.

Figure 2 Test Summary Report
Figure 2 Test Summary Report

Report Site Architecture

The design architecture used to build the report Web site is shown in Figure 3. The Web pages are built with ASP.NET Web controls that are bound to data from the results database. For all test reports, SQL Server Report Definition Language (RDL) is used to pull data from the database and create report pages. Several new stored procedures and views are created in the database and used by these Web controls. My site takes advantage of many features that SQL Server Reporting Services provides, such as exporting reports to other formats.

Figure 3 Report Site Architecture
Figure 3 Report Site Architecture

Even with this reporting site, you still need Visual Studio 2005 Team System to run load tests and configure the test run to log raw data into a centralized SQL database. However, by using this site architecture, you can easily customize the design of your own reporting pages by either modifying the sample report designs or creating your own reports.

The sample solution includes an ASP.NET site- setup project, WebSetup_PerfReport, from which you can install the Web site on any host server with IIS and ASP.NET 2.0. SQL Server RDL files are provided to build the test reports. But some setup steps are necessary before you can compile the sample projects and install the reporting Web site. (See the code download for details.)

After you have set up the reporting Web site to pull test result data from the SQL database and create test reports, you can test your reporting site and start writing Web tests or unit test cases and add them to a load test project. After running the load test project, check whether the test run shows up in the reporting site. As mentioned earlier, you can view the test results right after the test run, using Visual Studio 2005. The advantage of the reporting site is that you can get a summary report and detailed performance test reports simply by using a Web browser on any machine.

To write your own test report by accessing test data directly from the load test database, you need to understand the basic table structure, stored procedures, and views. Because the load test tool internally logs data into the tables, changing table design or structure could potentially break the test tool. Therefore, you need to adapt your approach to table structures required by Visual Studio. For example, in the sample reporting site, test runs for all test projects are organized in a tree view with a hierarchical structure like project name/scenario name/run ID. But the LoadTest database only contains the fields ScenarioName and RunId. Fortunately you can use the LoadTestName field to serve as the project name for reporting needs.

An example of using a customized stored procedure is shown on the right-hand side of the main page, shown in Figure 1, where the ASP.NET DataGrid control binds to test data returned from the following stored procedure:

CREATE PROCEDURE Prc_SelectLoadTestSummary
        @LoadTestName nvarchar(255)
AS
SELECT  ScenarioName, LoadTestRunId, Description, StartTime,
        EndTime, RunDuration, Outcome
FROM    dbo.LoadTestSummary
WHERE   LoadTestName = @LoadTestName

Load Test Run Management

Typically, you will conduct a number of performance test runs including several performance test scenarios for each project. This could result in a large number of test runs logged into the results database. Some of the results may not be useful, and you may want to remove them from the database. For more relevant test runs, you probably want to add analysis comments to summarize your observations and present them in test reports.

A nice feature of the new load test tool is that, for each performance test run, you can put your own analysis comment in the comment field of the table LoadTestRun. The comment is stored in the database after each test run, where you can analyze data and put summarized findings about the performance test run into it.

The page shown in Figure 4 was designed to allow users to modify test information through the reporting site. This page can be launched by clicking the Edit link on the main page. You can change the project name by entering a new one or choosing any existing project name from the dropdown control. You can also edit the test scenario name, test run description, and analysis comments in the same page. This function is especially useful when a certain test run is identified as the final test run and you want to share the results with your team. Once you click the Update button, the database will be updated with the new information.

Figure 4 Updating Test Run Information
Figure 4 Updating Test Run Information

If you find some test runs are not worth keeping in the database, there are several ways to remove them. One way is through the reporting site, where you can simply click the Delete link on the main page. The link internally calls a stored procedure Prc_DeleteLoadTestRun by passing a test RunId parameter corresponding to the clicked DataGrid row.

You need to pay attention to the delete action from the reporting site because any user accessing the page could accidentally remove test runs by clicking Delete. There are several things you can do to prevent this, though. For example, you can add authentication logic to restrict access to critical pages that offer the delete or edit actions.

You can remove multiple test runs at the same time by using your own stored procedure. One example of such a stored procedure is to remove test data for all test run IDs in a range (Prc_DeleteLoadTestRunFromIds):

CREATE PROCEDURE Prc_DeleteLoadTestRunFromIds
    @LoadTestRunStartId int,
    @LoadTestRunEndId int
AS
BEGIN
    DECLARE @LoadTestRunId int
    SET     @LoadTestRunId = @LoadTestRunStartId
    WHILE   @LoadTestRunId <= @LoadTestRunEndId
    BEGIN
        EXEC Prc_DeleteLoadTestRun @LoadTestRunId
        SET  @LoadTestRunId = @LoadTestRunId + 1
    END
END
GO
For example, to remove test results for ID 12 through 20, you'd simply run the following:
EXEC Prc_DeleteLoadTestRunFromIds 12, 20

SQL Server Reporting Pages

From the main test report page you can open three reporting pages for a test run by clicking Summary, LoadChart, or PageResults. The code behind the DataGrid control redirects the request to the corresponding reporting service page as shown in Figure 5. The three reports are designed with LoadTestSummary.rdl, PagePerformanceOverTime.rdl, and RequestPerformanceDetails.rdl in the sample project.

Figure 5 Redirect to SQL Reporting Pages (Close Figure 5)
private void DataGrid1_Item(object source,
    System.Web.UI.WebControls.DataGridCommandEventArgs e)
{
    try
    {
        string runID = e.Item.Cells[1].Text;
        switch(e.CommandName)
        {
            case "Results" :
                Response.Redirect(
                    "http://MySqlServer/ReportServer?" +
                    "%2fOcracokeReport%2fLoadTestSummary&" +
                    "TestRunId=" + runID, false);
                break;
            case "LoadChart" :
                Response.Redirect(
                    "http://MySqlServer/ReportServer?" +
                    "%2fOcracokeReport%2fPagePerformanceOverTime" +
                    "&TestRunId=" + runID, false);
                break;
            case "WebRequests" :
                Response.Redirect(
                    "http://MySqlServer//ReportServer?%2fOcracokeReport" +
                    "%2fRequestPerformanceDetails&TestRunId=" +
                    runID, false);
                break;
            case "Edit" :
                Session["RunId"] = runID;
                Response.Redirect("home.aspx", false);
                break;
        }
    }
    catch (Exception ex) { Debug.WriteLine(ex.ToString()); }
}

A key area of performance test reporting is system resource utilization on each test server. In the report summary page of Figure 2, you'll see there is a section called System Resource Usage. In this section, several important performance counter categories are reported. If you are interested in adding more pre-selected performance counter instances or categories in the summary report, you can modify the SQL query in the data source for report LoadTestSummary.rdl as shown in Figure 6.

Figure 6 Performance Counter Query (Close Figure 6)
SELECT *
FROM   LoadTestComputedCounterSummary
WHERE  loadTestRunId = @TestRunId AND
       ((CategoryName = ‘Processor’ AND
         CounterName = ‘% Processor Time’ AND
         InstanceName = ‘_Total’) OR
        (CategoryName = ‘Memory’ AND
         CounterName = ‘Available MBytes’) OR
        (CategoryName = ‘System’ AND
         CounterName = ‘Context Switches/sec’) OR
        (CategoryName = ‘.NET CLR Exceptions’ AND
         CounterName like ‘%Excep%’) OR
        (CategoryName = ‘.NET CLR Memory’ AND
         CounterName = ‘% Time in GC’) OR
        (CounterName = ‘Requests/Sec’ AND
         CategoryName Like ‘%ASP%’) OR
        (CounterName Like ‘%Queue%’ AND
         CategoryName Like ‘%ASP%’) OR
        (CategoryName = ‘Web Service’ AND
         CounterName = ‘Current Connections’) OR
        (CategoryName Like ‘%Network Interface%’ AND
         CounterName = ‘Bytes Total/sec’))
ORDER BY MachineName

In the System Resource Usage area, you can also click on each performance counter category name and jump to a more detailed report for the performance counter values in that category on all the test servers. This function lets you dig into more performance counter instances inside that category by showing the values for all the servers involved. For example, by clicking the System link, you get a detailed report page for system resources. The report page shows all the performance counter instances under the System category for all the involved test servers. This is especially useful when you want to analyze more detailed performance counter data for a specific performance counter category. This report is in SystemResourceUsageDetail.rdl.

If you have WebRequest in your load test project for Web page performance testing, you can view the Web page result report by clicking the PageResults link. This brings up a report page that lists all the test information about a certain Web request. You get multiple rows if you are performance testing multiple Web pages. The RDL file for this Web request report is RequestPerformanceDetails.rdl.

After you run a load test, the response time and latency numbers for test cases and transactions are shown in average values. The data in the Test Case column represents the elapsed time for any function in your code marked with the [TestMethod] attribute, as shown here:

[TestMethod]
public void SubmitEventWithEmail() {
    SubmitEvent(false);
}

[TestMethod]
public void SubmitEventWithPuid() {
    SubmitEvent(true);
}
Because there are two functions marked with the [TestMethod] attribute in this code, the latency data or response time for both of these functions is reported in the Test Case column. The response time represents the overall latency for the entire function.

It is important to measure latency for any sub-functional calls inside a large function. This can be implemented in your code by using BeginTimer and EndTimer functions to measure time being spent on any given functions. An example of this is shown in Figure 7. In this code, latency data will be collected for any functions surrounded by BeginTimer and EndTimer. The latency number is reported in the Transaction section of the page results.

Figure 7 Measuring Transaction Latency (Close Figure 7)

private void RunSinglePath()
{
    m_testContext.BeginTimer("GetChannels");
    string ch = _ch.GetChannels("/", _local, -1);
    m_testContext.EndTimer("GetChannels");

    m_testContext.BeginTimer("Add All - " +
        Items.Length.ToString() + " items");
    AddAll(false, false);
    m_testContext.EndTimer("Add All - " +
        Items.Length.ToString() + " items");
}

private TestContext m_testContext;

public TestContext TestContext
{
    get { return m_testContext; }
    set { m_testContext = value; }
}

Going Further

Once you are able to build your performance test reporting site, there are many things you can do to improve it. One report you can add is a SQL Trace report. If you enable SQL Trace Tracking in your load test project, you can get execution time information for SQL query commands or stored procedures in the LoadTest database. Based on SQL Trace data from a sample test run, you can create a report to display stored procedures ordered by latency, allowing you to point out the slowest stored procedure in the test run.

A side-by-side comparison report for two performance runs is also very beneficial, as it lets you see if a new version of a build has better performance than older versions. On a single Web page, you can display the performance differences (for, say, latency or CPU usage) between two versions. This way, you can instantly compare the performance of the two runs and find out the areas that require further investigation.

The reporting site I've discussed in this article displays average values for response time and performance counter. If you prefer performance test data represented in percentiles (rather than averages), refer to www.seanadams.com/95 for more information. Percentile data does have the benefit of providing statistical information.

With the default configuration of the load test tool, the percentile data is not collected. In order to enable the percentile calculation, you must change the property Timing Details Storage of Run Settings in the load test project. For more information about this setting and other useful techniques, visit Bill Barnett's blog. The percentile data is logged into a table called LoadTestTransactionSummaryData. You can modify the SQL reporting design schema to include percentile information on report pages. Note that enabling this will add extra data to the database and more disk space will be consumed.

Finally, unlike with functional test cases, identifying a performance test run as pass or fail doesn't always make sense. Sometimes you just want to collect benchmark performance numbers. In other cases, however, you probably have performance requirements in terms of acceptable and unacceptable latency or CPU usage. If the test run performance falls outside of the acceptable range, the test is essentially a fail. To identify whether a performance test run meets performance requirements, you can modify the current reporting design schema and Web controls to add this kind of result information to a specific test run.

Conclusion

Visual Studio 2005 Team System has integrated a handy new load test tool that allows you to run customizable tests and collect critical data. But other capabilities, such as the test reporting Web site feature, let you dig deeper and get more out of the built-in load testing. With a firm grasp of the test reporting Web site functionality, you can more easily share test results, analyze data, and manage stored test results.

Wen Ding works for Microsoft where he conducts software performance testing and optimization for many projects used on microsoft.com sites. Wen can be reached at dingwen@microsoft.com.


Editor's Note: New and Improved Team

Editor's Note
New and Improved Team

We've all been subconsciously conditioned through the years to turn a skeptical eye towards claims of being "new and improved." After all, the phrase often means that some favorite feature has been turned into a new deficiency. Who can forget a few years ago when a favorite brand of chocolate suddenly showed up on shelves with a "New size!" label? It turned out that they reduced the size of the bags from 14 ounces to 13—and you could experience this new size for the same old price. (Not to point any fingers, I'll call these candies "Unicode character U+041C" & "Unicode character U+041C"'s.) But sometimes a new and improved product really is new and improved. Take Visual Studio 2005 Team System, for example. It's packed with new features that create a virtual playground for the imagination (albeit without the dangerous Jungle Gym).

Visual Studio Team System is actually a family of products that includes specialized builds for testers, developers, and architects. (They are available as part of the complete Visual Studio Team System package or, with an MSDN Subscription, they can be obtained individually.) The core of each build is the same: you get Visual Studio 2005, a class designer, UML modeling with Visio, database tools, Visual SourceSafe, enterprise templates, modeling tools, and Microsoft Office programmability.

Beyond that, the edition you should work in depends on your job role. If you're an architect, for instance, you'll want to install the version that caters to your needs. For architects, this means getting an application designer, a deployment designer, and a logical infrastructure designer.

Developers, on the other hand, get some extra tools that make it far easier to understand, debug, and test their code. The additional tools let you perform unit testing, plot your code coverage, run code profiling, and make use of a dynamic code analyzer and static code analyzer.

Testers will find Visual Studio Team System particularly useful, with great attention given to making their jobs easier. The edition tailored for testing includes the unit testing and code coverage features found in the developer edition, but also adds tools for load testing, manual testing, test case management, and Web testing.

Visual Studio Team Suite, which includes all of this and then some, offers a lot of product in the box (or download). Some of the tools are similar to tools previously offered by ISVs in various forms. This past March, we spoke with a number of developers at SD West in Santa Clara and heard some frank concerns. In general, the ISVs were concerned that Visual Studio Team System would cannibalize their products. And they were worried that Visual Studio Team System would do everything, leaving no room for further innovation by third-party developers.

Far from being the end-all for everything, Visual Studio Team System introduces many concepts, like unit testing, to the package while leaving room for more comprehensive features. Over the next year to 18 months, we expect the market for add-on tools for Visual Studio to really blossom. Visual Studio has been designed with extensibility in mind, so third-parties have many opportunities to enhance the basic tools provided in the box.

In this issue, we've got an example of how you can extend Visual Studio Team System. Wen Ding shows you on page 123 how to enhance the load testing tool in Visual Studio Team System by creating Web-based reports. Your entire team will be able to see the latest results online as soon as the load test is complete.

If you want to step back and look at all that Visual Studio Team System has to offer, check out the overview Chris Menegay did in our special Visual Studio 2005 Guided Tour bonus issue. If you haven't had the chance to peruse this minimag, it is well worth your time. We've updated the articles to cover the release version of Visual Studio 2005, walking you through Visual Basic, C#, C++, ASP.NET 2.0, and much more. You can look it over at msdn.microsoft.com/msdnmag/issues/06/00/default.aspx and even download it. Drop on by and check it out!

—J.T.

Thanks to Thanks to the following Microsoft technical experts for their help with this issue: Ryan Cavalcante, Joe Duffy, Neil Enns, Mark Fussel, Lukasz Gwozdz, Brian Jones, Hao Kung, Ronald Laeremans, Michael Miele, Lukasz Pawlowski, Stefan Schackow, Maura Van Der Linden, and Mike Vernal.

Excel, IntelliSense, Internet Explorer, Microsoft, MSDN, PowerPoint, Visual Basic, Visual C++, Visual SourceSafe, Visual Studio, Windows, Win32, WinFX, Windows Server, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners. MSDN MAGAZINE DOES NOT MAKE ANY REPRESENTATION OR WARRANTY, EXPRESS OR IMPLIED WITH RESPECT TO ANY CODE OR OTHER INFORMATION HEREIN. MSDN MAGAZINE DISCLAIMS ANY LIABILITY WHATSOEVER FOR ANY USE OF SUCH CODE OR OTHER INFORMATION. MSDN and Microsoft are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. MSDN Magazine, MSDN, and Microsoft logos are used by CMP under license from owner. MSDN Magazine is published by CMP Media LLC. CMP Media LLC is an independent company not affiliated with Microsoft Corporation. Microsoft Corporation is solely responsible for the editorial contents of this magazine.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: Generate Office Documents, Monitor Event Logs, and More

Toolbox
Generate Office Documents, Monitor Event Logs, and More
Scott Mitchell

Contents
Create Native Word and Excel Docs
Monitor Your Event Logs
Using Source Control to Track Changes
The Bookshelf

Create Native Word and Excel Docs

Most data-driven Web sites are used as interfaces to collect, process, and summarize information. Reports that summarize the data can be presented to the user in a variety of formats—the most common way is to display the report directly in a Web page. However, in some scenarios Microsoft® Excel® and Word documents are a more ideal format. If the report is stored as an Excel document, for example, a user can easily e-mail it to a colleague, make her own modifications to the data, sort and filter the data without having to re-request the report from the Web site, and so on.

Generating Word and Excel documents on a Web server can be accomplished in a variety of ways. One option is to use the Office Web Components (OWC), which are a collection of COM components for generating Excel spreadsheets and charts. While OWC allows for creating more feature-rich Excel documents, it requires that Microsoft Office be installed on the Web server and introduces performance and scalability concerns.

In short, efficiently generating feature-rich Office documents in a server-side environment requires that you create the raw, binary version of the Office document on the server using simple file creation techniques. This approach offers the best of both worlds: native documents can be created with the full set of Office features, yet there's no requirement to have Office installed on the server.

OfficeWriter version 3.5, by SoftArtisans, allows for both Excel and Word documents to be created in this manner. With OfficeWriter, you start by creating an Excel or Word template on your desktop computer. The templates contain placeholders that indicate where the dynamic data will be placed. In addition, these templates can contain the full set of Office features and capabilities. Spreadsheet templates, for example, can include charts, pivot tables, formulas, macros, multiple sheets, and so on. When the template is complete, just upload it to the Web server and create an ASP.NET page that binds data to the template. This will create a new Excel or Word document, mashing together the data and template.

OfficeWriter allows for a variety of data sources to be bound to the underlying template, including results from a database query, and the resulting Office document can be saved to the Web server's file system or streamed to the client. OfficeWriter also provides integration with SQL Server™ Reporting Services, allowing for reports to be designed directly from within Word and for generated reports to be exported as Word documents or enhanced Excel spreadsheets.

Price: $1,495 per server for the Standard Edition.

www.softartisans.com

Monitor Your Event Logs

The Windows® Event Log serves as a centralized repository for security, system, and application-level information. While the Event Log contains information of interest primarily to system administrators (for example, failed login attempts, disk errors, unexpected shutdowns, and so on), it also contains database and application errors that concern developers. When these errors occur on production servers, it's important that the responsible developers are alerted immediately.

There are a number of products designed for system administrators to provide monitoring of the Event Log and notifications. One such tool is Event Sentry version 2.70, by NETIKUS.NET, which offers an easy-to-use, yet powerful interface for receiving notification when certain conditions are met in the Windows Event Log. Event Sentry provides filters that allow for only specific event conditions to trigger a notification. The filterable criteria include the event source, the event ID, the event severity, and the event text, among others.

You can also set a threshold indicating that you only want to be notified if an event occurs a certain number of times, like twice in the span of five minutes. Once a filter's criteria are met, a notification is generated. Event Sentry supports 12 forms of notification, including RSS feeds, sending e-mail, starting a process or script, and recording the results to a database or file.

For example, you might be using the Enterprise Library's ExceptionHandling feature to log unhandled exceptions to the server's Event Log with the Source "My Application". If you'd like to receive an e-mail notification when your application raises an unhandled exception, simply create a filter that looks for events from the "My Application" Source. Next, add an e-mail notification, specifying the SMTP server to use and the recipient list. Henceforth, any unhandled exceptions in your application will result in an e-mail notification in your Inbox!

Price: $69 per server (with discounts for multiple licenses).

www.eventsentry.com

Using Source Control to Track Changes

Every software company, whether it has one developer or teams of hundreds, should be using some form of source control. However, far too many one-developer companies, or companies whose primary business line is not in software, simply don't use source control. Such shortsightedness is trouble looking for a place to happen.

Microsoft has long offered developers Visual SourceSafe® as a source control tool. With Visual Studio® 2005 Team System, Microsoft has released another source control tool—one designed for supporting much larger developer teams. Despite these offerings, there is still a healthy third-party source control market. One such third-party tool is Vault from SourceGear LLC.

Like any source control tool, Vault facilitates storing source code in a centralized repository; maintains a history of code changes; allows for check out, modify, check in, and modify and merge workflows; offers branching and merging of source control trees; and so on. It was also designed with Visual SourceSafe users in mind. Vault will happily import your existing Visual SourceSafe database, and the check out, modify, check in workflow can be performed entirely through the Visual Studio IDE, just as with SourceSafe.

Unlike Visual SourceSafe, Vault uses Microsoft SQL Server as its centralized data store, making it easy to schedule backups, run queries directly against the source code repository, and so on. (Vault also offers a set of API libraries that allow you to programmatically access the repository.)

Vault was designed to work over the Internet, which is a boon for developers like myself who telecommute or work for remote clients. By simply installing Vault on a Web server, developers scattered around the globe can work on a project together. The requests to the repository hosted on the Web server are transparently handled via HTTP or HTTPS requests through the Vault client.

For those solo developer shops, there's absolutely no excuse not to be using source control as SourceGear offers a free single-user license. If you find yourself working on code without a source control tool in place, consider preventing that problem waiting to happen by using Vault.

Price: $289 per developer license.

www.sourcegear.com

The Bookshelf

In a September, 2004 column for the MSDN Web site, Eric Sink coined the term micro-ISV (Independent Software Vendor). As Eric describes it, a micro-ISV is one that is very small, "a company with less than 50 employees..., funded by its own revenues, and has not accepted investment from a venture capitalist." The majority of micro-ISVs were initially started as and continue to run as one-man (or one-woman) shops.

Many developers have fancied the notion of ditching their day job to pursue building and selling their own software product, but they don't know where or how to get started. Part of the challenge of starting a micro-ISV is that there's so much more to it than simply writing the next killer app.

If dreams of starting your own micro-ISV have floated through your head, check out Micro-ISV: From Vision to Reality (APress, 2006). In it, Bob Walsh explores the core challenges faced by micro-ISVs. The first hurdle to overcome is to have a well-defined idea for an application that solves a real customer's real pain. A product that doesn't solve a real problem or doesn't have a clear vision is one that's destined to fail. There are also unique challenges for micro-ISVs when designing the application.

In the corporate setting, rarely is a lone developer responsible for designing the user interface, crafting the application's architecture, deciding on the programming methodologies to be used, choosing the development tools, and writing all of the code. But as a micro-ISV, many, if not all, of those responsibilities and choices will fall squarely on your head. Deciding on a product and building it are just the first steps for a micro-ISV. There's still the much more important aspect: convincing people to pay for your product.

The many interviews sprinkled throughout the book provide a gamut of information including advice from successful micro-ISV founders with tips on envisioning, designing, developing, and marketing your product; discussions with payment processing Web sites on the options and challenges of accepting customer payments online; great productivity tips; and food for thought from an attorney regarding partnership and liability issues that businesses must consider. These interviews greatly enhance the information presented throughout this book, and the advice from successful micro-ISV founders serves as a source of inspiration.

Price: $29.99

www.appress.com

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


Advanced Basics: Setting Word Document Properties the Office 2007 Way

Advanced Basics
Setting Word Document Properties the Office 2007 Way
Ken Getz

Code download available at: AdvBasics2006_06.exe (167KB)
Contents
Introducing the Open XML File Formats
Investigating the Sample Project
Wrapping Up

The last time I wrote this column (March 2006), I shared an application that allows you to update all the Microsoft® Word documents in a folder and its subfolders. Each time the application finds a document in the specified path, it updates the document properties to match those you specified in the application. I use this application myself. I often need to change a group of Word documents, setting all the properties to match, and performing this task by hand is a real chore. I'm revisiting this topic with a new version of the application. For information on the prior version, check out my previous column. That column explains the application and discusses its plumbing (which includes Windows® Forms data binding and automation of Microsoft Word 2003).

The original application has some shortcomings. It requires each document to be loaded into Microsoft Word, where the properties are manipulated and the document is saved, and then it moves on. Obviously, the application requires a copy of Word on the local computer. In addition, the application incurs the extra overhead of using COM interop to interact with Microsoft Word, and the inherent security issues involved in using unmanaged code as part of a managed application.

In the March column, I promised to revisit the issue. This time I've taken advantage of technology that is new to the Microsoft Office arena. I've rebuilt the application using the new Open XML file formats that are used by the upcoming release of 2007 Microsoft Office system (formerly code-named Office "12"). As you'll see, the new application allows you to modify the contents of Word documents without a copy of Word and without any COM interop involvement. There are also significant speed improvements. The new Open XML file format really makes manipulation fast and lightweight. Whereas the old application would read an entire Word document into memory, the new one simply loads the portion of the document necessary to make and save the changes. Figure 1 shows the finished Office 2007 version of the application.

Figure 1 Modify Core Document Properties
Figure 1 Modify Core Document Properties

Before getting started, I should point out that if you want to play with the sample associated with this column, you'll need to have Microsoft Office 2007 Beta 2 (or later) available so you can create documents using the new file format—this version of the sample won't work at all with files created in earlier versions of Microsoft Word, or with files using the old file format. In addition, you'll need a copy of Visual Studio® 2005. Finally, the classes that provide .NET Framework support for the Open XML file formats are only available starting with the latest builds of WinFX®, so you will also need at least the February 2006 community technology preview (CTP) version of WinFX.

Introducing the Open XML File Formats

In the past, each of the Microsoft Office products maintained data in its own proprietary format. I remember clearly, back in the late eighties, the pain involved in writing a file format converter to convert files from the word-processing product I was working on as a developer into a very early version of Microsoft Word. Each of the other applications, as well, was written to store its data in a proprietary format. These formats allowed each app to optimize its serialization/deserialization techniques specifically for its particular file format. But the closed nature of the formats made it difficult for "outsiders" to work with the data natively. You could use COM Automation to load a copy of the product, load the document, and then programmatically modify the document as necessary. This sort of work is clumsy and slow, and taking on this task in a server situation was nearly impossible due to licensing and threading limitations.

But this is a problem of the past now that Microsoft has opened up the Office file formats in Office 2007. Word 2007, Excel® 2007, and PowerPoint® 2007 lead this initiative, and other Office applications may follow suit in future versions. Each of these products saves its data natively in an XML-based format, making it reasonably easy for your applications to interact with these files, using tools provided by just about any programming language. Microsoft is even submitting the new file formats as a European Computer Manufacturers Association (ECMA) standard. But that's a story for a different article. For more information on the new file formats, visit the Microsoft Office Developer Center, and check out the overview.

The new file formats require a new file extension. If you work with Office 2007, you'll find *.docx, *.xlsx, and *.pptx files. And if you add VBA code to your documents, you'll find *.docm, *.xlsm, and *.pptm files. Templates and other files have their own new extensions, as well, but in reality, all these files are document parts that are grouped and saved in a standard ZIP file format. (By the official release of the upcoming version of Microsoft Office, you'll be able to download converters that enable Microsoft Office 2003 products to convert files from the new format, as well).

The sidebar "Get Inside a Word File" shows you how to manually look at and modify the various elements of a Word 2007 file (you needn't take any of the steps described in the sidebar if you're manipulating the files programmatically). As you can see, there is a lot more involved than you'd like to have to do in order to modify a document. However, WinFX provides support for opening the compressed Open XML file as a System.IO.Packaging.Package object, allowing you to programmatically investigate the relations and parts within the package. That's exactly how the sample for this column works.

Get Inside a Word File

Basically, a Word 2007 file is all about parts. To get a glimpse of what’s going on in there, follow this guide to peer inside a simple Word 2007 file.

With Word 2007 running, create a simple document containing a paragraph or two. Then save the document in a known location, such as C:\Sample.docx. Use Windows® Explorer to browse to the location where you saved the file and rename it using the .zip extension (C:\Sample.docx.zip, for example). Windows complains, of course, but just persevere and let Windows know who’s boss.

In Windows Explorer, right-click the renamed file and select Extract All. Windows will extract the contents of the compressed file, as shown in Figure A.

Figure A Extracting the Group Files
Figure A Extracting the Group Files

Take a moment to investigate the various folders. The_rels folder contains a file named .rels, which contains XML information that defines the contents of the entire document. The information in the .rels file looks like the code in Figure B.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<Relationships xmlns="http://schemas.openxmlformats.org/
                      package/2006/relationships">
    <Relationship Id="rId3" Type="http://schemas.openxmlformats.org/
         officeDocument/2006/relationships/extended-properties"
         Target="docProps/app.xml"/>
    <Relationship Id="rId2" Type="http://schemas.openxmlformats.org/
         package/2006/relationships/metadata/core-properties"
         Target="docProps/core.xml"/>
    <Relationship Id="rId1" Type="http://schemas.openxmlformats.org/
         officeDocument/2006/relationships/officeDocument"
         Target="word/document.xml"/>
    <Relationship Id="rId4" Type="http://schemas.openxmlformats.org/
         officeDocument/2006/relationships/custom-properties"
         Target="docProps/custom.xml"/>
</Relationships>

The Target attribute in the file’s XML content provides references to the three main parts of the Word file: the application-specific properties, the core Office properties, and the document itself. It’s the docProps/core.xml part that the sample in this article manipulates.

When you look in the docProps and word folders, you’ll see that each contains various parts of the document (yes, the pieces that make up the document are really called "parts"). The docProps folder contains the XML files mentioned in the .rels file; the word folder contains the document, a folder (named rels) that contains another file (document.xml.rels) that lists the relations between the document and all its subsidiary pieces, a styles folder, and all the subsidiary parts (comments, settings, styles, and more).

Go to the docProps folder and open core.xml in a text editor. You’ll find XML that looks something like the code shown in Figure C. Modify one of the elements in this file, and then save the XML document.

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<cp:coreProperties xmlns:cp="http://schemas.openxmlformats.org/
                             package/2006/metadata/core-properties"
        xmlns:dc="http://purl.org/dc/elements/1.1/"
        xmlns:dcterms="http://purl.org/dc/terms/"
        xmlns:dcmitype="http://purl.org/dc/dcmitype/"
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <dc:title>Doc title </dc:title>
    <dc:subject>MSDN Magazine: Fix Word Properties</dc:subject>
    <dc:creator>Ken Getz</dc:creator>
    <cp:keywords>Articles</cp:keywords>
    <dc:description>These are Peter’s comments.</dc:description>
    <cp:lastModifiedBy>Ken Getz</cp:lastModifiedBy>
    <cp:revision>4</cp:revision>
    <dcterms:created xsi:type="dcterms:W3CDTF">
          2005-12-21T22:18:00Z
    </dcterms:created>
    <dcterms:modified xsi:type="dcterms:W3CDTF">
      2006-03-29T16:45:00Z
    </dcterms:modified>
    <cp:category>Sample</cp:category>
</cp:coreProperties>

You can easily modify the content of the Word document as well, simply by changing text within the XML file. To do this, browse to the word folder and open document.xml in a text editor. In the XML file, you’ll see a lot of XML content. Scroll down until you find the text you originally entered. Simply modify the text and save the XML file.

Close the document, browse to the root of the folder, right-click on the folder, and select Send To | Compressed (zipped) Folder from the context menu. Delete the folder that contained the decompressed file, and rename the newly created ZIP file to a file with the .docx extension.

When you open your document in Word, you’ll see the modifications you’ve just made. Check out the document properties, and you will see any changes you made there, as well.

A Word 2007 document is indeed the sum of its parts.

The concept behind the Open XML file format is discoverability. You should be able to programmatically dive into a file and find the "start" part (the document itself) and all the subsidiary information. Simply put, everything is out in the open. All the information you need about the document is stored in XML content inside the Open XML file.

Investigating the Sample Project

If you compare the form shown in Figure 1 to the sample shown in my March 2006 column, you'll see that the new version contains slightly fewer properties. The difference is that in Office 2007, the document properties have been split into multiple parts: app.xml, core.xml, and custom.xml. In an attempt to concentrate on the technology, as opposed to fixating on functionality, I am going to focus only on the properties stored in core.xml. These include Title, Creator, Subject, Keywords, Comments, and a few others. You can easily modify the code you'll find here to work with other properties and other document parts. (Note that in order to work with the sample form within Visual Studio 2005, you may find it necessary to first open the Data Sources window.)

To give the application a try, create a group of Word 2007 documents, and store them in a folder, possibly with subfolders. Run the sample application in Visual Studio 2005 and select the root folder for the documents. If you want to recurse into subfolders, check the Include Subfolders checkbox. Click Make Changes when you're ready to roll and stand back. The application walks through all the files it finds, doing its work in a background thread and reporting on its progress in the status bar. The form saves and restores settings, including all your choices within the form (See the previous article for information on the behavior of the form—that's not the point of this article.)

In order to work with the Open XML file formats in your own projects, you'll need to have WinFX installed on the computer (the sample requires at least the February 2006 CTP version I mentioned earlier) and you'll need to set a reference to the appropriate assembly within WinFX. When I wrote this sample, this step required adding a reference within Visual Studio 2005 by browsing to the C:\Program Files\Reference Assemblies\Microsoft\WinFx\v3.0 folder and selecting the WindowsBase.dll assembly. This assembly contains the classes from the System.IO.Packaging namespace that are required in order to manipulate the files.

From here, the sample boils down to investigating the code. Besides handling the parts and the relations between the parts, most of the code you'll find in this application (and in any application that manipulates Office 2007 files) centers around XML handling. In this case, the application's goal is to find the docProps/core.xml part, make the necessary modifications, and then save the part.

If you open the sample project and load the form's code into the code editor, you'll see that most of the code in the top half of the form's class hasn't changed since the previous version. All the code that deals with the form's properties, saving user settings, and running a background thread is identical to the previous sample. Near the top of the file, however, you'll find the first major change, which involves System.IO.Packaging:

Imports System.IO.Packaging

The System.IO.Packaging namespace makes it possible to work with the compressed Open XML file formats, taking advantage of classes within the namespace. Applications that work with Open XML files need to interact with the relations between parts—a large part of the functionality that the System.IO.Packaging namespace provides—and this application is no exception. You shouldn't count on any particular folder layout for Open XML files: you should always use the relationships described in the file to find the part you need. This application is interested solely in the docProps/core.xml part, so it doesn't need to investigate or create relations.

Near the middle of the file, you'll find a comment indicating that the remaining code is specific to Office 2007. Immediately below the comment, you'll find three constants and three class-level variables, as shown here:

Private Const corePropertiesSchema As String = _
    "http://schemas.openxmlformats.org/" & _
    "package/2006/metadata/core-properties"
Private Const dcPropertiesSchema As String = _
    "http://purl.org/dc/elements/1.1/"
Private Const corePropertiesRelationshipType As String = _
    "http://schemas.openxmlformats.org/" & _
    "package/2006/relationships/metadata/core-properties"

Private wdPackage As Package
Private corePropertiesPart As PackagePart
Private nsManager As XmlNamespaceManager

The constants provide information about the namespaces used by core.xml, and the relationship type you'll use to find core.xml. When searching for elements later on, using the XmlDocument class's SelectSingleNode method, you'll need this namespace information. The variable of type System.IO.Packaging.Package, wdPackage, maintains a reference to the entire package (that is, the .docx file). The System.IO.Packaging.PackagePart variable, corePropertiesPart, refers to the actual core.xml part, while the variable of type System.Xml.XmlNamespaceManager, nsManager, maintains the System.Xml.NameTable object containing information about prefixes and namespaces for the XML manipulation.

The GetWordCorePropertyPart procedure retrieves the core.xml part and stores its XML content in a System.Xml.XmlDocument object for later manipulation (see Figure 2).

Figure 2 GetWordCorePropertyPart Procedure (Close Figure 2)
Private Function GetWordCorePropertyPart( _
        ByVal docName As String) As XmlDocument

    wdPackage = Package.Open( _
         docName, FileMode.Open, FileAccess.ReadWrite)
    ' Get the main document part (document.xml).

    ' Use the package relations to find the core
    ' properties part. There’s only one of these,
    ' so you can exit the loop immediately upon
    ' finding the part:
    For Each relation As PackageRelationship In _
      wdPackage.GetRelationshipsByType(corePropertiesRelationshipType)
        Dim corePropertiesUri As Uri = _
            PackUriHelper.ResolvePartUri( _
                New Uri("/", UriKind.Relative), relation.TargetUri)
        corePropertiesPart = wdPackage.GetPart(corePropertiesUri)
        ' There’s only one of these, so get out now.
        Exit For
    Next

    ' Manage namespaces to perform Xml XPath queries.
    Dim nt As New NameTable()
    nsManager = New XmlNamespaceManager(nt)
    nsManager.AddNamespace("cp", corePropertiesSchema)
    nsManager.AddNamespace("dc", dcPropertiesSchema)

    ' Get the properties from the package.
    ' Load the XML in the part into an XmlDocument instance:
    Dim xDoc As New XmlDocument(nt)
    Using partStream As Stream = corePropertiesPart.GetStream()
        xDoc.Load(partStream)
    End Using
    Return xDoc
End Function

The code starts by opening the package, with read/write access. The System.IO.Packaging.Package WinFX class provides the necessary Open method:

wdPackage = Package.Open( _
  docName, FileMode.Open, FileAccess.ReadWrite)

To retrieve a part from a package, you must ask the document to use its .rels part (which contains relation information for the document and properties parts) to retrieve the URI (Uniform Resource Identifier) of the core properties part. The code calls the GetRelationshipsByType method, which returns a collection of PackageRelationship objects. (There's no way to directly index into the relationships this method returns—you must iterate through the collection, although you can exit after the first item.) The relationship provides a TargetUri property, and the code uses the PackUriHelper class's ResolvePartUri method to retrieve a Uri object relative to the root of the document. Given this Uri, the code can get the core properties part:

For Each relation As PackageRelationship In _
        wdPackage.GetRelationshipsByType(corePropertiesRelationshipType)
    Dim corePropertiesUri As Uri = _
        PackUriHelper.ResolvePartUri( _
            New Uri("/", UriKind.Relative), relation.TargetUri)
    corePropertiesPart = wdPackage.GetPart(corePropertiesUri)
    ' There's only one of these, so get out now.
    Exit For
Next

Now that you've got the PackagePart in memory, you need to focus on its XML content. Because the application must search for items within the XML, it needs an XmlNamespaceManager to pass to the XmlDocument.SelectSingleNode method. The code creates the namespace manager, adds the two namespaces that the core.xml part uses, and creates a new XmlDocument instance using the System.Xml.NameTable instance associated with the namespace manager:

Dim nt As New NameTable()
nsManager = New XmlNamespaceManager(nt)
nsManager.AddNamespace("cp", corePropertiesSchema)
nsManager.AddNamespace("dc", dcPropertiesSchema)
Dim xDoc As New XmlDocument(nt)

Finally the code calls the PackagePart.GetStream method to load the contents of the part into memory, loading the XmlDocument with the contents of the stream. It then returns the new XmlDocument instance, as shown here:

Using partStream As Stream = corePropertiesPart.GetStream()
    xDoc.Load(partStream)
End Using
Return xDoc

Clicking the button on the form triggers the DoWork event of the BackgroundWorker component, and the DoWork event handler calls the HandleDocs method to search for and handle the documents, as shown in the following:

Private Sub bgw_DoWork( _
 ByVal sender As System.Object, _
 ByVal e As System.ComponentModel.DoWorkEventArgs) _
 Handles bgw.DoWork

    ' Handle the documents
    ' in the specified folder.
    Try
        HandleDocs(folderLabel.Text, e)
    Catch ex As Exception
        MessageBox.Show(ex.Message)
    End Try
End Sub

The HandleDocs procedure (covered in the March 2006 column) loops through folders, recursing into subfolders if requested. For each matching file the procedure finds, the OnFileFound procedure that you see in Figure 3 is called.

Figure 3 OnFileBound Procedure (Close Figure 3)
Private Sub OnFileFound(ByVal fi As FileInfo)
    Try
        bgw.ReportProgress(0, fi.Name)

        Dim xmlDoc As XmlDocument = _
            GetWordCorePropertyPart(fi.FullName)

        ' Title is a special case:
        If useTitleCheckbox.Checked Then
            Dim title As String
            If useDocNameCheckBox.Checked Then
                title = Path.GetFileNameWithoutExtension(fi.Name)
            Else
                title = titleTextBox.Text
            End If
            SetWordCoreProperty(xmlDoc, "dc:title", title)
        End If

        ' Handle all the rest of the properties:
        SetProperty(useSubjectCheckBox, xmlDoc, _
            "dc:subject", subjectTextBox)
        SetProperty(useAuthorCheckBox, xmlDoc, _
            "dc:creator", authorTextBox)
        SetProperty(useCategoryCheckBox, xmlDoc, _
            "cp:category", categoryTextBox)
        SetProperty(useKeywordsCheckBox, xmlDoc, _
            "cp:keywords", keywordsTextBox)
        SetProperty(useCommentsCheckBox, xmlDoc, _
            "dc:description", commentsTextBox)
        SaveWordCorePropertyPart(xmlDoc)

    Catch ex As Exception
        MessageBox.Show("Unable to change properties: " & _
            ex.Message)
    Finally
        If wdPackage IsNot Nothing Then wdPackage.Close()
    End Try
End Sub

After reporting progress (displaying the file name in the status bar within the BackgroundWorker component's ProgressChanged event handler) and loading the core properties part as shown in the previously discussed procedure, the code sets the various properties stored within the part. The code handles the Title property specially because it may use the document file name instead of the value you've supplied. Each property setting ends up calling the SetWordCoreProperty procedure, as shown in Figure 4.

Figure 4 SetWordCoreProperty Procedure (Close Figure 4)
Private Function SetWordCoreProperty( _
  ByVal xmlDoc As XmlDocument, ByVal propertyName As String, _
  ByVal propertyValue As String) As String
    ' Set the value of the specified property, within the XML
    '  document. Return the current value of the property.
    ' Note that property names are case-sensitive.

    Dim retval As String

    Dim searchString As String = _
      String.Format("//cp:coreProperties/{0}", propertyName)
    Dim xNode As XmlNode = xmlDoc.SelectSingleNode( searchString, nsManager)
    If xNode Is Nothing Then
        xNode = xmlDoc.CreateElement(propertyName, corePropertiesSchema)
        xNode.InnerText = propertyValue
        xmlDoc.DocumentElement.AppendChild(xNode)
        retval = String.Empty
    Else
        ' Get the current value:
        retval = xNode.InnerText
    End If
    If xNode IsNot Nothing Then
        ' Now update the value:
        xNode.InnerText = propertyValue
    End If
    Return retval
End Function

Investigating the SetWordCoreProperty procedure verifies that most of the work that you'll do with the Open XML file formats boils down to XML manipulation. This procedure sets up an XPath search string to look for an element with the property name, located beneath the cp:coreProperties element:

Dim searchString As String = _
  String.Format("//cp:coreProperties/{0}", propertyName)
Dim xNode As XmlNode = xmlDoc.SelectSingleNode( _
  searchString, nsManager)

Note that the XmlNamespaceManager instance maps the "cp" prefix used in this corresponding namespace within the part, and that you must supply the namespace prefix for each property you set. If the search turns up empty, the code creates the new node, using standard XmlDocument manipulations. If the search finds the node, it sets the variable that contains the procedure's return value to the current contents of the element:

If xNode Is Nothing Then
    xNode = xmlDoc.CreateElement(propertyName, _
      corePropertiesSchema)
    xNode.InnerText = propertyValue
    xmlDoc.DocumentElement.AppendChild(xNode)
    retval = String.Empty
Else
    ' Get the current value:
    retval = xNode.InnerText
End If

Finally, the procedure sets the inner text for the element and returns the original value:

If xNode IsNot Nothing Then
    ' Now update the value:
    xNode.InnerText = propertyValue
End If
Return retval

After modifying all the properties, the OnFileFound procedure calls the SaveWordCorePropertyPart procedure, which saves the XML document back to the core.xml property part:

Private Sub SaveWordCorePropertyPart( _
  ByVal xdoc As XmlDocument)
    ' Save the properties XML back to its part.
    Using infoStream As Stream = _
      corePropertiesPart.GetStream(FileMode.Create, _
      FileAccess.Write)
        xdoc.Save(infoStream)
    End Using
End Sub

Conveniently, the PackagePart class provides a GetStream method, which makes for an easy way to save the contents of the XML document back to the part.

Wrapping Up

That's all the Open XML code you'll need to accomplish the task for this particular case.

However, late in the process of writing this column, I discovered that the Package class provides a PackageProperties property that allows access to the exact properties the core.xml part provides. This means that for this particular scenario, you don't need to modify the XML. If you need to set or retrieve a particular core property, you can simply use the Package.PackageProperties property. On the other hand, because the point of this column is to demonstrate how you can manipulate the Office 2007 file formats directly, I left the code as is. Feel free to modify it to use PackageProperties instead. But keep in mind that if you need to work with document or custom properties you'll need to use code similar to the code presented in this column.

If you want to become proficient with the file formats, you'll need to read the documentation on the schemas for the various parts. And as Office 2007 matures, more information will be available on the MSDN Office developer site.

For more information on the Open XML file formats and all the Office gossip you could want, drop by Brian Jones's blog or Kevin Boske's blog.

I've spent a great deal of time over the past few months writing code samples using the new file formats. Although I will attest that it's easier to write VBA (Visual Basic for Applications) code to perform the same tasks, the flexibility provided by the Open XML file formats, combined with the fact that any vendor can now write applications that read and write Word, Excel, and PowerPoint files, makes the concept a winner. (Of course, you can still use VBA to program the content, just as you could in previous versions.)

Start digging into the file formats, investigating the schemas and the content of the various parts. You'll also need to beef up your XML programming skills. We're on the verge of a new way of interacting with Office files and the new formats open up entirely new avenues of interaction.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a senior consultant with MCW Technologies, and a courseware author for AppDev (www.appdev.com). He is coauthor of ASP .NET Developers Jumpstart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2002), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


CLR Inside Out: Windows Vista Globalization Features

CLR Inside Out
Windows Vista Globalization Features
Shawn Steele

Contents
Custom Locales and Custom Cultures
Windows-Only Cultures and Regions
No More LCIDs?
.NET Encodings and Windows Code Pages
.NET Encoding Classes and Windows Code Page Functions
Windows Vista and Managed IDN Mapping APIs
Conclusion

Windows XP and the Microsoft .NET Framework both have APIs that support globalization. Windows Vista™ will further extend globalization support by introducing several new features.

When you begin to think about Windows Vista, you'll probably want to know how cultures in the .NET Framework compare to locales in Windows Vista, what the difference is between a managed encoding and a Windows®-based code page, and why Windows Vista has Internationalized Domain Name (IDN) risk mitigation functions that aren't available in managed code. This month's column will answer those and other questions about globalization features in the .NET Framework 2.0 and Windows Vista.

Custom Locales and Custom Cultures

The .NET Framework uses the term "culture" (or "region") to refer to what Windows refers to as a "locale." In the October 2005 issue of MSDN®Magazine, Michael Kaplan and Cathy Wissink discussed the need for user-defined culture data in the .NET Framework (see msdn.microsoft.com/msdnmag/issues/05/10/Globalization). Custom cultures provide such a mechanism for defining data for the CultureInfo and RegionInfo objects used across the machine. Using the new .NET Framework 2.0 CultureAndRegionInfoBuilder class, you can create new cultures or modify existing ones and register them for use on the machine.

All of the scenarios that require .NET Framework custom cultures can also apply to Windows Vista. For example, a company in the United States may prefer a particular short date format such as yyyy/mm/dd instead of the U.S. default (mm/dd/yyyy). In such a case, you can create a custom culture that replaces the existing en-U.S. culture data with this specific short date format. That customized culture format can then be installed on all machines in the company to support this corporate preference.

Another scenario is to derive a custom culture from another culture. For example, you could use an English language culture as a template to create a new English language culture for India. Many fields, such as month and day names, would remain the same, but by changing the currency symbol, region names, and other information you could create a culture appropriate for English in India.

For completely new cultures you could create a new custom locale entirely from scratch, specifying all of the region and culture data appropriate to that locale. When registered on the machine, all applications would then have access to this completely new locale by using the normal CultureInfo constructor.

.NET-based cultures and regions are comparable to locales in Windows. There is also a need for custom cultures for Windows locales. The .NET Framework 2.0 CultureAndRegionInfoBuilder fulfills these needs for Windows Vista, as well. Any custom cultures registered by the .NET Framework are automatically shared as custom locales in Windows Vista. This ability to use .NET-based custom cultures as Windows Vista custom locales gives you more flexibility.

In Windows Vista a custom locale created or modified by the .NET Framework 2.0 CultureAndRegionInfoBuilder can be selected by a user as his default locale. This gives legacy unmanaged applications access to custom locale data when it is the user's default (see Figure 1). The locale in Figure 2 requires a custom font and keyboard, but shows that even complex custom locale solutions can now be implemented.

Figure 1 Custom Locale in Windows Vista
Figure 1 Custom Locale in Windows Vista

Replacement .NET Framework cultures now become replacement Windows Vista locales. This allows users to modify the short time format or other values for a particular locale on the entire system. Those culture/locale changes will be respected by both managed and unmanaged applications. Modifying the data of existing locales can be fairly straightforward, as you can see in Figure 3.

Figure 3 Modifying Existing Locales (Close Figure 3)

Code
Imports System
Imports System.Globalization

Public Class Example
    Public Shared Sub Main()
        ‘ Show a date format before our changes
        Console.WriteLine("Before Modification:")
        Console.WriteLine(DateTime.Now.ToString("d", _
            New CultureInfo("en-US", False)))

        ‘ Make a replacement for en-US
        Dim carib As New CultureAndRegionInfoBuilder("en-US", _
            CultureAndRegionModifiers.Replacement)

        ‘ Modify the short date format
        Dim dtfi As DateTimeFormatInfo = carib.GregorianDateTimeFormat
        dtfi.ShortDatePattern = "yyyy/MM/dd"
        carib.GregorianDateTimeFormat = dtfi

        ‘ Make it available on this machine
        ‘ note that this impacts all applications
        carib.Register()

        ‘ Show a date format after our changes, specify no user
        ‘ override to demostrate that this comes from the custom
        ‘ culture, not a user override
        Console.WriteLine("After Modification:")
        Console.WriteLine(DateTime.Now.ToString("d", _
            New CultureInfo("en-US", False)))

        ‘ Remove the replacement
        CultureAndRegionInfoBuilder.Unregister("en-US")
    End Sub
End Class
Output
> ubc /r:sysglobl.dll example.ub
> example.exe
Before Modification:
2/13/2006
After Modification:
2006/02/13
Figure 1 Complex Custom Locale Solution
Figure 1 Complex Custom Locale Solution

Some locales may have multiple acceptable spellings or other user preferences dictated by regional dialects or other cultural differences. Occasionally the generic data provided by Windows Vista must choose one form which, although technically correct, may not be appropriate for all users in that locale. Users with expectations that differ from where the data was collected can now use the CultureAndRegionInfoBuilder to change both the .NET Framework and Windows Vista culture data to fit their preferences, even if it's a minority preference.

Microsoft® Locale Builder for Windows Vista will help users build these custom locales by providing a user-friendly tool that will allow even novices to customize their locale data. Even though it modifies Windows Vista locale data, the tool itself is written in C# and relies on the .NET Framework 2.0 CultureAndRegionInfoBuilder class to create these custom locales. See the sidebar "Some Custom Culture Gotchas" for more information.

Historically, Microsoft has shipped the most recent culture or locale data available, whether in the .NET Framework or Windows. Unfortunately, the data doesn't always stay the same. Cultures continue to be added, cultural changes can occur (such as the adoption of the Euro), governments can change standards, and cultural variations can come into favor. Since .NET and Windows have been shipped at different times, they are usually out of sync.

By using the CultureAndRegionInfoBuilder class to build custom cultures/locales, data can be shared between the .NET Framework 2.0 and Windows Vista, ensuring the consistency of culture data between managed and unmanaged applications.

Windows-Only Cultures and Regions

Microsoft continues to expand the list of locales supported by Windows and the .NET Framework. The .NET Framework 2.0 shipped with the locales that were available in Windows XP SP1. Since that time, Microsoft has continued to add locales to Windows. With Windows Vista, even more locales will be added. The .NET Framework 2.0 can access all locales that it finds in Windows by synthesizing a "Windows-only" culture and region from the data returned by Windows APIs. The .NET Framework 2.0 provides all of the data for this Windows-only culture by querying Windows for all values, including data formats, strings, sort behavior, and every other value related to this Windows-only culture or region.

One interesting difference between the Windows-only cultures and the built-in framework cultures is that Windows does not support region-neutral or language-neutral regions. Region-neutral cultures are cultures without a specific region. In the Framework, each specific culture, such as en-US, has a neutral parent culture such as en, which then has the Invariant locale as its parent. Language-neutral regions such as US do not exist in Windows.

In Windows, locales with only a language name do not exist. When the Framework cannot find a region-neutral parent for a specific Windows-only culture, the Invariant culture is used instead for that culture's parent. Managed applications need to use a specific full culture name to take advantage of these synthetic Windows-only cultures.

Region-only information is similarly unavailable to Windows and Windows-only regions. Microsoft recommends that users construct RegionInfo objects from a complete culture name (like en-US) rather than just a region name (like US). Using a full region name is also necessary to get a culturally appropriate language for the region name, such as United States for en-US and Estados Unidos for es-US. In some cases this difference in region names is a geopolitical issue, so a specific culture should always be used when possible.

The Windows-only cultures provide a way for the .NET Framework to support the same set of locales that the operating system supports. One caveat is that .NET installations on one machine may not have the same set of Windows-only cultures as another machine. For example, one machine could have a different version of Windows or service pack than another computer. The CultureInfo.GetCultures method can filter on the type of culture desired, and the CultureInfo.CultureTypes property will include the WindowsOnlyCultures value for Windows-only cultures.

No More LCIDs?

It's probably a little early to say that locale identifiers (LCIDs) are no longer needed, but I can imagine such a future. Windows has historically relied on the LCID to identify locales. From its inception, the .NET Framework has been more flexible, allowing the use of names to identify cultures. .NET still allows CultureInfo construction by LCID, but names are preferred, and with custom locales names are required.

With the advent of custom cultures, names have become even more important since LCIDs can only be assigned by Microsoft. This is as true for Windows Vista locales as for the .NET Framework. Since C code doesn't allow for overloading, Ex versions have been created to accept locale names for all of the National Language Support (NLS) APIs in Windows Vista. So GetLocaleInfoEx, EnumSystemLocalesEx, and so on have been created to allow querying locale data by name identifiers. These functions will allow native applications to access data in new .NET Framework custom cultures.

It is expected that custom cultures/locales will become very important to Windows and the .NET Framework. Ethnologue lists 6,912 languages in the world, with 162 spoken in the U.S. alone. Microsoft is adding over 60 new locales to Windows Vista on top of the 137 available for Windows XP. This includes new support for emerging markets in Africa, South America, and Asia, increased support for regional languages in established markets in Europe, and support for Spanish (U.S.). However, sourcing and maintaining data for all the language and country combinations in the world would be very difficult, particularly in emerging markets that are newer to technology. Even identifying a required locale set is challenging since everyone's locale is important to them and it is difficult to get a clear perspective on the needs of so many different groups.

For interoperability with legacy applications, data, and standards, Microsoft has added LCIDToLocaleName and LocaleNameToLCID functions to Windows Vista. Combined with the CultureInfo.LCID property, applications should have the tools necessary to migrate to locale names even in mixed managed and unmanaged code environments.

While the LCID will need to remain for legacy applications and other cases in the near future, culture names (and locale names) are now predominant. The .NET Framework 2.0 has enabled a plethora of custom cultures/locales, and the Microsoft Locale Builder for Windows Vista (shown in Figure 4) will make it even easier to create and distribute custom locale solutions. Users can select their own name identifiers without having to rely on Microsoft to assign an LCID, and they can use the new name-based APIs to query their data. The process started by .NET is actually hastening the demise of the LCID.

Figure 4 Microsoft Locale Builder for Windows Vista
Figure 4 Microsoft Locale Builder for Windows Vista

Remember, though, that without LCIDs you need to use culture and locale names. Microsoft recommends getting culture and region data by full culture name, like en-US. Also, keep in mind that Microsoft .NET culture names respect the syntax of the Internet Engineering Task Force (IETF) standard names. When names include language and region, they are of the form language-region (for example, en-US, fr-FR). Some locales also include the script used in the locale. For example, there are two locales for Uzbek-Uzbekistan: uz-Cyrl-UZ for the Cyrillic script locale and uz-Latn-UZ for the Latin script locale. Additionally, some IETF names have unusual codes. For example, en-029 is the IETF code for the Windows English (Caribbean) locale. Unmanaged applications should be using GetLocaleInfoEx and related functions that accept names.

.NET Encodings and Windows Code Pages
Some Customer Culture Gotchas

The custom cultures/locales feature allows for a plethora of variations to the default CultureInfo and locale data. New locales can be added, causing different machines to have different sets of supported locales. Replacement locales can be created, causing unexpected field values. Users should be aware that installing replacement locales will hide the default data shipped by Microsoft for all users, and that the Regional Options is the right way to customize many fields for individual users.

Application developers should be aware that any culture or locale data can change, avoiding dependencies on particular locale data for particular locales. For persisting data, the best choices are binary formats, application-provided format strings, and the use of CultureInfo.InvariantCulture as an IFormatProvider. You need to consider problems such as what happens if mm/dd/yyyy suddenly becomes dd-mm-yy. Will your application still work? What about the other culture and region data fields? What if "January" becomes "yraunaJ" for en-US? The list goes on.

Code pages, encodings, and character sets map character data from one form, such as Unicode, to another, such as ASCII. Most code pages support only a subset of the characters supported by the .NET Framework or Windows. Unfortunately, there are variations in how those subsets are implemented. The easiest way to avoid unexpected conversion behavior is to use standard Unicode encodings such as UTF-8 or UTF-16.

The .NET Framework uses encoding and the Encoding class to refer to the mapping of the internal UTF-16 Unicode representation to a different byte representation. Windows uses the term code page to refer to the mapping tables used by its APIs to do the same thing.

Most encodings only handle a subset of Unicode. ASCII, for example, can only encode 128, or 0.1 percent, of the more than 95,000 characters in Unicode. Depending on the mechanism used, the remaining 99.9 percent of Unicode would be ignored, converted to another character like ?, or result in errors. Other single-byte code pages, like Windows-1252, only raise that figure to about .25 percent. Some common double-byte code pages in Windows still only accurately convert about 10 percent of the characters in Unicode.

Another issue with encodings is that new standards have been created and existing code pages have been extended, corrected, or otherwise changed. Some encodings have unassigned code points that can be used by applications for their own behavior. As the character set of those encodings increase, sometimes those code points are reused for new characters. Sometimes a private use region becomes well-known and accepted as part of the standard. Those changes can break applications or data that depend on the old behavior.

Historically, different operating systems, vendors, and applications have had unique algorithms for mapping characters between various encodings, often because different versions of encoding standards were adopted by different vendors. In other situations vendors may have had needs that differed slightly from existing standards. Sometimes there are even unintentional bugs in encoding behavior.

These differences between encoding implementations can cause data to be corrupted. Fortunately, those corruptions are usually in edge cases or different revisions of standards; however, that does not help people using data that needs to use a corrupted character. Round-tripping from Unicode to an encoding and back to Unicode is usually less lossy if the same API set is used for both encoding and decoding the data.

All of these complexities mean that even though a character may encode and decode correctly using one set of APIs, it may not correctly round-trip when it shifts between platforms. Everyone has probably seen square boxes instead of rich quotes on well-known Web sites or in popular publications. This is often because the computer, application, or browser client hasn't interpreted the character data in the intended way, even if it thought it was using the correct encoding. Plus, in some languages it is common for certain characters to become corrupted on Web pages because of inconsistent interpretations between vendors, changes to a code page standard, or mislabeled character set tags.

At Microsoft, the philosophy has been that if a user saves a document using a set of APIs, he should be able to get the same data back with the same set of APIs. Historically, some code pages have encoded characters not yet assigned by a standard. Other standards have evolved to include new characters, and Microsoft has provided varying levels of support for other standards. Newer standards may be more correct than those provided by Microsoft, but the extension of a standard must be balanced with the cost of causing customers to lose data they saved several years ago.

.NET Encoding Classes and Windows Code Page Functions

In Windows there are two primary ways to convert character data: MultiByteToWideChar/WideCharToMultiByte and MLang. In the .NET Framework there are the Encoding classes. All three have their own unique quirks, which means that none of them can read the data written by the others with 100 percent accuracy.

MLang has several features designed to work with international code pages. In general, MultiByteToWideChar is preferred to calling the MLang conversion functions. MLang is used by Microsoft Internet Explorer® for its code page detection and conversion. The .NET Framework Encoding class provides similar conversion functionality, although some code points may convert differently in some code pages. For example, in ASCII, MLang maps characters with the eighth bit set by just dropping the eighth bit, which could allow spoofing. Internally, MLang converts from Unicode to a specific code page and then tries to adapt that result to the target code page. Its algorithms sometimes cause random mappings, particularly of undefined sequences in a code page. The .NET Framework has cleaner implementations without the random mappings. The .NET Encoding class is also more consistent about restoring the state to a baseline when writing multiple sequences. MLang could cause inadvertent mode changes, such as reading single byte data as double byte data. For more information on MLang, see msdn.microsoft.com/workshop/misc/mlang/mlang.asp.

MultiByteToWideChar and WideCharToMultiByte are the workhorses of the Windows APIs, and are used internally by most of Windows. They are also called by MLang. Some code pages differ from the .NET Encoding class versions, particularly the iso-2022-xx code pages. MultiByteToWideChar and WideCharToMultiByte also expect data to be converted in its entirety, whereas the .NET Framework provides the Encoder and Decoder classes. Calling MultiByteToWideChar from unmanaged C code is fairly straightforward, as you see here:

int   length;
WCHAR wcTemp[BUFFER_SIZE];
BYTE* bytes = "Input byte string";
Length = MultiByteToWideChar(CP_UTF8, 0, bytes, -1,
                             wcTemp, BUFFER_SIZE);

The Encoding classes are used by managed code and include features not present in the Windows APIs. The Encoder and Decoder classes can be used to provide encoding or decoding of large buffers in multiple steps. Additionally the .NET Framework 2.0 has support for fallbacks, which provide user-specified behavior for unknown or corrupted characters.

Here's a simple UTF-8 example using the default behavior:

Dim bytes() As Byte = Encoding.UTF8.GetBytes( _
    "Convert this string to UTF8")

Here's one using the constructor that selects an exception fallback to throw on illegal input data:

Dim bytes() As Byte = New UTF8Encoding(true, true).GetBytes("Convert " & _
    "this string and throw on illegal characters like " & ChrW(&hD800))

And here's an example using custom fallback to replace unknown characters with {unknown}:

Dim utf8 As Encoding = CType(Encoding.UTF8.Clone(), Encoding)
utf8.EncoderFallback = New EncoderReplacementFallback("{unknown}")
Dim bytes() As Byte = utf8.GetBytes( _
   "Convert and replace illegal " & ChrW(&hD800) & " with {unknown}")

.NET-based applications that communicate with native Windows-based applications, or with applications on other platforms for that matter, need to be aware that encoding differences can occur. The preferable way to exchange data is to use Unicode, since that is the exact set of the characters supported by Windows or .NET. Unfortunately, some protocols rely on other standards and a conversion is necessary. As more and more applications move to Unicode, these cases should be limited. Fortunately, many newer or updated standards have some provision for Unicode.

In recognition of the importance of Unicode in both Windows and the .NET Framework, the .NET Framework 2.0 team put serious effort into optimizing the UTF-8 and UTF-16 Encodings. In nearly all cases UTF-8 is now significantly faster than most of the non-Unicode encodings. Similar effort was spent on Windows Vista to improve the speed of UTF-8 used by the MultiByteToWideChar functions. Incompatibilities between unmanaged and managed character conversion algorithms can be avoided by using these Unicode encodings.

Windows Vista and Managed IDN Mapping APIs

Windows Vista has also added unmanaged versions of the Unicode Normalization and IDN mapping APIs. As mentioned briefly in the aforementioned article by Michael Kaplan and Cathy Wissink, the IDN standard exposes a potential for spoofing attacks by way of the confusion some characters cause. Windows Vista has added APIs that can help mitigate some of those concerns. These APIs are not available to managed code, so developers of such code should be aware of potential security issues.

The primary concern with the IDN standard is that the character repertoire of a host name is increased greatly. That increases the possibilities of domain name spoofing through homographs. A homograph is a word that looks like another word but isn't. A common example is the creation of a paypal.com URL that is spelled pаypal.com instead of paypal.com. Obviously, it is very difficult for a human to see that the first "a" here in pаypal.com has been changed to a Cyrillic "а", code point U+0430, from a Latin "a", code point U+0061.

With the addition of these "new" characters to Unicode, phishing attacks could use homographs to spoof legitimate domains and gather personal information from users, to slander a legitimate business by making a similar site with a homograph name, or do other nefarious things. It should be noted however that these attacks aren't new to IDN: rnicrosoft.com looks a lot like microsoft.com. In some fonts I (i) looks like l (L) or 1 (one). Phishing attacks don't require an exact homograph name, either. The Web sites paypal.com.safesecure.netor mine.com/paypal.com/safe/logon can also trap some users.

One approach to mitigate homographs is to detect the permissible scripts in a domain name. The idea is that domain names are probably restricted to a single script such as Latin or Cyrillic, but not both. So if a domain name is encountered that uses more scripts than expected, it is considered suspect. In some locales it is common to see Latin or other scripts mixed with the native script, such as in the name of a local division of a foreign company. In other cases, a single script can have many characters that appear similar, particularly if rendered in small fonts in a browser address bar. So this mitigation is not perfect, but can be used as part of a broader strategy.

The script detection approach is the primary idea in the Windows Vista mitigation APIs. These APIs allow for the detection and comparison of the scripts present in an IDN. The new unmanaged Windows Vista GetStringScripts function will return a list of scripts present in a string. GetLocaleInfoEx can now query for LOCALE_SSCRIPTS to discover the scripts expected for a particular locale, and VerifyScripts compares the two previous results to determine if the expected scripts are within the set expected by the locale (see Figure 5).

Figure 5 Verifying Scripts (Close Figure 5)

Code
Imports System
Imports System.Runtime.InteropServices

Public Class Example
    Public Shared Sub Main()
        TestScripts("en-US", "microsoft.com")
        TestScripts("en-US", "m" & ChrW(&h456) & "crosoft.com")
        TestScripts("ru-RU", "m" & ChrW(&h456) & "crosoft.com")
        TestScripts("ja-JP", ChrW(&h65E5) & ChrW(&h672C) & _
                             ".microsoft.com")
    End Sub

    Public Shared Sub TestScripts(
            ByVal locale As String, ByVal test As String)
        Dim result(79) As Char
        Console.WriteLine("Checking scripts for string {0}" & _
            " in locale {1}:", test, locale)

        ‘ Get the scripts expected for this locale
        Dim size As Int32 = GetLocaleInfoEx(locale, LOCALE_SSCRIPTS, _
            result, 80)
        If size = 0 Then
            ‘ Unexpected Error reported in GetLastError()
            Console.WriteLine("Unexpected Error: {0}", _
                Marshal.GetLastWin32Error())
            Return
        End If
        Dim localeScripts As New String(result, 0, size)
        Console.WriteLine("Expected Locale Scripts: {0}", localeScripts)

        ‘ Get the scripts found in this string
        size = GetStringScripts(0, test, test.Length, result, 80)
        If size = 0 Then
            ‘ Unexpected Error reported in GetLastError()
            Console.WriteLine("Unexpected Error: {0}", _
                Marshal.GetLastWin32Error())
            Return
        End If
        Dim stringScripts As New String(result, 0, size)
        Console.WriteLine("Found Scripts in String: {0}", stringScripts)

        ‘ Are the scripts in the string allowable for that locale?
        Dim allowedBool As Int32 = VerifyScripts(0, localeScripts, _
            localeScripts.Length, stringScripts, stringScripts.Length)
        Dim allowed As Boolean = (allowedBool <> 0)
        Console.WriteLine("Are scripts allowed?     {0}", allowed)

        ‘ Are they allowed if Latn; is explicitly allowed?
        allowedBool = VerifyScripts(VS_ALLOW_LATIN, localeScripts, _
            localeScripts.Length, stringScripts, stringScripts.Length)
        allowed = (allowedBool <> 0)
        Console.WriteLine("Allowed if Latn allowed? {0}", allowed)
        Console.WriteLine()
    End Sub

    <DllImport("kernel32.dll", _
     CharSet:=CharSet.Unicode, SetLastError:=True)> _
    Private Shared Function GetLocaleInfoEx( _
        ByVal lpLocaleName As String, ByVal LCType As Int32, _
        ByVal lpLCData As Char(), ByVal cchData As Int32) As Int32
    End Function

    <DllImport("kernel32.dll", _
     CharSet:=CharSet.Unicode, SetLastError:=True)> _
    Private Shared Function GetStringScripts(ByVal dwFlags As Int32, _
        ByVal lpString As String, ByVal cchString As Int32, _
        ByVal lpScripts As Char(), ByVal cchScripts As Int32) As Int32
    End Function

    <DllImport("kernel32.dll", _
     CharSet:=CharSet.Unicode, SetLastError:=True)> _
    Private Shared Function VerifyScripts(ByVal dwFlags As Int32, _
        ByVal lpLocaleScripts As String, _
        ByVal cchLocaleScripts As Int32, ByVal lptestScripts As String, _
        ByVal cchTestScripts As Int32) As Int32
    End Function

    Private Const LOCALE_SSCRIPTS As Int32 = &h6C
    Private Const VS_ALLOW_LATIN As Int32 = &h01
    Private Const GSS_ALLOW_INHERITED_COMMON As Int32 = &h01
End Class
Output
> ubc example.ub
> example.exe
Checking scripts for string microsoft.com in locale en-US:
Expected Locale Scripts: Latn;
Found Scripts in String: Latn;
Are scripts allowed?     True
Allowed if Latn allowed? True

Checking scripts for string mcrosoft.com in locale en-US:
Expected Locale Scripts: Latn;
Found Scripts in String: Cyrl;Latn;
Are scripts allowed?     False
Allowed if Latn allowed? False

Checking scripts for string mcrosoft.com in locale ru-RU:
Expected Locale Scripts: Cyrl;
Found Scripts in String: Cyrl;Latn;
Are scripts allowed?     False
Allowed if Latn allowed? True

Checking scripts for string .microsoft.com in locale ja-JP:
Expected Locale Scripts: Hani;Hira;Kana;
Found Scripts in String: Hani;Latn;
Are scripts allowed?     False
Allowed if Latn allowed? True

Since the homograph concerns with IDN were raised after the .NET Framework 2.0 was released, there are not any managed equivalents to these Windows Vista APIs. As mentioned earlier, remember that these APIs cannot solve all of the security concerns regarding IDN names by themselves. Trusted managed applications could use P/Invoke to call these native Windows Vista APIs.

Conclusion

Windows and the .NET Framework support globalization needs in complementary ways. Each has its own strengths, but together they can provide excellent customer solutions—for example, the shared custom culture and locale behavior in Windows Vista. Each also approaches the tasks in different ways, as seen with the encoding and code page behavior. Therefore, a certain amount of care needs to be taken when creating solutions that utilize both managed and unmanaged functionality.

If you'd like to do further reading, check out the Unicode Standard, Michael Kaplan's blog, and my blog. Additional information can be found at the Microsoft Global Development and Computing Portal, Dr. International's columns, and in the Windows Vista and Microsoft .NET Framework 2.0 SDKs.

Send your questions and comments to clrinout@microsoft.com.

Shawn Steele is a Software Design Engineer at Microsoft. He works on the Windows operating system and also the .NET Framework. He is primarily responsible for culture/locale data (naDev tlhInganpu' tu'lu', Qapla'!), code pages/encodings, normalization, and IDN. Shawn blogs at blogs.msdn.com/shawnste and can be reached at shawnste@microsoft.com.


Data Points: Designing Reports with SQL Server Reporting Services 2005

Data Points
Designing Reports with SQL Server Reporting Services 2005
John Papa

Code download available at: DataPoints2006_06.exe (206KB)
Contents
Report Definition Language
Data Sources and Deployment
Data Preparation Considerations
DataSets and Parameters
Multivalued Parameters
Expressions
Wrapping Up

Many applications require some degree of integration with a reporting tool. A good solution, SQL Server™ Reporting Services 2005, provides Web-based reports and can be integrated into both Windows® Forms and Web-based applications. Plus, it introduces several enhancements over its predecessor, Reporting Services 2000, that make designing effective reports simpler than ever.

In this month's column I will show you how to design reports effectively using Reporting Services 2005, which can manipulate, sort, and group rowsets of SQL data. However, there are times when it is beneficial to prepare the data first using SQL Server, so I'll look at what to consider when that's necessary. I will also show how multivalued parameters (new to Reporting Services 2005) work and how you can easily parse the multivalued parameters using a SQL Server function. In addition, I'll discuss Report Definition Language (RDL), designer tools, and more.

Report Definition Language

The XML-based RDL defines how all aspects of a report should behave, including (but not limited to) the data source, query, report parameters, report layout, external code, and color schemes (read the RDL spec at microsoft.com/sql/technologies/reporting/rdlspec.mspx). The schema for RDL is well defined and is openly available, allowing third-party applications to interact with and even build reports to work with Reporting Services.

Microsoft offers a few tools that can build reports including Visual Studio® 2005 and Business Intelligence Studio, which comes with SQL Server 2005.

Since RDL defines all aspects of a report, you can also open the RDL file using an XML editor or a basic text editor such as Notepad. I have often found it very useful to set some properties of a report in one of the designer tools like Business Intelligence Studio and then look at what changes were made to the RDL. For example, I recently wrote several reports that all had the same set of report parameters. The reports were very different in almost every way except that a subset of the report parameters were exactly the same. The parameters had a predefined list of acceptable values, captions, and default value settings. I was able to copy and paste this small portion of functionality from one report to the others within the RDL. Figure 1 shows what I mean. In this particular case I was able to save time by editing the RDL directly. In other situations, viewing the RDL can be helpful to see what code has been injected into a report.

Figure 1 Report Parameters Shown in RDL (Close Figure 1)
<ReportParameters>
  <ReportParameter Name="startDate">
    <DataType>DateTime</DataType>
    <DefaultValue>
      <Values>
        <Value>=Today</Value>
      </Values>
    </DefaultValue>
    <Prompt>Start Date</Prompt>
  </ReportParameter>
  <ReportParameter Name="endDate">
    <DataType>DateTime</DataType>
    <DefaultValue>
      <Values>
        <Value>=DateAdd("d", 7, Today())</Value>
      </Values>
    </DefaultValue>
    <Prompt>End Date</Prompt>
  </ReportParameter>
  <ReportParameter Name="status">
    <DataType>Integer</DataType>
    <DefaultValue>
      <Values>
        <Value>=1</Value>
      </Values>
    </DefaultValue>
    <Prompt>Status</Prompt>
    <ValidValues>
      <ParameterValues>
        <ParameterValue>
          <Value>1</Value>
          <Label>Online</Label>
        </ParameterValue>
        <ParameterValue>
          <Value>2</Value>
          <Label>Offline</Label>
        </ParameterValue>
        <ParameterValue>
          <Value>3</Value>
          <Label>Archived</Label>
        </ParameterValue>
      </ParameterValues>
    </ValidValues>
  </ReportParameter>
</ReportParameters>

Data Sources and Deployment

Once you create a reporting solution and add your first report to it using either Visual Studio or Business Intelligence Studio, the usual next step is to prepare the data for the report. To get the data into a report you must create a report data source and link it to the report. The data source tells your report what server, database, and credentials to use in connecting to the database. You can add a new data source by right-clicking on the Shared Data Sources node in the Solutions Explorer window and choosing Add New Data Source. The data source will ask you to provide the connection string information to use and a name (I named mine SampleDataSource). Later, when you create a DataSet for your report to use, you will specify the name of the data source.

The data source you use in your development environment will likely differ from the one you use in production. If so, you would not want your local data source to be deployed to the production server and thus have production reports pointing to your development database server. Fortunately, report designer tools allow you to specify whether or not you want your data sources to be deployed to the target server and overwrite the data sources on the target server.

Figure 2 shows the properties dialog of the sample project, which indicates that I have chosen not to overwrite the target server's data sources. (Also note that in my sample project, I am using the debug configuration and have specified that my target server is my local installation of Reporting Services 2005. You can create multiple configurations, which is what I do when I want to be able to deploy to different servers based on need.)

Figure 2 Deployment Options
Figure 2 Deployment Options

Data Preparation Considerations

When designing reports you can use either embedded SQL queries within the report or you can invoke a stored procedure. The decision is often predicated on your development department's standards; however, my preference is to use stored procedures, if for nothing else, for the ability to make minor modifications to the SQL without having to open a report.

Reports generally gather data from a relational database to be displayed in a more user-friendly manner, such as tables, hierarchical drill downs, matrices, and charts. In most cases the data stored in the database is not stored the way that users want to view it. This means that the transformation of the data into the desired format to be shown on a report must be done somewhere. One way is to let the report format the data the way the user wants it. Another method is to massage the data in a stored procedure before it gets to the report. There are pros and cons to each approach and sometimes a combination is best.

In general, grouping and other aggregation features are performed faster in a database management system than in a report. On the other hand, providing the data in a raw format to the report offers the flexibility of changing the grouping within the same report.

The upside to filtering in the report is that you can grab a lot of data once from the database and then apply various filters in the report without hitting the database again. The problem with this technique is that if you grab too much data you are really slowing down the initial load of the report only to show a fraction of the data you return. A general rule of thumb I use with report filters is to avoid them and instead first try filtering in the stored procedure (or in the SQL). Only if the performance tests prove this to be too slow do I opt to use report filters.

DataSets and Parameters

The Data tab is where you set up the DataSet(s) that a report will use. My sample report, which is named BasicProcParametersReport.rdl, is going to get a list of customers and their orders, so I created the stored procedure shown in Figure 3.

Figure 3 Gathering the Data (Close Figure 3)
CREATE PROCEDURE prCustomerOrders
    @startDate DATETIME, @endDate DATETIME
AS
SELECT
        c.CustomerID, c.CompanyName,
        c.City, c.ContactName,
        o.OrderID, o.OrderDate,
        od.UnitPrice, od.Quantity,
        od.UnitPrice * od.Quantity AS ExtendedPrice
FROM
        Customers c
INNER JOIN
        Orders o ON c.CustomerID = o.CustomerID
INNER JOIN
        [Order Details] od ON o.OrderID = od.OrderID
WHERE
        o.OrderDate BETWEEN @startDate AND @endDate
ORDER BY
        c.CompanyName,    o.OrderDate

I create a DataSet on the Data tab and tell it to use the data source I already created (SampleDataSource) and the stored procedure prCustomerOrders. When you enter this information on the query tab of the Dataset dialog window and click OK, the report designer grabs the fields that the stored procedure returns as well as a list of the parameters required by the stored procedure. It automatically creates report parameters to match them and associates the stored procedure's parameters with the report parameters, as shown in the Parameters tab of the Dataset dialog window (see Figure 4).

Figure 4 Dataset Parameters
Figure 4 Dataset Parameters

There are many ways you can set the values of parameters. There is no rule that says you have to have a report parameter that matches the parameter in your SQL. This comes in handy if you want to use an existing stored procedure but you do not want the report users to be able to choose the values for all of the parameters. If you want to predefine the value of a parameter to a stored procedure, you can go to the parameters tab of the Dataset dialog window and enter the value manually.

The technique I like to use when I want to set a value of a parameter to a stored procedure without allowing the user of the report to change it is to keep the matching report parameter but set it to hidden. You can do this from the Layout tab by right-clicking and choosing Report Parameters from the menu. From the Report Parameters dialog window you will see the two parameters that were previously created to match the stored procedure's parameters (see Figure 5). From here you can set the default value for a parameter using an expression such as the following, which calculates the date one year from today:

    =DateAdd("yyyy", 1, Today())

Figure 5 Report Parameters
Figure 5 Report Parameters

You can also specify a list of values users can choose from. This can be a hardcoded list (non-queried) or a queried list derived from a stored procedure. I could create a Country report parameter that is populated from a query. To do this, I'd create a second DataSet called CountryDataSet that gets a list of countries to populate the parameter's dropdown list. For a working example, see SimpleReport.rdl included in the download on the MSDN®Magazine Web site. SimpleReport.rdl gets a list of customers from the specified country dropdown list parameter. The list of countries is retrieved from a second DataSet defined in the report.

Multivalued Parameters

The Report Parameters dialog window lets you set basic properties like the name of the parameter, its data type, and the caption. You can also set whether or not the parameter is hidden from the user and if the parameter allows multiple values. This latter feature is new to Reporting Services 2005 and is very useful for letting a user select more than one value from a list.

For example, I could modify prCustomerOrders from Figure 3 to accept a list of customers (from a dropdown list with checkboxes) as a parameter and then set up the corresponding report parameter to be a multivalued parameter (MVP). The MVP lets the user select more than one value and then passes those values to the stored procedure as a comma-delimited string. Of course, if you use the MVPs, you need to account for parsing out the parameter values in your stored procedure.

The example report for MVPs in the downloadable code is named MVPReport.rdl. This report uses the stored procedure prCustomerOrders_SelectCustomers which, in addition to accepting date parameters, also accepts a list of CustomerID fields. To parse out the list of CustomerIDs I created a SQL Server function named Split (also included in the code download). The Split function accepts a delimited list of values and a delimiter. It parses the list of values and returns a table with a single column called Item. In my example, each row of the resulting table will contain a CustomerID from the comma-delimited list. Another option, if using SQL Server 2005, would be to use the common language runtime (CLR) and a managed stored procedure. However, this Split function will work in either SQL Server 2000 or SQL Server 2005. I added the following line of code to the stored procedure's WHERE clause to match any of the customer IDs that are passed in the list:

AND c.CustomerID IN
    (SELECT Item FROM dbo.Split(@customerIDs, ','))

Figure 6 shows the results of the multivalued parameter report (MVPReport.rdl) after selecting several customers from its MVP. Notice that you can select one or more customers from the list or you can choose the Select All option.

Figure 6 Multivalued Parameters
Figure 6 Multivalued Parameters

Expressions

The expression builder in Reporting Services 2000 left a lot to be desired. While you could type in a slew of Visual Basic® .NET-compliant methods and operations to create expressions, there was no palette of functions and operations to choose from. Reporting Services 2005 addresses this issue as it has more features built into its expression builder, including a categorized list of available functions and operators that can be used to build expressions. The expression builder also provides IntelliSense® in its code window to assist with building expressions.

Figure 7 shows the expression builder with an expression that calculates the date one year from today. Basic logic, including if statements, can be performed through its code window. However, if the logic gets more complicated than a line or two, consider encapsulating the code into either the report's code window or in an external assembly.

Figure 7 Expression Builder
Figure 7 Expression Builder

Expressions are quite valuable and offer a great deal of flexibility to reports. For example, they can be used to set report properties such as the values of textboxes, tooltip text, visibility of controls, toggle appearance, fonts, and countless other properties. They can also be used to set a parameter's default value, the color of the bars on a chart control, or even a value to group the data in a report. In fact, most places inside a report can accept an expression to set its value. Anywhere you see the function button (circled in Figure 5) you can set an expression. In addition, you can enter an expression just about anywhere you can type in a value even when there is no explicit button to open the expression builder.

Wrapping Up

Before designing the report using the Visual Studio or Business Intelligence tools, it pays to do a little design preparation. The tools available in Reporting Services 2005 expose a wealth of functionality that can help you build your applications faster and with more flexibility. In the next installment of Data Points, I will demonstrate how to use report links, how to embed code within a report, and how to access custom external .NET assemblies from a report. I will also explore how to make more advanced reports using the chart control and some of the other controls available for Reporting Services 2005.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a Senior .NET Consultant with ASPSOFT and a baseball fanatic who spends most of his summer nights rooting for the Yankees with his family and his faithful dog, Kadi. John, a C# MVP, has authored books on ADO, XML, and SQL Server, and can often be found speaking at industry conferences such as VSLive or blogging at codebetter.com/blogs/john.papa.


Test Run: Five Ways to Emit Test Results as XML

Test Run
Five Ways to Emit Test Results as XML
James McCaffrey

Code download available at: TestRun2006_06.exe (164KB)
Contents
The Problem
Technique 1: Using the XmlTextWriter Class
Technique 2: Using the XmlDocument Class
Technique 3: Using the XPathDocument Class
Technique 4: Using the DataSet Class
Technique 5: Using the XmlSerializer Class
Summing It Up

The use of XML files in software testing has steadily increased over the past few years. Test case data, test harness configuration information, and test result data are now stored as XML. Recently I was writing some .NET test automation that saved test results to XML storage, and in conversations with colleagues I discovered that there are a variety of opinions on the best way to emit XML files. I set out to enumerate the various ways that exist to emit XML in a .NET testing environment and to understand the pros and cons of each technique. The "best" technique depends primarily on how your test result data is stored in memory, the type of processing you are performing on the data, and to some extent the style of programming you prefer.

In the sections that follow, I will demonstrate how to emit test result data to an XML file using five different techniques: XmlTextWriter, XmlDocument, XPathDocument, DataSet, and XmlSerializer. Each is based on a different Microsoft® .NET Framework class and its associated methods. These five techniques are by no means the only ways to emit test results to XML, but they are fundamental and will be a valuable addition to your .NET skill set.

The Problem

Let's begin by examining the basis for each of the five. Suppose you are developing a dice game application named Boatzee, as shown in Figure 1. Behind the scenes of the application there is a library named BoatzeeLib that contains classes such as a Die class to represent a single die, and a Hand class to represent a set of five dice.

Figure 1 Application Under Test
Figure 1 Application Under Test

One of the methods in the BoatzeeLib library you need to test is Hand.GetHandType, which returns a member of an enumeration. For example, if a Hand object, h, has the five dice shown in Figure 1 (2, 6, 6, 1, 6), then h.GetHandType returns an enumeration with string representation "ThreeOfAKind".

Now suppose you have test case data stored in a simple text file named TestCases.txt:

    0001:3:3:3:3:3:Boatzee
    0002:5:5:2:2:2:FullHouse
    0003:6:5:4:3:1:SmallStraight

Each line of the test case data represents a single test case. The first field is a test case ID, the second through sixth fields are inputs to the Hand constructor, and the last field is an expected result when Hand.GetHandType is called. Of course, in a real-life situation, your test case data would probably be stored in an XML file, and you would have many hundreds of test cases. I'm using a text file for test case storage just to avoid confusion with the XML test result files. I will demonstrate several ways to emit test results to an XML file using this test case data. Each of the techniques will produce the XML test results file shown in Figure 2.

Figure 2 XML Test Result File (Close Figure 2)
<?xml version="1.0" encoding="utf-8" ?>
<TestResults>
  <result id="0001">
    <input>[3] [3] [3] [3] [3]</input>
    <expected>Boatzee</expected>
    <passfail>Pass</passfail>
  </result>
  <result id="0002">
    <input>[5] [5] [2] [2] [2]</input>
    <expected>FullHouse</expected>
    <passfail>Pass</passfail>
  </result>
  <result id="0003">
    <input>[6] [5] [4] [3] [1]</input>
    <expected>SmallStraight</expected>
    <passfail>Pass</passfail>
  </result>
</TestResults>

Technique 1: Using the XmlTextWriter Class

The most traditional technique is to use the XmlTextWriter class. The basic plan is to write XML elements (and attributes) to the output stream as the data for the XML elements becomes available to the test harness. The example code is shown in Figure 3. When run, the output will be as shown in Figure 2.

Figure 3 Using XmlTextWriter (Close Figure 3)
using(StreamReader sr = new StreamReader("..\\..\\TestCases.txt"))
{
    string line, id, expected;
    string[] tokens;

    XmlTextWriter xtw = new XmlTextWriter("..\\..\\TestResults1.xml",
        Encoding.UTF8);
    xtw.Formatting = Formatting.Indented;
    xtw.WriteStartDocument();
    xtw.WriteStartElement("TestResults");

    while((line=sr.ReadLine()) != null)
    {
        tokens = line.Split(‘:’);
        id = tokens[0];
        expected = tokens[6];
        Hand h = new Hand( new Die(int.Parse(tokens[1])),
            new Die(int.Parse(tokens[2])),
            new Die(int.Parse(tokens[3])),
            new Die(int.Parse(tokens[4])),
            new Die(int.Parse(tokens[5])));

        xtw.WriteStartElement("result");
        xtw.WriteAttributeString("id", id);
        xtw.WriteElementString("input", h.ToString());
        xtw.WriteElementString("expected", expected);
        xtw.WriteElementString("passfail",
            h.GetHandType().ToString() == expected ? "Pass" : "*FAIL*");
        xtw.WriteEndElement(); // </result>
    }

    xtw.WriteEndElement(); // </TestResults>
    xtw.Close();
}

I start coding by creating an XmlTextWriter object, then use the WriteStartDocument method to write an XML declaration, and the WriteStartElement method to write the <TestResults> root element to the output stream. Next I iterate through each line in the test case data file and parse out the test case ID, the five inputs to the Die constructor, and the expected result.

Inside the main test loop, I instantiate a Hand object using the test case inputs. Then I use WriteStartElement to create a <result> tag, and the WriteAttributeString method to insert the test case ID into the <result> tag as an attribute. I use the WriteElementString method to add the entire <input> and <expected> elements. Then I call the Hand.GetHandType method under test, fetch the actual return value, and compare it to the expected value to determine a pass/fail test result.

Once I know the pass/fail result, I can create the entire <passfail> element and add the closing </result> tag to the output stream. After all the test cases have been executed, I add the final </TestResults> tag using the WriteEndElement method. Finally, I close the XmlTextWriter object, which causes any internally buffered output to be written to the file.

Emitting test results to XML with XmlTextWriter has a traditional, pre-.NET feel. As each piece of test result data becomes available, you add it to the output stream. Using XmlTextWriter is simple and straightforward, and is appropriate when test result data is streamed to your test harness. Because the various Write methods of the XmlTextWriter class are forward-only mechanisms, using XmlTextWriter to emit test results is not usually appropriate in situations where the data to write is not streamed. For example, in a testing situation where a particular test result cannot be fully determined until after some subsequent test case executes, you are probably better off using one of the other four techniques presented in this column. Compared to the other techniques covered here, XmlTextWriter operates at a lower level of abstraction, meaning it is up to you as a programmer to keep track of where you are in the output XML file, and write elements and attributes correctly. Instead of using the methods in the XmlTextWriter class, you could simply write all output using a WriteLine method, along the lines of the following, but you would lose a lot of error-checking, power, flexibility, and clarity:

Console.WriteLine("<?xml version ='1.0' ?>");
Console.WriteLine("<TestResults>");
...

Technique 2: Using the XmlDocument Class

The second way to emit test result data to an XML file is to use the XmlDocument class. The plan is to create a complete in-memory representation of the entire XML results file, and then write it to a file (see Figure 4). When run, the output will be exactly the same as the output from the previous XmlTextWriter technique, as shown in Figure 2.

Figure 4 Using XmlDocument (Close Figure 4)
using(StreamReader sr = new StreamReader("..\\..\\TestCases.txt"))
{
    string line, id, expected;
    string[] tokens;

    XmlDocument xd = new XmlDocument();
    XmlElement root = xd.DocumentElement;
    XmlDeclaration xdec = xd.CreateXmlDeclaration("1.0", "utf-8", null);
    xd.InsertBefore(xdec, root);

    XmlElement TestResults = xd.CreateElement("TestResults");

    while((line=sr.ReadLine()) != null)
    {
        tokens = line.Split(‘:’);
        id = tokens[0];
        expected = tokens[6];
        Hand h = new Hand( new Die(int.Parse(tokens[1])),
            new Die(int.Parse(tokens[2])),
            new Die(int.Parse(tokens[3])),
            new Die(int.Parse(tokens[4])),
            new Die(int.Parse(tokens[5])));

        XmlElement resultEl = xd.CreateElement("result");
        resultEl.SetAttribute("id", id);

        XmlElement inputEl = xd.CreateElement("input");
        inputEl.InnerText = h.ToString();
        resultEl.AppendChild(inputEl);

        XmlElement expectedEl = xd.CreateElement("expected");
        expectedEl.InnerText = expected;
        resultEl.AppendChild(expectedEl);

        XmlElement passfailEl = xd.CreateElement("passfail");
        passfailEl.InnerText = h.GetHandType().ToString() == expected ?
            "Pass" : "*Fail*";
        resultEl.AppendChild(passfailEl);

        TestResults.AppendChild(resultEl);
    }
    xd.InsertAfter(TestResults, xdec);
    xd.Save("..\\..\\TestResults2.xml");
}

I start by instantiating an XmlDocument object. Next I obtain a reference to the virtual root element which is returned from the XmlDocument.DocumentElement property. The real purpose of this root element is to provide an anchor with which to insert an XmlDeclaration object with the InsertBefore method (because an XML declaration must come before the root element). Next I create the actual <TestResults> root element using the CreateElement method. After these preliminaries, I process the test cases exactly as I do in the previous section: read a test case, parse the fields, call the method under test, and determine a pass/fail result.

My next step is to create the <input>, <expected>, and <passfail> elements using the CreateElement method, and supply values using the XmlElement.InnerText property. I use the SetAttribute method to insert the test case ID attribute into each <result> element. Each element is added to the in-memory XmlDocument object by using the AppendChild method or the InsertAfter method, as appropriate.

The XmlDocument class is modeled on the W3C XML Document Object Model (XML DOM) and may have a different feel to it than non-XML .NET Framework classes that you are familiar with. Using the XmlDocument class to emit test results to XML is appropriate if you need to construct result data in a nonsequential manner, as you can use InsertBefore, InsertAfter, AppendChild and other methods to place data anywhere in the in-memory XML document representation.

Using XmlDocument to emit test results is also appropriate if you are already using XmlDocument objects (for test case data, for example) and want to maintain a consistent feel in your test code. Because using XmlDocument builds the entire XML file in memory, using it here may not be a good choice if your XML result set is very large.

Let me note that in discussions with my colleagues, there was some minor confusion about the role of the XmlDataDocument class. It is derived from the XmlDocument class and is intended for use in conjunction with DataSet objects. So, in this example, I could have used the XmlDataDocument class but would not have gained any significant advantage by doing so.

Technique 3: Using the XPathDocument Class

The third technique uses the XPathDocument class. The basic plan is to start with an XPathDocument, which is an in-memory representation of an XML document, and use an XslTransform object in conjunction with an XSL stylesheet template to emit results. The following example emits XML using XPathDocument:

XPathDocument xpd = new XPathDocument("..\\..\\TestResultsOrig.xml");
XslTransform xslt = new XslTransform();
xslt.Load("..\\..\\TransformSheet.xsl");
XmlTextWriter xtw = new XmlTextWriter("..\\..\\TestResults3.xml",
  Encoding.UTF8);
xtw.Formatting = Formatting.Indented;
xtw.WriteStartDocument();
xslt.Transform(xpd, null, xtw, null);
  xtw.Close();

The associated XSL stylesheet is shown in Figure 5. When both of these are run, the resulting XML file will be the same as the one you saw in Figure 2. Before I explain the mechanics of the XPathDocument technique, let me first describe a few scenarios you may encounter.

Figure 5 XSL Stylesheet for XPathDocument Technique (Close Figure 5)
<xsl:stylesheet
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
  version="1.0">

  <xsl:template match="TheTestResults">
    <TestResults>
      <xsl:apply-templates select="aresult" />
     </TestResults>
  </xsl:template>

  <xsl:template match="aresult">
    <result>
      <xsl:attribute name="id"><xsl:value-of select="@id"/>
           </xsl:attribute>
       <xsl:apply-templates select="theinput" />
       <xsl:apply-templates select="expected" />
       <xsl:apply-templates select="passfail" />
    </result>
  </xsl:template>
  <xsl:template match="theinput">
    <input><xsl:value-of select="." /></input>
  </xsl:template>

  <xsl:template match="expected">
    <expected><xsl:value-of select="." /></expected>
  </xsl:template>

  <xsl:template match="passfail">
    <passfail><xsl:value-of select="." /></passfail>
  </xsl:template>

</xsl:stylesheet>

An XPathDocument object is read-only; this means that you will never directly create an XPathDocument of test results from a test harness. The only practical way to get an XPathDocument into memory is to load it from an existing XML file or stream when you call the XPathDocument constructor. The purpose of an XPathDocument is to provide a highly efficient means of searching an XML file using XPath syntax. So the situations in which you emit test result data to XML using XPathDocument are very limited. The most common testing scenario I have encountered goes like this. You create an XML test results file using one of the other four techniques presented in this column (XmlTextWriter, XmlDocument, DataSet, or XmlSerializer). Later, you find that you must perform some analysis of the original XML test results file where search-efficiency is important, and you also find that the format of the original XML file is not precisely what you need. So, you read the original XML file into memory as an XPathDocument object, perform your read-only analysis, and finally emit the XPathDocument to a new XML file with the desired format.

Now let me explain the mechanics of using an XPathDocument object. First I create an XPathDocument object by passing an existing XML file to the XPathDocument constructor. Next I create an XslTransform object and pass an existing XSL stylesheet to the constructor. Creating the XSL stylesheet is the key to the whole technique. Once again, I want the new XML file to look exactly like the one shown in Figure 2. As I just described, you only need to use this technique if your original XML results file is not in the correct format. Here I assume that file TestResultsOrig.xml looks like this:

<?xml version="1.0" encoding="utf-8"?>
<TheTestResults>
  <aresult id="0001">
    <theinput>[3] [3] [3] [3] [3]</theinput>
    <expected>Boatzee</expected>
    <whenrun>05/12/2006</whenrun>
    <passfail>Pass</passfail>
  </aresult>
   ...
</TheTestResults>

In other words, the root node of the original XML file is <TheTestResults> instead of the desired <TestResults>, the original file has <aresult> instead of <result>, <theinput> instead of <input>, and a <whenrun> element, which is not wanted. You must construct an XSL stylesheet to act as a mapping template from the original XML file to the desired XML file. I start building up the stylesheet by looking at the innermost elements in the original XML file that actually hold data I want to keep. In this example, those are the <theinput>, <expected>, and <passfail> elements. For each of these elements I create a matching template which identifies the original element, specifies the new element name, and assigns a value to the new element name like this:

<xsl:template match="theinput">
  <input><xsl:value-of select="." /></input>
</xsl:template>

Notice that I've placed these <xsl:template> tags at the bottom of my XSL stylesheet, so I am building my stylesheet from the bottom up. After doing this for each innermost data element, I move up one level of nesting. Here, that is the <aresult> element:

<xsl:template match="aresult">
  <result>
    <xsl:attribute name="id"><xsl:value-of
        select="@id"/></xsl:attribute>
      <xsl:apply-templates select="theinput" />
      <xsl:apply-templates select="expected" />
      <xsl:apply-templates select="passfail" />
  </result>
</xsl:template>

I specify the match criteria in the original XML file, and specify what to place inside the target XML file—in this case, the results of the innermost templates. The @ character in @id instructs the parser to retrieve the value of an attribute named id. I finish by moving up to the highest level of nesting, which is the root node in this example:

<xsl:template match="TheTestResults">
  <TestResults>
    <xsl:apply-templates select="aresult" />
  </TestResults>
</xsl:template>

If you haven't created XSL stylesheets before, you may find the process a bit bizarre at first. However, after you make one or two stylesheets, you'll get used to it. After constructing the XSL stylesheet, emitting XML results from an XPathDocument is easy. I create an XmlTextWriter object, and call the Transform method of the XslTransform class.

There is a completely different approach to emitting test results from an XPathDocument to an XML file. As I mentioned, XPathDocument objects exist to promote efficient searching. Searching is most often performed using an XPathNodeIterator object. You can traverse through each node of an XPathDocument object using an XPathNodeIterator object and the MoveNext method, get a reference to each node as you traverse, and then write each node, modifying if necessary, using an XmlTextWriter object. This technique requires much more coding than using an XslTransform object, but if you want to avoid XSL stylesheets, this offers a good alternative approach.

Technique 4: Using the DataSet Class

The basic plan this time is to create an in-memory DataSet object, store all test result data into the DataSet, then emit the entire DataSet to an XML file. The example code, shown in Figure 6, will yield the same XML file as that produced by all the other techniques presented here.

Figure 6 Using DataSet (Close Figure 6)
using(StreamReader sr = new StreamReader("..\\..\\TestCases.txt"))
{
    string line, id, expected;
    string[] tokens;

    DataSet ds = new DataSet("TestResults");
    DataTable dt = new DataTable("result");
    dt.Columns.Add("id");
    dt.Columns.Add("input");
    dt.Columns.Add("expected");
    dt.Columns.Add("passfail");

    dt.Columns["id"].ColumnMapping = MappingType.Attribute;

    ds.Tables.Add(dt);

    while((line=sr.ReadLine()) != null)
    {
        tokens = line.Split(‘:’);
        id = tokens[0];
        expected = tokens[6];
        Hand h = new Hand( new Die(int.Parse(tokens[1])),
            new Die(int.Parse(tokens[2])),
            new Die(int.Parse(tokens[3])),
            new Die(int.Parse(tokens[4])),
            new Die(int.Parse(tokens[5])));

        object[] values = new object[4];
        values[0] = id;
        values[1] = h.ToString();
        values[2] = expected;
        values[3] = h.GetHandType().ToString() == expected ?
            "Pass" : "*FAIL*";

        ds.Tables[0].Rows.Add(values);
    }
    ds.WriteXml("..\\..\\TestResults4.xml");
}

The key to this technique is to set up a DataSet object with a format that maps to your desired XML file format. First I create a DataSet object called TestResults, which will become the XML root element. Next I create a DataTable object with the name result. Each table name will become a parent element name when the DataSet is emitted to XML. Next I add columns named id, input, expected, and passfail. Each column name will become a data-element name or data-attribute name. By default, column names become element names. In order to force a column name into becoming an attribute, as I want to do with the test case ID value, I use the DataColumn.ColumnMapping method and set its MappingType property to MappingType.Attribute. After setting up the DataTable object, I add it into my DataSet object with the DataTables.Add method.

Once the DataSet is ready, I process my test cases just as with the XmlTextWriter and XmlDocument techniques: read test case data, parse, call method under test, and determine pass/fail result. I store each test case id, input, expected, and passfail value into an object array, and then pass that array to the DataRow.Add method to insert the result data into the DataSet object. After all test cases have been processed, emitting the test result data to an XML file is simply a matter of calling the DataSet.WriteXml method.

Using this DataSet technique is most appropriate when your test harness naturally uses a DataSet to store test results. For example, storing test results into a DataSet is particularly useful if you need to perform processing of you results—if your results are in a DataSet, you can easily filter and process aggregate results such as the number of test cases of a particular type that failed.

Using the DataSet technique to emit test results to XML is often not a good choice if you have a predetermined XML format you must conform to, and that format is deeply nested. Each nested level of an XML file maps to a DataTable object. If, for example, your target XML format looked something like this:

<TestResults>
  <result>
    <alpha>
      <beta>
        <data>5</data>
      </beta>
    </alpha>
  </result>
  etc.
</TestResults>

then you would need a column named data in a DataTable named beta, and you'd need a second DataTable named alpha, and also a third DataTable named result. These three DataTable objects would have to be connected through DataRelation objects.

Technique 5: Using the XmlSerializer Class

The last fundamental way to emit test result data as an XML file I'll discuss is to use the XmlSerializer class. The basic plan is to store test results in memory in an array or ArrayList object, and then serialize the entire object to an XML file. The example code is shown in Figure 7. When this is run, the XML file will be the same as that produced by all the other techniques in this column.

Figure 7 Using XmlSerializer (Close Figure 7)
using(StreamReader sr = new StreamReader("..\\..\\TestCases.txt"))
{
    string line, id, expected;
    string[] tokens;

    XmlSerializer s = new XmlSerializer(typeof(TestResults));
    XmlTextWriter xtw = new XmlTextWriter("..\\..\\TestResults5.xml",
        Encoding.UTF8);
    xtw.Formatting = Formatting.Indented;

    TestResults testResults = new TestResults();

    while((line=sr.ReadLine()) != null)
    {
        tokens = line.Split(‘:’);
        id = tokens[0];
        expected = tokens[6];
        Hand h = new Hand( new Die(int.Parse(tokens[1])),
            new Die(int.Parse(tokens[2])),
            new Die(int.Parse(tokens[3])),
            new Die(int.Parse(tokens[4])),
            new Die(int.Parse(tokens[5])));

        result r = new result();
        r.id = id;
        r.input = h.ToString();
        r.expected = expected;
        r.passfail = h.GetHandType().ToString() == expected ?
            "Pass" : "*Fail*";

        testResults.Results.Add(r);
    }

    s.Serialize(xtw, testResults);
    xtw.Close();
}

Before I write the serialization code, I have to set up two utility classes to hold my test result data. The first class, named result, holds an individual test result:

public class result
{
    [XmlAttribute]
    public string id;

    [XmlElement("input")]
    public string input;

    public string expected;
    public string passfail;
}

Notice that I decorate the id field with an [XmlAttribute] attribute. This instructs the common language runtime (CLR) to interpret the id field as an attribute when an XmlSerializer object serializes a result object. I don't pass in a name argument to XmlAttribute so the id field will be serialized with name id. I decorate the input field with an [XmlElement] attribute. This is the default interpretation so I could have left [XmlElement] out altogether; I put it in so you can see how to rename element fields if necessary.

My second utility class, which is called TestResults, is a collection of result objects:

public class TestResults // becomes root node
{
    [XmlElement(ElementName = "result", Type = typeof(result))]
    public ArrayList Results = new ArrayList();
}

The class is really just a wrapper around an ArrayList object. The class name, TestResults, becomes the root element in the resulting XML file after serialization. I decorate the ArrayList field with an [XmlElement] attribute to instruct the CLR that each item in the ArrayList contains a result type, and that each item should produce an XML element named result.

When you are using the XmlSerializer technique, any field you want to decorate must be public. In a testing scenario this is usually not too much of a problem, but if you need to serialize private fields, you can create public get and set properties, and then decorate those properties.

Using XmlSerializer to emit test result data to an XML file is most appropriate when your test harness naturally requires a class that encapsulates test results. XML serialization is an elegant mechanism but it is sometimes tricky to set up the storage classes so that you get a particular XML format after serialization. As a general rule, you are better off letting the requirements of your particular testing situation determine the design of your storage classes. You can use serialization on those classes and then, if your resulting XML file format is not quite what you want, use an XSL transform to get to the desired format. For example, the previous TestResults class definition is somewhat artificial. A more natural design is:

public class TestResults // becomes root node
{
  [XmlArray("TestResults")]
  [XmlArrayItem(Type=typeof(result))]
  public ArrayList Results = new ArrayList();
}

The attribute decorations to the ArrayList object tell the CLR to interpret the ArrayList as an array-derived type where each item is a result object. With this class, the previous code will produce an XML file like this:

<?xml version="1.0" encoding="utf-8"?>

<TestResults xmlns:xsd="http://www.w3.org/2001/XMLSchema"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <TestResults>
        <result id="0001">
            <input>[3] [3] [3] [3] [3]</input>
            <expected>Boatzee</expected>
            <passfail>Pass</passfail>
        </result>
        ...
    </TestResults>
</TestResults>

There is nothing at all wrong with this format; it just doesn't exactly match the format of the other four techniques presented in this column.

XML serialization tends to have slower performance than the other four techniques presented here, so serialization may not be appropriate if performance is a major concern.

Summing It Up

XML data files are a key component of the Microsoft .NET development environment. When testing in a .NET environment, you will often want to save test result data to an XML file. I have described five different ways to do this. Using XmlTextWriter is most appropriate when your test result data becomes available as a stream. Using XmlTextWriter is least appropriate when you need to perform some aggregate analysis of your test results. Using XmlDocument is useful when your test result data must be constructed nonsequentially. The XmlDocument technique, along with all techniques other than XmlTextWriter, is not a good choice if your result data is very large. The XPathDocument technique will be your least commonly used approach because it is useful only when you start with an in-memory XPathDocument object and you need to emit XML results in a different format. Using a DataSet object to emit XML is most appropriate when your test harness naturally stores result data into a DataSet, such as when you need to aggregate analysis of your results. Using a DataSet object is not appropriate when you have a required XML format that is deeply nested. The XmlSerializer technique is useful when your test harness has one or more classes that store test result objects, but is less useful when performance is a primary concern.

You can also combine these techniques in many ways. For example, if your test result data is stored in a DataSet object, you can iterate through the object's DataTable and write to a file using an XmlTextWriter object. Or if your test results are stored in an ArrayList object, you can iterate through the list and build up an XmlDocument object in memory, and then emit the document to file.

The code in this column was designed to work with the .NET Framework versions 1.1 and 2.0. If you are working in a strictly .NET Framework 2.0 environment, you can take advantage of some neat new features. For example, the DataTable class now has many methods that previously were available only to the DataSet class, including a WriteXml method. Additionally, a DataTable object can now be serialized by itself, so you aren't forced to create a DataSet with a single table to serialize test result data. There is also support for reading and writing of XML where a single table or XML element is nested inside two or more parent tables.

.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Cutting Edge: A Provider-Based Service for ASP.NET Tracing

Cutting Edge
A Provider-Based Service for ASP.NET Tracing
Dino Esposito

Code download available at: CuttingEdge2006_06.exe (190KB)
Contents
Tracing in ASP.NET
The <trace> Element
Connecting to Diagnostic Listeners
Taking Control of the Trace
The MyTracer HTTP Module
Tracing as a Provider-Based Service
Creating a Section in Web.config
The Trace Provider Interface
Devising the Service Class
Wrapping the Standard Tracer
A Richer ASP.NET Tracer

When it comes to catching programming errors, the debugger is a developer's best friend. ASP.NET tracing, however, is a nice complement to the debugger and shouldn't be overlooked. It enables your ASP.NET code to emit messages during processing, offering valuable information that can prove quite useful for detecting problems.

The two techniques are a bit different. Using a debugger is an inherently interactive technique that relies on being able to pause execution and examine current state. Tracing is less intrusive, simply tracking information being emitted by the code (similar to the classic "printf" style of debugging). At the end of the execution, you take the output produced by the tracing activity and analyze it.

The ASP.NET tracing subsystem exposes an API through which any control, page, or component can add its own trace output. You can output any information available at run time that makes sense to you.

In ASP.NET, tracing is tightly integrated with the page and the runtime pipeline. You can enable tracing by turning on an attribute in individual pages. To enable tracing for all pages in an application, you change a value in the configuration file. (More on this in a moment.) This means it's possible, for example, to enable tracing for an application after it has been deployed.

In this month's column, I'll briefly review ASP.NET tracing basics and discuss what's new in ASP.NET 2.0. Finally, I'll explore ways to customize the appearance of traced text in ASP.NET pages.

Tracing in ASP.NET

Tracing is built right into the infrastructure of ASP.NET as a subsystem that provides applications with diagnostic information about individual requests. Through tracing, you can analyze data communicated between a browser and a Web server, including request details, timing information, server variables, cookies, headers, session state, and so on.

When tracing for a page is enabled, ASP.NET can display this information in one of two ways: it can append diagnostic information to the page's output, and it can render the information through a built-in trace viewer application. To see trace information appended to the bottom of a particular page, you simply set the Trace attribute of the @Page directive to true, like so:

<%@Page trace="true" ...  %>

Note that in this case the trace information is sent as part of the page and as such it displays through the browser. This is not an issue when you're creating and testing the page. However, it becomes a nasty bug if you forget to reset a Trace attribute in one of your pages before deployment, thereby allowing the world to see this (potentially sensitive) information.

For the most part, the content and layout of trace information is fixed, but page and control authors can customize the output to some extent. In ASP.NET 2.0, the trace output is divided into 12 distinct sections, as outlined in Figure 1.

Figure 1 Trace Output Sections (Close Figure 1)

Section Name | What It Displays
Request Details | Information about the current request, such as status code, verb, session ID, and timestamp
Trace Information | System- and user-defined messages sorted by time or category
Control Tree | The hierarchy of the controls in the page, including each corresponding view state size
Session State | All the items stored in Session
Application State | All the items stored in the Application intrinsic object
Request Cookies Collection | Name and content of all cookies attached to the request (ASP.NET 2.0 only)
Response Cookies Collection | Name and content of all cookies attached to the response (ASP.NET 2.0 only)
Headers Collection | Name and content of all request headers
Response Headers Collection | Name and content of all headers attached to the response (ASP.NET 2.0 only)
Form Collection | Name and content of all input fields packed in the HTTP response
Querystring Collection | Name and content of all query string parameters
Server Variables | Name and content of all server variables available

In ASP.NET 1.x there was only one Cookies Collection object and it included both request and response cookies. There was also only one Headers Collection, and it included both request and response headers.

You can use the methods of the TraceContext class to emit specific messages and information in the Trace Information section. You use the following code:

' In a Page-derived type
Trace.Write("Your message")

' Anywhere in an ASP.NET application
HttpContext.Current.Trace.Write("Your message")

An instance of the TraceContext class is exposed to developers through the Trace property of the HttpContext class and is mirrored by the Trace property on the Page class.

The message is composed in the trace output based on the time it is executed. For example, if you emit a message in the Page_Load event of a page, the message will appear as shown in Figure 2. The TraceContext class also features a Warn method. Warn differs from Write in that it renders its text in red. Both the Write and Warn methods have a few overloads:

' Write overloads
Public Sub Write(String)
Public Sub Write(String, String)
Public Sub Write(String, String, Exception)

' Match Warn overloads
Public Sub Warn(String)
Public Sub Warn(String, String)
Public Sub Warn(String, String, Exception)

The first overload writes the specified text in the Message column (see Figure 2). The second overload accepts the name of the category and the message. The category name, which can be used to sort trace information, can be any string that makes sense to the application and the message. The third overload adds an extra Exception object in case the message is tracing an error. In this case, the text in the Message column also includes the exception's message. No HTML formatting is accepted in the trace output as the text is escaped before rendering to the browser.

Figure 2 Trace Output in an ASP.NET Page
Figure 2 Trace Output in an ASP.NET Page

The TraceContext class also features a couple of properties: IsEnabled and TraceMode. IsEnabled is a Boolean property that gets and sets whether tracing is on. TraceMode property gets and sets the order in which the traced rows will be displayed in the page. The property is of type TraceMode—an enumeration that includes values such as SortByCategory and SortByTime. You can enable tracing programmatically by setting IsEnabled. In ASP.NET 2.0, you can also use the new property, TraceEnabled, on the Page class. This property just wraps Trace.IsEnabled.

The <trace> Element

As mentioned, most of the tracing attributes can be controlled from within the configuration file. There is an option that allows you to capture trace information without emitting text in the page served to the browser. This feature is extremely useful for tracking data inconsistencies, monitoring the flow, and asserting conditions on applications already deployed in a production environment. The configuration section of interest is <trace>, which looks like this in ASP.NET 2.0:

<trace
   enabled="true|false"
   localOnly="true|false"
   mostRecent="true|false"
   pageOutput="true|false"
   requestLimit="integer"
   traceMode="SortByTime|SortByCategory"
   writeToDiagnosticsTrace="true|false" />

The various attributes are detailed in Figure 3.

Figure 3 Attributes of the <trace> section (Close Figure 3)

Section Name | What It Indicates
enabled | Whether tracing is enabled through the trace.axd viewer. This attribute alone is not sufficient to append trace information to all pages. The default is false.
localOnly | Whether the trace.axd viewer should be available only on the local Web server. The default is true.
mostRecent | The tracing system only maintains tracing information on a specific number of requests, as determined by requestLimit. When mostRecent is true, information from newer requests overwrites that from older requests when the limit has been reached. The default is false, meaning that trace data is recorded only until the limit is reached, causing subsequent traces to be lost (ASP.NET 2.0 only).
pageOutput | Whether trace output is rendered inside each page of the application. If false, trace output is only accessible through the trace.axd viewer. The default is false.
requestLimit | The number of trace requests to store on the server. The default is 10; the maximum is 10,000.
traceMode | The order in which trace information is displayed: sorted by category or by time. The default is SortByTime.
writeToDiagnosticsTrace | Whether trace output should be forwarded to any .NET trace listeners that are registered. The default is false (ASP.NET 2.0 only).

ASP.NET also supports application-level tracing through a built-in trace viewer tool—trace.axd. Note that the enabled attribute of the <trace> section activates this tool, but doesn't activate trace text in each page. This was done intentionally to let you to turn on tracing in production without affecting the output of pages served to users.

You might want to leave tracing statements in your published application but turn off Trace attributes in individual pages and disable tracing in the web.config file. The application won't incur noticeable overhead in normal conditions, and this leaves you the option to turn on tracing at any time by tweaking the web.config file as follows:

<trace enabled="true" pageOutput="false" />

Once tracing has been enabled for the application, each page request routes all the page-specific trace information to the viewer. Trace.axd (shown in Figure 4) is a system-defined HTTP handler that retrievs the internal cache of trace information and displays it in a Web interface. You can access the trace viewer by requesting trace.axd from the root application directory. You can also directly access the trace information of a particular request by adding the id=XXX parameter to the query string. XXX indicates the 0-based index of the requests where 0 represents the oldest.

Figure 4 Trace Viewer
Figure 4 Trace Viewer

In ASP.NET 1.x, only 10 requests are traced by default (the default value of requestLimit in <trace>). In ASP.NET 2.0, you can keep the 10 most recent, overwriting older traces (or whatever value is specified by requestLimit).

All the trace information is cached inside the TraceContext class using a DataSet object. An instance of the TraceContext class is created upon the first initialization of the HttpRuntime object for the application's AppDomain. As a result, when trace.axd is enabled, the more trace information you maintain, the more you pay in terms of memory consumption.

A new feature of ASP.NET 2.0 lets you subscribe to the TraceContext.TraceFinished event, which is raised when all request information has been captured. An event handler for TraceFinished is provided with a TraceFinishedEventArgs, which exposes a collection of TraceRecord objects. Each of these stores the information from an individual ASP.NET trace message and any associated data, allowing you to capture all of this information and do with it as you please, showing it on the page, storing it to your own database, and so on.

Connecting to Diagnostic Listeners

Tracing in ASP.NET is somewhat orthogonal to tracing in the .NET Framework. The Systems.Diagnostics namespace defines two classes, Trace and Debug, whose methods are used to trace the code. The Trace and Debug classes are essentially identical and work on top of more specialized modules known as listeners. A listener collects and stores messages in the Windows event log, a text file, or some other similar file. Each application can have its own set of listeners; all registered listeners receive all emitted messages. The tracing subsystem of ASP.NET 2.0 has been enhanced to include support for forwarding ASP.NET tracing information to any registered .NET trace listeners. You enable this feature by turning on the writeToDiagnosticsTrace attribute in the <trace> section of the web.config file, like so:

<trace enabled="true" pageOutput="false"
       writeToDiagnosticsTrace="true" />

Setting the writeToDiagnosticsTrace attribute is not enough, though. You also need to register one or more listeners for the Web application. Use the <trace> section under the <system.diagnostics> section to do that, as shown here:

<system.diagnostics>
  <trace autoflush="true">
    <listeners>
      <add name="myListener"
           type="System.Diagnostics.TextWriterTraceListener"
           initializeData="c:\myListener.log" />
    </listeners>
  </trace>
</system.diagnostics>

There are two important things to note here. First, only text written through either Write or Warn is forwarded to the listeners. That is, the listener won't store a copy of the standard ASP.NET trace information; the information tracked is limited to the contents of the Trace Information section. The second thing is that you can write custom listeners to send trace information to the medium of your choice—for example, a SQL Server database. All you have to do is inherit a class from System.Diagnostics.TraceListener and override a few methods. Next, you register the new class with the web.config file of the Web application you plan to trace. (For more information on writing custom TraceListener types, see msdn.microsoft.com/msdnmag/issues/06/04/CLRInsideOut.) One interesting thing to note is that System.Web.dll now includes a TraceListener-derived type, WebPageTraceListener, whose job is to serve as the counterpart to writeToDiagnosticsTrace. When an instance of WebPageTraceListener is added to the trace listeners collection, messages written to the .NET diagnostics trace will also be written to the ASP.NET trace.

Finally, you should be aware that in ASP.NET 2.0 a new <deployment> section has been added to the Web configuration section related to tracing.

<deployment retail="true|false" />

When the retail attribute is set to true, the ASP.NET runtime disables trace output, custom errors, and debug capabilities. The <deployment> section can only be set in the machine.config file.

Taking Control of the Trace

The standard trace text that ASP.NET emits in the page contains twelve sections (as in Figure 1) whose overall size is about 20KB of additional data. The size and format of this data is typically not an issue, since you'll be tracing pages only during the testing and development stages. However, a bit of control over the format of the traced text can be helpful, allowing you to increase your ability to quickly read that data.

ASP.NET trace data, however, is hard to catch. The data is collected during the request processing and cached internally by the TraceContext class. When the request processing is completed, the trace information is output. Tracing is controlled by the ProcessRequest method of the Page class—the central method of any ASP.NET page request. Basically, the trace information is flushed only when ProcessRequest is finished with the page markup. How can you capture the trace text? How can you post-process trace information to append additional data, remove unnecessary sections, or just format it differently?

In my September 2004 column, I provided a page-specific solution that was difficult to extend to all pages in the application. The idea is quite simple. You wrap the response stream with a custom stream and give yourself a chance to post-process everything being sent through the page's output stream—the page body, but also the traced text.

To obtain this, you need to set the Filter property of the HttpResponse object to a custom stream object. The Filter property can be used to set a wrapping filter object to modify the HTTP body before transmission. In the September 2004 column I just did this from within a sample page, and it worked fine. But what if you want to enable this feature for all pages in the application? What if you want to automatically filter the trace text in all the pages where tracing is enabled? And what if you want to do this without touching the source code of the page? Read on.

The MyTracer HTTP Module

The idea is to install an HTTP module to overwrite the Filter property of all response objects belonging to pages where tracing is enabled. An HTTP module is a class that implements the IHttpModule interface and is registered under the <httpModules> section of the application's web.config file. Figure 5 shows the source code of the MyTracer module I'm using to lay the foundations of a custom ASP.NET trace service.

Figure 5 MyTracer HTTP Module (Close Figure 5)
Namespace MyTracer
    Public Class TracerModule : Implements IHttpModule
        Public ReadOnly Property ModuleName() As String
            Get
                Return "MyTracerModule"
            End Get
        End Property

        Sub Init(ByVal app As HttpApplication) Implements _
            IHttpModule.Init
            AddHandler app.PostRequestHandlerExecute, _
                AddressOf OnPostRequestHandlerExecute
        End Sub

        Public Sub Dispose() Implements IHttpModule.Dispose
        End Sub

        Private Sub OnPostRequestHandlerExecute( _
                ByVal source As Object, ByVal e As EventArgs)
            Dim application As HttpApplication = _
                DirectCast(source, HttpApplication)
            Dim context As HttpContext = application.Context
            If context.Trace.IsEnabled Then
                Dim response As HttpResponse = context.Response
                If Not response Is Nothing Then
                    response.Filter = New TraceStream(response.Filter)
                End If
            End If
        End Sub
    End Class
End Namespace

The module listens to an application-level event named PostRequestHandlerExecute. That event occurs immediately after the request processing has terminated. More precisely, the event is notified immediately after the ProcessRequest call to the HTTP handler in charge of the page request has returned. When this happens, the trace information has been generated and flushed but the page response is still held on the server and not sent to the browser. The module registers the event handler in its Init method.

As you can see in Figure 5, the event handler checks the value of the IsEnabled property on the Trace object and, if tracing is enabled for the request, sets the Filter property of the response to a custom stream object. The net effect of this operation is that all the response text passes through the custom stream object and can be read and modified at will. Why can't you just use the original Filter stream object, reading from it and writing back to it, to do the required manipulation? For performance reasons, the stream object returned by the Filter property is write-only and throws an exception as soon as you try to call any of the read methods.

Figure 6 shows the source code of the TraceStream sample class. It is a memory stream class that overrides the Write method to post-process the byte array being sent to the browser. The byte array is transformed into a string, parsed as necessary, and, when done, written back to the browser. The heart of the new, smarter tracing service is the TraceOutputService class.

Figure 6 Custom TraceStream Class (Close Figure 6)
Public Class TraceStream : Inherits MemoryStream
    Private _outputStream As Stream

    Public Sub New(ByVal outputStream As Stream)
        _outputStream = outputStream
    End Sub

Public Overrides Sub Write( _
            ByVal buffer As Byte(), ByVal offset As Integer, _
            ByVal count As Integer)

        Dim enc As New ASCIIEncoding
        Dim pageResponse As String = enc.GetString(buffer, offset, count)

        Dim newPageResponse As String = _
            TraceOutputService.Parse(pageResponse)

        Dim data() As Byte = enc.GetBytes(newPageResponse)
        _outputStream.Write(data, 0, data.Length)
    End Sub
End Class

Before I delve deeply into the TraceOutputService class, let's briefly recap. Take a look at the following script:

<httpModules>
  <add name="MyTracerModule"
       type="MsdnMag.CuttingEdge.MyTracer.TracerModule, MyTracer" />
</httpModules>

By simply adding this to the web.config file, you apply an output filter to the page output, enabling yourself to edit virtually everything—including the trace information.

Tracing as a Provider-Based Service

To process the trace information, you might want to separate the regular page output from the standard trace information and then process the latter at will. Here's where the aforementioned TraceOutputService class comes into play.

The idea is to turn this feature into a provider-based service. You call a static method on the service class. The method, in turn, locates the default provider for the tracing service and invokes it. By registering distinct providers, you can process the trace information in any number of different ways—excluding some sections from view, collapsing other sections, adding new sections, displaying everything through a popup window, saving sections to a database, and so on.

There are various levels of complexity and flexibility to choose from when implementing such a service. The simplest option requires you to read a line from web.config or a custom XML file and call Activator.CreateInstance to instantiate the specified object. You then use a contracted interface on the object to post-process the trace information. Another option is to supply an explicit implementation of the factory pattern—more or less what happens in ADO.NET 2.0 with the DbProviderFactories class.

But of all the possibilities, the most ASP.NET-like approach is to devise the feature as a service and apply the provider model. The whole operation can be separated into three main steps: define a web.config section to collect configuration data, define a base provider class to implement the service interface, and design a service class to invoke the selected provider.

More information on how to design and author a custom provider-based service can be found at msdn.microsoft.com/library/en-us/dnaspp/html/ASPNETProvMod_Prt8.asp.

Creating a Section in Web.config

In ASP.NET 2.0, you write code to handle custom sections in the web.config file using a new and different approach than you used in ASP.NET 1.x. The idea is that you derive a class from ConfigurationSection, which is located in the System.Configuration namespace. From its parent, the new class will inherit methods and logic to parse the content of the section(s) it is bound to. Public properties on this class decorated with the ConfigurationProperty attribute are the only valid attributes in the section. To build a section for a list of providers, you can rely on some system-provided classes such as ProviderSettingsCollection (see Figure 7).

Figure 7 Custom ConfigurationSection Class (Close Figure 7)
Public Class TraceOutputServiceSection : Inherits ConfigurationSection
    <ConfigurationProperty("providers")> _
    Public ReadOnly Property Providers() As ProviderSettingsCollection
        Get
            Return DirectCast(MyBase.Item("providers"), _
                ProviderSettingsCollection)
        End Get
    End Property

    <ConfigurationProperty("defaultProvider", _
     DefaultValue:="StandardTraceOutputProvider")> _
    Public Property DefaultProvider() As String
        Get
            Return DirectCast(MyBase.Item("defaultProvider"), String)
        End Get
        Set(ByVal value As String)
            MyBase.Item("defaultProvider") = value
        End Set
    End Property
End Class

The TraceOutputServiceSection class serves a block of XML that has a subelement named <providers> and a defaultProvider attribute that has a default value of StandardTraceOutputProvider. Needless to say, a custom section—TraceOutputService in this case—must be explicitly registered with the web.config file to prevent schema complaints:

<configSections>
  <sectionGroup name="system.web">
    <section name="traceOutputService"
             type="MsdnMag.TraceOutputServiceSection, MyTracer"
             allowDefinition="MachineToApplication" />
  </sectionGroup>
</configSections>

At this point, the following configuration script is perfectly legal and correctly handled by the system:

<traceOutputService defaultProvider="RichTraceOutputProvider">
   <providers>
      <add name="StandardTraceOutputProvider"
           type="MsdnMag.StandardTraceOutputProvider, MyTracer" />
      <add name="RichTraceOutputProvider"
           type="MsdnMag.RichTraceOutputProvider, MyTracer"
           sections="All"
           collapseAll="true" />
   </providers>
</traceOutputService>

Each item in the <providers> subsection is represented with a ProviderSettings object. This object has a name/value collection property called Parameters, which lets you freely define any set of attributes to characterize your provider's type. For more information on configuration in the .NET Framework 2.0, see Bryan Porter's article on the subject in this issue of MSDN®Magazine.

The Trace Provider Interface

A trace provider is a component that exposes a fixed interface to its caller so that the caller can polymorphically invoke any registered provider without knowing any of its implementation specifics. The provider's public contract is defined through a base class that inherits from ProviderBase:

Public MustInherit Class TraceOutputProvider : Inherits ProviderBase
    Public MustOverride Property ApplicationName As String
    Public MustOverride Function PostProcess(  _
       ByVal tracedText As String) As String
End Class

Note that you make this base class inherit from ProviderBase if you want to stick to the pure ASP.NET provider model. This pattern can be applied regardless of predefined ASP.NET facilities.

Devising the Service Class

Each provider-based service features an entrypoint class with a number of static methods mapped more or less directly to the members of the selected provider. The TraceOutputService class has only one purpose—parsing the page trace and applying some custom logic. For this reason, I defined a Parse static method:

Public Shared Function Parse(ByVal pageResponse As String) As String

The Parse method receives the full page content (body and trace). It first separates body from trace, and then loads the selected provider and invokes the PostProcess method on it. The return value of PostProcess is combined with the page body, returned to the stream, and then served to the user.

The TraceOutputService class features a Provider property pointing to a running instance of the selected provider and a Providers collection property that gathers all registered providers for the feature. The internal members behind the properties are static members. The class loads all providers only once for the whole lifetime of the application. Subsequently, all concurrently running requests share the same instance of the providers. That's why you must properly lock any write access to a provider's instance. Figure 8 lists the full source code of the TraceOutputService class.

Figure 8 Trace Output Service Class (Close Figure 8)
Friend Class PageOutputInfo
    Public PageContent As String
    Public PageTrace As String
End Class

Public Class TraceOutputService
    Private Shared _provider As TraceOutputProvider = Nothing
    Private Shared _providers As TraceOutputProviderCollection = Nothing
    Private Shared _internalLock As New Object

    Public ReadOnly Property Provider() As TraceOutputProvider
        Get
            Return _provider
        End Get
    End Property

    Public ReadOnly Property Providers() As TraceOutputProviderCollection
        Get
            Return _providers
        End Get
    End Property

    Public Shared Function Parse(ByVal pageResponse As String) As String
        Dim info As PageOutputInfo = SplitPageResponse(pageResponse)

        LoadAllProviders()

        If String.IsNullOrEmpty(info.PageTrace) Then
            Return info.PageContent
        End If

        return info.PageContent & _
            _provider.PostProcess(info.PageTrace)
    End Function

    Private Shared Sub LoadAllProviders()
        If Not _provider Is Nothing Then Return
        SyncLock _internalLock
            If Not _provider Is Nothing Then Return

            Dim name As String = "system.web/traceOutputService"
            Dim section As TraceOutputServiceSection = _
                DirectCast(WebConfigurationManager.GetSection(name), _
                TraceOutputServiceSection)

            _providers = New TraceOutputProviderCollection()
            ProvidersHelper.InstantiateProviders(section.Providers, _
                _providers, GetType(TraceOutputProvider))

            _provider = _providers(section.DefaultProvider)
            If _provider Is Nothing Then
                Throw New ProviderException(
                    "Couldn’t load the default TraceOutputProvider")
            End If
        End SyncLock
    End Sub

    Private Shared Function SplitPageResponse( _
            ByVal pageResponse As String) As PageOutputInfo
        Dim info As New PageOutputInfo

        Dim beginWith As String = "<div id=""__asptrace"">"
        Dim pos As Integer = pageResponse.IndexOf(beginWith)
        If (pos < 0) Then
            info.PageContent = pageResponse
            info.PageTrace = String.Empty
            Return info
        End If

        info.PageTrace = pageResponse.Substring(pos)
        info.PageContent = pageResponse.Substring(0, pos - 1)
        Return info
    End Function
End Class

You populate the Providers collection and then retrieve the current provider object by name, as shown here:

_providers = New TraceOutputProviderCollection()
ProvidersHelper.InstantiateProviders(section.Providers, _
    _providers, GetType(TraceOutputProvider))
_provider = _providers(section.DefaultProvider)

The TraceOutputProviderCollection class inherits from a system-provided class, called ProviderCollection. ProvidersHelper is a built-in helper class located in the System.Web.Configuration namespace. It features two essential methods—InstantiateProviders and InstantiateProvider. The former creates instances of all registered providers; the latter creates the instance of a particular provider. Both methods retrieve from the configuration section the details of the providers or provider to be created. These methods relieve you of most of the burden involved in instantiating an ASP.NET provider. These classes, in fact, are extensively used within the built-in implementation of the provider model. The InstantiateXXX methods end up calling into Activator.CreateInstance to get an instance of the specified type.

Wrapping the Standard Tracer

Now that you have a functional infrastructure for a provider-based trace service, it is time to start adding a few sample providers. The first provider you should try is one that simply mirrors the standard output, adding a signature to signify that the new infrastructure is being used.

The StandardTraceOutputProvider inherits from TraceOutputProvider and overrides the Initialize and PostProcess methods. In the Initialize method StandardTraceOutputProvider doesn't do much more than invoke the base method and read the application name, if any, from the configuration. (Check the source code in this month's download for more details.)

Public Overrides Function PostProcess( _
       ByVal tracedText As String) As String
    Return GetTopSignature() & tracedText & GetBottomSignature()
End Function

You register the trace provider with a script like so:

<traceOutputService defaultProvider="StandardTraceOutputProvider">
   <providers>
      <add name="StandardTraceOutputProvider"
           type="MsdnMag.StandardTraceOutputProvider,
           MyTracer"
      />
   </providers>
</traceOutputService>

At this point, add any pages to a sample ASP.NET application and turn on page output tracing by setting Trace=true in either the @Page directive or in web.config.

A Richer ASP.NET Tracer

Now let's craft a richer trace provider that will display only a selected subset of sections (that is, only Trace Information) and will include collapse/expand functionality that makes reading data easier.

To accomplish this, you need to map the traced text to an internal representation—a list of custom TraceBlock structures—and, based on the configuration, decide which ones to insert in the returned stream. In addition, you'll emit a bit of JavaScript code in each block used to hide/display the block on demand. Figure 9 shows the final result.

Figure 9 TraceProvider with Collapse/Expand Functionality
Figure 9 TraceProvider with Collapse/Expand Functionality

The original trace text is a collection of <table> elements, each representing a section (Request Details, Trace Information, and so on). The PostProcess method of the RichTraceOutputProvider object walks through the original text and populates a list of TraceBlock objects. Here's what a TraceBlock looks like in C#:

internal class TraceBlock
{
    public TraceSections Code;
    public string Name;
    public string ID;
    public string Body;
}

Here's TraceBlock in Visual Basic:

Friend Class TraceBlock
  Public Code As TraceSections
  Public Name As String
  Public ID As String
  Public Body As String
End Class

The Code field indicates the type of the section from the TraceSections enum (see the source code in this month's code download). The Name field is the title of the section. The ID field is the name-based ID used to script the section. And the Body field contains the original text for the section block.

The provider also defines a couple of new virtual properties: Sections and CollapseAll. The former is a TraceSections value that indicates which traced sections should be displayed to the user. The CollapseAll property is a Boolean value that indicates whether the sections should be initially displayed in the collapsed state. The trace output in Figure 9 originated from this configuration script:

<add name="RichTraceOutputProvider"
     type="MsdnMag.CuttingEdge.RichTraceOutputProvider, MyTracer"
     sections="RequestDetails,TraceInformation,SessionState"
     collapseAll="true" />

The original HTML text for each section is prefixed with a client button. If the button is visible, the table with trace data is hidden and vice-versa. Both the button and the table are bound to a click event that toggles visibility on and off. The button markup and any necessary script code is added in the PostProcess method of the provider. The following script code is associated with each button and table:

function Toggle(showInfo, tableName, buttonName) {
   var table = document.getElementById(tableName);
   var button = document.getElementById(buttonName);
   if (showInfo) {
    table.style["display"] = 'none';
    button.style["display"] = '';
   } else {
    table.style["display"] = '';
    button.style["display"] = 'none';
   }
}

By default, buttons are hidden and tables are shown. By setting CollapseAll to true, you get the opposite. As in Figure 9, to expand a collapsed trace section you click the button; to collapse an expanded section, you click anywhere in the table.

Note that the preceding script works with the Mozilla Firefox and Netscape 7 browsers, as well as with Internet Explorer.

ASP.NET tracing is a powerful feature that lets developers audit the run-time behavior of pages and applications. In this column I demonstrated how to build a non-obtrusive and easy-to-use infrastructure to capture the standard traced text and either replace or modify it before display.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Service Station: WSE 3.0, SOAP Transports, and More

Service Station
WSE 3.0, SOAP Transports, and More
Aaron Skonnard

Code download available at: ServiceStation2006_06.exe (175KB)

It's that time again. Time to answer some of the questions I get on a regular basis. This month I'll look at service orientation and policy-based compatibility, SOAP's transport-neutral design, and Web Services Enhancements (WSE) 3.0.

Q I know SOAP was designed as a transport-neutral messaging framework. Is it really possible to design WSE/Windows® Communication Foundation applications without taking into account transport specifics?

I know SOAP was designed as a transport-neutral messaging framework. Is it really possible to design WSE/Windows® Communication Foundation applications without taking into account transport specifics?

A SOAP was indeed designed as a transport-neutral XML messaging framework. One of its primary goals was to enable the packaging of protocol information within XML messages so that such information would not have to be carried at the transport level. This allows XML protocols (such as those defined by WS-* in the areas of security, reliable messaging, and transactions) to be used across a variety of transport communication scenarios.

A typical SOAP stack provides a few different layers of code (Figure 1 shows an example). The topmost layer is the application code. This is either the client application that consumes a service or the service business logic itself, depending on which side of the wire you're talking about. (Most stacks provide a programming model that makes it easy to program and consume services.) Then there's the SOAP messaging layer. You can think of this layer as the infrastructure that implements SOAP and the various WS-* protocols. This code processes the various SOAP headers using interception techniques—the process is completely hidden from you at the application layer. (Most stacks also provide a layer that abstracts away transport specifics, shielding the layers above from such communication details.) Finally, there's the physical network where bytes meet the wire. This layering model holds true for both clients and services.

Figure 1 SOAP Stack
Figure 1 SOAP Stack

With the layering shown in Figure 1, the code you write at the application layer is completely shielded from the underlying transport details. Given this design, you shouldn't have to modify the application code in order to use a different transport, as long as the new transport is compatible with your service requirements. Of course, the feasibility of such flexibility depends largely on the implementation stack and the issues surrounding "compatibility."

WSE 2.0 was the first Microsoft product to bring SOAP's promise of transport-neutral XML messaging to life when it introduced support for TCP as an alternative to HTTP. WSE 2.0 relied on a layered approach similar to the one shown in Figure 1. It allowed you to write services that could run over either TCP or HTTP without changes to the service logic. WSE 3.0 builds on the WSE 2.0 foundation, making a few changes to the topmost layers in order to simplify the developer experience.

The Microsoft next-generation Web services platform, Windows Communication Foundation, was designed according to the general architecture shown in Figure 1. It too allows a single service to run over multiple transports, at the same time even, and it supports more protocols than just HTTP and TCP (MSMQ and P2P, for instance). Windows Communication Foundation also provides a more sophisticated SOAP messaging layer, which supports more WS-* specifications. With all of these stacks, developers can write custom transports and plug them in, providing truly unlimited transport possibilities.

Now, let's get back to your question. Is it really feasible to write WSE/Windows Communication Foundation services without worrying about transport specifics so administrators can choose the transports at run time? It's plausible, but achieving success with such an approach is highly doubtful. (It's probably as feasible as the promise that ODBC will allow administrators to swap out an application's underlying database after development.)

When designing services, there are numerous issues related to the communication transport that must be considered earlier rather than later. For example, if your service code requires request-response semantics and, therefore, a two-way transport, an administrator can't configure it to use the Microsoft® Message Queue (MSMQ) transport, which is inherently one-way. Or imagine a transport developed for a special channel that provides numerous optimizations but also requires all messages to be smaller than 10KB in size. An administrator might then configure a service with this special transport (to speed things up). While this configuration might pass initial QA tests, it would almost certainly fail down the road once it encounters messages that exceed the size limit. Only the developer can ensure success with the special transport by making the appropriate design considerations up front. In general, administrators will only be able to use transports that are completely compatible with your service design/implementation.

Developers simply cannot ignore transport details when designing services. Likewise, administrators cannot ignore the service design when choosing a transport. In fact, developers should be able to impose requirements that influence the type of transport used by administrators at run time. This would help administrators make better choices. Windows Communication Foundation helps facilitate this via special attributes that developers can apply to their code (the [DeliveryRequirements] attribute, for example). At run time, Windows Communication Foundation validates that the delivery requirements (specified by the developer) match the actual service configuration during startup. Developers can even write custom Windows Communication Foundation validators. These techniques are helpful because they encourage communication between developers and administrators.

Despite this reality, separating the SOAP programming model from the underlying transport layer is still a good thing. It's helpful to both developers and administrators. It makes everyone's life a bit easier. Think of it as a way to increase productivity and flexibility. One could argue that this particular SOAP design goal has been realized to as great a degree as possible.

Q One of the tenets of service orientation is "policy-based compatibility." This seems to suggest that policies should be used to negotiate behavior. Is that actually the case with WSE or Windows Communication Foundation?

A A policy is an ordered set of assertions that describe a service's requirements or capabilities. Policies describe semantic requirements or capabilities that cannot be expressed in structural contract languages like XML Schema Definition (XSD) and Web Services Description Language (WSDL). For example, a particular security policy may include assertions that specify the type of token required for client authentication as well as integrity and privacy requirements for each message exchange. The assertions might specify exactly what portions of each message need to be signed and/or encrypted along with what key to use for each task.

The tenet you refer to states that service compatibility should be determined based on policy information. This implies that policy information must be shared between services and clients, which may be implemented using different stacks. In order for this to work, the industry needs a common framework for expressing policy in a platform-neutral manner. WS-Policy exists for this purpose.

In addition to a common policy framework, standard policy assertions are also necessary to negotiate compatibility and protocols in specific areas. WS-PolicyAssertions and WS-SecurityPolicy are examples of specifications that define standard policy assertions that all Web service frameworks can agree on. These specifications define the structure of each policy assertion along with its meaning. Similar specifications that focus on different functional areas will surely appear over time.

In theory, a service could publish a policy containing predefined assertions and somehow make the policy available to interested clients. A client would then use the service policy to determine whether it is capable of meeting the service's requirements (based on its understanding of the assertions). When the client is compatible it could even use the policy to automate its support for the specified requirements and protocols. The key is that code generators need to become policy-aware so they can automate the client experience when the service is using more than just the WS-I basic profile.

Although policy has long been touted as a necessary piece of the service-oriented puzzle, support for exchanging policies and policy-aware code generation hasn't materialized in today's stacks. Developers working with WSE 3.0, for example, must manually define the policy used within each application, even in clients. You may be able to manually reuse the same policy file within a service and a client (assuming you're using WSE 3.0 in both applications) but it requires a human to copy the file and to configure each application individually. Today's code generators (wsdl.exe and wsewsdl3.exe) don't look at policy information—they look only at XSD and WSDL to generate structure contracts.

In reality, WSE 3.0 uses policy as a configuration language and not as an exchange format for negotiating behavior. There is still value in this declarative approach since it decouples common messaging needs from the service logic itself. This separation gives developers the freedom to modify the policy (configuration) file to change service behavior without having to recompile.

The good news is that Windows Communication Foundation is poised to change this process. Windows Communication Foundation doesn't use policy as a configuration format; it defines a much simpler configuration language for developers to use (via the concept of bindings).

Windows Communication Foundation translates binding information into WS-Policy statements embedded within the service's WSDL definition. This makes it easy to share the complete service description with other parties. As long as consumers understand how to process the embedded WS-Policy statements, service compatibility can actually be negotiated (and even automated). When you generate a Windows Communication Foundation proxy using svcutil.exe, it automatically builds the appropriate client configuration based on the policy statements embedded within the WSDL or it informs you of any potential incompatibilities. It would be nice if similar support would come to fruition across non-Microsoft frameworks as well.

Q How will I port a WSE 3.0 filter to Windows Communication Foundation? Is the Windows Communication Foundation notion of binding equivalent to the WSE 3.0 notion of policy?

A WSE 3.0 makes use of a "pipeline" to implement the SOAP processing layer you saw back in Figure 1. The WSE pipeline consists of an ordered sequence of SoapFilter objects, each of which gets a chance to process the SOAP message en route. You can customize message processing by injecting your own SoapFilters into the pipeline. This is done via the WSE 3.0 policy framework. For example, you could inject a custom filter to handle validation, security, or logging needs.

Moving SoapFilters forward to Windows Communication Foundation will require some significant reworking. Windows Communication Foundation provides a user-friendly programming model similar to what's offered by ASMX. In Windows Communication Foundation parlance, we refer to this as the "service model" layer (because the root namespace is called System.ServiceModel). It also provides a sophisticated messaging layer and the underlying transport layers. Windows Communication Foundation provides two extensibility pipelines: one within the service model layer (on the dispatcher/proxy) and one within the messaging layer (the channel stack). This is a significant difference from the WSE 3.0 architecture.

When working with Windows Communication Foundation, you influence the channel stack by choosing (configuring) a binding. The channel stack is concerned with what goes on the wire, which requires agreement and coordination with the other side. Hence, Windows Communication Foundation translates binding information into policy statements (which show up in the auto-generated WSDL) so the other applications can build equivalent channel stacks in their environments. The policy is no longer just a configuration mechanism (like it was in WSE 3.0) but rather a product of the system. You can think of a binding as the Windows Communication Foundation configuration mechanism, which results in a policy when the endpoint description is shared via WSDL.

You configure the service model dispatcher/proxy using contracts. The dispatcher/proxy also has an associated pipeline of "behaviors" that influence local execution details. You configure behaviors using either custom attributes in your code or custom elements in your configuration file.

When moving WSE 3.0 SoapFilter code forward to Windows Communication Foundation, you must first decide whether the functionality belongs within the channel stack or the dispatcher/proxy. This largely depends on whether it requires coordination with the other side. If your SoapFilter requires some level of agreement between the client and the service (like security), you would implement the SoapFilter as a custom Windows Communication Foundation channel along with a custom binding element. You would use the custom binding element on your endpoint to inject the functionality. This allows your binding element to produce policy statements for inclusion in the generated WSDL. You can then configure svcutil.exe to recognize the policy statements and produce your binding element in the generated proxy configuration. This approach is required for any application-level protocol that requires coordination across the wire.

If your SoapFilter does not require coordination between both sides (as with validation or logging), you'd implement the SoapFilter as a Windows Communication Foundation behavior that inserts the appropriate interface in the proxy/dispatcher. Windows Communication Foundation provides hooks for many types of behaviors including parameter inspection, message formatting, message inspection, operation selection, and operation invocation. The difference is that these extensibility points only affect the local execution behavior of the Windows Communication Foundation stack.

Moving lower-level extensibility code almost always requires some significant reworking and this particular transition is no exception to the rule.

Q When using the WSE 3.0 turnkey security assertions, what must I do to integrate future Windows Communication Foundation applications?

A ASMX, WSE 3.0, and Windows Communication Foundation will all be compatible on the wire. ASMX and WSE 3.0 clients will be able to consume Windows Communication Foundation services as long these services don't use any WS-* protocols that aren't supported by ASMX or WSE. Likewise, Windows Communication Foundation clients will be able to consume ASMX and WSE 3.0 services. Obviously, Windows Communication Foundation provides more functionality and support for WS-* than its predecessors, which means you'll need to pay close attention to your configuration in order to achieve compatibility.

If you're trying to integrate Windows Communication Foundation with existing ASMX or WSE applications that conform to the basic profile, use BasicHttpBinding within Windows Communication Foundation. If, however, you're using message-based security via the WSE 3.0 turnkey assertions, you need to make sure you use a compatible binding configuration. For example, when using the UsernameOverTransport assertion in WSE 3.0, you should configure your Windows Communication Foundation binding in the following manner:

<basicHttpBinding>
   <binding name="UsernameOverTransportBinding">
      <security mode="TransportWithMessageCredential">
         <message clientCredentialType="UserName" />
      </security>
   </binding>
</basicHttpBinding>

If you happen to be using the CertificateMutualAuthenticationProfile, you should still use BasicHttpBinding, but instead configure it this way:

<basicHttpBinding>
   <binding name="CertificateMutualAuthenticationProfileBinding">
      <security mode="Message">
         <message clientCredentialType="Certificate"
            negotiateServiceCredential="false"/>
      </security>
   </binding>
</basicHttpBinding>

In Windows Communication Foundation, WSHttpBinding provides support for more sophisticated security scenarios. You need to use this when integrating with WSE 3.0 applications using the other turnkey security assertions—you have to customize the binding differently for each turnkey security assertion. For example, if you're using the UsernameOverCertificate assertion on the WSE 3.0 side, you must configure the WSHttpBinding binding, as shown here:

<wsHttpBinding>
   <binding name="UsernameOverCertificateBinding">
      <security mode="Message">
         <message clientCredentialType="UserName"
                  negotiateServiceCredential="false"/>
      </security>
   </binding>
</wsHttpBinding>

Microsoft will provide more guidance on integrating Windows Communication Foundation with existing ASMX and WSE 3.0 applications as it gets closer to shipping.

Q Is Microsoft planning to provide any tools for analyzing a security policy?

A Microsoft Research is working on a project, code-named "Samoa,"for building tools to analyze security policies. The goal of the project is to "exploit recent theoretical advances in the analysis of security protocols in the practical setting of XML Web services." The group has produced a tool called the Policy Advisor that works with both WSE 2.0 and WSE 3.0. In fact, it actually ships with WSE 3.0 as a sample (you'll find it in the default samples directory).

The Policy Advisor examines the configuration and policy files for one or more WSE endpoints and produces a human-readable report which highlights typical security risks and provides some basic advice. You can use this tool to review your existing WSE 3.0 applications without making any changes to your application code. The generated report provides valuable fodder for security reviews and related discussions.

The tool doesn't require any installation because it's simply an XSLT file that processes configuration files and WSE 3.0 policy cache files, looking for potential security issues to consider. In order to use the tool, you must first author a WSE endpoint file that defines a list of endpoints you'd like the tool to inspect. The code in Figure 2 illustrates the format of such a file.

Figure 2 WSE Endpoint File (Close Figure 2)
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet href="http://PolicyAdvisor.xml" type="text/xsl"?>
<endpoints xmlns="http://schemas.microsoft.com/pa/2005/10/endpoints">
  <endpoint name="Stock Service client"
      path="StockServiceClient\"
      config="app.config"/>

  <endpoint name="Stock Service service"
      path="StockService\"
      config="Web.config"/>

  <endpoint name="SC client"
      path="SecureConversationPolicyClient\"
      config="app.config"
      policyCache="wse3policyCache.config"/>

  <endpoint name="SC server"
      path="SecureConversationPolicyService\"
      config="Web.config"
      policyCache="wse3policyCache.config"/>
  ...
</endpoints>

Notice that the file contains an xml-stylesheet processing instruction that associates the file with the PolicyAdvisor.xml XSLT file. Then it simply contains a list of <endpoint> elements, each of which defines the path to the project, the name of the configuration file, and the name of the WSE policy file.

In order to run the policy advisor, simply open the endpoint file in Internet Explorer®, which causes the XSLT processing to start. You'll see the report displayed in the browser (see Figure 3). If you'd like more information on the Samoa project, you should check out research.microsoft.com/projects/samoa.

Figure 3 Policy Advisor for WSE
Figure 3 Policy Advisor for WSE

Q How do you change aspects of a policy that are not exposed by the WSE 3.0 policy wizard?

A The WSE 3.0 security policy wizard provides step-by-step guidance for building a security policy with the new turnkey security policy assertions. Hence, the wizard only lets you make changes related to your security policy—it's not a generic policy editor. If you need to modify a policy that was built with the policy wizard and you need to add/change elements not related to the security assertions, you need to use an XML editor to work on the policy.

Luckily, the WSE 3.0 policy format is much simpler than WS-Policy (used in WSE 2.0) and isn't hard to work with directly. Plus, if you're working with policy files in Visual Studio® 2005, you'll notice helpful XML IntelliSense® that makes working directly with XML policies even easier.

Q Is it possible to use the WSE configurations tool without running Visual Studio?

A Yes, with WSE 3.0, you can now run the WSE configuration tool outside of Visual Studio 2005. You used to have to open the Visual Studio project and select WSE Settings from the project menu to launch the tool. This didn't allow you to modify configuration settings in deployment environments where Visual Studio wasn't installed.

Once installed, you can launch the tool from the start menu (Start | All Programs | Microsoft WSE 3.0 | Configuration Tool. The executable is named WseConfigEditor3.exe and it's found in the Tools subfolder of the installation directory. After launching the tool, you can choose either to create a new configuration file or open an existing one from the File menu (see Figure 4). After you've made your changes you can save the file.

Figure 4 WSE 3.0 Configuration Tool
Figure 4 WSE 3.0 Configuration Tool

Q If WSE 3.0 supports both HTTP and TCP, why doesn't the following line of code work?

SoapReceivers.Add("http://localhost:8080/math/", typeof(MathService));

A WSE 3.0 was designed to make use of the IIS/ASP.NET infrastructure for receiving HTTP messages. Most developers configure WSE 3.0 on existing ASMX implementations deployed in IIS. However, WSE also provides some messaging classes (SoapReceiver/SoapService) that implement IHttpHandler in order to integrate WSE services with IIS/ASP.NET without using ASMX. But in all cases, WSE requires the use of IIS/ASP.NET for receiving incoming HTTP messages.

If you need to host a WSE service within a non-IIS application (such as a Windows service or a rich client), you need to use the self-hosting technique I discussed earlier (SoapReceivers). But when using self-hosting, WSE 3.0 only supports the TCP transport. WSE can spin up a TCP listener within any application to wait for incoming messages, but the same is not true for HTTP since it relies on IIS.

The introduction of HTTP.SYS (with Windows XP SP2 and Windows Server™ 2003) makes it possible to support self-hosting over HTTP in future releases, and this is precisely what Windows Communication Foundation does. With Windows Communication Foundation, you can spin up an HTTP listener in any .NET-based application and wait for incoming HTTP messages. Since Windows Communication Foundation is already on the doorstep, it's unlikely that WSE will ever provide this feature. Nevertheless, if you really want this capability in WSE, the WSE 3.0 extensibility points make it possible to implement a custom HTTP.SYS transport on your own.

The .NET Framework 2.0 provides a System.Net.HttpListener class for easily integrating with the underlying HTTP.SYS driver. All you need to do is write the code to integrate the HttpListener class with the SOAP transport layer provided by WSE 3.0 (that is, you need to write a SoapTransport class). Once the custom transport is written, you configure your applications to use it (instead of the default HTTP transport tied to IIS/ASP.NET), as shown here:

<configuration>
  <microsoft.web.services3>
    <messaging>
      <transports>
        <add scheme="http"
         type="Skonnard.Samples.HttpSys.HttpSysTransport,
               Skonnard.Samples.HttpSys"/>
      </transports>
    </messaging>
  </microsoft.web.services3>
</configuration>

With this configuration in place, the line of code shown in your question will work (assuming, of course, you have a functional HTTP.SYS transport implementation).

I've provided a complete sample that illustrates how to implement a custom HTTP.SYS transport. This is nothing more than a sample and is by no means production ready, but it's a good starting point for those who need to investigate this approach further. You can download the complete sample from the MSDN®Magazine Web site.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a cofounder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight's Applied Web Services 2.0, Applied BizTalk Server 2006, and Introducing Windows Communication Foundation courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at pluralsight.com/aaron.


Wicked Code: Three Cures for Common Site Map Ailments

Wicked Code
Three Cures for Common Site Map Ailments
Jeff Prosise

Code download available at: WickedCode2006_06.exe (390KB)
Contents
Persistent TreeViews. Really?
Don't Supersize—Atlasize!
SiteMapPaths and Unmapped Pages
Multiple Site Maps

Data-driven site navigation is among the niftiest and most useful features in ASP.NET 2.0. To get it working, all you do is create an XML site map file (or a SQL site map if you're using the MSDN®Magazine SqlSiteMapProvider), add a SiteMapDataSource, and bind a TreeView or Menu to the SiteMapDataSource. Just like that, you have a data-driven navigation UI. If the structure of your site changes, simply update the site map and the navigation UI changes to match. For good measure, you can throw in a SiteMapPath control to implement a bread crumb element showing the path to the current page.

In many cases, building a navigation UI really is that easy. But it's also easy to run into snags using the ASP.NET 2.0 data-driven site navigation infrastructure and find yourself up against a wall searching for a way out. One of the most common ailments is TreeView branches that don't stay expanded or collapsed as you click around a site. Another is unmapped pages—pages that don't appear in the site map and therefore don't display in SiteMapPath controls (even though you want them to). A third involves pages that contain multiple navigation controls: a TreeView on the left and a Menu at the top, for example. It's easy enough to bind both controls to a SiteMapDataSource. But once you do, how do you configure them to show one set of links in the TreeView and another set in the Menu?

I chose these three problems to tackle because they seem to eat a lot of dev cycles. Fortunately, the solutions are easy. A little know-how coupled with an understanding of how site maps—and site map providers—work can go a long way, as the sample code (from the MSDN Magazine Web site at msdn.microsoft.com/msdnmag/code06.aspx) will show. Throw in a little Atlas to spice things up, and the results are nothing short of magical. But don't just take my word—read on.

Persistent TreeViews. Really?

The number-one problem that bedevils developers who combine site maps with expandable TreeView controls is the fact that the TreeView loses its expanded/collapsed state as you navigate among pages. The TreeView may be declared in a Master Page, but it's still instantiated for each and every request. The next TreeView knows nothing about the previous one. View state doesn't help here because view state preserves state across postbacks. When you click a link in a TreeView, you're not posting back; you're navigating to a whole new page. State pertaining to the previous page is lost.

One solution, which really isn't a solution at all if you want to support interactive expanding and collapsing, is to "fix" the TreeView problem by setting its ShowExpandCollapse property to false. Another approach is to serialize the TreeView's view state into session state and then deserialize it into the TreeView on the next page. But that's inefficient, because a TreeView stores a lot more in view state than simply a record of which branches are expanded and which aren't.

A better solution is the one exemplified in Figure 1. The code comes from the Master Page of the sample site named WickedSiteMap that accompanies this article. When the TreeView fires a DataBound event indicating it has been populated from the site map, the event handler (TreeView1_DataBound) checks to see if session state contains an item named "TreeViewState." If there is no such item, TreeView1_DataBound calls a helper method named SaveTreeViewState to create a List<> containing an entry for each expandable branch that's currently expanded. Then it saves the List<> in session state. If session state already contains a saved List<>, TreeView1_DataBound retrieves it and calls a helper method named RestoreTreeViewState to restore the TreeView to the state recorded in the List<>. RestoreTreeViewState walks the TreeView item by item and expands expandable branches recorded in the List<>, while collapsing expandable branches that are not in the list.

Figure 1 Site.master.cs (Close Figure 1)
using System;
using System.Web.UI.WebControls;
using System.Collections.Generic;

public partial class Site_master : System.Web.UI.MasterPage {
    protected void Page_Load(object sender, EventArgs e) {
        // Disable ExpandDepth if the TreeView’s expanded/collapsed
        // state is stored in session.
        if (Session["TreeViewState"] != null)
            TreeView1.ExpandDepth = -1;
}

    protected void TreeView1_DataBound(object sender, EventArgs e)
    {
        if (Session["TreeViewState"] == null)
        {
            // Record the TreeView’s current expanded/collapsed state.
            List<string> list = new List<string>(16);
            SaveTreeViewState(TreeView1.Nodes, list);
            Session["TreeViewState"] = list;
        }
        else
        {
            // Apply the recorded expanded/collapsed state to
            // the TreeView.
            List<string> list = (List<string>)Session["TreeViewState"];
            RestoreTreeViewState(TreeView1.Nodes, list);
        }
    }

    protected void TreeView1_TreeNodeCollapsed(
        object sender, TreeNodeEventArgs e)
    {
        if (IsPostBack) {
            List<string> list = new List<string>(16);
            SaveTreeViewState(TreeView1.Nodes, list);
            Session["TreeViewState"] = list;
        }
    }

     protected void TreeView1_TreeNodeExpanded(
        object sender, TreeNodeEventArgs e)
    {
        if (IsPostBack) {
            List<string> list = new List<string>(16);
            SaveTreeViewState(TreeView1.Nodes, list);
            Session["TreeViewState"] = list;
        }
    }

    private void SaveTreeViewState(
            TreeNodeCollection nodes, List<string> list) {
        // Recursively record all expanded nodes in the List.
        foreach (TreeNode node in nodes)
        {
            if (node.ChildNodes != null && node.ChildNodes.Count != 0) {
                if (node.Expanded.HasValue && node.Expanded == true &&
                    !String.IsNullOrEmpty(node.Text))
                        list.Add(node.Text);
                SaveTreeViewState(node.ChildNodes, list);
            }
        }
    }

    private void RestoreTreeViewState(
            TreeNodeCollection nodes, List<string> list) {
        foreach (TreeNode node in nodes)
        {
            // Restore the state of one node.
            if (list.Contains(node.Text)) {
                if (node.ChildNodes != null &&
                    node.ChildNodes.Count != 0 &&
                    node.Expanded.HasValue &&
                    node.Expanded == false)
                    node.Expand();
            }
            else if (node.ChildNodes != null &&
                    node.ChildNodes.Count != 0 &&
                    node.Expanded.HasValue &&
                    node.Expanded == true)
            {
                    node.Collapse();
            }

            // If the node has child nodes, restore their states, too.
            if (node.ChildNodes != null && node.ChildNodes.Count != 0)
                RestoreTreeViewState(node.ChildNodes, list);
        }
    }
}

The DataBound event handler ensures that when you click an item in the TreeView to go to another page, the TreeView in the receiving page assumes the configuration of the TreeView in the sending page. But it doesn't take into account the fact that the TreeView's state might have changed—that is, that the user might have expanded or collapsed some branches—while the page was displayed. That scenario is handled by the TreeView1_TreeNodeCollapsed and TreeView1_TreeNodeExpanded event handlers, which update the List<> stored in session state each time a branch is collapsed or expanded.

The final piece of the puzzle is the Site.master.cs Page_Load method. If the TreeView's ExpandDepth property is set, you want to honor that property the first time the TreeView is displayed, but override it with the saved state of the TreeView in subsequent requests. The Page_Load method in Figure 1 does just that. If session state contains a saved TreeView state, Page_Load sets the TreeView's ExpandDepth property to -1, effectively disabling any ExpandDepth property value specified in the <asp:TreeView> tag.

One implementational detail of interest is how SaveTreeViewState and RestoreTreeViewState identify the expanded TreeView nodes. When SaveTreeViewState finds an expanded node, it writes the value of the node's Text property into the List<>. Similarly, when RestoreTreeViewState traverses a TreeView, expanding nodes recorded in the List<>, it compares the Text properties of the nodes to the Text properties in the List<>. Consequently, a node won't retain its expanded state unless it has a non-empty Text property, and that property should be unique among all the nodes in the TreeView. (In other words, if two nodes are named "Foo" and one is expanded but the other is not, both will expand when you click a link and jump to another page.) If that's a problem, you can modify my code to apply additional criteria to node identification.

Don't Supersize—Atlasize!

The chief downside to processing a TreeView's TreeNodeCollapsed and TreeNodeExpanded events has to do with postbacks. An unfortunate side effect of registering handlers for these events is that the TreeView will now post back to the server each time a branch is expanded or collapsed. An elegant way to eliminate the postbacks while retaining the ability to record changes to the TreeView state on the server is to incorporate asynchronous XML-HTTP callbacks (a technology known as Asynchronous JavaScript and XML, or AJAX). The ASP.NET TreeView has XML-HTTP support built in, but rewiring it to convert expand/collapse postbacks into callbacks would require nontrivial changes to the control.

I considered making those changes, but elected not to for a simple reason: Atlas. Atlas is the code name for the forthcoming AJAX framework for ASP.NET developed by Microsoft. The Atlas UpdatePanel control makes it simple, almost trivial, to convert postbacks into asynchronous XML-HTTP callbacks. I didn't want to bring Atlas into the sample site because Atlas isn't a shipping product yet, and as of this writing, you're not allowed to use it in production Web apps. But the future has Atlas written all over it, and I'll have much more to say about it—and UpdatePanels in particular—in a future installment of Wicked Code.

The good news is that MSDN Magazine is in the business of showing you what the future holds. So I built a separate version of the site (included in the download, of course) named AtlasSiteMap that uses Atlas to convert the TreeView's expand and collapse postbacks into XML-HTTP callbacks. The result is magical. A few lines of markup (no additional code!) and expand/collapse postbacks simply disappear; you can verify this by running AtlasSiteMap in Internet Explorer® and observing that no progress bar appears in the browser's status bar when you expand or collapse a branch of the TreeView—a clear indication that the TreeView is no longer posting back to the server. Yet thanks to the brilliance of the UpdatePanel, your server-side event handlers are still being called, as evidenced by the fact that the TreeView retains its expanded/collapsed state as you click around the site.

Figure 2 shows the changes required to Site.master to Atlas-ize the TreeView. (Note, though, that no changes were made to Site.master.cs!) First I added an Atlas ScriptManager control. Among other things, that control downloads the JavaScript files needed to support Atlas on the client. Then I wrapped the TreeView in an Atlas UpdatePanel control. UpdatePanel effectively subsumes the TreeView's TreeNodeCollapsed and TreeNodeExpanded events and converts them into asynchronous XML-HTTP callbacks. The UpdatePanel <Triggers> element identifies the TreeView events I want to subsume.

Figure 2 Convert TreeView Postbacks into Callbacks (Close Figure 2)
<head runat="server">
      ...
    <atlas:ScriptManager ID="ScriptManager1"
        EnablePartialRendering="true" runat="server" />
</head>
...
<atlas:UpdatePanel ID="UpdatePanel1" runat="server">
    <Triggers>
        <atlas:ControlEventTrigger ControlID="TreeView1"
            EventName="TreeNodeCollapsed" />
        <atlas:ControlEventTrigger ControlID="TreeView1"
            EventName="TreeNodeExpanded" />
    </Triggers>
    <ContentTemplate>
        <asp:TreeView ID="TreeView1" runat="server" ...>
          ...
        </asp:TreeView>
    </ContentTemplate>
</atlas:UpdatePanel>

AtlasSiteMap was built and tested against the January release of Atlas, better known as the January CTP (Community Technology Preview). Now the March CTP is available (see microsoft.com/downloads/details.aspx?familyid=b01dc501-b3c1-4ec0-93f0-7dac68d2f787). AtlasSiteMap may or may not require modification to work with newer releases.

SiteMapPaths and Unmapped Pages

A second site map problem seeking a solution is exemplified by a situation I recently found myself in. I was building a site that presents newspaper-style content. Articles are displayed by passing an article ID in a query string to a page that fetches articles from a back-end database and renders them into HTML. That page doesn't appear in the site map because there's no good reason to go to that page without an article ID. Yet when the page is viewed, I want it to appear in the SiteMapPath, and I want the node text in the SiteMapPath to be the article headline.

The problem with this scenario is twofold. First, a SiteMapPath control doesn't know what to do with URLs that aren't included in the site map, so it displays nothing. Second, a hook is needed to modify—on the fly—the text displayed by SiteMapPath for the current node. While you can use a SiteMapPath control's ItemCreated and ItemDataBound events to modify node text on the fly, SiteMapPath doesn't fire these events for nodes that don't appear in the site map.

Enter the little-known SiteMap.ResolveSiteMap event. The documentation describes it this way: "Subscribers attach a SiteMapResolveEventHandler object to the static SiteMapResolve event to receive notification when the CurrentNode property is accessed. This enables the user to implement custom logic when creating a SiteMapNode representation of the currently executing page without requiring a custom provider implementation."

Behind the scenes, the SiteMapPath control reads the SiteMap.CurrentNode property to determine what to render out. Then SiteMap.CurrentNode, in turn, reads the default site map provider's CurrentNode property. In System.Web.SiteMapProvider, which is the base class for site map providers, CurrentNode's get accessor is implemented this way:

HttpContext context = HttpContext.Current;
SiteMapNode result = ResolveSiteMapNode(context);
if (result == null) result = FindSiteMapNode(context);
return ReturnNodeIfAccessible(result);

The helper method ResolveSiteMapNode fires a ResolveSiteMap event. By processing these events, you can add nodes to site maps on the fly and customize their properties to match the context of the current request—for example, set the node's Title property to the article headline that corresponds to the article ID in the query string. Furthermore, you don't have to remove these nodes when you're finished because ASP.NET considers them to be temporary nodes and they are not joined to the actual site map.

Figure 3 shows you how the sample site handles ResolveSiteMap events. The Global.asax Application_Start method, which executes when the application starts up, registers a static handler for ResolveSiteMap events. The handler is named HandleUnmappedNodes.

Figure 3 SiteMap.ResolveSiteMap Events in Global.asax (Close Figure 3)
<%@ Application Language="C#" %>

<script runat="server">
    void Application_Start(object sender, EventArgs e)
    {
        // Register a handler for SiteMap.SiteMapResolve events.
        SiteMap.SiteMapResolve +=
            new SiteMapResolveEventHandler(HandleUnmappedNodes);
    }

    public static SiteMapNode HandleUnmappedNodes(
        object sender, SiteMapResolveEventArgs e)
    {
        HttpContext context = HttpContext.Current;

        // Create a custom SiteMapNode for Unmapped.aspx.
        if (context.Request.Path.ToLower().Contains("unmapped.aspx"))
        {
            string param = context.Request["param"];
            string title = String.IsNullOrEmpty(param) ?
                "Unmapped Page" : param;
            SiteMapNode node = new SiteMapNode(SiteMap.Provider,
                "~/Unmapped.aspx", "~/Unmapped.aspx", title);
            node.ParentNode = SiteMap.Provider.RootNode;
            return node;
        }

        return null; // Do nothing for other URLs.
    }
</script>

The job of HandleUnmappedNodes is simple. If the current request targets a page named Unmapped.aspx, HandleUnmappedNodes creates a SiteMapNode representing the page and parents it to the site map's root node. In addition, the handler dynamically sets the SiteMapNode's Title property to "Unmapped page" if the request doesn't contain a query string parameter named param, or to the value of that parameter if it does. Therefore, if you request UnmappedPage.aspx, the SiteMapPath will show "Unmapped Page" for the current node. But if you request UnmappedPage.aspx?param=Microsoft, the SiteMapPath will show "Microsoft" for the current node.

You can see this for yourself in the sample site by typing some text into the TextBox in Default.aspx and clicking the "Go to UnmappedPage.aspx" button. Because the text you typed is passed to UnmappedPage.aspx in a query string, it will appear in the SiteMapPath control. On the other hand, if you click the button with the TextBox empty, there will be no query string and the SiteMapPath will simply show "Unmapped Page."

Multiple Site Maps

A third aspect of the sample site that you'll really be interested in if you want to get a handle on site maps is the manner in which it provides data to the two navigational elements: the TreeView on the left and the Menu across the top. Each presents the user with a set of links that is different from (and independent of) the other.

The secret to hosting two independent navigation controls in one page is to provide two site maps. By default, XmlSiteMapProvider looks for your site map in a file named Web.sitemap. You can change that by registering an independent instance of XmlSiteMapProvider and using the siteMapFile configuration attribute to identify the site map file. Figure 4 shows how. This snippet from web.config registers not one, but two instances of XmlSiteMapProvider, one that retrieves a site map from TreeView.sitemap, and another that retrieves a site map from Menu.sitemap. It also makes the former instance of XmlSiteMapProvider—the one named TreeViewSiteMapProvider—the default site map provider to tell the SiteMapPath control which site map to use. For good measure, it also deregisters the default instance of XmlSiteMapProvider since it won't be used anyway.

Figure 4 Registering Multiple Site Map Providers (Close Figure 4)
<siteMap defaultProvider="TreeViewXmlSiteMapProvider">
  <providers>
    <add name="TreeViewXmlSiteMapProvider"
         type="System.Web.XmlSiteMapProvider, System.Web,
               Version=2.0.0.0, Culture=neutral,
               PublicKeyToken=b03f5f7f11d50a3a"
         siteMapFile="TreeView.sitemap"/>
    <add name="MenuXmlSiteMapProvider"
         type="System.Web.XmlSiteMapProvider, System.Web,
               Version=2.0.0.0, Culture=neutral,
               PublicKeyToken=b03f5f7f11d50a3a"
         siteMapFile="Menu.sitemap"/>
    <remove name="AspNetXmlSiteMapProvider" />
  </providers>
</siteMap>

Now that the site map is split into two independent site map files, the next step is to bind the TreeView to one site map and the Menu to the other. That's accomplished with SiteMapDataSource's SiteMapProvider property. In the Site.master snippet shown here the TreeView control is bound to one SiteMapDataSource (TreeViewSiteMapDataSource) and the Menu control is bound to another (MenuSiteMapDataSource):

<asp:SiteMapDataSource ID="MenuSiteMapDataSource" runat="server"
  SiteMapProvider="MenuXmlSiteMapProvider" ... />
<asp:Menu DataSourceID="MenuSiteMapDataSource" runat="server" ... />
...
<asp:SiteMapDataSource ID="TreeViewSiteMapDataSource" runat="server"
  SiteMapProvider="TreeViewXmlSiteMapProvider" ... />
<asp:TreeView DataSourceID="TreeViewSiteMapDataSource"
  runat="server" ... />

Each SiteMapDataSource is bound to one of the XmlSiteMapProvider instances declared in web.config. The TreeView control is bound to TreeViewSiteMapDataSource, which is bound to TreeViewXml-SiteMapProvider, which is bound to TreeView.sitemap. The Menu control is bound to MenuSiteMapDataSource, which is bound to MenuXmlSiteMapProvider, which is bound to Menu.sitemap. As a result, the TreeView gets its data from TreeView.sitemap, and site map data for the Menu comes from Menu.sitemap.

The technique of splitting the navigation UI into two or more independent site maps is powerful for a couple of reasons. One, it isn't limited to two controls: you can partition a site map into two site maps or 100 site maps (although admittedly, 100 site maps and 100 navigation controls would make for a very confusing user interface!). Two, it circumvents the fact that XmlSiteMapProvider requires every URL in a given site map to be unique. You can't partition a site map internally and divvy up different parts of it to different controls if one URL appears in both halves of the site map. But you can duplicate URLs all day long if the URLs live in different site maps. Although the sample site's TreeView and Menu don't duplicate any links, you could share as many URLs as you like between them and XmlSiteMapProvider wouldn't complain since TreeView.sitemap and Menu.sitemap are two different site maps.

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (Microsoft Press, 2002). He's also a cofounder of Wintellect, a software consulting and education firm specializing in Microsoft .NET.


Concurrent Affairs: Reader/Writer Locks and the ResourceLock Library

Concurrent Affairs
Reader/Writer Locks and the ResourceLock Library
Jeffrey Richter

Contents
My ResourceLock Library

If multiple threads concurrently execute code that writes to or modifies a resource, then obviously the resource must be protected with a thread synchronization lock to ensure that the resource doesn't get corrupted. However, it is common to have a resource that is occasionally written to but frequently read from. If multiple threads are concurrently reading a resource, using a mutual exclusive lock hurts performance significantly because only one thread at a time is allowed to read from the resource. It's far more efficient to allow all the threads to read the resource simultaneously, and it's fine to do this if all of the threads treat the resource as read-only and do not attempt to write to or modify it.

A reader/writer synchronization lock can and should be used to improve performance and scalability. A reader/writer lock ensures that only one thread can write to a resource at any one time and it allows multiple threads to read from a resource simultaneously as long as no thread is writing at the same time.

The Microsoft® .NET Framework Class Library includes a ReaderWriterLock class in the System.Threading namespace that lets you obtain multiple-reader/single-writer semantics. While it is nice that a class like this exists, there are several problems with its implementation and I recommend you do not use it:

Performance Even when there is no contention for a ReaderWriterLock, its performance is very slow. For example, a call to its AcquireReaderLock method takes about five times longer to execute than a call to Monitor's Enter method.

Policy If a thread completes writing and there are both reader and writer threads waiting, the ReaderWriterLock releases the reader threads instead of the writer threads. But if you have a resource that is always being written to, you should use a mutually exclusive lock to guard access. And if you have a resource that's always being read from, you don't need a thread synchronization lock at all. So, the reason to use a reader/writer lock is if you expect a lot of reader threads and very few writer threads. Because the ReaderWriterLock releases any waiting readers when a writer releases the lock, writer threads may get queued up and take an unusually long time to get serviced. In fact, I know of some Web sites that were architected around this lock and had problems. Users would go to a Web page, submit a form to modify their profile data, then on the server the thread would call the ReaderWriterLock's AcquireWriterLock method to update the profile data. But other threads were reading from the data, so the writer thread wasn't allowed access in a reasonable amount of time and the client's browser would time out.

On the other hand, the policy of releasing reader threads when a writer thread releases the lock ensures that threads are always making progress and do eventually get serviced. Imagine a reader/writer lock that has a different policy: when a writer thread releases the lock, another waiting writer thread is allowed to own the lock. With this policy, if a lot of writer threads happen to show up, then reader threads are starved. I generally prefer a reader/writer lock that favors writers and if I expect a lot of writers, I'd use a mutually exclusive lock instead. In my opinion, the ReaderWriterLock that ships with the .NET Framework defeats the main purpose of using a reader/writer lock.

Recursion The ReaderWriterLock class supports recursion. This means it remembers which threads currently own the lock and if an owning thread attempts to acquire the lock recursively, it allows the thread to acquire the lock and increments a counter for the thread's ownership. The thread must then release the lock the same number of times so that the thread doesn't own the lock anymore. Although this seems like a nice feature, it comes at a very high cost. First, since multiple reader threads can own the lock simultaneously, the lock must keep the counter on a per-thread basis and this requires additional memory and time to update the counter. This feature contributes significantly to the ReaderWriterLock's poor performance. Second, it is sometimes useful to design an architecture where you need to acquire a lock in one thread and release the lock in another thread. Because of its recursion feature, the ReaderWriterLock prohibits this kind of application architecture.

Resource Leak Prior to version 2.0 of the .NET Framework, the ReaderWriterLock had a bug that caused it to leak some of the kernel objects it was using. These objects would be reclaimed only after process termination. Fortunately, this bug has been fixed in the .NET Framework 2.0.

For all these reasons, I never use the ReaderWriterLock that ships with the .NET Framework. Instead, I have created my own reader/writer lock that is super fast, favors writer threads over reader threads, and does not support recursion. My lock is called OneManyResourceLock because you can use it either to allow just one thread or to allow many threads access to a resource. The object model of my class looks like this:

public sealed class OneManyResourceLock : ResourceLock {
   public IDisposable WaitToRead()  { ... }
   public void        DoneReading() { ... }
   public IDisposable WaitToWrite() { ... }
   public void        DoneWriting() { ... }
}

I've written many reader/writer locks and all of them exhibited the convoy problem I discussed in my March 2006 column. I decided to fix this in my new OneManyResourceLock. Implementing my new version turned out to be fairly complicated. To give you a sense of my new lock, I created some state diagrams showing how WaitToWrite (Figure 1), WaitToRead (Figure 2), DoneWriting (Figure 3), and DoneReading (Figure 4) work.

Figure 1 WaitToWrite
Figure 1 WaitToWrite

To understand the state diagrams, you first need to know that the OneManyResourceLock has an Int32 field that maintains the state of the lock. An instance of the lock also holds one semaphore that waiting reader threads wait on and another that waiting writer threads wait on. The bytes of the Int32 lock state field represent different parts of the lock's state and are always manipulated using the interlocked methods I described in my first Concurrent Affairs column. The first byte represents the number of writers waiting (WW), the second represents the number of readers waiting (RW), the third represents the number of readers reading (RR), and the last byte represents the current disposition of the lock, which can be any of the following:

  • Free: no thread owns the lock.
  • Owned by writer (OBW): one writer thread owns the lock.
  • Owned by readers (OBR): one or more reader threads owns the lock.
  • Owned by readers and a writer is pending (OBRAWP): one or more reader threads own the lock but a writer thread is waiting. In this state, a new reader thread cannot own the lock.
  • Reserved for writer (RFW): the lock enters this state when a writer thread is waiting, when one leaves the lock with another writer thread waiting, or when the last reader thread leaves the lock and a writer thread is waiting.

Figure 2 WaitToRead
Figure 2 WaitToRead

The OneManyResourceLock class is much more complex than the Optex class described in my previous column, but it is quite useful and can greatly improve the performance and scalability of your application code.

Figure 3 DoneWriting
Figure 3 DoneWriting

Notice that the WaitToRead and WaitToWrite methods have a return type of IDisposable. I did this to offer the convenience of calling these methods with a C# using statement. In other words, you can write code like this:

public void ModifyResource() {
   using (m_OneManyResourceLock.WaitToWrite()) {
      // The code to modify the resource goes here
   }
}

Figure 4 DoneReading
Figure 4 DoneReading

and the C# compiler will compile the code as if you had written code like this:

public void ModifyResource() {
   IDisposable temp = m_OneManyResourceLock.WaitToWrite();
   try {
      // The code to modify the resource goes here
   }
   finally {
      if (temp != null) temp.Dispose();
   }
}

When WaitToWrite (or WaitToRead) is called, a reference to an object that implements IDisposable is returned. When Dispose is called on this object, the Dispose method internally calls DoneWriting (or DoneReading). This makes it very convenient to use any of the ResourceLock-derived types in your code. Note that I create just one Disposable object for any given lock instance; I don't create a new IDisposable object every time WaitToRead or WaitToWrite is called, thereby improving performance and decreasing memory consumption.

It would be pretty easy to add additional features to the OneManyResourceLock class. For example, you could add recursion so that an owning thread could acquire the lock multiple times. You could add TryWaitToRead/TryWaitToWrite methods that take timeout values rather than having the calling threads wait infinitely to gain access to the resource. You could change the policy of the lock to release any waiting readers when a writer releases the lock. In fact, you could record the time of the longest waiting reader and use this information to make the policy decision very intelligent. Of course, you could modify the WaitToRead/WaitToWrite methods so that they spin in user mode a few times before transitioning to kernel mode to wait on the semaphore.

In addition, if you know that the work performed by the threads once they own the lock is very short and of a finite duration, you could even create a reader/writer lock that doesn't need semaphores and never transitions into kernel mode. It spins entirely in user mode until the lock can be obtained by the calling thread. I have implemented a lock like this that I call the OneManySpinResourceLock. Be careful when you use it though, because calling threads never relinquish the CPU, so if there is lengthy contention, you can waste a lot of CPU time.

My ResourceLock Library

By now, you're aware of the various locks that ship with the .NET Framework that can be used to allow threads to access a shared resource in a thread-safe way: Monitor, ReaderWriterLock, Mutex, Semaphore, and EventWaitHandle. I've introduced a few others: SpinLock, Optex, OneManyResourceLock, and OneManySpinResourceLock. So now, the question is, with all these thread synchronization locks available, how do you know which is best for any given situation? The answer is that you probably don't know. It fact, it really depends on how your application is being used in the field. It could be that one customer typically writes to the data and only infrequently reads the data. Another customer might write to the data initially and subsequently considers the data read-only.

Is the resource generally accessed for a long or short period of time? Does the machine running your application have one CPU or more than one? Are the CPUs hyper-threaded? Are they multicore? All of these variables have some impact on your application and should influence the exact lock implementation you use in your code. But, how can you code your application so the thread synchronization locks are selectable at run time? Well, I have a proposal—and an implementation.

I have defined an abstract base class, called ResourceLock. This class has a bunch of virtual methods: WaitToRead, DoneReading, WaitToWrite, and DoneWriting. It offers Close and Dispose methods as well. ResourceLock is the base of many concrete classes I've also defined. The table in Figure 5 shows the set of classes (sorted alphabetically) I've derived from my ResourceLock base class.

Figure 5 ResourceLock-Derived Classes (Close Figure 5)

ResourceLock-Derived Type | Wrapped Thread Synchronization Lock | Exclusive | Recursion | Spins
ResourceLock-Derived Type | Wrapped Thread Synchronization Lock | Exclusive | Recursion | Spins
ExclusiveSpinResourceLock | My SpinLock |  |  |
MonitorResourceLock | The .NET Framework Monitor |  |  |
MutexResourceLock | The .NET Framework/Windows Mutex |  |  |
NullResourceLock | None (provides no synchronization at all) | N/A | N/A |
OneManyResourceLock | My reader/writer lock |  |  |
OneManySpinResourceLock | My reader/writer spin lock |  |  |
OptexResourceLock | My Optex lock |  |  |
ReaderWriterResourceLock | The .NET Framework ReaderWriterLock |  |  |
SemaphoreResourceLock | The .NET Framework/Windows Semaphore |  |  |

Each type represents an existing thread synchronization lock you should already be familiar with. But, the class wraps the locks so they all offer the same reader/writer programming interface. You'll find the code for all of these classes in my PowerThreading library, which can be downloaded from my Web site (see my biographical information at the end of this column).

When your application starts up, you construct an instance of the ResourceLock-derived type you desire and assign the object's reference to a variable of type ResourceLock (the abstract base class). In your source code, always use reader/writer semantics and call the WaitToRead, DoneReading, WaitToWrite, DoneWriting methods using the ResourceLock variable. Note that the table shows the different capabilities of each lock.

  • A lock with a [image] in the Exclusive column identifies a lock that allows only one thread at a time to access the resource. So, even if two threads call WaitToRead, the lock will only let one thread at a time access the resource.
  • A lock with a [image] in the Recursion column identifies a lock that allows a thread to own the lock multiple times. The thread must release the lock the same number of times before it completely releases the lock.
  • A lock with a [image] in the Spin column identifies a lock that spins the calling thread in user mode while it waits to own the lock. Calling threads never transition to kernel mode. Locks that spin should be used with great caution and only when the guarded work performed by the thread is known to be of very short duration. You also need to make sure that all waiting threads are of the same priority and that priority boosting is disabled for these threads by calling the Win32® SetProcessPriorityBoost or SetThreadPriorityBoost functions.

It's clear from the table that some permutations of Exclusive, Recursion, and Spin do not have corresponding locks. While it would be possible to create such locks, I haven't found them necessary in my own work so I haven't gone to the trouble. If you're doing any cross-AppDomain or cross-process synchronization, though, you really need a lock that is a wrapper around a kernel object such as a mutex or semaphore. My MutexResourceLock and SemaphoreResourceLock classes don't offer constructors that let you set a string name for the underlying mutex and semaphore objects, but modifying the code to support this should be trivial.

There's one more thing I should point out: the actual class you construct when your application starts can be determined dynamically. For example, when your application starts up, it could determine how many CPUs are in the machine and whether those CPUs are hyper-threaded or multicore and then use this information to decide which specific ResourceLock-derived class to construct. You could also have an application setting in an XML file or equivalent that you use to determine what class to create, which would let you try different locks without modifying and recompiling the source code. In addition, you could instrument your application's code and monitor the behavior the application exhibits when deployed in the field. Based on the results, the application could choose a specific lock for its next run, allowing your application to fine-tune itself to each customer!

Just for fun, I decided to run some performance tests on these different locks. My test application had just one thread that first called WaitToRead/DoneReading two million times and then called WaitToWrite/DoneWriting two million times. In these tests there was no contention for the lock because only one thread was accessing the lock object. Then I ran the test again, this time with four threads calling both sets of methods two million times. I used a 2.4GHz AMD Athlon 64 X2 dual-core CPU system for the tests. The results in Figure 6 should give you a feel for the performance variations of the locks. Obviously, the results will be different on differently configured machines.

Figure 6 Lock Performance (in seconds) (Close Figure 6)

ResourceLock-Derived Type | 1 Thread Reading | 1 Thread Writing | 4 Threads Reading | 4 Threads Writing
NullResourceLock | 0.055 | 0.060 | 0.085 | 0.077
OneManySpinResourceLock | 0.096 | 0.125 | 1.088 | 1.952
OptexResourceLock | 0.105 | 0.102 | 1.211 | 1.076
OneManyResourceLock | 0.125 | 0.163 | 1.676 | 1.588
MonitorResourceLock | 0.144 | 0.141 | 0.503 | 0.496
ExclusiveSpinResourceLock | 0.149 | 0.113 | 1.743 | 1.380
ReaderWriterResourceLock | 0.487 | 0.522 | 1.782 | 2.187
MutexResourceLock | 3.841 | 3.802 | 24.094 | 24.052
SemaphoreResourceLock | 3.941 | 3.771 | 22.738 | 22.915

The rows are sorted using the 1-Thread Reading time, from best performing (NullResourceLock) to worst performing (SemaphoreResourceLock). Notice that, for each lock, the 1-Thread times for reading and writing are about the same. For example, OptexResourceLock shows 0.105 seconds for reading and 0.102 seconds writing. This is expected because, without contention, there isn't much difference between a thread acquiring/releasing a reader lock versus acquiring/releasing a writer lock.

In the 4-Thread Reading and Writing columns, you'll notice that the time to read versus time to write is also about the same for all the exclusive locks. For example, the MonitorResourceLock shows 0.503 seconds for reading and 0.496 seconds for writing; the 0.007 second difference here is obviously just noise. The nearly identical times are also expected because these locks let only one thread at a time own the lock.

However, notice the disparity between reading and writing using four threads for a nonexclusive lock. For example, the ReaderWriterResourceLock (see Figure 7) shows 1.782 seconds for reading and 2.187 seconds for writing; writing took more than 20 percent longer than reading. That's not surprising. Since multiple threads can access the resource simultaneously for reading whereas only one thread at a time can access the resource for writing, we'd expect the reading time to be substantially less, proving that the lock is doing what it is supposed to be doing.

Figure 7 ReaderWriterResourceLock (Close Figure 7)
public sealed class ReaderWriterResourceLock : ResourceLock {
    private readonly ReaderWriterLock m_lockObj = new ReaderWriterLock();

    public ReaderWriterResourceLock() { }

    protected override void OnWaitToWrite() {
        m_lockObj.AcquireWriterLock(-1);
    }
    protected override void OnDoneWriting() {
        m_lockObj.ReleaseWriterLock();
    }

    protected override void OnWaitToRead() {
        m_lockObj.AcquireReaderLock(-1);
    }
    protected override void OnDoneReading() {
        m_lockObj.ReleaseReaderLock();
    }
}

Finally, my code also includes an IResourceLock interface, which is implemented by my ResourceLock abstract base class. The interface lets you define a type and make that type follow the same reader/write lock programming pattern discussed here. Suppose, for example, you have a CustomerOrder class that implements my IResourceLock interface. Now you can use reader/writer semantics to access an instance of the CustomerOrder class. When you define your CustomerOrder class, most likely you would have a private ResourceLock field that will be initialized to refer to one of the ResourceLock-derived types. Of course, your CustomerOrder class would then have to implement the IResourceLock methods and, internally, you'd have each of the methods delegate to the corresponding method. Here is an example:

public sealed class CustomerOrder : IResourceLock {
   private ResourceLock m_rl = new OneManyResourceLock();

   public IDisposable WaitToRead()  { return m_rl.WaitToRead(); }
   public void        DoneReading() { m_rl.DoneReading(); }
   public IDisposable WaitToWrite() { return m_rl.WaitToWrite(); }
   public void        DoneWriting() { m_rl.DoneWriting(); }
   ...
}

I've been using my library now for several years with great results. I love the fact that it lets me separate the kind of locking I need to do from the exact lock I decide to use. Now, when I write code, I always think about reader/writer locking for a shared resource. After seeing how the code performs, I decide exactly which lock to use. In fact, I can even use the NullResourceLock to turn off locking altogether if I determine it's not necessary to have concurrent access to a resource.

Send your questions and comments for Jeffrey to mmsync@microsoft.com.

Jeffrey Richter is a cofounder of Wintellect (www.Wintellect.com), a training and consulting firm. He is the author of several books, including Applied Microsoft .NET Framework Programming (Microsoft Press, 2002). Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.


.NET Matters: Parameterized ThreadStart, EventWaitHandle, and More

.NET Matters
Parameterized ThreadStart, EventWaitHandle, and More
Stephen Toub

Q The ThreadPool.QueueUserWorkItem method lets me pass information to the delegate that will be executed on the worker thread. That's great, but I don't want to use a thread from the pool for this purpose as my scenario really requires a dedicated thread. Unfortunately, the ThreadStart delegate I use to start a new thread is parameterless. How do I pass data to a new thread?

A There are several common solutions when using the Microsoft® .NET Framework 1.x, and another, easier solution for the .NET Framework 2.0. All of the solutions for the .NET Framework 1.x are variations on the same theme, based on the fact that you can pass to a Thread's constructor a ThreadStart delegate that wraps an instance method of a class. This lets you create a new instance of a class that contains as member variables the state you want to be available to the background thread. That class also contains the entry method for the thread (the method to be wrapped by the ThreadStart delegate). Since this method is part of the same class that contains the relevant state, it has access to the state.

With this in mind, Figure 1 shows a general solution to the problem. The WorkerThreadState class contains two data members, a WaitCallback (a delegate with a single object parameter) and an object that represents the state to be passed to the background thread. It also exposes a parameterless public method that simply executes the stored delegate, passing in the background state. Since the Execute method takes no parameters, it can be wrapped in a ThreadStart delegate, and thus it can be used as the entry point with a new Thread. This makeshift solution thus allows you to pass methods to ThreadStart that take a single object parameter, albeit with a riff on the original syntax:

static void Main() {
    string state = "Hello, background world!";
    Thread t = new Thread(new ThreadStart(new WorkerThreadState(
        new WaitCallback(MyMethod), state).Execute));
    t.Start();
}

static void MyMethod(object state) {
    Console.WriteLine("State: " + state);
}
Rather than wrapping MyMethod with a ThreadStart delegate (which can't be done given the difference in signature), MyMethod is wrapped with a WaitCallback (which has the correct signature). That WaitCallback and the state to be supplied to it are wrapped in a new WorkerThreadState, and that WorkerThreadState's parameterless Execute method is wrapped with a ThreadStart delegate. A bit of indirection, to be sure, but it accomplishes the goal. If you're using C# 2.0, you can even take advantage of delegate inference to simplify the call:
Thread t = new Thread(new WorkerThreadState(MyMethod, state).Execute);
Of course, if you're using C# 2.0, you're also using the .NET Framework 2.0, and there's a much simpler solution. The main purpose of a large class library, such as the .NET Framework base class library (BCL), is to provide functionality you can take advantage of in your code. This prevents you and everyone else from having to reinvent the wheel, which helps alleviate code bloat, decreases the amount of code you need to test (since the library has already been tested), and decreases the time to market for your code. Given how frequently used the previous pattern was in applications written for the .NET Framework 1.x, the .NET Framework 2.0 includes a baked-in solution so you don't have to implement a solution like the one in Figure 1. Not only is there a constructor for Thread that accepts a ThreadStart delegate, Thread in the .NET Framework 2.0 exposes another constructor that accepts a ParameterizedThreadStart delegate. ParameterizedThreadStart has the same signature as WaitCallback, and can thus be used with methods that accept a single object parameter. The Thread class's Start method has a parallel overload that accepts an object as a parameter, the state to be passed to the method wrapped with the ParameterizedThreadStart. So, in the .NET Framework 2.0, the previous code snippet could be written simply as:
string state = "Hello, background world!";
Thread t = new Thread(new ParameterizedThreadStart(MyMethod));
t.Start(state);
Again, due to delegate inference, the Thread's creation can be simplified even further:
Thread t = new Thread(MyMethod);
Certainly this is a nice reduction in the amount of code you have to write!

Figure 1 Passing State to a New Background Thread (Close Figure 1)
class WorkerThreadState
{
    private WaitCallback _callback;
    private object _data;

    public WorkerThreadState(WaitCallback callback, object data)
    {
       if (callback == null) throw new ArgumentNullException("callback");
       _callback = callback;
       _data = data;
    }

    public void Execute()
    {
       _callback(_data);
    }
}

One thing to be careful of when using ParameterizedThreadStart is another constructor now exposed on Thread:

public Thread(ParameterizedThreadStart, int);
If your method wrapped in a ParameterizedThreadStart is expecting an integer as its state, you might inadvertently attempt to pass that state as the second parameter to this constructor, rather than to the Start overload:
int state = 42;
Thread t = new Thread(MyMethod, state);
t.Start();
This will not produce the effect you intend. That second parameter is not the state for Thread, but rather the requested size for the thread's stack. The ability to set a thread's stack size is not new to Win32® developers, as the CreateThread function exposes that capability, but it is new to managed threads with the .NET Framework 2.0. Just because the option is exposed doesn't mean you should use it, though. The default stack size of 1MB should be appropriate for almost any application you're developing. If it's not, chances are you have a bug or poor design in your code. That said, there are valid circumstances where you might want to change the stack size.

An easy way to see the effect of this parameter is to create and start as many threads as you can. Suppose you have two gigabytes of virtual address space. If all of that space is available for thread creation and if each thread reserves 1MB of memory, you should be able to create around 2000 threads. Figure 2 shows a simple test harness for counting the number of threads that can be created and started with a specific stack size. Sure enough, calling it three times with starting stack sizes of 512KB, 1MB, and 2MB, respectively, yields the following output:

After 3828 threads: OutOfMemoryException
After 1917 threads: OutOfMemoryException
After 956 threads: OutOfMemoryException
I can't think of any scenarios in which you'd want to create so many threads; it would almost certainly wreck your performance. The point here is that changing the stack size of each thread changes the number of threads you can create, which is limited by available memory.

Figure 2 Creating as Many Threads as Memory Allows (Close Figure 2)
static void TestCreateThreads(int stackSize)
{
    ManualResetEvent mre = new ManualResetEvent(false);
    List<Thread> threads = new List<Thread>();

    for (long i = 0; ; i++)
    {
        try
        {
            Thread t = new Thread(
                delegate() { mre.WaitOne(); }, stackSize);
            t.Start();
            threads.Add(t);
        }
        catch (Exception exc)
        {
            Console.WriteLine("After " + i + " threads: " +
               exc.GetType().Name);
            break;
        }
    }

    mre.Set();
    foreach (Thread t in threads) t.Join();

    GC.Collect();
    GC.WaitForPendingFinalizers();
    GC.Collect();
}

Q I have two String variables referencing two different String instances. I'd like to query the first variable for the String it references, while at the same time copying the reference from the second String variable to the first. The trouble is that I'm doing this in a concurrent application, and I want to make sure this set of operations is atomic. The Interlocked.Exchange method seems to be exactly what I need, but the compiler complains when I try to use this overload:

object Exchange(ref object location1, object value1);
The error I receive from the compiler states that it can't convert from "ref string" to "ref object". Why not? And if that's not possible, what good is Interlocked.Exchange?

A Think about the contract you're creating when you author a method with a pass-by-reference parameter (signaled by the ref keyword in C#). For example, consider the following method:

void DoSomething(ref object data)
{
    data = "a string";
    data = 42;
    data = WebRequest.Create("http://msdn.microsoft.com/msdnmag");
}
In this code, the data argument can be set to anything at all within the body of DoSomething. It could be set to a string, a boxed integer, an HttpWebRequest, or anything else that is or that derives from System.Object, which in .NET is pretty much everything. This means that the variable passed by reference to this method must be capable of holding any managed type, since the method could attempt to store any type of object into it. And since the type safety rules of .NET prevent a variable of a certain type from storing a value or a reference to an object of an incompatible type (one that isn't the same type or that derives from that type), the variable you pass by reference to DoSomething must be capable of storing an instance of any type. The only type in the .NET Framework that fits that bill is System.Object.

Interlocked.Exchange suffers from the same burden. Its first parameter is an object reference passed by reference, and Exchange needs to be able to set this to an instance of any type, especially since half the purpose of Exchange is to set this parameter equal to the second parameter, and the second parameter can be anything (since it's of type System.Object). Thus, the first parameter passed by reference to Exchange must be of type System.Object, nothing more and nothing less (there is nothing less). In the .NET Framework 1.x this makes Interlocked.Exchange useless for variables with types other than the core numerical types like System.Int32 (each of which have their own Exchange overloads) and System.Object itself.

The .NET Framework 2.0 helpfully addresses this limitation with a new generic overload of Interlocked.Exchange:

public static T Exchange<T>(ref T location1, T value) where T: class
This overload lets you do exactly what you want:
private string str1 = ...;
private string str2 = ...;
...
string oldStr1Value = Interlocked.Exchange<string>(ref str1, str2);
And through the magic of type inference, the C# compiler allows you to omit generic type parameters where they can be inferred from method arguments. Thus, you could simply use:
string oldStr1Value = Interlocked.Exchange(ref str1, str2);
As an aside, I've heard several rumblings from the general development community that generics are good for little more than collections. This is a perfect example of how generics can be used to solve a number of problems unrelated to collections.

Q I noticed that there's a new EventWaitHandle class in the .NET Framework 2.0. How does this differ from AutoResetEvent and ManualResetEvent?

A Win32 exposes manual-reset and auto-reset events as a single type of event object, created using the CreateEvent function. This function accepts a Boolean bManualReset argument, which dictates whether the ResetEvent function must be used to change the event object to a nonsignaled state or whether the object is automatically reset to a nonsignaled state after a single waiting thread has been released. For the .NET Framework 1.x, this functionality was split into two separate managed classes, AutoResetEvent and ManualResetEvent, which simply wrap CreateEvent, supplying the appropriate bManualReset argument. Like Mutex, both AutoResetEvent and ManualResetEvent derived from WaitHandle, obscuring their relationship and leaving only their names to suggest their shared heritage.

EventWaitHandle remedies this in the .NET Framework 2.0. A concrete class, EventWaitHandle contains all of the functionality (and more) that AutoResetEvent and ManualResetEvent exposed in .NET Framework 1.x. All of EventWaitHandle's public constructors accept an argument of type EventResetMode:

public enum EventResetMode {
    AutoReset, ManualReset
}
Just like the bManualReset argument to CreateEvent, the EventResetMode tells the new EventWaitHandle whether to act as a manual-reset or an auto-reset event object. AutoResetEvent and ManualResetEvent now both derive from EventWaitHandle, highlighting their relationship as siblings. In fact, these two classes now contain nothing more than constructors used to pass the appropriate EventResetMode to their base class's constructor:
public sealed class AutoResetEvent : EventWaitHandle
{
    public AutoResetEvent(bool initialState) :
        base(initialState, EventResetMode.AutoReset) {}
}

public sealed class ManualResetEvent : EventWaitHandle
{
    public ManualResetEvent(bool initialState) :
        base(initialState, EventResetMode.ManualReset) {}
}
If you're using AutoResetEvent or ManualResetEvent in code you're upgrading to the .NET Framework 2.0, don't fret about this change. Unless your code is using reflection to examine the immediate base class and will freak out if it finds something other than WaitHandle, you should be fine. In the .NET Framework 1.x, the only public methods exposed from AutoResetEvent and ManualResetEvent were Set and Reset, and both of these are available on EventWaitHandle with the exact same signature. Your code should compile and run just fine.

EventWaitHandle is more than just the result of refactoring. It exposes several frequently asked-for features not provided in the .NET Framework 1.x. This new functionality is best exemplified by one of the public constructors on EventWaitHandle:

public EventWaitHandle(
    bool initialState, EventResetMode mode,
    string name, out bool createdNew,
    EventWaitHandleSecurity eventSecurity);
The initialState parameter serves the same purpose as it does with AutoResetEvent and ManualResetEvent, determining whether the initial state of the event is signaled or nonsignaled, and the mode parameter determines the type of the event, as already discussed. The third parameter, name, allows for the creation of named events. Specifying null for this parameter reverts to the same unnamed event functionality available in the .NET Framework 1.x. But specifying a non-null string allows you to create non-local events, events that are systemwide and that can be accessed by other processes on the machine (in the .NET Framework 1.x, named synchronization primitives like this were available only in the form of a Mutex). This lets you coordinate resource usage across process boundaries. You can open an existing named event using the static EventWaitHandle.OpenExisting method, but you can also use the EventWaitHandle constructor. When you create a named event, the fourth parameter (the createdNew output Boolean) will tell you whether you created a new event with this name or opened an existing event with the same name.

The final parameter, eventSecurity, hints at a new feature that spans all of the kernel objects exposed through the .NET Framework 2.0: the ability to use Windows® access control security with named kernel objects. With this feature you can restrict which users on the system have access to the named event, as well as what they're able to do with it. It also gives you control over security auditing on the event.

Q I'm writing a class library that needs to perform some work asynchronously. I want to fire off some events from the background thread doing the actual work, but I want the event handlers for the event to be raised on an appropriate thread for the current execution environment. So, for example, if the library is being used in a Windows Forms-based application, I'd like the events to be raised on the UI thread. How can my library determine the environment and then do this marshaling correctly?

A For Windows Forms, you'd typically use the ISynchronizeInvoke interface of one of the controls created on the UI thread. This would marshal the call to the UI thread, making the executed method safe for UI control manipulation. Of course, if you don't have a reference to a valid control, as might be the case in a library you're writing for use in a variety of contexts, it is more difficult to use this technique.

The .NET Framework 2.0 introduces the SynchronizationContext class to address this and related scenarios. You can think of SynchronizationContext as ISynchronizeInvoke on steroids, providing the same functionality but at a higher level of abstraction, allowing it to work with environments other than Windows Forms. SynchronizationContext provides two methods you'll use most frequently, Send and Post. Send corresponds to ISynchronizeInvoke.Invoke, while Post corresponds to ISynchronizeInvoke.BeginInvoke. The idea here is that from your library, you can retrieve the current SynchronizationContext from the SynchronizationContext.Current static property. You can then use its Post or Send method to marshal your invocation to the proper thread.

Though not abstract, SynchronizationContext by itself isn't all that useful. The default implementation of the Send and Post methods are as follows:

public virtual void Send(SendOrPostCallback d, object state)
{
      d(state);
}

public virtual void Post(SendOrPostCallback d, object state)
{
      ThreadPool.QueueUserWorkItem(new WaitCallback(d.Invoke), state);
}
public delegate void SendOrPostCallback(object state); Both Send and Post accept a SendOrPostCallback (a delegate with a single object parameter) and an object instance to be supplied to the target method. The default implementation of Send simply invokes the delegate, passing to it the state object. The default implementation of Post simply queues the delegate and the state to the ThreadPool. Not very interesting. What is interesting is that both of these methods are virtual, the idea being that specialized SynchronizedCallback-derived classes will be created for each environment that has specific needs when it comes to synchronizing invocations across threads.

Windows Forms is, of course, one such environment. It provides a class derived from SynchronizationContext: WindowsFormsSynchronizationContext. (Tricky, huh?) An instance of WindowsFormsSynchronizationContext is created before the application's main message loop is started and is stored into the ExecutionContext for the UI thread (instances of WindowsFormsSynchronizationContext may also be created when controls are instantiated from non-UI threads). When instantiated, WindowsFormsSynchronizationContext queries the current Application's thread context for the control to use for marshaling. It uses this control in its overrides of Send and Post, which look something like the following:

public override void Send(SendOrPostCallback d, object state)
{
    controlToSendTo.Invoke(d, new object[] { state });
}
public override void Post(SendOrPostCallback d, object state)
{
    controlToSendTo.BeginInvoke(d, new object[] { state });
}
As you can see, WindowsFormsSynchronizationContext handles all of the marshaling for you. So, if your library will be executing some code on a background thread and needs to marshal some code to a safe UI context, all you need to do is retrieve the current WindowsFormsSynchronizationContext, store it, and then from the background thread use either Post or Send. This is exactly what the BackgroundWorker component does.

BackgroundWorker is a new class in the .NET Framework 2.0 that makes it easy to execute long-running tasks from a background thread, executing progress updates and completion notifications on the UI thread. You register with the BackgroundWorker instance's DoWork event a DoWorkEventHandler for the code you want to be run on the background thread. You can then also register event handlers with the ProgressChanged and RunWorkerCompleted events. To start the background task running, you use the RunWorkerAsync method. Before using delegate asynchronous invocation to start the background task, however, BackgroundWorker retrieves and stores the current SynchronizationContext (though not directly, but rather through AsyncOperationManager.CreateOperation, which indirectly wraps SynchronizationContext.Current). The code running on the background thread uses the ReportProgress method to execute code back on the UI thread; under the covers, BackgroundWorker is simply using the stored SynchronizationContext and its Post method to marshal the call.

Windows Forms isn't the only environment that provides a SynchronizationContext-derived class. ASP.NET also provides one, AspNetSynchronizationContext, though it's not public and is not meant for external consumption. Rather, it is used under the covers by ASP.NET to facilitate the asynchronous pages functionality in ASP.NET 2.0 (for more information, see msdn.microsoft.com/msdnmag/issues/05/10/WickedCode). This implementation allows ASP.NET to prevent page processing completion until all outstanding asynchronous invocations have been completed.

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is the Technical Editor for MSDN Magazine.


Netting C++: The .NET Wrap

Netting C++
The .NET Wrap
Stanley B. Lippman

Code download available at: NettingC2006_06.exe (165KB)
Contents
Compiling Native Code to .NET
Wrapping Native Code to Publish within .NET

This month, we are changing the column name from Pure C++ to Netting C++ to better reflect our focus on Microsoft® .NET programming using C++/CLI, the .NET extensions to Visual C++® that are supported in Visual Studio® 2005. In this first Netting C++ column, and in a number of upcoming ones, I'll look at moving a working native C++ application onto .NET through a series of incremental transformations using the C++/CLI language extensions of Visual C++ 2005. The application is a Text Query Language (TQL) for processing natural language texts, first developed back in 1996 for the third edition of my C++ Primer. It serves as a good illustration of object-oriented design (OOD), using the Standard Template Library (STL), and string manipulation in general. It should make a good vehicle for drilling down into .NET-based programming.

Here's the scenario: my company has a working native C++ application, TQL, that is the foundation of the business. We at the company realize it needs to be extended to make it Web-sensitive and to provide XML support. Moreover, it's currently a console application (it began life as a UNIX command-line application), and we want to front it with a cutting-edge GUI, as well as add more functionality that users have requested.

However, before doing all this neat stuff, the application must be brought over to .NET. Because the application works, and by some accounts works well, I don't necessarily want to change it, at least not without some profit from the change. So, the first step in netting TQL is to get it to compile as a .NET executable. That's the focus of the first section, "Compiling Native Code to .NET."

Though the process I describe enables TQL to execute under .NET, it doesn't use any of the .NET runtime services or provide interoperability with any other .NET-compliant languages. I'll get to this in the second section, "Wrapping Native Code to Publish within .NET," where wrapping TQL with a C++/CLI proxy class will provide the necessary elbow room for adding to and replacing pieces of the existing TQL application.

Subsequent columns will focus on mapping the native types to types supported by the .NET Common Type System (CTS) and examining the performance characteristics of the application as it transitions. We'll also look at the type information available to the runtime, using the ildasm command to explore the Common Intermediate Language (CIL) into which all .NET-based languages are compiled.

When that's done, we'll explore multithreading, Web services, cross-language interoperability with a C# ASP.NET front, XML support, and integrating with Windows Vista™. So, we have our work cut out for us. (Note that all the development work for this column is done through the Visual C++ IDE. I won't directly address compiler switches, therefore, but speak in terms of Projects.)

Compiling Native Code to .NET

The ultimate goal is to get the native code compiled as a .NET executable. To start out, I selected a Visual C++ | CLR | CLR Console Application from the New Project wizard, which generated the following code shell:

// Project wizard generated code
#include "stdafx.h"

using namespace System;

int main()
{
    Console::WriteLine(L"Hello World");
    return 0;
}
(I simplified it slightly by removing the array access to the command-line arguments as I'm not yet ready to use this.)

The TQL application has a tripartite design:

  1. A Query class hierarchy to support user-directed Boolean queries, such as AND (&&), OR (||), and NOT(!), against user-specified text. (You'll see an example of a user query session in Figure 1.)
  2. A UserQuery class that handles all aspects of the interactive user text query session.
  3. A TextQuery class that handles all processing of the actual text, from reading in the file(s) to building up an internal normalized structure against which to search to displaying the results.

Figure 1 TQL Query (Close Figure 1)
please enter file name: alice_emma

Enter a query -- please separate each item by a space.
Terminate query (or session) with a dot( . ).

==> fiery .
        fiery ( 1 ) lines match.
                Location(s): [ { 3, 3 } { 3, 9 } ]

Requested query: fiery

( 3 ) like a fiery bird in flight. A beautiful fiery bird, he tells her,

Enter a query -- please separate each item by a space.
Terminate query (or session) with a dot( . ).

==> beautiful && fiery .
        beautiful ( 1 ) lines match.
                Location(s): [ { 3, 8 } ]

        fiery ( 1 ) lines match.
                Location(s): [ { 3, 3 } { 3, 9 } ]

        beautiful && fiery ( 1 ) lines match.
                Location(s): [ { 3, 8 } { 3, 9 } ]

Requested query: beautiful && fiery

( 3 ) like a fiery bird in flight. A beautiful fiery bird, he tells her,

Enter a query -- please separate each item by a space.
Terminate query (or session) with a dot( . ).

==> fiery || magical .
        fiery ( 1 ) lines match.
                Location(s): [ { 3, 3 } { 3, 9 } ]

        magical ( 1 ) lines match.
                Location(s): [ { 4, 1 } ]

        fiery || magical ( 2 ) lines match.
                Location(s): [ { 3, 3 } { 3, 9 } { 4, 1 } ]

Requested query: fiery || magical

( 3 ) like a fiery bird in flight. A beautiful fiery bird, he tells her,
( 4 ) magical but untamed. "Daddy, shush, there is no such creature,"

Each part corresponds to a pair of files: Query.h and Query.cpp; UserQuery.h and UserQuery.cpp; and TextQuery.h and TextQuery.cpp. These are the six native C++ files I'll import into my Project and ideally compile without having to make significant changes to the code. That, at least, is one of the primary ideas behind the C++/CLI extensions.

One strategy, of course, is to simply import all the files at once, hit build, and see what happens. Though this may work with a small number of files, generally it doesn't scale because warnings and errors tend to cascade. I prefer to add files incrementally, one at a time. And I'd rather compile the header files first, since usually you have to work through inclusion issues. Besides, they're generally easier to compile than program files.

So that's the process I used, and it was almost totally painless except for two compiler warnings resulting from comparing a signed and unsigned int:

// add existing header files from TQL one at a time
// 1. no compilation errors ...
#include "Query.h"

// 2. two warnings: signed/unsigned comparisons
#include "TextQuery.h"

// 3. again, two warnings on signed/unsigned comparisons
#include "UserQuery.h"
The warnings stem from this line:
if ( _paren < _current_op.size() )
where _current_op is an STL stack class, size() returns an unsigned int, and _paren is an int counter of parentheses in a user-query such as "alice | ( hair & ( magical | fiery) )". The compiler correctly advises:
warning C4018: '<' : signed/unsigned mismatch
In the application itself, however, _paren can't be either negative or so large as to invalidate the comparison, so there's no semantic danger in the comparison, only the possible performance cost of promoting the int to unsigned. When we look at performance issues, we can decide if this actually matters. For now, just live with the warnings. The idea is to get the code transitioned.

Once all the header files compile, I add the program files, again one at a time. The only error, it turns out, was my failure to add precompiled header support. For example:

// 4. add Query.cpp
// compiler message:
//   fatal error C1010: unexpected end of file while looking for
//   precompiled header.   Did you forget to add '#include "stdafx.h"' //   to your source?
In fact, that's exactly what I did, not just with Query.cpp, but with TextQuery as well:
// 5. add TextQuery.cpp
// a. same fatal error C1010 ... and same solution ...
// b. compiled fine ...
Happily, by UserQuery.cpp, I had been trained:
// 6. add UserQuery.cpp and add #include "stdafx.h"
If only porting were so simple! One design decision the Visual C++ team made to minimize the potential impact of introducing new keywords into the language was to make them contextual rather than reserved. A contextual keyword has a special meaning only within specific program contexts. Within the general program, for example, sealed is treated as an ordinary identifier. However, when sealed occurs within the declaration portion of a class or virtual function, it is treated as a keyword only within the context of that declaration.

A spaced keyword is a special case of a contextual keyword. It literally pairs an existing keyword with a contextual modifier separated by a space. The pair, such as "ref class", is treated as a single unit rather than as two separate keywords. For example:

ref class Foo   // spaced contextual keyword
    { ... } ^ref;  // non-contextual use as identifier
In this case, the first use of ref indicates that Foo is a .NET reference class type. The second use of ref is the identifier of the Foo object. The hat (^) symbol identifies the object type to be a .NET reference type.

Now that the source code compiles, the next step is to replace the canned Hello, World greeting with an invocation of the application, as shown in the following:

#include "stdafx.h"
#include "Query.h"
#include "TextQuery.h"
#include "UserQuery.h"

using namespace System;

int main()
{
    TextQuery tq;

    tq.build_up_text();
    tq.query_text();

    return 0;
}
When compiled and executed, the application just works, which is pretty cool. The code in Figure 1 shows a slightly cleaned up session, with the input in red.

This ability of Visual C++ to turn native code into .NET without significant changes to the source code was one of the primary requirements of both the original Managed Extensions for C++ released with Visual Studio .NET back in 2001, and for the C++/CLI extensions that replace the Managed Extensions with Visual Studio 2005.

Wrapping Native Code to Publish within .NET

Now the application is up and running under .NET, but its type hierarchy is opaque both to the runtime and to other languages because the original source code, not surprisingly, is not built within the CTS. So, the next step is to integrate some managed code into the application.

What I want to do is open up TQL to .NET, but without investing heavily in either programmer resources or schedule. My preferred strategy is to wrap the native application in a CTS reference class type—in TQL's case, this means writing a reference class that holds a TextQuery object.

So, the first question is, how does a reference class type declare a native class object as a member? The jokey answer is, very carefully! An actual object of a native class cannot be stored directly within a reference class type; rather, you can only store a pointer to the native class object. So, the initial class will look something like the following:

#include "TextQuery.h"

ref class TQL {
private:
      // our native object through which to invoke our application
    TextQuery *pTQuery;
};
The TextQuery pointer member, as in native C++, is best initialized within each associated constructor; similarly, I can free it within the destructor, as follows:
ref class TQL {
public:
    // constructor creates a TextQuery object on native heap
    // destructor frees it
    TQL()  { pTQuery = new TextQuery; }
    ~TQL() { delete pTQuery;          }
};
This sort of maneuver—acquiring resources through initialization and freeing them within the destructor—should look familiar to the ISO-C++ programmer. Things are a bit more subtle, though, because of how the .NET managed heap handles garbage collection. But I'll save that discussion until my next column.

So far, the class has been concerned only with maintaining the lifetime of the native class object within the managed reference class. What the native application needs now is some form of publication for its interface within .NET. Initially, I'll simply provide a one-to-one mapping of the public operations available to TQL users. To simplify things, I've included just the two main functions:

ref class TQL {
public:
    // the operations we wish to publish within .NET
    // these will generally be inlined by the compiler

    void build_up_text(){ pTQuery->build_up_text(); }
    void query_text()   { pTQuery->query_text();    }
};
In practice, a one-to-one mapping is likely to be far too expensive since each invocation represents a boundary crossing between .NET and native code. But again, the goal is just to get up and running.

The TQL class definition is placed in a header file called TQL.h, which will replace TextQuery.h in the include file list. Now, I'll revise the main function to use the TQL class rather than the native TextQuery class, like so:

// Our Managed TQL Wrapper Put to Work ...
int main()
{
    TQL ^tq = gcnew TQL;
    tq->build_up_text();
    tq->query_text();
    return 0;
}
Notice the gcnew keyword. You use this within C++/CLI when you want to allocate an object from the .NET managed heap (but continue to use new to allocate a non-CTS type within the native heap). The declaration
TQL ^tq
declares tq to be a tracking handle to a TQL reference class object. The "tracking" adjective underscores the idea that a reference type sits on the managed heap and can therefore transparently move locations during garbage collection heap compaction. A tracking handle is automatically updated by the runtime during the execution of the program.

Note that tracking handles use the arrow member selection operator (->) rather than the dot member selection operator (.) to invoke member functions or access class data members. When compiled and executed, the application works exactly as in the first section.

So that's a wrap, folks! I've taken a 10-year old native application and stubbed it into .NET with nary a whimper. That's the good news. The not-so-good news is that this implementation has a number of problems reflecting subtleties of programming in .NET. I'll address these in detail in the next column.

Send your questions and comments for Stanley to purecpp@microsoft.com.

Stanley B. Lippman began working on C++ with its inventor, Bjarne Stroustrup, in 1984 at Bell Laboratories. Later, Stan worked in feature animation both at Disney and DreamWorks and served as a Software Technical Director on Fantasia 2000. He has since served as Distinguished Consultant with JPL, and an Architect with the Visual C++ team at Microsoft.


C++ at Work: Managed Code in Visual Studio 2005

C++ At Work
Managed Code in Visual Studio 2005
Paul DiLascia

Code download available at: CatWork2006_06.exe (262KB)
Contents
Minor Language Changes
Migrating to the Managed World
Running ManWrap
Donning a New Hat
Two Minor Gripes

Many of you are no doubt in the process of upgrading to Visual Studio® 2005, so I thought now would be a good time to relate some of my own experiences with the new compiler. What took me so long? Hey, I'm a retro kind of guy! Better late than never!

The first thing you'll notice about Visual Studio 2005 is that it has a version manager that looks at your project and decides which version of itself to launch. You can install Visual Studio 2005 alongside Visual Studio .NET 2003; the two will cohabit your computer peacefully, which is great if you want to upgrade your projects at your own leisure. If you open an older project, Visual Studio 2005 prompts you to save a copy before converting, then generates an XML report that describes whatever issues it found.

Minor Language Changes

To take Visual Studio 2005 for a spin, I compiled a half dozen or so projects from past columns. They all required minor edits, mostly to accommodate a handful of language-conformance changes that make Visual Studio 2005 a modern C++ compiler. Most of the "new" rules have been in the C++ standard for a while, but only now does Visual Studio enforce them. The two most common language mutations I saw are for-loop scoping and default int types.

Local variables inside for loops no longer have scope outside the loop. Before, it was possible to write:

for (int i=0; i<max; i++) {
   // do something
}
if (i>0) {
   // do something else
}

In this snippet, the variable i is scoped inside the for statement and used outside it. Officially, C++ doesn't allow this, so now you need to rewrite your code like so:

int i; // move outside for loop
for (i=0; i<max; i++) {
   // do something
}
if (i>0) {
   // do something else
}

Undeclared static variables, both local and global, no longer default to integer types. Previously, you could write

const BUFLEN=255;

and the compiler would implicitly assign BUFLEN type int. Implicit ints are now forbidden. You must declare your type like so:

   const int BUFLEN=255;

This applies to variables of all kinds—static, global, data members, and function return types. If you leave out the int, you'll get "error C4430: missing type specifier - int assumed. Note: C++ does not support default-int".

Another whole category of C/C++ changes relates to the new Safe C and Safe C++ libraries. These libraries provide more secure versions for many of the oldfangled C runtime (CRT) functions you know and love: strcpy, fopen and others. I plan to write more about Safe C++ in a future column. If you can't wait, read Martyn Lovell's excellent article, "Safe! Repel Attacks on Your Code with the Visual Studio 2005 Safe C and C++ Libraries" in the May 2005 issue (msdn.microsoft.com/msdnmag/issues/05/05/SafeCandC).

So much for C++, what about MFC? Visual Studio 2005 introduces no major changes to MFC which, as I've said before, is a good thing. It means MFC is stable. I did notice, however, that the return type for CWnd::OnNcHitTest has morphed from UINT to LRESULT. There may be other minor tweaks, but you shouldn't run into anything major that would break your MFC apps.

Migrating to the Managed World

C++ and MFC are stable programming systems, so I didn't expect any upgrade glitches in those areas. The big changes betweenVisual Studio .NET 2003 and Visual Studio 2005 lie in the realm of managed code. There's the new C++/CLI syntax you've been reading about—think of it as Managed Extensions V2. If you're not ready to wrap your brain around tracking handles just yet, you can use /clr:oldSyntax to stick with the old Managed Extensions. That's what you get by default when you upgrade a managed/mixed project from Visual Studio .NET 2003 to Visual Studio 2005.

To test the new compiler on managed code, I used the ManWrap library that I created for my April 2005 article "Wrappers: Use Our ManWrap Library to Get the Best of .NET in Native C++ Code". ManWrap comprises a wrapper DLL (RegexWrap.dll) and three test programs: RegexTest, RegexForm, and WordMess (see Figure 1). ManWrap is a good bellwether because, while small, it's fairly sophisticated in some of the things it does—like mixing managed and unmanaged code in a DLL. RegexWrap is a native DLL that wraps the common language runtime (CLR) Regex class.

Figure 1 Testing Managed Code
Figure 1 Testing Managed Code

So I kept the old syntax and let the compiler rip... In seconds, a flurry of errors scrolled across my screen: "C3395 ... __declspec(dllexport) cannot be applied to a function with the __clrcall calling convention." Say whaaaat?

ManWrap is a library that lets you wrap managed classes in purely native C++ so you can call the CLR from native C++ code that's compiled without /clr. For example, say you have a legacy app that uses the old Visual C++® 6.0 compiler, but you want to add a feature that calls the CLR. You can't call managed classes directly from C++ without throwing the /clr switch (and Visual C++ 6.0 doesn't have one to throw!), so the only way you can reach managed classes is to wrap them in a DLL with native entry points. ManWrap provides a general mechanism for writing such wrappers.

The trick at the heart of ManWrap is to use the special predefined preprocessor symbol _MANAGED to generate different code for internal and external consumption. Each wrapper class holds a single data member, a handle to the managed object:

#ifdef _MANAGED
# define GCHANDLE(T) gcroot<T*>
#else
# define GCHANDLE(T) intptr_t
#endif

Wrapper classes use GCHANDLE to declare their object handles like so:

// wrapper for managed Object
class CMObject {
   GCHANDLE(Object) m_handle;
};

The header file with CMObject is compiled two ways. When you build the wrapper DLL, you compile with /clr, so _MANAGED is defined and the compiler sees m_handle as gcroot<Object*>. When you build the native app that calls the wrapper, you compile without /clr, so _MANAGED is undefined and the compiler sees m_handle as intptr_t. This works because gcroot<Object*> is guaranteed to have the same size as intptr_t. Only the wrapper DLL knows what the handle is. To the outside (native) world, m_handle is just a magic cookie—similar to an HWND, HINSTANCE or any other kind of handle. The only catch is that the copy constructors and assignment operator must be true functions, not inline—so they call into the wrapper, which manipulates the handles as the true objects they are. (You can't copy intptr_t handles, you need to go through gcroot.)

So in addition to holding the handles, ManWrap wrappers define the usual methods to construct and copy objects. Each wrapper class also defines a ctor and operator-> so the wrapper can create and access native wrapper objects from their corresponding managed types. For example, there's a constructor to create a native CMRegex from a managed Regex. The wrapper classes use the ctor and operator-> internally. Figure 2 shows a code snippet from ManWrap.h, with the managed methods inside the #ifdef _MANAGED block. Note that the entire class is exported with WREXPORT, which expands to __declspec(dllexport) when building the DLL. This is what causes the C3395 error. You can't export managed methods (methods with managed parameters) using __declspec(dllexport) because native and managed functions use different calling conventions. Well, that makes sense—why would I attempt to export managed functions from a native DLL in the first place? But I'm not really exporting them. They're all defined inline. The managed methods are not required in the native interface, nor are they visible to it; but the compiler doesn't know this. Apparently Visual Studio 2005 isn't as smart as the old compiler, which let me dllexport the entire class. Or else it's too smart, because under most circumstances it makes no sense to export managed methods from native code. Whatever. The point is that Visual Studio 2005 won't let you export a class if it has any managed methods, period.

Figure 2 Exporting a Class Containing Managed Methods (Close Figure 2)
//////////////////////////////////////////////////////////////////
// Snippet from old (Visual Studio .NET 2003) version of ManWrap.h
// Note entire class declared using WREXPORT.

//////////////////
// Macros to control DLL. Use BUILD_MANWRAP_DLL when you’re writing
// your own wrapper. Use USE_MANWRAP_DLL if you’re writing an app that
// links to RegexWrap.dll. Don’t use anything if you link the code
// directly into your app, but then you have to compile this and
// ManWrap.cpp with /clr.
//
#if defined(BUILD_MANWRAP_DLL)
#  define WREXPORT __declspec ( dllexport )
#elif defined(USE_MANWRAP_DLL)
#  define WREXPORT __declspec ( dllimport )
#else
#  define WREXPORT
#endif

//////////////////
// Handle to managed object.
// Appears as gcroot<T> wrapper to managed code, intptr_t to native code.
//
#ifdef _MANAGED
# define GCHANDLE(T) gcroot<T>
#else
# define GCHANDLE(T) intptr_t
#endif

class WREXPORT CMObject {
protected:
   // handle to underlying managed object
   GCHANDLE(Object*) m_handle;
public:

#ifdef _MANAGED
   // visible to managed clients only: anything that deals with __gc
   // objects
   CMObject(Object* o) : m_handle(o) { }
   Object* ThisObject() const { return (Object*)m_handle; }
   Object* operator->() const { return ThisObject(); }
#endif

   // visible to all clients
   CMObject();
   CMObject(const CMObject& o);
   CMObject& operator=(const CMObject& r);
   ...
};

So what to do? What I'd like to do is tell the compiler, "export the entire class except these three methods." That is, I'd like a way to suppress __declspec(dllexport) for specific methods. Alas, there's no such option. Absent that, there are only two solutions I can think of: remove WREXPORT from the class declaration and add it to each native method or remove the offending methods entirely. Plan A is simpler, and that's the path I chose. Moving WREXPORT to the method declarations is a bore and somewhat error prone because it's easy to forget WREXPORT when you add new methods. If you do, though, the compiler will remind you.

Plan B, if you really, really want to dllexport the entire class, is to remove the offending methods: the managed copy constructor and dereference operator->. Then instead of writing

// inside wrapper class for MClass
(*this)->ManagedMethod();

which silently invokes the now-defunct operator->, you'd have to write:

(static_cast<MClass*>((Object*)m_handle))->ManagedMethod();

What a mouthful! You could introduce a macro to save typing:

THISOBJ(MClass*)->ManagedMethod();

But it's kind of clunky and you still have to deal with the constructor. If you're totally lost (and I expect many readers will be), don't worry, I chose Plan A, the simple approach. Figure 3 shows the revised code with WREXPORT on all the methods. After I made this change, ManWrap compiled just fine.

Figure 3 Exporting Managed Methods Individually (Close Figure 3)
////////////////////////////////////////////////////////////////
// New version of ManWrap.h for Visual Studio 2005.
// Now only the native methods get WREXPORT.

//////////////////
// Macros to control DLL. Use BUILD_MANWRAP_DLL when you’re writing
// your own wrapper. Use USE_MANWRAP_DLL if you’re writing an app that
// links to RegexWrap.dll. Don’t use anything if you link the code
// directly into your app, but then you have to compile this and
// ManWrap.cpp with /clr.
//
#if defined(BUILD_MANWRAP_DLL)
#  define WREXPORT __declspec ( dllexport )
#elif defined(USE_MANWRAP_DLL)
#  define WREXPORT __declspec ( dllimport )
#else
#  define WREXPORT
#endif

//////////////////
// Handle to managed object.
// Appears as gcroot<T> wrapper to managed code, intptr_t to native code.
//
#ifdef _MANAGED
# define GCHANDLE(T) gcroot<T*>
#else
# define GCHANDLE(T) intptr_t
#endif

//////////////////
// Wrapper for Object, base of .NET hierarchy.
// CMObject doesn’t use DECLARE_WRAPPER because it has no base class.
// For Visual Studio 2005, must explicitly export each method--can’t use
// WREXPORT on entire class.
//
class CMObject {
protected:
   GCHANDLE(Object) m_handle; // handle to underlying managed object
public:

#ifdef _MANAGED
   // visible to managed clients only: anything that deals with __gc
   // objects
   CMObject(Object* o) : m_handle(o) { }
   Object* ThisObject() const { return (Object*)m_handle; }
   Object* operator->() const { return ThisObject(); }
#endif

   // visible to all clients
   WREXPORT CMObject();
   WREXPORT CMObject(const CMObject& o);
   WREXPORT CMObject& operator=(const CMObject& r);

   WREXPORT BOOL operator==(const CMObject& r) const;
   WREXPORT BOOL operator!=(const CMObject& r) const { return ! operator==(r); }
   WREXPORT CString ToString() const;
   WREXPORT CString TypeName() const;
   WREXPORT BOOL IsNull() const;
};

Running ManWrap

Getting past the compiler is one thing, getting the code to run is something else. This is where I hit a wall—and almost hit my computer, too—in the form of an ASSERT bomb way down in dbgheap.c:

ASSERTE(_CrtIsValidHeapPointer(pUserData));

Yikes! The stack trace was mostly useless, full of system DLLs with no debug symbols. This is the worst kind of bug, the hardest to track down: your program dies somewhere deep in the system and you have no clue why. Well, almost no clue. Closer examination of the stack trace revealed about 50 frames back that my code was attempting to access a static ATL variable called g_Allocator. Aha! That was the subtle clue that broke the case.

g_Allocator is a static global. Initializing C++ static variables is always a delicate affair, particularly in DLLs. The compiler has to generate code to call the CRT initialization function that initializes your static variables before calling DllMain. Everything works fine in native land, but if your DLL calls managed classes, you can end up with loader-lock problems: Windows® attempts to load your DLL, which attempts to load the CLR, which in turn attempts to load your DLL—and, your app grinds to a halt in what's known as loader lock. In general, you can't load (require) a DLL while your DLL is loading. A common example of this occurs when you try to call ::MessageBox to display a diagnostic from your DllMain or static object constructor—it doesn't work.

To avoid loader lock, Visual Studio .NET 2003 mandated that managed DLLs should be /NOENTRY DLLs (DLLs with no DllMain entry point). That insures against loader lock, but it's like throwing the proverbial baby out with the bathwater: now static variables don't get initialized because your DLL doesn't get _DllMainCRTStartup, the magic CRT function that initializes them. My February 2005 column describes the conundrum in gory detail (msdn.microsoft.com/msdnmag/issues/05/02/CATWork). Not having static objects is a big deal because both ATL and MFC need them. Without DllMain you couldn't write a mixed DLL that uses ATL or MFC. Since that's unacceptable, the friendly Redmondtonians provided a special file <_vcclrit.h> with functions __crt_dll_initialize and __crt_dll_terminate() that you could call to initialize your static variables.

If all that sounds like a Rube Goldberg machine, it is. You'll be happy to learn that Visual Studio 2005 fixes the mixed assembly loader-lock problem. You don't need _vcclrit.h or /NOENTRY, as you can compile mixed-mode DLLs the normal way. For details, see "Initialization of Mixed Assemblies" at msdn2.microsoft.com/ms173266.aspx.

So why did ManWrap crash in dbgheap.c? Because I still had /NOENTRY left over from my old project. Of course my program croaked. It didn't have a DllMain. My static objects, including ATL's g_Allocator, never got initialized. Once I removed /NOENTRY, ManWrap ran just fine. Whew!

Donning a New Hat

One of the blessings of bogglesome bugs is how great you feel when you finally fix them. After successfully compiling and running all three ManWrap test programs (RegexTest, RegexForm, and WordMess), I was quite the happy managed camper. Why not shoot for the moon, embrace the new syntax? In case you've been hibernating for the past year, the heart of C++/CLI is the introduction of a new type called tracking handle that's identified by the symbol ^ (hat). So I got rid of /clr:oldSyntax (see Figure 4) and changed one character in ManWrap.h, from * to ^:

#ifdef _MANAGED
# define GCHANDLE(T) gcroot<T^>
#else
# define GCHANDLE(T) intptr_t
#endif

Figure 4 Using the New Syntax
Figure 4 Using the New Syntax

After making this tiny change, I fed ManWrap to the compiler and fixed the errors one-by-one as they came belching out. Mostly it's a matter of changing Mumble* to Mumble^ everywhere there's a managed Mumble. Of course, there are other syntactical slings and arrows to dodge. Here's an idiosyncratic list of syntax snafus I encountered with ManWrap. They'll seem old-hat (pun intended) to readers already fluent with C++/CLI, so feel free to skim if you're a CLI-savant.

•Managed classes must now be declared with ref or value instead of __gc or __value. In general, all __managed keywords have been replaced with contextually sensitive keywords.

•Default indexers are now called "default" instead of Item. Instead of writing:

x = m->Item[name];

you now must write:

x = m->default[i];

This works even when you have multiple indexers, as with the MatchCollection in the Regex library.

MatchCollection* mc;
mc->default[0];       // int
mc->default["alpha"]; // string

•Managed objects must be allocated with gcnew. Anywhere you allocate a managed object, change new to gcnew.

•Certain conversions are no longer implicit. Consider the following snippet:

// native entry
void Foo(LPCTSTR lpsz)
{
   // managed ctor takes a String
   Mumble *m = new Mumble(lpsz);
}

With the old Managed Extensions, the compiler would implicitly create a new managed String initialized from lpsz in order to construct Mumble. With Visual Studio 2005, you must allocate the String explicitly, like so:

void Foo(LPCTSTR lpsz)
{
   Mumble ^m = gcnew Mumble(gcnew String(lpsz));
}

It's more typing, but it's also more intelligible. I like gcnew because it forces you to know when you're allocating from the managed heap. Implicit conversions make the code appear clean and simple—deceptively simple. C++ is a relatively low-level language (I count that a blessing), so better to keep things visible and not perform too much magic behind the curtain. Since RegexWrap creates Strings all over, I wrote a macro to save typing.

#define S(s) (gcnew String(s))

This is intended to resemble the S modifier for managed strings, as in S"Hello, world." Now I can write:

Mumble ^m = gcnew Mumble(S(lpsz));

C++/CLI has a new syntax for managed arrays. Instead of

ManagedType* myarray[];

you now must write:

array <ManagedType^>^ myarray;

The second hat threw me at first. If you leave it out, you'll get "error C3149: cannot use this type here without a top-level '^'", which seems cryptic (top-level of what?), but makes perfect sense once you know the rule. I'm not sure why the top-level hat (top hat?) is required, since the compiler knows the array is managed by virtue of the "array" keyword—but I'm sure the language cops have a reason and besides, it does make sense. Managed things wear hats. You need the top hat for managed arrays of primitive types, too. For example:

// managed array of ints
array<int>^ foo;

•Instead of Count, use Length to get the length of an array.

•If you need to check for managed NULL pointers, use nullptr instead of NULL.

•Templates work better with managed classes. This is one of the most important reasons to use C++/CLI. Because managed and native classes each have their own private syntax (instead of sharing the overloaded *), the template generator can easily tell them apart.

There are more syntax changes I haven't mentioned. No doubt you'll discover them on your own. For an overview, see "Hello C++/CLI" by Stan Lippman in our special Visual Studio 2005 issue of MSDN Magazine (see msdn.microsoft.com/msdnmag/issues/06/00/PureC). I also recommend Stan's article, "A Baker's Dozen: Thirteen Things You Should Know Before Porting Your Visual C++ .NET Programs to Visual Studio 2005" (see msdn.microsoft.com/library/en-us/dnvs05/html/BakerDozen.asp).

Don't let the new syntax scare you. As soon as I changed * to ^ and /clr:oldSyntax to /clr, it was merely a matter of fixing each error as the compiler complained. Once ManWrap got the compiler's seal of approval, it ran without a hitch. The Redmondtonians should take pride in that—any time the compiler can ensure correctness, it's a win. The smooth switch to hats made me so cheerful I thought of renaming GCHANDLE to MANHANDLE.

Nah.

Two Minor Gripes

Overall, the transition from Visual Studio .NET 2003 to Visual Studio 2005 was fairly painless. I don't have much to say about the IDE since I'm a dyed-in-the-wool text hacker who wouldn't consider editing my code with anything less than Emacs. That said, I have two small gripes with Visual Studio 2005. First, it insists on polluting "My Documents" with all sorts of folders for stuff I never use or need. By spelunking, I was able to discover some registry keys to remap all but a couple of folders to a TEMP directory where out of sight is out of mind. I take great care to organize my folders the way I like, so it irks me when authoritarian programs take over my disk. Let this be a warning: if your app needs folders, make sure you let users decide where they go.

My other complaint is that Visual Studio 2005 no longer supports sound schemes. Boo hoo! I'm not a big fan of noisy apps, but this is one place where sounds are really useful. With Visual Studio .NET 2003, I'd start a build, then go off to work in another window. When the compiler finished, I'd know instantly from the happy chirp or grumpy bonk whether my build was a success or failure. With Visual Studio 2005, I have to actually read the output screen—what a bore! There's another lesson here: never remove features.

Aside from these minor bumps and the /NOENTRY snafu, upgrading to Visual Studio 2005 was a smooth ride. If you haven't already made the switch, go for it—especially if you're writing mixed/managed assemblies. The new syntax is just better. As with any change, it takes a little adjustment. But once you get the hang of hats, they're way cool.

You can download the latest ManWrap from the MSDN Magazine Web site. The download comprises three complete versions: the original one for Visual Studio .NET 2003 and new versions for Visual Studio 2005 using both old and C++/CLI syntax. Happy programming!

.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.


{ End Bracket }: Singularity

{End Bracket}
Singularity
James Larus, Galen Hunt, and David Tarditi

When the C and C++ programming languages were invented, computers were slow, memory was limited, and compilers were simple and memory challenged, so a practical language could be little more than a veneer for assembly language. Times change, though, and so do the limits on software development. Today, program performance is seldom limited by raw processor speed. Instead, latency—memory, disk, network, database—often determines performance. Moreover, software dependability—an amalgam of reliability, availability, safety, and security—has become a challenge that often dominates software development.

Safe programming languages can increase dependability by preventing (or, at least detecting) many common programming errors. Safety has two parts: type and memory. Type safety means that a program can't treat an object of type A as if it was of an unrelated type (unchecked casts in C/C++ offer this "feature"). Memory safety means that a program can't reference beyond the bounds of an object or construct a pointer to an object, for example, by casting some random integer to a pointer.

No doubt you are thinking: "yeah, yeah, I've heard all that before, but safe languages are too slow or hog too much memory for my application." In fact, your problem may have more to do with the implementation of your programming language or operating system than with language safety.

To explore these issues, our team in Microsoft Research built a new system to study trade-offs in the pervasive use of safe languages and to demonstrate that they need not incur large performance penalties. Singularity is a new operating system, written almost entirely in C#, which executes only verifiably safe programs. Singularity's kernel and runtime libraries are the only parts of the system containing unsafe code and, even in those sections, most code is written in safe C#.

Singularity achieves good performance by reinventing the environment in which code executes. In existing systems, safe code is an exotic newcomer who lives in a huge, luxurious home in an elegant, gated community with its own collection of services. Singularity, in contrast, has architected a single world in which everyone can be safe, with performance comparable to the unsafe world of existing systems.

A key starting point is Singularity processes, which start empty and add features only as required. Modern language runtimes come with huge libraries and expressive, dynamic language features such as reflection. This richness comes at a price. Features such as code access security or reflection incur massive overhead, even when never used.

A Singularity application specifies which libraries it needs, and the Bartok compiler brings together the code and eliminates unneeded functionality through a process called "tree shaking," which deletes unused classes, methods, and even fields. As a result, a simple C# "Hello World" process in Singularity requires less memory than the equivalent C/C++ program running on most UNIX or Windows® systems. Moreover, Bartok translates from Microsoft® intermediate language (MSIL) into highly optimized x86 code. It performs interprocedural optimization to eliminate redundant run-time safety tests, reducing the cost of language safety.

Aggressive interprocedural optimization is possible because Singularity processes are closed—they do not permit code loading after the process starts executing. This is a dramatic change, since dynamic code loading is a popular, but problematic, mechanism for loading plug-ins. Giving plug-ins access to a program's internals presents serious security and reliability problems (did you know that 85 percent of blue screens in Windows are caused by third-party plug-ins and device drivers?). Dynamic loading frustrates program analysis in compilers or defect-detection tools, which can't see all code that might execute. To be safe, the analysis must be conservative, which precludes many optimizations and dulls the accuracy of defect detection.

Plug-ins in Singularity execute in their own processes and communicate with carefully verified communication channels. Plug-ins can fail without killing the other. This architecture is practical because Singularity processes are inexpensive to create and communicate between since they rely on language safety, not virtual memory hardware, to enforce isolation.

Rethinking the environment in which a program executes can dramatically reduce the penalty of modern, safe programming languages. Singularity is reinventing systems from the hardware up so programmers no longer need to sacrifice safety for speed.

James Larus, Galen Hunt, and David Tarditi are researchers who lead the Singularity project in Microsoft Research. Jim and David are programming and languages guys, while Galen is the black sheep of the group, coming from the OS community.