 MSDN Magazine March 2007

MSDN Magazine March 2007

IIS 7.0: Explore The Web Server For Windows Vista And Beyond

Mike Volodarsky .NET Security: Support Certificates In Your Applications With The .NET Framework 2.0

Dominick Baier Digital Media: Add Video To Controls And 3D Surfaces With WPF

Lee Brimelow WiX Tricks: Automate Releases With MSBuild And Windows Installer XML

Sayed Ibrahim Hashimi ASP.NET 2.0: Manage Web Users With Custom Profile Providers

Jason N. Gaylord Editor's Note: Bring On the Swag

Joshua Trupin Toolbox: Share Your Desktop, Lightweight Code Editing, and More

Scott Mitchell CLR Inside Out: .NET Application Extensibility, Part 2

Jack Gudenkauf and Jesse Kaplan Test Run: Testing Custom Transform Streams

Dr. James McCaffrey Cutting Edge: Validating ASP.NET Query Strings

Dino Esposito Wicked Code: Scalable Apps with Asynchronous Programming in ASP.NET

Jeff Prosise Service Station: BizTalk Server 2006 Web Services

Aaron Skonnard Bugslayer: GUI Control to Major Tom

John Robbins Foundations: Tracking Services in Windows Workflow Foundation

Matt Milner Concurrent Affairs: Implementing the CLR Asynchronous Programming Model

Jeffrey Richter { End Bracket }: Developing the Virtual Earth 3D Control

Duncan Lawler


IIS 7.0: Explore The Web Server For Windows Vista And Beyond

IIS 7.0

Explore The Web Server For Windows Vista And Beyond

Mike Volodarsky
This article discusses:
  • Modular Web server functionality
  • Simplified deployment and configuration
  • Extensibility and ASP.NET integration
  • Improved security, performance, and compatibility
This article uses the following technologies:
Windows Vista, IIS
Contents
Modular Web Server
Simplified Deployment and Configuration
Improved Administration
.NET Framework and Scripting
Building Web Server Features
ASP.NET Integration
Improved Security
Improved Diagnostics
Diagnosing Errors without Debugging
Improved Performance
Backward Compatibility
Conclusion

I
often hear people-both inside and outside of Microsoft-refer to the new IIS 7.0 Web server as one of the most important developments coming out of Microsoft over the past several years. This is a rather significant statement, given the impressive lineup of technologies released by Microsoft recently, including Windows Vista™!

The release of IIS 7.0 coincides with the ten-year anniversary of the release of the first version of IIS, in Windows NT® 4.0. In 2001, four versions later, IIS 5.0 became the most prevalent Web server on the Internet, though months later it fell victim to the infamous Code Red and Nimbda worms. IIS 6.0, released in Windows Server® 2003, was a major rewrite of the server, focused entirely on improving security, reliability, and performance. Since then, IIS 6.0 has proven to be a rock-solid Web server, achieving high reliability and security track records with only a single critical security bulletin since its release (which was not exploitable remotely).

In this article, I want to take the opportunity to present the major reasons why the next-generation IIS 7.0 Web server makes such a big difference to developers and administrators alike-and give you a head start in using many of its new features.

The vision for IIS 7.0 was to take the speed, reliability, and security of the IIS 6.0 codebase and turn it into a highly extensible and manageable Web server platform powerful enough to run Web applications of the future. The result is the most ambitious Microsoft Web server yet, delivering the largest number of architectural improvements in the history of IIS.

At the core of the IIS 7.0 release is a completely modular Web server, comprised of more than 40 features that can be assembled into small-footprint Web servers optimized for the desired role in your application topology. These features are built on top of a new extensibility layer that allows developers to extend or replace virtually any aspect of the server, in native code or with the Microsoft® .NET Framework. IIS 7.0 offers extensibility throughout its runtime, management, and operational features to help you build end-to-end solutions for your specific needs. On top of the core platform, IIS 7.0 tackles many of the problems associated with the manageability and operation of the server. It features a brand new configuration system that enables fully delegated management of sites and finally makes xcopy deployment of Web applications a reality. The new management APIs and diagnostic features make deployment, administration, and troubleshooting of the server significantly easier and more convenient than ever before.

But why should you start thinking about IIS, a server application, before the next version of Windows Server, code-named "Longhorn," is even near final release? It's important to start thinking about it now because Windows Vista ships with the same full-featured IIS 7.0 bits that are expected to be released in Windows Server "Longhorn." This means you can immediately take advantage of the new IIS 7.0 features to build your personal Web site and host it on Windows Vista. What's more, you can get a head start in developing and testing your production Web applications and Web server infrastructure on the same IIS platform you will be deploying them on when Windows Server "Longhorn" ships.

Intrigued? Let's dive into the details.

Modular Web Server

IIS 7.0 breaks up the Web server into a lightweight server core and more than 40 feature modules that can be plugged into this core. These modules -like StaticFileModule, which allows downloads of static Web content, or WindowsAuthModule, which supports integrated NTLM authentication -can be independently installed on the server to provide the exact functionality you need.

These modules can be completely uninstalled from the server at any time (see Figure 1) or specifically disabled for a particular application where they aren't needed. This allows server administrators to deploy minimal footprint servers quickly with dramatically reduced attack surface area and to significantly improve performance by executing only the required code.

Figure 1 Use Only the Features You Want
Figure 1 Use Only the Features You Want(Click the image for a smaller view)
Figure 1 Use Only the Features You Want
Figure 1 Use Only the Features You Want (Click the image for a larger view)

The componentized architecture is a critical property of IIS 7.0, leading to reduced security risks and minimized patching requirements. It also enables specialized server deployments, which combine select IIS features and custom components optimized for a specific server role in your application topology, such as reverse proxies and cache servers, HTTP protocol load balancers, or SSL and security sentinel servers.

All of the server features that ship with IIS 7.0 are built on top of new public extensibility APIs. As a developer, you have the ability to replace any of the existing server features with your own or build a new module to add to the IIS 7.0 feature set. Want to replace the built-in authentication mechanisms with a custom authentication module or provide a new form of response compression? Go ahead.

The new extensibility APIs are a fundamental improvement over the previous ISAPI extensibility model, enabling you to enhance the server with much more flexibility and ease. Virtually every single aspect of the server, starting with the core server and continuing to configuration, management, and diagnostics, provides extensibility to allow you to extend and tailor the server to your own needs. More about extensibility later in the article.

Simplified Deployment and Configuration

The centralized configuration store of the previous IIS releases, known affectionately as the metabase, is gone. IIS 7.0 features a new delegated configuration system based on a hierarchy of distributed XML configuration files. This hierarchy is comprised of a global applicationHost.config file, which contains server-level configuration defaults, and distributed web.config files within the application's directory structure. These are the same web.config files that are used by the ASP.NET application framework to store application settings in a portable way. This allows the side-by-side storage of IIS and ASP.NET configuration, using clean and strongly structured XML directives. Here's an example:

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <system.web>
        <customErrors mode="Off" />
    </system.web>
    <system.webServer>
      <directoryBrowse enabled="true" />
    </system.webServer>
</configuration>

In the past, IIS application settings had to be explicitly configured in the machine-level metabase repository before the application could function correctly. With distributed web.config files, applications encapsulate the required server configuration within their directory structure. This dramatically simplifies deployment, allowing self-contained applications to be simply copied to the target server's application directory and thus be up and running immediately with the desired settings.

The new configuration system also gives comprehensive control to the server administrators, allowing them to delegate certain configuration options to the application while maintaining control over others for security or business reasons. In this way, applications on hosted servers can set essential configuration directly in their application without requiring the calling of the server administrator for help or using an external configuration panel.

In the true spirit of IIS 7.0, the configuration system is completely extensible. New modules can add their own configuration schema, enabling applications to configure their features side-by-side with the IIS and ASP.NET configuration:

<configuration>
    <system.webServer>
      <directoryBrowse enabled="true" />
    </system.webServer>
    <myBandwidthThrottler enabled="true" />
</configuration>

The custom configuration sections use the same configuration schema as the configuration for the IIS 7.0 features, taking advantage of the strongly typed attribute values, collection syntax, and hierarchical overriding and locking semantics.

IIS 7.0 continues to support existing setup code that uses the Admin Base Object (ABO) APIs to write to the legacy metabase or scripts that use the higher-level Active Directory® Service Interfaces (ADSI) and Windows Management Instrumentation (WMI) objects to configure IIS. It does this by providing a compatibility layer that emulates the ABO APIs, on top of which all other legacy configuration APIs are based, allowing such scripts to read and change configuration just like they did in the previous releases of IIS. (For more information about metabase compatibility, see the "Improved Performance" and "Backward Compatibility" sections towards the end of this article.) While the new structured XML configuration format makes it easier to work with configuration in your favorite text editor, IIS also provides a host of management tools and APIs for administrators to simplify server management and enable automated configuration and deployment.

Improved Administration

IIS 7.0 offers a rich set of administration features that make managing the server possible in a wide range of scenarios. The new graphical IIS Manager administration tool, which replaces the InetMgr.exe MMC snap-in, makes manual server administration very simple with its task-based management interface (see Figure 2).

Figure 2 IIS Manager Provides Graphical Administration Tools
Figure 2 IIS Manager Provides Graphical Administration Tools(Click the image for a smaller view)
Figure 2 IIS Manager Provides Graphical Administration Tools
Figure 2 IIS Manager Provides Graphical Administration Tools (Click the image for a larger view)

IIS Manager allows you to administer most IIS 7.0 features and monitor the server's operation. The tool supports remote administration over a firewall-friendly HTTP/SSL connection, with an option to support both Windows-based and other credentials for authentication.

What's more, the tool enables delegated management, letting application owners manage their applications remotely without having administrative access to the server computer. Armed with this capability, users of hosted services can run the administration tool on their home desktop and connect remotely to manage their applications on the hosted server. Server administrators, of course, have complete control over what management features are delegated to application owners.

Finally, the Administration tool is completely extensible, building on top of configuration system extensibility to allow custom management UI to be added to the tool. You can learn more about the IIS Manager tool and how to add your own administration plug-ins at iis.net/default.aspx?tabid=7&subtabid=73.

For more flexible command-line administration, IIS 7.0 offers the appcmd.exe command-line tool (see Figure 3). This tool provides a comprehensive set of management functionality and better support for bulk operations than the UI. This powerful utility makes it easy to read and write configuration, access site and application pool state information, and perform almost any other management task, all from the command prompt.

Figure 3 Appcmd.exe Command-Line Administration for IIS 7.0
Figure 3 Appcmd.exe Command-Line Administration for IIS 7.0(Click the image for a smaller view)
Figure 3 Appcmd.exe Command-Line Administration for IIS 7.0
Figure 3 Appcmd.exe Command-Line Administration for IIS 7.0 (Click the image for a larger view)

With appcmd.exe, you can create and configure sites, apps, application pools, and virtual directories. You can start and stop sites, recycle application pools, list running worker processes, examine currently executing requests, and search Failed Event Request Buffering (FREB) trace logs. You can also search, edit, export, and import IIS and ASP.NET configuration data.

The tool is built to allow flexible searches for supported server objects, for example, letting you quickly find sites with a particular setting set or application pools that are stopped. When performing a search, you can use any number of conditions on any of the object's properties, including using numeric ranges and simple wildcard string matching.

Appcmd also supports linked operations like those found in Windows PowerShell™, allowing multiple operations on a related set of objects to be performed together from a single command line. For example, you can find and recycle all application pools that host applications for a certain site in a single command. To learn how to manage IIS with AppCmd, check out iis.net/default.aspx?tabid=2&subtabid=25&i=954&p=1.

.NET Framework and Scripting

In addition to the manual server administration with IIS Manager or the appcmd.exe command-line tool, IIS 7.0 provides a wealth of options for programmatic administration. First, you can utilize the Microsoft.Web.Administration API to manage the server from .NET applications. Or you can use the new COM API to manage the IIS configuration system directly, or access it from a scripting environment such as ASP or Windows® Script Host (WSH). There is also a new WMI provider and support for legacy WMI and ADSI providers via the metabase compatibility layer.

Microsoft.Web.Administration, the new .NET Administration API, makes it easy for managed code applications to programmatically provision IIS sites and applications, access important state and diagnostic information, and otherwise configure the server. Enabling .NET Framework-based applications to easily access IIS configuration and state information opens up a world of possibilities for writing .NET-based setup and management applications or even carrying out management tasks directly from ASP.NET pages.

As an example, Figure 4 shows a small C# program that uses Microsoft.Web.Administration to create a new Web site from the command line.

Figure 4 Provisioning Sites with Microsoft.Web.Administration (Close Figure 4)
using System;
using Microsoft.Web.Administration;

class CreateASite
{
    static void Main(string[] args)
    {
        ServerManager serverManager = new ServerManager();
        Site mySite = serverManager.Sites.Add(
            "MySite", "d:\\inetpub\\mysite", 8080);
        mySite.ServerAutoStart = true;
        serverManager.CommitChanges();
    }
}

Microsoft.Web.Administration makes IIS operations and configuration tasks a breeze, right inside your application in a .NET-compliant language of your choice. It also makes it easy to access runtime state information about the server, such as the running worker processes or currently executing requests.

The Microsoft.Web.Administration API serves as the foundation for accessing custom configuration inside custom .NET server modules and UI plug-ins for the IIS Manager tool. For an example of an end-to-end server package, including an image copyright handler to enhance the Web server and associated configuration and management components, take a look at iis.net/default.aspx?tabid=2&subtabid=25&i=1076.

In the Windows Server "Longhorn" timeframe, the IIS team is creating a unified extensibility model for adding custom management objects or extending existing ones, which will enable custom management functionality to be automatically exposed through the different management features including scripting and the Microsoft.Web.Administration API. While you cannot add or extend management objects in Windows Vista, you can use Microsoft.Web.Administration and other APIs to access and manipulate your custom configuration sections just like you can do with existing IIS configuration sections.

Building Web Server Features

IIS 7.0 enables you to mold the server into whatever you need it to be, letting you add or replace any feature in the server in order to provide the functionality you need. At the heart of this capability is a brand new Web server extensibility API on top of which all existing IIS 7.0 HTTP features are built. This API is public, which means that you can implement any of the features that ship with IIS 7.0. This is a first for IIS and a fundamental improvement over the previously limited ISAPI extensibility model.

The new extensibility API is a set of intuitive C++ classes that define the Web server object model and enable a module to provide request processing services on IIS. These classes are defined in the \inc\httpserv.h header file in the Windows Vista SDK.

Compared to ISAPI, these APIs are both more powerful and a lot easier to use. How is this possible? First, the new API features a type-safe, well-encapsulated object model. Development is much easier with the new server object model, which provides specialized interfaces for all of the basic server objects and tasks. This includes:

  • Inspecting the request with the IHttpRequest class
  • Manipulating the response with the IHttpResponse class
  • Consuming useful utility functionality from the IHttpServer class
  • Providing authentication with the IHttpUser class
  • Accessing your module's custom configuration section with the configuration APIs

These classes expose a much greater set of the server's functionality than before (more than needed to build all of the features that ship with IIS), but are also much easier to use then the loosely typed ISAPI interfaces.

Developers will also benefit from improved patterns for memory and state management. Most of the IIS 7.0 server APIs use server-managed memory for data they return, instead of requiring you to allocate and manage the buffers as does ISAPI and the majority of existing Win32® APIs. In the past, this has been one of the most error-prone and tedious areas for ISAPI development. The new API also simplifies many complex request processing tasks, such as response buffering, authentication, and preparing response data for the client. A few months ago, I started a series of posts of my blog that explain the critical improvements and patterns in the new programming model. If you are considering C++ development for IIS, check it out at mvolo.com/blogs/serverside/archive/2006/10/07/10-reasons-why-server-development-is-better-with-IIS7.aspx.

IIS 7.0 also provides a fully integrated .NET Framework API for extending the server. What's more, this is the same API that's been provided by ASP.NET for building ASP.NET modules and handlers since the days of ASP.NET 1.0 on Windows 2000. But don't be fooled-while the familiar ASP.NET model allows existing ASP.NET modules and handlers to keep working on your IIS 7.0 server, it is actually far from the same old technology.

In IIS 7.0, ASP.NET comes in two versions: Classic mode and Integrated mode. Classic mode works exactly the same way as it did in previous versions of IIS. Integrated mode, the new platform default, uses a brand new engine to provide unparalleled integration with the IIS Web server. In Integrated mode, ASP.NET APIs can be used to develop IIS 7.0 modules that integrate directly with the Web server and are capable of providing virtually all of the services that are possible with the underlying C++ API.

This is basically the best of both worlds-the familiar interfaces and convenience of the .NET Framework and ASP.NET 2.0 application services like Membership and Role Management, plus the raw power to extend the server that was previously only available to C-based ISAPI components.

In addition to being able to write new ASP.NET modules that build on the specific benefits of the Integrated mode, many legacy ASP.NET modules can be made a lot more powerful just by changing a few configuration options in your web.config file.

ASP.NET Integration

With IIS 7.0, ASP.NET 2.0 becomes more than just an excellent framework for building dynamic applications. It becomes a platform for extending the IIS Web server, enabling ASP.NET components to become full members of the IIS request processing pipeline. Here's how it works.

In IIS releases up to version 6.0, ASP.NET connected to the Web server as a standalone application framework. It was responsible for processing the request extensions that were registered to it, typically .aspx and few others, and for those requests it offered powerful features like Forms Authentication, Response Output Caching, and others including services provided by custom ASP.NET modules. Because of this, only content types registered to ASP.NET could benefit from these services. Others, including ASP pages, PHP pages, images, and CGI applications could not. In addition, even for ASP.NET resources, certain Web server functions were not possible in ASP.NET because of runtime limitations. For example, it was not possible to inspect the set of outgoing HTTP response headers and modify them before being sent to the client.

When running in Integrated mode in IIS 7.0, ASP.NET modules run in the unified request processing pipeline side-by-side with native C++ IIS modules (see Figure 5). This means that existing ASP.NET services like Output Caching, URL Rewriting, and any others provided by your custom ASP.NET modules can now apply to any content type. Better runtime integration also enables ASP.NET modules to access previously unavailable server functionality, removing the need to write native IIS extensibility in most cases.

Figure 5 Integration with ASP.NET in IIS 6.0 and IIS 7.0
Figure 5 Integration with ASP.NET in IIS 6.0 and IIS 7.0(Click the image for a smaller view)
Figure 5 Integration with ASP.NET in IIS 6.0 and IIS 7.0
Figure 5 Integration with ASP.NET in IIS 6.0 and IIS 7.0 (Click the image for a larger view)

Finally, in Integrated mode, ASP.NET provides a small number of new APIs that expose additional functionality available due to tighter integration with IIS. These include the ability to review all of the response headers regardless of who generated the response and the ability to fully rewrite request execution to another URL, among other features.

An existing application can often take advantage of Integrated mode without requiring new ASP.NET modules that use Integrated mode-specific functionality. Simply by changing configuration, an application can do things such as using ASP.NET Forms Authentication and URL Authorization to protect your entire Web site with user security or using ASP.NET URL Mapping to rewrite URLs in your application. For an example of leveraging Intergrated mode to stop Web leeches from hot-linking to your site's images, check out the sample ASP.NET module to do this on mvolo.com/2006/11/10/stopping-hotlinking-with-iis-and-aspnet.aspx. This is a good example of how you can get more bang out of existing third-party ASP.NET modules by using them in Integrated mode.

For a detailed walkthrough on taking advantage of Integrated mode for existing applications, check out my article at iis.net/default.aspx?tabid=2&subtabid=25&i=1081&p=1.

Improved Security

IIS 7.0 builds upon the IIS 6.0 code base, which has a proven security track record thanks to careful coding practices and secure-by-default design principles. On top of this, IIS 7.0 introduces a few architectural changes to provide even stronger security and a number of features to help you build secure Web applications.

Reducing the attack surface area is one of the fundamental principles of designing and deploying secure systems. Taking the lockdown-by-default approach of IIS 6.0 to the next level, IIS 7.0 installs fewer features by default, yielding an even more locked down server. By further taking advantage of the modular nature of the server to remove all unused features, you can reduce your server's attack surface to a minimum-significantly reducing the risk of your server being compromised by an attacker.

If a vulnerability is found in any of the components not in use on your server, you do not need to take the server out of rotation in order to prevent exploits or patch the vulnerable component immediately. This can result in increased availability of your application and lower patch management costs.

In addition to the core security improvements, IIS 7.0 offers a number of security features you can use to further lock down and deploy secure apps on your server. IIS has always provided strong support for protecting application content with authentication. Now, with ASP.NET Integrated mode, you can use the popular ASP.NET security features such as Forms Authentication, Membership, and Login controls to provide a complete authentication and access control solution for your entire app. You can often do this in minutes and without writing a single line of code.

The new URL Authorization feature, inspired by the ASP.NET URL Authorization feature, can be used to configure declarative access control rules for your entire application. The access rules can be used to allow or deny access to URLs within your application based on user names and roles. URL Authorization integrates seamlessly with the ASP.NET 2.0 Membership and Role Management features and can be used effectively with the ASP.NET Forms Authentication and Login controls to quickly enable user security for your applications.

The new request filtering feature provides powerful lockdown functionality, part of which was available in the popular URLScan tool. You can use request filtering to further lock down your site by rejecting requests containing suspicious data, protecting sensitive resources, or enforcing aggressive request limits.

IIS 7.0 also introduces a number of changes meant to make deployment and management of security settings easier. The new IIS_IUSR anonymous account is built in, which means that it is not affected by password expirations and does not require password synchronization between machines. The new IIS_IUSRS group, which replaces the IIS_WPG group, is automatically injected into the worker process' identity at run time, which alleviates the need to manually add the worker process identity to the group when using custom accounts.

Because of the built-in IIS_USR account and IIS_USRS group, application content that specifies access control lists (ACLs) for the anonymous IIS account and group can be simply copied from one IIS server to another without any extra steps required to preserve the security settings. This dramatically simplifies application deployment across the development-test-production cycle.

The distributed configuration system discussed earlier allows application owners to manage the necessary Web server settings directly in their application without having administrative access to the server. Application administrators can specify the required configuration in the web.config files inside their application content when uploading their applications to the server or use the IIS Manager tool in order to configure their applications remotely.

The IIS Manager tool offers secure remote administration via firewall-friendly HTTPS connections. With the ability to authenticate application administrators as either Windows users or custom user accounts via the Membership service, the Administration tool allows remote application management without the owner having any Windows access on the server.

As a server administrator, you have complete control over what settings are configurable by the application via the flexible locking support in the configuration system. Likewise, you can control which IIS Manager tool features are available to the application administrators managing their applications remotely.

Improved Diagnostics

Between all the new features supported in Windows, IIS 7.0, and your Web application, a Web server is a very a complicated system that often takes a lot of effort to troubleshoot. IIS 7.0 introduces a number of new features to help you monitor the operation of the server and debug application problems.

First, IIS 7.0 lets you peer into the real-time state of the server. This feature, called the Runtime State and Control API, or RSCA (pronounced "reeska"), exposes the active state of sites and application pools, running worker processes, and even allows you to view the currently executing requests on the server! It also enables you to control the state of the server, such as starting and stopping sites, or recycling application pools. In Windows Vista, you can access this information in the IIS Manager, via the appcmd.exe command-line tool, or programmatically using the Microsoft.Web.Administration API.

For example, you can view currently executing requests and the server stages where they are stuck. This lets you quickly resolve hanging request problems and track down which script is spinning your CPU (see Figure 6).

The RSCA capabilities come in very handy when you are investigating server issues or tuning server performance, both for being able to quickly see what's going on in the system and to control the server when performing troubleshooting. I often end up using appcmd.exe to view the state of application pools, examine worker processes, and start or stop offending applications pools to quickly zero in on the problem when investigating bugs in the office.

Figure 6 Tracking Down Stuck Scripts in IIS Manager
Figure 6 Tracking Down Stuck Scripts in IIS Manager(Click the image for a smaller view)
Figure 6 Tracking Down Stuck Scripts in IIS Manager
Figure 6 Tracking Down Stuck Scripts in IIS Manager (Click the image for a larger view)

When errors occur in your Web application, they can be due to incorrect server configuration, application errors, or various environmental factors. The status code and standard error messages provide little clue to what went wrong and can make troubleshooting the server a nightmare. IIS 7.0 provides detailed error information about most errors, telling you exactly what went wrong, why, and how to fix it (see Figure 7).

Figure 7 Error Details Indicate the Problem and the Solution
Figure 7 Error Details Indicate the Problem and the Solution(Click the image for a smaller view)
Figure 7 Error Details Indicate the Problem and the Solution
Figure 7 Error Details Indicate the Problem and the Solution (Click the image for a larger view)

The detailed errors follow a security scheme similar to the ASP.NET detailed errors. By default, you will only get the detailed information when browsing the Web site from the local machine. As before, you can also configure custom error pages for different error codes or redirect to a custom URL. The detailed error pages are also now localized, offering the error description in the preferred language of the client, if the language pack for the corresponding language is installed.

Diagnosing Errors without Debugging

What if the error condition you are experiencing is unknown or caused by a complex interplay of multiple Web server components? Not to worry-IIS 7.0 provides a comprehensive tracing mechanism that generates a verbose paper trail for each request, which can be used to track down the problem quickly.

IIS 7.0 adds more informational events on top of the Event Tracing for Windows (ETW) events added to IIS 6.0 in Windows Server 2003 Service Pack 1 (SP1). These events contain useful information about each stage of server processing you can examine to back-trace through request execution and pinpoint the place where things went wrong. These events can be routed to the Windows tracing infrastructure, which allows multiple Windows components including ASP.NET and SQL Server™ to link their tracing information into a single, logical execution trace for the request.

They can also be routed to the new Failed Request Tracing feature (also called FREB), which saves the trace logs into XML log files that can be viewed with the provided XSLT stylesheet (see Figure 8) or consumed programmatically.

Figure 8 Viewing XML Log Files
Figure 8 Viewing XML Log Files(Click the image for a smaller view)
Figure 8 Viewing XML Log Files
Figure 8 Viewing XML Log Files (Click the image for a larger view)

The coolest thing about Failed Request Tracing is that it can be left enabled on the server. It allows you to capture trace logs for requests that have encountered a configurable failure condition automatically while avoiding the performance penalty of saving the trace logs for requests that have completed successfully. For example, you can turn it on for requests that result in server errors or take more than a certain amount of time to complete.

With Failed Request Tracing, you can always capture the valuable tracing information when errors occur, even if they are intermittent or hard to reproduce. This can help diagnose and fix difficult problems that previously required hardcore debugging.

The underlying tracing infrastructure is exposed to IIS modules via the server extensibility model, allowing all server components, whether they ship with IIS or are developed by third parties, to emit detailed tracing information during request processing. IIS 7.0 tracing integrates with ASP.NET tracing via the System.Diagnostics API and the ASP.NET page trace, allowing managed modules to take advantage of the unified tracing model. To take things even further, you can write your own tracing modules that provide new ways to process and output the tracing information. For example, you can be the first one to write a module to save IIS tracing information to SQL Server or to a text file.

Improved Performance

While Windows Vista is a client operating system release and not intended for high-throughput production deployment (IIS on Windows Vista is limited to 10 concurrent requests at a time), it already showcases some of the major architectural improvements aimed at significantly increasing Web application performance. Coupled with the extensive performance work we are doing in the Windows Server "Longhorn" timeframe, these improvements will help IIS 7.0 elevate the performance of your server.

The first one is, of course, componentization. The modular nature of the server allows administrators to remove unneeded server features, saving memory and CPU usage during request processing. This can result in significant improvements in throughput and capacity on the machine. The ability to enable features in a granular way (turning them on and off for each application on your server) further enables applications to achieve performance gains in the case where only certain parts of your site require a particular feature.

The other notable performance feature in IIS 7.0 is the new IIS Output Cache. This feature provides support for re-using responses to expensive dynamic pages on the server, alleviating the need to perform the expensive rendering and database transactions in order to return the response to the client. The IIS Output Cache is a faster alternative to the existing rich output caching feature in ASP.NET, supporting a smaller set of caching functionality but providing enough flexibility to cache dynamic content in a performance-enhancing way.

By output caching your dynamic content, whether it is ASP.NET pages, PHP scripts, or CGI applications, you can frequently get a 5-10 times performance boost with much less load on your disk and database.

Backward Compatibility

IIS 7.0 should be able to run most existing applications without modification. This was a major feat given the extent of the architectural changes that were needed to support the innovation in this release. The configuration system has undergone the most changes, moving away from a centralized loosely typed configuration store to a delegated XML configuration file hierarchy. Both the structure and the storage of the configuration information are completely different from the IIS 6.0 metabase and do not enable access via the legacy configuration APIs.

IIS 7.0 solves this problem by providing an emulation layer for the metabase, which does on-the-fly translations between the underlying data in the configuration system and the interfaces exposed by the metabase ABO APIs. This, in turn, allows code written for the metabase to work correctly when accessing it via ABO, or higher-level WMI or ADSI scripts. Be sure to install the compatibility setup components to get this functionality, though.

While IIS 7.0 provides a new extensibility model for developing IIS components, ISAPI components are still supported. If you install the ISAPI Extensions and ISAPI Filters setup components, you will be able to run your extensions and filters as before. However, if you are developing new components, you should make sure to use the new extensibility model for a much more powerful and improved development experience.

A small percentage of ASP.NET applications that have runtime incompatibilities with the Integrated mode may have to be moved to an application pool running in Classic mode. In this case, you can run multiple applications in both modes side-by-side on the same server by placing them in separate application pools. For a complete list of ASP.NET breaking changes and general ASP.NET compatibility information on IIS 7.0, be sure to check out the ASP.NET compatibility whitepaper at iis.net/default.aspx?tabid=2&subtabid=25&i=1223.

Conclusion

The IIS 7.0 release in Windows Vista aims to provide the best architectural foundation for the next-generation Web application platform, focusing on the right core architecture, extensibility, and management platform for the Web server. Windows Vista gives you the ability to develop and test your applications on the same server platform as will be used to deploy it when the server version of Windows Vista becomes available.

As IIS 7.0 is released in Windows Vista, the Web Platform and Tools team's focus is shifting to getting the Web server ready for production environments and improving stability and performance for production scenarios. However, the core development and management features that ship in Windows Vista will remain the same-and when the server version of IIS 7.0 is complete, it's improvements are expected to become available for Windows Vista via a service pack. At that point, your client and server machines will once again be running the same exact version of IIS so you can continue to develop and test Web applications on your desktop running Window Vista.

To get started with IIS 7.0, be sure to check out the large number of exceedingly useful resources available on the Web, starting with the iis.net Web site, which is the new home of the IIS team. Our new site contains a portal for all things IIS 7.0, including detailed articles and walkthroughs for all IIS 7.0 features. Be sure to use the forums to ask questions and discuss problems with the IIS team and the IIS community.

You can also find in-depth coverage of IIS 7.0 and inside information on my blog, www.mvolo.com. Be sure to stop by. And do let me know what your favorite IIS 7.0 topics are and I will do my best to cover them in the blog.

Mike Volodarsky is a technical Program Manager on the Web Platform and Tools Team at Microsoft. He owns the core server infrastructure for ASP.NET and IIS. Michael is now focusing on improving the Web application platform in the next-generation Web server, IIS 7.0.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


.NET Security: Support Certificates In Your Applications With The .NET Framework 2.0

.NET Security

Support Certificates In Your Applications With The .NET Framework 2.0

Dominick Baier
This article discusses:
  • The Windows Certificate Store
  • Certificate classes in .NET
  • Validation, SSL, Web services, and code signing
  • Signing and encrypting data
This article uses the following technologies:
.NET Framework 2.0
Code download available at: Certificates2007_03.exe (160KB)
Contents
How to Get a Certificate
The Windows Certificate Store
Working with Certificates
Accessing Certificates
Display Certificate Details and the Certificate Picker
Validating Certificates
SSL Support
Web Service Security
Security Policy and Code Signing
ClickOnce Manifests
Signing and Encrypting Data
Decrypting Data and Verifying Signatures
Putting It All Together

C
ertificates are used in many places across the Microsoft® .NET Framework, from secure communication to code signing to security policies. The .NET Framework 2.0 introduced revamped support for certificates and it added a completely new namespace for standards-compliant cryptographic operations with certificates. In this article, I will discuss the background for certificates and the Windows® Certificate Store. I'll also show you how to work with the certificate APIs and how they are used by the Framework to implement security features.

A "certificate" is really an ASN.1 (Abstract Syntax Notation One) encoded file that contains a public key and additional information about that key and its owner. In addition, a certificate has a validity period and is signed with another key (the so-called issuer) which is used to provide an authenticity guarantee of those attributes and, most importantly, the public key itself. You can think of ASN.1 as a sort of binary XML. Like XML, it also has encoding rules, strong types, and tags; however, these are binary values that often don't correspond to any printable character.

For such a file to be interchangeable between systems, a standard format is needed. This is X.509 (currently version 3), which is described in RFC 3280 (tools.ietf.org/html/rfc3280). X.509 doesn't dictate the type of key embedded in the certificate, but the RSA algorithm is currently the most popular asymmetric cryptographic algorithm in use.

I'll start with a little history. The name RSA is an acronym for the surnames of three inventors of this algorithm: Ron Rivest, Adi Shamir, and Len Adleman. They formed a company, RSA Security, which published several standard documents called Public Key Cryptography Standards (PKCS). These documents describe several aspects of cryptography.

One of the most popular of these documents, PKCS #7, defines a binary format for signed and encrypted data called the Cryptographic Message Syntax (CMS). CMS is now used in many popular security protocols, including secure sockets layer (SSL) and Secure Multipurpose Internet Mail Extensions (S/MIME). Since it is a standard, it is also the format of choice for when applications need to exchange signed and encrypted data between several parties. The PKCS documents are available on the RSA Laboratories Web site (www.rsasecurity.com/rsalabs/node.asp?id=2124).

How to Get a Certificate

There are several ways to acquire a certificate. When files are being exchanged, certificates will usually appear in one of two formats. Files with the .cer extension are signed ASN.1 files in the X.509v3 format. They contain a public key and the extra information I mentioned earlier. This is what you give to business partners or friends so they can use the public key to encrypt data for you.

You may also encounter files with a .pfx (Personal Information Exchange) extension. A .pfx file contains a certificate and the corresponding private key (the format is described in the PKCS #12 standard). Such files are highly sensitive and are typically used to import key pairs on a server or for backup purposes. When exporting key pairs, Windows offers to encrypt the .pfx file with a password; you have to provide this password again when importing the key pair.

You can also generate your own certificates. How you generate them usually depends on how they will be used. For normal Internet scenarios, where you don't know who your peers are, you typically request a certificate from a commercial certification authority (CA). This approach has the advantage that these known CAs are already trusted by Windows and any other OS (and browser) that supports certificates and SSL. As a result, you don't have to do a CA key exchange.

For B2B and intranet scenarios, you can use an internal CA. Certificate Services are included in Windows 2000 and Windows Server® 2003. Combined with Active Directory®, this functionality lets you easily distribute certificates across an organization. (I will show you how to request certificates from a private CA in a moment.)

Sometimes during development you might be in a situation where the approaches just described don't work. For instance, if you need a certificate quickly for testing purposes, you can use makecert.exe. Included in the .NET Framework SDK, this tool generates certificates and key pairs. There is a similar tool, called selfssl.exe, in the IIS Resource Kit; it is specialized for creating SSL key pairs, and it can also configure IIS with such a key pair in a single step.

The Windows Certificate Store

Certificates and their corresponding private keys can be stored on a variety of devices, such as hard disks, smartcards, and USB tokens. Windows provides an abstraction layer, called the certificate store, to unify how you access certificates regardless of where they are stored. As long as the hardware device has a Windows-supported cryptographic service provider (CSP) you can access the data stored on it using the Certificate Store API.

The certificate store is buried deep in the user profile. This allows use of ACLs on the keys for a specific account. Every store is partitioned into containers. For instance, there's a container called Personal where you store your own certificates (the ones that have an associated private key). The Trusted Root Certification Authorities container holds the certificates of all CAs that you trust. The Other People container holds the certificates of people you securely communicate with. And so on. The easiest way to get to your certificate store is to run certmgr.msc.

There is also a machine-wide store, which is used by the Windows machine accounts (NETWORK, LOCAL SERVICE, and LOCAL SYSTEM) or if you want to share certificates or keys across accounts. ASP.NET applications always use the machine store; for desktop applications, you typically install certificates in the user store.

Only administrators can manage the machine and service account stores. For this purpose, you have to start the Microsoft Management Console (mmc.exe) and add the Certificates snap-in. There you can choose the store to administer. Figure 1 shows a screenshot of the MMC snap-in.

Figure 1 Certificates MMC Snap-In
Figure 1 Certificates MMC Snap-In(Click the image for a smaller view)
Figure 1 Certificates MMC Snap-In
Figure 1 Certificates MMC Snap-In (Click the image for a larger view)

Besides allowing you to import, export, and search for certificates, the snap-in also lets you request certificates from an internal enterprise CA. Just right-click the personal container and select All Tasks | Request Certificate. The local machine then generates an RSA key pair and sends the public key portion to the CA for signing. Windows adds the signed certificate to the certificate store and the corresponding private key to a key container. The certificate gets linked to the key container via a storage attribute.

The private key containers are strongly ACL'd for either the corresponding account or LOCAL SYSTEM. This is a problem when you want to access keys stored in the machine profile from ASP.NET or from other user accounts. I wrote a tool that you can use to modify the ACLs of the container file (it's available at www.leastprivilege.com/HowToGetToThePrivateKeyFileFromACertificate.aspx).

Commercial and Windows CAs also have Web interfaces for requesting certificates. For these scenarios, typically an ActiveX® control in Internet Explorer® generates the keys and imports them into the store of the current user. As a general rule, when you want to make a certificate accessible to a user or service, you have two choices: either import it into his store or request it while being logged on as that user.

Working with Certificates

Certificates are used in various places in the .NET Framework, and at some level all of this functionality relies on the X509Certificate class from the System.Security.X509Certificates namespace. If you take a closer look, you'll also find a certificate class ending with a 2. This is because the .NET Framework 1.x had a representation of X.509 certificates called X509Certificate. This class had limited functionality and no support for cryptographic operations. In version 2.0, a new class was added called X509Certificate2. This is derived from X509Certificate and adds many capabilities. You can convert back and forth between them as necessary, but whenever possible you should use the latest version.

Accessing Certificates

You can retrieve certificates from the file system directly. However, it is better to retrieve them from the certificate store. To create an X509Certificate2 instance from a .cer file, simply pass the file name to the constructor:

X509Certificate2 cert1 = new X509Certificate2("alice.cer");

You can also load certificates from .pfx files. However, as I mentioned earlier, .pfx files can be password protected, and you should supply this password as a SecureString. SecureString encrypts the password internally and tries to minimize exposure of it in Memory, page files, and crash dumps. For this reason, you can only add a single (value type) char at a time to the string. The code in Figure 2, which disables the console echo and returns a SecureString, is useful if you want to ask your users for a password from the console.

Figure 2 Requesting a Password from the Console (Close Figure 2)
private SecureString GetSecureStringFromConsole()
{
    SecureString password = new SecureString();

    Console.Write("Enter Password: ");
    while (true)
    {
        ConsoleKeyInfo cki = Console.ReadKey(true);

        if (cki.Key == ConsoleKey.Enter) break;
        else if (cki.Key == ConsoleKey.Escape)
        {
            password.Dispose();
            return null;
        }
        else if (cki.Key == ConsoleKey.Backspace)
        {
            if (password.Length != 0)
                password.RemoveAt(password.Length - 1);
        }
        else password.AppendChar(cki.KeyChar);
    }

    return password;
}

In the article "Credential Management with the .NET Framework 2.0" (available at msdn.microsoft.com/library/en-us/dnnetsec/html/credmgmt.asp), Kenny Kerr included code to convert the result of the usual Windows credentials dialog into a SecureString. Regardless of how you obtain it, the SecureString can then be passed to the X509Certificate2 constructor to load the .pfx file, like so:

X509Certificate2 cert2 = new X509Certificate2("alice.pfx", password);

To access the Windows certificate store, you use the X509Store class. In its constructor you provide the store location (current user or machine) and the store name. You can use either a string or the StoreName enumeration to specify the container you want to open. Be aware that the internal names don't always match the names you find in the MMC snap-in. The Personal container maps to the name My, whereas Other People becomes AddressBook.

Once you have a valid instance of X509Store, you can search for, retrieve, delete, and add certificates. With the exception of deployment scenarios, you will probably use the search functionality most often. You can search for certificates on a variety of criteria, including subject name, serial number, thumbprint, issuer, and validity period. If you programmatically retrieve certificates in your applications from the store, you should use a unique property-the subject key identifier, for instance. The thumbprint is also unique, but keep in mind that this is a SHA-1 hash value of the certificate and will change if, for example, the certificate gets renewed. The code in Figure 3 shows a generic way to search for certificates.

Figure 3 Searching for Certificates (Close Figure 3)
static void Main(string[] args)
{
    // search for the subject key id
    X509Certificate2 cert = FindCertificate(
      StoreLocation.CurrentUser, StoreName.My,
      X509FindType.FindBySubjectKeyIdentifier,
      "21f2bf447298e83056a69eb02ebe9085ed97f10a");
}

static X509Certificate2 FindCertificate(
    StoreLocation location, StoreName name,
    X509FindType findType, string findValue)
{
    X509Store store = new X509Store(name, location);
    try
    {
        // create and open store for read-only access
        store.Open(OpenFlags.ReadOnly);

        // search store
        X509Certificate2Collection col = store.Certificates.Find(
          findType, findValue, true);

        // return first certificate found
        return col[0];
    }
        // always close the store
    finally { store.Close(); }
}

Once you have an instance of X509 Certificate2, you can inspect the various properties of the certificate (such as the subject name, expiration dates, issuer, and the friendly name). The HasPrivateKey property tells you if there is an associated private key. The PrivateKey and PublicKey properties return the corresponding key as an RSACryptoServiceProvider instance.

To import a certificate, you call the Add method on the X509Store instance. When you specify a store name that doesn't exist in the constructor of the store, a new container will be created. Here's how you would import a certificate in a file named alice.cer into a new container called Test:

static void ImportCert()
{
    X509Certificate2 cert = new X509Certificate2("alice.cer");
    X509Store store = new X509Store("Test", StoreLocation.CurrentUser);
    try
    {
        store.Open(OpenFlags.ReadWrite);
        store.Add(cert);
    }
    finally { store.Close(); }
}

Display Certificate Details and the Certificate Picker

Windows offers two standard dialogs for working with certificates: one for showing certificate details (properties and certification path) and one for letting users pick a certificate from a list. You can access these dialogs from the two static methods of the X509Certificate2UI class: SelectFromCollection and DisplayCertificate.

To show a list of certificates you have to populate an X509Certificate2Collection and pass that to SelectFromCollection. It is very common to let a user choose from one of his personal certificates in the store. For this, you simply pass in the Certificates property of an opened X509Store. You can also control the dialog caption, a message, and whether multiple selections are allowed. The DisplayCertificate method shows the same dialog you see when double-clicking on a .cer file in Windows Explorer. Figure 4 shows the dialog used for picking a certificate and Figure 5 provides the corresponding code.

Figure 5 Code for Choosing a Certificate (Close Figure 5)
private static X509Certificate2 PickCertificate(
  StoreLocation location, StoreName name)
{
    X509Store store = new X509Store(name, location);
    try
    {
        store.Open(OpenFlags.ReadOnly);

        // pick a certificate from the store
        X509Certificate2 cert =
            X509Certificate2UI.SelectFromCollection(
                store.Certificates, "Caption",
                "Message", X509SelectionFlag.SingleSelection)[0];

        // show certificate details dialog
        X509Certificate2UI.DisplayCertificate(cert);
        return cert;
    }
    finally { store.Close(); }
}
Figure 4 Dialog for Choosing Certificates
Figure 4 Dialog for Choosing Certificates(Click the image for a smaller view)
Figure 4 Dialog for Choosing Certificates
Figure 4 Dialog for Choosing Certificates (Click the image for a larger view)

Validating Certificates

There are a few criteria to consider when validating a certificate, especially the issuing party (generally, you only trust certificates that were issued by a CA in your trusted CA list) and its current validity (certificates can become invalid, such as when they expire or are revoked by the issuing CA). You can use the X509Chain class to check these various properties. Using this class, you can specify a policy for validity checking-for example, you can demand a trusted root CA or specify whether to check online or local revocation lists. If you need to check certificates that were used to sign data, it is important to check whether the certificate was valid when the signature was computed-for this, X509Chain allows you to change the verification time.

After constructing a policy, you call the Build method to get information about the validation outcome on the ChainStatus property. If there are multiple validation errors, you can iterate over the ChainElement collection to get more details. Figure 6 shows how to perform a strict validation of a certificate and its issuer against offline and online revocation lists.

Figure 6 Strict Certificate Validation (Close Figure 6)
static void ValidateCert(X509Certificate2 cert)
{
    X509Chain chain = new X509Chain();

    // check entire chain for revocation
    chain.ChainPolicy.RevocationFlag = X509RevocationFlag.EntireChain;

    // check online and offline revocation lists
    chain.ChainPolicy.RevocationMode =
        X509RevocationMode.Online | X509RevocationMode.Offline;

    // timeout for online revocation list
    chain.ChainPolicy.UrlRetrievalTimeout = new TimeSpan(0, 0, 30);

    // no exceptions, check all properties
    chain.ChainPolicy.VerificationFlags = X509VerificationFlags.NoFlag;

    // modify time of verification
    //chain.ChainPolicy.VerificationTime = new DateTime(1999, 1, 1);

    chain.Build(cert);

    if (chain.ChainStatus.Length != 0)
        Console.WriteLine(chain.ChainStatus[0].Status);
}

SSL Support

The SSL authentication protocol relies on certificates. Support for SSL in the .NET Framework consists of two parts. The special (but most widely used) case of SSL over HTTP is implemented by the HttpWebRequest class (this is also ultimately used for Web service client proxies). To enable SSL, you don't have to do anything special besides specify a URL that uses the https: protocol.

When connecting to an SSL secured endpoint, the server certificate is validated on the client. If validation fails, by default the connection is immediately closed. You can override this behavior by providing a callback to a class called ServicePointManager. Whenever the HTTP client stack does certificate validation, it first checks if a callback is provided-if that's the case, it executes your code. To hook up the callback, you have to provide a delegate of type RemoteCertificateValidationCallback:

// override default certificate policy
// (for example, for testing purposes)
ServicePointManager.ServerCertificateValidationCallback =
    new RemoteCertificateValidationCallback(VerifyServerCertificate);

In your callback, you get the server certificate, an error code, and a chain object passed in. You can then do your own check and return true or false. It can be helpful to turn off one of these checks if, for instance, your certificate has expired during development or testing. On the other hand, this also allows you to implement stricter validation policies than provided by default. Figure 7 provides a sample validation callback.

Figure 7 Validation Callback (Close Figure 7)
private bool VerifyServerCertificate(
    object sender, X509Certificate certificate,
    X509Chain chain, SslPolicyErrors sslPolicyErrors)
{
    if (sslPolicyErrors == SslPolicyErrors.None) return true;

    foreach (X509ChainStatus s in chain.ChainStatus)
    {
        // allows expired certificates
        if (string.Equals(s.Status.ToString(), "NotTimeValid",
            StringComparison.OrdinalIgnoreCase))
                return true;
    }

    return false;
}

SSL also supports client authentication using a certificate. If the Web site or service you want to access mandates a client certificate, both the Web service client proxy and HttpWebRequest provide a ClientCertificates property of type X509Certicate:

  proxy.Url =
    "https://server/app/service.asmx";
  proxy.ClientCertificates.Add(
    PickCertificate(...));

In addition, the .NET Framework 2.0 introduces a new class called SslStream. This lets you layer SSL on top of any stream, not just HTTP, which makes it possible to SSL-enable a custom socket based protocol. SslStream makes use of the standard .NET certificates support in multiple ways, for example, using the validation callback mechanism I discussed:

public SslStream(Stream innerStream, bool leaveInnerStreamOpen,
    RemoteCertificateValidationCallback ValidationCallback) {...}

And to start an SSL authentication with SslStream, you pass an X509Certificate to its AuthenticateAsServer method:

ssl.AuthenticateAsServer(PickCertificate(...));

Web Service Security

The WS-Security standard specifies client and server authentication and secure communication using certificates. Toolkits like the Web Services Enhancements (WSE) for the .NET Framework and technologies like the Windows Communication Foundation fully support this. Again, this boils down to supplying a certificate either in code or through configuration. The following snippet shows how to add a client certificate to a Web service proxy using WSE3:

X509SecurityToken token = new X509SecurityToken(PickCertificate(...));
proxy.RequestSoapContext.Security.Tokens.Add(token);

With Windows Communication Foundation, you typically provide a reference to a certificate store in a configuration file (see Figure 8). As you can see, all configuration attributes map directly to the enums used earlier in code.

Figure 8 Providing Certificate Reference in WCF (Close Figure 8)
<system.serviceModel>
  <behaviors>
    <serviceBehaviors>
      <behavior name="ServiceBehavior">
        <serviceCredentials>
          <serviceCertificate storeLocation="LocalMachine"
            storeName="My" x509FindType="FindBySubjectKeyIdentifier"
            findValue="1a7b..." />
        </serviceCredentials>
      </behavior>
    </serviceBehaviors>
  </behaviors>
</system.serviceModel>

Security Policy and Code Signing

Certificates are also used in Authenticode® code signing. By signing a binary, you can add information about the publisher and make sure the signed file can be reliably validated after it has been signed. You can use the signtool.exe tool from the .NET Framework SDK to sign .exe and .dll files. Afterwards, you can verify the signature and view the certificate using the properties dialog in Windows Explorer. Note that if both Authenticode and strong name signatures are going to be used, the strong name signature needs to be applied first. Additionally, Authenticode signed assemblies can experience delays at load time, which translates to a longer application startup time if it's the entry point executable that's been signed.

Signed files can also be used for security policies. Using software restriction policies, you can restrict execution of unmanaged executables based on signatures or the absence of signatures (see microsoft.com/technet/prodtechnol/winxppro/maintain/rstrplcy.mspx). And the .NET Framework code access security (CAS) policy supports code groups based on a publisher certificate.

To create a CAS policy, you use mscorcfg.msc to create a new code group based on a publisher membership condition. You can then assign a permission set to all applications signed by that publisher (see Figure 9).

Figure 9 Assigning Permissions to a Publisher
Figure 9 Assigning Permissions to a Publisher(Click the image for a smaller view)
Figure 9 Assigning Permissions to a Publisher
Figure 9 Assigning Permissions to a Publisher (Click the image for a larger view)

ClickOnce Manifests

Another technology that uses certificates for publisher information is ClickOnce. When you publish a ClickOnce application, you have to sign the deployment and application manifest. This again adds publisher information to the application and ensures that sensitive information in the manifests (such as the security policy and application dependencies) cannot be modified without invalidating the signature. ClickOnce makes the publisher information available to clients during the installation so they can make intelligent decisions about the trustworthiness of the application. Depending on the certificate (and its validation outcome) the ClickOnce installer also uses different visual cues. Figure 10 shows the Visual Studio® manifest signing dialog.

Figure 10 Visutal Studio Manifest Signing Dialog
Figure 10 Visutal Studio Manifest Signing Dialog(Click the image for a smaller view)
Figure 10 Visutal Studio Manifest Signing Dialog
Figure 10 Visutal Studio Manifest Signing Dialog (Click the image for a larger view)

Signing and Encrypting Data

So far, I've focused on the fundamental certificate-related APIs and how other technologies make use of them. Now I want to discuss cryptographic operations, like encrypting and signing data with certificates, and the new PKCS #7 implementation found in the .NET Framework 2.0.

Protecting data is always a two-step process. First you sign the data to make it tamper-proof. Then you encrypt the data to protect it from disclosure. Before you can perform any cryptographic operation with the PKCS #7 classes, however, you first have to wrap the data in a ContentInfo object, representing a CMS data structure. From there you can transform the data into signed or encrypted data, represented respectively by the SignedCms and EnvelopedCms classes.

Technically, a digital signature is the hash of the data that's then encrypted with your private key. This means you need a certificate with an associated private key or a .pfx file. Based on such a certificate, you can create a CmsSigner object, which represents the signer of the data. The SignedCms class in turn computes the signature and outputs a PKCS #7, CMS-compliant byte array. Figure 11 shows the corresponding code. The encoded byte array contains your data, the signature, and the certificate used to sign the data.

Figure 11 Outputting CMS-Compliant Byte Array (Close Figure 11)
byte[] Sign(byte[] data, X509Certificate2 signingCert)
{
    // create ContentInfo
    ContentInfo content = new ContentInfo(data);

    // SignedCms represents signed data
    SignedCms signedMessage = new SignedCms(content);

    // create a signer
    CmsSigner signer = new CmsSigner(signingCert);

    // sign the data
    signedMessage.ComputeSignature(signer);

    // create PKCS #7 byte array
    byte[] signedBytes = signedMessage.Encode();

    // return signed data
    return signedBytes;
}

This might not be critical if you are signing large amounts of data, but if the amount of data is small, this adds some overhead. For example, signing a 10 byte array with a 2KB public key results in a roughly 2,400 byte array. Keep that in mind if you want to store the signed data in, say, a database. An alternative approach is to use a so-called detached signature. This lets you remove your data from the signature and store it separately. You could, for example, first combine multiple small pieces of data and sign them altogether. To create a detached signature you have to pass an additional true to the SignedCms constructor, as shown in Figure 12.

Figure 12 Creating Detached Signature (Close Figure 12)
byte[] SignDetached(byte[] data, X509Certificate2 signingCert)
{
    // create ContentInfo
    ContentInfo content = new ContentInfo(data);

    // pass true to the constructor to indicate
    // we want to sign detached
    SignedCms signedMessage = new SignedCms(content, true);

    // these steps are the same
    CmsSigner signer = new CmsSigner(signingCert);
    signedMessage.ComputeSignature(signer);
    byte[] signedBytes = signedMessage.Encode();

    // return only the signature (not the data)
    return signedBytes;
}

Once you've signed the data, you can encrypt it. You'll need the public keys of the recipients that should be able to decrypt the data. You usually get these from your Other People store (or from a .cer file if you don't want to use the certificate store). This time the EnvelopedCms class does all the heavy lifting. You specify the public keys used for encryption in a CmsRecipientCollection, which you pass into the Encrypt method. As with SignedCms, here the Encode method creates the PKCS #7, CMS-compliant byte array (see Figure 13).

Figure 13 Encode Method (Close Figure 13)
byte[] Encrypt(byte[] data, X509Certificate2 encryptingCert)
{
    // create ContentInfo
    ContentInfo plainContent = new ContentInfo(data);

    // EnvelopedCms represents encrypted data
    EnvelopedCms encryptedData = new EnvelopedCms(plainContent);

    // add a recipient
    CmsRecipient recipient = new CmsRecipient(encryptingCert);

    // encrypt data with public key of recipient
    encryptedData.Encrypt(recipient);

    // create PKCS #7 byte array
    byte[] encryptedBytes = encryptedMessage.Encode();

    // return encrypted data
    return encryptedBytes;
}

Internally, EnvelopedCms generates a random session key with which the data is symmetrically encrypted. Afterwards, the session key is encrypted with the public key of each recipient. Thus, you don't need a separate encrypted version of the data for each of your recipients. In addition, some extra information is embedded, allowing the recipient to find the matching private key for decryption in his certificate store.

Decrypting Data and Verifying Signatures

At the receiving end, the whole process is reversed. That means you first decrypt the data and then validate the signature and the signing certificate. In code, you first have to call the Decode methods of the SignedCms and EnvelopedCms classes to deserialize the CMS byte array back to an object representation. Then you can call Decrypt and CheckSignature, respectively.

The process looks in the encrypted package to see if the session key can be decrypted by searching for a corresponding private key in the certificate store. Afterwards, the decrypted session key is used to decrypt the actual data. You can also supply a list of additional certificates that should be taken into consideration during decryption in case the private key is not stored in the certificate store:

static byte[] Decrypt(byte[] data)
{
    // create EnvelopedCms
    EnvelopedCms encryptedMessage = new EnvelopedCms();

    // deserialize PKCS#7 byte array
    encryptedMessage.Decode(data);

    // decryt data
    encryptedMessage.Decrypt();

    // return plain text data
    return encryptedMessage.ContentInfo.Content;
}

Verifying the data is a two-step process. You first make sure the signature is valid, which means the data has not been tampered with. You then check the signing certificate. The CheckSignature method of the SignedCms class lets you do both steps at once. In this case, the certificate is validated against the default system policy. If you want more control over that process, you can do your own checking using an X509Chain object and code like that in Figure 6. And Figure 14 shows the code used to check and remove a signature, while Figure 15 provides the code used for detached signature validation.

Figure 15 Detached Signature Validation (Close Figure 15)
static bool VerifyDetached(byte[] data, byte[] signature)
{
    ContentInfo content = new ContentInfo(data);

    // pass true for detached
    SignedCms signedMessage = new SignedCms(content, true);

    // deserialize signature
    signedMessage.Decode(signature);

    try
    {
        // check if signature matches data
        // the certificate is also checked
        signedMessage.CheckSignature(false);
        return true;
    }
    catch { return false; }
}
Figure 14 Verify and Remove a Signature (Close Figure 14)
byte[] VerifyAndRemoveSignature(byte[] data)
{
    // create SignedCms
    SignedCms signedMessage = new SignedCms();

    // deserialize PKCS #7 byte array
    signedMessage.Decode(data);

    // check signature
    // false checks signature and certificate
    // true only checks signature
    signedMessage.CheckSignature(false);

    // access signature certificates (if needed)
    foreach (SignerInfo signer in signedMessage.SignerInfos)
    {
        Console.WriteLine("Subject: {0}",
          signer.Certificate.Subject);
    }

    // return plain data without signature
    return signedMessage.ContentInfo.Content;
}

Putting It All Together

When working with security policies or communication protocols, you will encounter certificates in all kinds of situations. In this article I explained the basic APIs used to retrieve and search certificates and showed you how to use them for encryption and digital signatures. In addition, I provided some examples of higher level application services that require you to understand the Windows certificate store and the relationship between public and private keys.

The source code for this article, available for download from the MSDN®Magazine Web site, includes a small Windows Forms application that supports signing and encrypting files. It uses many of the techniques I've discussed in this article, like selecting certificates from different stores and protecting/verifying data using encryption and signatures.

Dominick Baier is an independent security consultant in Germany. He helps companies with secure design and architecture, content development, penetration testing, and code auditing. He is also the security curriculum lead at DevelopMentor, a Developer Security MVP, and author of Developing More-Secure Microsoft ASP.NET 2.0 Applications (Microsoft Press, 2006). His blog is located at www.leastprivilege.com.


Digital Media: Add Video To Controls And 3D Surfaces With WPF

Digital Media

Add Video To Controls And 3D Surfaces With WPF

Lee Brimelow
This article discusses:
  • Windows Presentation Foundation digital media basics
  • Using MediaElement and MediaPlayer
  • Video embedded in WPF controls
  • Mapping video onto 3D surfaces
This article uses the following technologies:
.NET Framework 3.0
Contents
Technical Overview
Using the MediaElement Control
Using the MediaPlayer Class
Embedding Video inside WPF Controls
The Wet Floor Technique
Mapping Video onto 3D Surfaces
Rendering Bitmaps from Video Frames
Additional Audio Capabilities in WPF
Summary

W
hen the buzz surrounding Windows Presentation Foundation (WPF) began circulating throughout the Adobe Flash community, my initial reaction was one of skepticism. That Microsoft had introduced a competing technology led many Flash developers, including myself, to lash out against the company in support of our beloved platform. Then I received an e-mail message saying that Microsoft was sending a technical evangelist to the frog design studio in San Francisco to demo Windows® Presentation Foundation, and I saw it as a perfect opportunity to show everyone why Flash was superior in every way.

Halfway through Karsten Januszewski's presentation, he showed off the now-famous North Face demo (channel9.msdn.com/Showpost.aspx?postid=116327), which features a 3D carousel in which curved 3D mesh objects are mapped with high-quality Windows Media® Video (WMV) clips. The demo was created jointly by Fluid, an interactive studio also located in San Francisco, and members of the Microsoft Windows Presentation Foundation team. My prejudices were obliterated as reality set in. This type of presentation would be difficult or impossible to accomplish with Flash-or any other platform for that matter.

Not long after that session, I swallowed my pride and downloaded the Microsoft® .NET Framework 3.0 runtime, thus beginning my career as a Windows Presentation Foundation interactive designer. Since that time, the whole Flash versus Windows Presentation Foundation debate has dissipated due to the fact that there aren't many areas in which the two technologies actually compete. The debate may start raging again when Microsoft releases its latest solution for delivering cross-platform Web pages that include graphics, video, animation, and audio. This upcoming release is codenamed "WPF/E" and a preview is available on the "WPF/E" developer center at msdn2.microsoft.com/bb187358.aspx.

The ability to map video to the 3D surfaces certainly provided the necessary eye candy to grab the attention of interactive designers. But this feature only scratches the surface of what's possible when integrating audio and video into Windows Presentation Foundation. This article will get you up to speed with Windows Presentation Foundation media integration and give you the knowledge you need to add media files and that optional visual interest to your own applications.

Technical Overview

The two most common ways of adding digital media files to Win32® applications today are by using the DirectShow® API or by embedding the Windows Media Player ActiveX® control. Using DirectShow gives developers ultimate control over media integration, but is much more complex than simply embedding the Windows Media Player ActiveX control.

Using DirectShow is still a viable option for Windows Presentation Foundation applications that need fine-grained control over media display and management. You might also implement DirectShow in a Windows Presentation Foundation application when an existing Win32 app is being ported to Windows Presentation Foundation and the .NET Framework 3.0.

Playing media using the Windows Media Player ActiveX control is still an option in Windows Presentation Foundation, but it makes much more sense to use the new MediaElement control. It allows you to do essentially the same thing, as you'll see in a few moments; however, it provides better integration with Windows Presentation Foundation. Specifically, it gives you the ability to composite media with alpha-blending, and it also lets you render onto 3D surfaces.

Audio and video integration with Windows Presentation Foundation gives developers multiple options to choose from, each with its own levels of complexity and control over media files. Regardless of the control or class used, what happens behind the scenes remains pretty much the same. The general idea is that when your application is told to play a media file in Windows Presentation Foundation, it launches the Windows Media Player runtime in the background to play the file. The fact that Windows Presentation Foundation uses Windows Media Player and does not contain its own media rendering engine presents some advantages and disadvantages. Piggybacking on Windows Media Player is somewhat analogous to using the ActiveX control.

Since Windows Media Player is responsible for Windows Presentation Foundation media playback, it's easy to talk about which media formats are compatible with Windows Presentation Foundation. Basically, any formats an end user can play through an existing Windows Media Player installation are the formats available to Windows Presentation Foundation applications. In order to add support for an additional media type, the user would have to install a new video codec.

Be aware that since the Windows Media Player runtime has to launch to play media files, there will be a delay in the media's initial playback as the runtime is loaded in the background. This can make precise control over media difficult, especially when trying to do things like syncing audio or video to animations or other events in Windows Presentation Foundation. One way to get around this is by preloading media files and keeping them in a paused state until you are ready to use them. Another options is that by using slip in the animation engine, you can make other animations dependent on the MediaElement, so at least everything will delay equally.This is not an optimal workflow and the Windows Presentation Foundation media team is working on a solution for the next version. Luckily, for audio playback there's an alternative-the System.Media.SoundPlayer class that allows for simple playback of WAV files without the need for Windows Media Player. We'll look at using this class later in this article.

Using the MediaElement Control

By far the easiest way to add media to a Windows Presentation Foundation application is by using the new MediaElement control. This control, like all others in Windows Presentation Foundation, can be added using either XAML or in the codebehind file using either C# or Visual Basic®. MediaElement enables basic playback and control over media files and can be run in one of two modes: Independent or Clock.

Independent mode, which is the default, works most closely like a traditional media player:

   <MediaElement Source="myMedia.wmv" Width="320"
     Height="240" LoadedBehavior="Manual" />

You set the Source property, which will be a URI to a media file. What happens when the media file is loaded depends on the LoadedBehavior property. This property must be set to a value from the MediaState enumeration: Close, Manual, Pause, Play, or Stop. In order to control media playback using the Play, Pause, or Stop methods of the MediaElement control, the LoadedBehavior property must be set to Manual.

Figure 1 shows an example of a simple media player using the MediaElement control running in Independent mode. Figure 2 shows the XAML and C# code used to implement this control.

Figure 2 Implementing the MediaElement Control (Close Figure 2)

XAML
XAML
<Window x:Class="VideoProject.Window1"
    xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    Title="VideoProject" Height="800" Width="600">
<Grid>
    <StackPanel HorizontalAlignment="Center" VerticalAlignment="Center">
        <MediaElement Name="myMedia" Source="myMedia.wmv"
            LoadedBehavior="Manual" Width="320" Height="240" />
        <StackPanel Orientation="Horizontal" Margin="0,10,0,0">
            <Button Content="Play" Margin="0,0,10,0"
                Padding="5" Click="mediaPlay" />
            <Button Content="Pause" Margin="0,0,10,0"
                Padding="5" Click="mediaPause" />
            <Button x:Name="muteButt" Content="Mute"
                Padding="5" Click="mediaMute" />
        </StackPanel>
    </StackPanel>
</Grid>
</Window>

C#
C#
using System;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Data;
using System.Windows.Documents;
using System.Windows.Input;
using System.Windows.Media;
using System.Windows.Media.Imaging;
using System.Windows.Shapes;

namespace VideoProject
{
    public partial class Window1 : System.Windows.Window
    {
        public Window1()
        {
            InitializeComponent();
            myMedia.Volume = 100;
            myMedia.Play();
        }

        void mediaPlay(Object sender, EventArgs e)
        {
            myMedia.Play();
        }

        void mediaPause(Object sender, EventArgs e)
        {
            myMedia.Pause();
        }

        void mediaMute(Object sender, EventArgs e)
        {
            if (myMedia.Volume == 100)
            {
                myMedia.Volume = 0;
                muteButt.Content = "Listen";
            }
            else
            {
                myMedia.Volume = 100;
                muteButt.Content = "Mute";
            }
        }
    }
}
Figure 1 A Simple MediaElement Control
Figure 1 A Simple MediaElement Control

Running the MediaElement control in Clock mode enables you to control media playback using the robust Windows Presentation Foundation animation engine. Clock mode can be enabled either by targeting the MediaElement as part of a MediaTimeline or by explicitly creating a MediaClock from a MediaTimeline and assigning it to the MediaElement (though that's not a recommended approach). This may sound confusing, but essentially it just means you can attach the MediaElement to a Storyboard and animate its position using the Windows Presentation Foundation animation engine. This allows you to sync up media playback to other animations in your application.

Here's an example of the XAML code needed to playback media in Clock mode using a Storyboard:

<MediaElement Name="myMedia" Width="320" Height="240" />
<Grid.Triggers>
<EventTrigger RoutedEvent="Grid.Loaded">
  <EventTrigger.Actions>
    <BeginStoryboard>
      <Storyboard Storyboard.TargetName="myMedia">
        <MediaTimeline Source="myMedia.wmv"
         BeginTime="00:00:00" Duration="00:05:00" />
      </Storyboard>
    </BeginStoryboard>
  </EventTrigger.Actions>
</EventTrigger>
</Grid.Triggers>

Note that the source of the media file is set in MediaTimeline rather than in MediaElement. MediaTimeline targets the MediaElement control via the Storyboard.TargetName attached property. Clock mode can be confusing if you're not familiar with the Windows Presentation Foundation animation engine, so make sure you're comfortable with basic Windows Presentation Foundation animation before venturing down this path.

After you've set up MediaElement and configured its mode, you can then read and write to a whole slew of properties to further enhance the media playback experience. To get or set the position of the media file, you can simply set the Position property with a valid TimeSpan object. You can display the buffering and playback progress to the user by reading the BufferingProgress and DownloadProgress properties. For audio control, the Balance and Volume properties can be set to customize or animate a media's audio track. Now think about if these values were databound to a couple of sliders-you could give the user complete control over the media's aural experience!

MediaElement also fires a number of useful events. MediaOpened and MediaEnded are handy for triggering other parts of your application based on whether the media has started or ended. The BufferingStarted and BufferingEnded events allow you to display feedback to the user when the video freezes during a buffering operation. If used in conjunction with the BufferingProgress property, they let you display a buffering percentage to the user similar to what is seen when viewing streaming video with Windows Media Player.

Based on my own experience-and reinforced by conversations with the Windows Presentation Foundation media team-I recommend you normally use the MediaElement control in Independent mode unless Clock mode is absolutely necessary. For standard media playback and control, there's no need to deal with Storyboards or MediaTimelines. And MediaElement should be your default choice for media integration since the ability to add it using XAML lets designers implement media, leaving the developer free to do other things.

Using the MediaPlayer Class

If the MediaElement control doesn't provide enough functionality and control over playback for your application, the next option for you to consider is using the MediaPlayer class located in the System.Windows.Media namespace. Since this is not a Windows Presentation Foundation control element, the only way to implement a MediaPlayer is to create it in the codebehind file using either C# or Visual Basic. Here's a simple example that shows how to implement a MediaPlayer using C#:

MediaPlayer mp = new MediaPlayer();
Try {
    mp.Open(new Uri("myMedia.wmv"));
}
catch(Exception ex) {
    MessageBox.Show(ex.Message);
}

The properties and methods available in the MediaPlayer class are strikingly similar to those of the MediaElement control. This is not merely a coincidence-the MediaElement is essentially a high-level wrapper of the MediaPlayer class. The major caveat with using MediaPlayer is that it doesn't have any direct visual representation and can't be added directly to the visual tree. To play video you need to draw the MediaPlayer onto a visual surface and then add that surface to the visual tree. This makes MediaPlayer a good choice for playing audio as, obviously, no visual representation is needed. For straight video playback, the MediaElement control is a much better option since it inherits from the UIElement class and thus can be directly added to the visual tree in your applications.

Just like the MediaElement control, you can run a MediaPlayer instance in either Independent or Clock mode. As for playback control, again this is essentially a carbon copy of the MediaElement control. So what's different about the MediaPlayer? Well, for one, there are no LoadedBehavior or UnloadedBehavior properties to deal with, as these are unique to the MediaElement control. If you are planning to draw media to 3D surfaces using the VideoDrawing class, then you must use MediaPlayer because the VideoDrawing's Player property won't accept a MediaElement control as its value. (Don't worry-I'll show you how to draw video to 3D and other surfaces a little later.)

Another advantage (or disadvantage, depending on your situation) is that since the MediaPlayer is not implemented via XAML, it doesn't become part of the visual tree. Not being a part of the visual tree can be beneficial if your media file doesn't need visual representation. There are also some differences in how you load a media file into your application. In the MediaElement control you set its Source property, but when using the MediaPlayer class you call the Open method and pass in a valid Uri object.

So if your application doesn't require media files to be implemented using XAML or to have an automatic visual representation, the Media Player is probably your best option since it gives you all the capabilities of the MediaElement control without the dreaded additional overhead.

Embedding Video inside WPF Controls

Since the MediaElement control inherits from the System.Windows.UIElement class, it can be placed anywhere any other UIElement control can be. One such example would be nesting a MediaElement control inside a Button control, thus creating a video button. This can be accomplished using this XAML code:

  <Button Name="myVideoButton" Click="playVideo"
          Width="320" Height="240">
      <MediaElement Name="myVid" Source="myMedia.wmv"
          LoadedBehavior="Manual" Width="320"
          Height="240" />
  </Button>

In this code I'm embedding MediaElement inside of a Button and setting its LoadedBehavior to Manual. The click event of the Button calls a method (not shown) that tells the MediaElement to begin playing. With a few lines of code you've created an interactive video thumbnail. All kinds of creative possibilities open up due to the fact that Windows Presentation Foundation treats the MediaElement control just like any other UI element in the visual tree. As you'll see in the next section, you can paint the video playing in the MediaElement onto other Windows Presentation Foundation objects to create incredibly rich user experiences.

The Wet Floor Technique

In recent times, glass and reflections have taken over the interfaces of everything in sight. Without a doubt, someday you will be asked to deliver such a technique in Windows Presentation Foundation, and fortunately it's easy. The extensive brush system in Windows Presentation Foundation allows you paint one control onto another with ease. In this example, I'll paint the video playing in a MediaElement control onto a Rectangle using a VisualBrush, creating a perfect reflection of the video. In the design world, this effect is known as a wet floor.

The easiest way to create a wet floor effect is by placing your MediaElement control in the first slot of a StackPanel control. Below that you then create a Rectangle with the same dimensions as the video. Next, use a VisualBrush as the fill for the Rectangle and databind its Visual property to the MediaElement control. Finally, flip the reflection to create a mirror image and apply an OpacityMask to fade out the reflection.

Figure 3 shows the completed effect, while Figure 4 shows the XAML code. This type of visual manipulation would have been incredibly complicated in traditional Win32 programming. And, thanks to the powerful databinding features in Windows Presentation Foundation, this is a live reflection that automatically updates whenever your video does.

Figure 4 XAML Used to Create the Wet Floor (Close Figure 4)
<Grid x:Name="myGrid">
<StackPanel VerticalAlignment="Center">
    <MediaElement Name="myVid" Source="myMedia.wmv"
        LoadedBehavior="Play" Width="320" Height="240" />
    <Rectangle Width="320" Height="240">
        <Rectangle.Fill>
            <VisualBrush Visual="{Binding ElementName=myVid}" />
        </Rectangle.Fill>
        <Rectangle.OpacityMask>
            <LinearGradientBrush StartPoint="0.5,0" EndPoint="0.5,1">
                <GradientStop Color="#AA000000" Offset="1" />
                <GradientStop Color="#00000000" Offset="0" />
            </LinearGradientBrush>
        </Rectangle.OpacityMask>
        <Rectangle.RenderTransform>
            <TransformGroup>
                <ScaleTransform ScaleY="-1" />
                <TranslateTransform Y="242" />
            </TransformGroup>
        </Rectangle.RenderTransform>
    </Rectangle>
</StackPanel>
</Grid>
Figure 3 The Completed Wet Floor
Figure 3 The Completed Wet Floor

Mapping Video onto 3D Surfaces

If I had to pick the one area that seems to have created the biggest buzz around Windows Presentation Foundation, it would be the ability to map video files onto 3D mesh objects. The North Face demo showed just how powerful this could be in helping to create a visually stunning user interface. While many people have written off this effect as either too gimmicky or too resource-intensive, I feel that, when done in moderation, it can turn an ordinary application into one that provides an outstanding experience for users.

As for performance, of course there's quite a bit of overhead involved with displaying and animating 3D meshes. If you add to that displaying full-motion video that is being mapped onto the 3D meshes, you are talking about a ton of overhead. Just as with drinking mocha frappucinos, the key here is moderation. Overall, I've found the 3D performance in Windows Presentation Foundation excellent, so long as the complexity (polygon count) of the 3D meshes is kept to the minimum necessary. The Windows Presentation Foundation media team is committed to reducing the expense of 3D media in future versions, although it will always be more expensive than 2D media.

Before looking at how to map video onto 3D objects, let's first review what it takes to set up a 3D scene in Windows Presentation Foundation. First and foremost, you need a 3D object to place into the scene. There are no 3D primitive classes that are a part of the .NET Framework 3.0. But someone at Microsoft created a class called Mesh3DObjects that can create a wide range of primitive objects. (You can find more information about this class in the tutorial by Karsten Januszewski at msdn.microsoft.com/library/en-us/dnlong/html/avalon2d-3d.asp.)

There are certain sets of elements required in order to create a 3D scene in Windows Presentation Foundation. First, you need the 3D object, and this object needs to have some kind of material in order to be viewable in the scene. Materials are created using brushes like SolidColorBrush, LinearGradientBrush, ImageBrush, or a VisualBrush. You'll see in a few moments how a VisualBrush can be used to map video, or any other visual element for that matter, onto a 3D surface. The technique is essentially the same as the wet floor technique that I discussed earlier.

The 3D mesh and materials need to be grouped inside of a GeometryModel3D element, and doing so creates a valid Windows Presentation Foundation 3D object. For a 3D scene to be visible, you'll need to add one or more lights to the scene. Then, just as a photograph isn't possible without a camera, a 3D scene in Windows Presentation Foundation isn't either. Finally, to complete the scene, all of these items need to be wrapped in a Viewport3D control in order to seal the deal.

Once you have a functional 3D scene in place, you can start having fun mapping video and other media to the 3D meshes. This is accomplished using a VisualBrush to draw the video onto the 3D surface. In the following code snippet, I am applying a VisualBrush to a 3D object's DiffuseMaterial:

<GeometryModel3D.Material>
<DiffuseMaterial>
    <DiffuseMaterial.Brush>
        <VisualBrush>
            <VisualBrush.Visual>
                <MediaElement Source="myMedia.wmv" />
            </VisualBrush.Visual>
        </VisualBrush>
    </DiffuseMaterial.Brush>
</DiffuseMaterial>
</GeometryModel3D.Material>

Figure 5 shows a single frame of the video mapped onto the Windows Presentation Foundation 3D surface.

Figure 5 Mapped to a WPF 3D Surface
Figure 5 Mapped to a WPF 3D Surface

There's another alternative you can use to map video onto 3D surfaces in Windows Presentation Foundation. The VideoDrawing class, which is a part of the System.Windows.Media namespace, does precisely what its name implies. It can draw video coming from a MediaPlayer instance via a DrawingBrush onto any brush-compatible surface. In order to achieve this, you need to set the Drawing property of the DrawingBrush to an instance of the VideoDrawing class. This VideoDrawing instance can render video by setting its Player property to a MediaPlayer instance containing a video file. Finally, the DrawingBrush can be applied to a 3D object's DiffuseMaterial's Brush property. The following code snippet shows an example implementation using C#:

MediaPlayer mp = new MediaPlayer();
mp.Open(myValidMediaUri);
VideoDrawing vd = new VideoDrawing();
vd.Player = mp;
DrawingBrush db = new DrawingBrush();
db.Drawing = vd;
diffuseMat.Brush = db;
vd.Player.Play();

One of the major differences between using a VisualBrush or the VisualDrawing class is that a VisualDrawing must be created and implemented in the codebehind file (using C# or Visual Basic). Furthermore, the VideoDrawing class will only work with MediaPlayer instances and not with the MediaElement control. When it comes to performance, there isn't much difference between the two methods since both require Windows Presentation Foundation to create an intermediary render surface in order to correctly map the video to 3D.

Rendering Bitmaps from Video Frames

A video application may need to essentially take a screen grab of the video stream and render it to a bitmap. One such example is seen in some commercial DVD playback software where the user can click on a camera icon at any time during playback to take a screen grab from the video. Another use could be for creating thumbnails of a set of video files if you didn't want all of them actually connected to MediaElement controls. There is also some potential for creating runtime video effects, although for this type of thing you'd be much better served by diving into the DirectShow API.

Like most things in Windows Presentation Foundation, there are multiple ways of accomplishing a task, and rendering bitmaps of video frames is no different. No matter what method is used, the core functionality for rendering bitmaps of visual elements in Windows Presentation Foundation is handled by the RenderTargetBitmap class, which is a part of the Windows.Media.Imaging namespace. This amazing class can create bitmaps from any item in the visual tree of a Windows Presentation Foundation application.

To render a bitmap using this class, first you need to create a new DrawingVisual object, which gets passed to the RenderTargetBitmap.Render method. Next you create a new DrawingContext object that you'll pass to the DrawingVisual.Render method. With this DrawingContext instance, you can draw all kinds of graphics into the DrawingVisual object. To draw a frame of video, you can call the DrawingContext.DrawVideo method and pass in a MediaPlayer instance that is currently playing the video file. You'll also need to pass in a Rect value that determines the area of the video file that will be captured.

At this point, you can actually render a bitmap from the DrawingVisual instance that now contains your captured video frame. To do this, you just need to pass it to the RenderTargetBitmap.Render method. Now your RenderTargetBitmap instance contains the video frame, but it won't be a bitmap until you pass it to the BitmapFrame.Create method. To see the bitmap frame, you can simply set this method call as the value of an Image control's Source property. The following snippet shows a C# implementation of this frame-capture technique:

RenderTargetBitmap rtb = new RenderTargetBitmap(320, 240, 1 / 200,
    1 / 200, PixelFormats.Pbgra32);
DrawingVisual dv = new DrawingVisual();
DrawingContext dc = dv.RenderOpen();
dc.DrawVideo(myMediaPlayer, new Rect(0, 0, 320, 240));
dc.Close();
rtb.Render(dv);
Image im = new Image();
im.Source = BitmapFrame.Create(rtb);

The target bitmap's format is set by passing a valid PixelFormats enumeration value to the RenderTargetBitmap constructor. The ability to render bitmaps from video frames may at first seem like a niche feature, but there are actually many uses for this type of functionality beyond the simple video screen capture tool.

Additional Audio Capabilities in WPF

Most everything I've discussed so far about Windows Presentation Foundation media integration holds true for both video and audio. There are, however, several additional classes available for handling audio playback in Windows Presentation Foundation. As I mentioned earlier, media played in Windows Presentation Foundation is actually driven by a set of Windows Media Player runtimes that need to be loaded before the file can be played back. This can cause major issues when you're trying to synchronize media files to user events or animations, though it can be mitigated with slip behavior in timelines. When it comes to audio, there are some alternatives that can bypass this problem.

The SoundPlayer class, which is a part of the System.Media namespace, provides a simple interface for loading and playing back audio WAV files. These WAV files can be external, streaming, or embedded resources. This method of audio playback has much lower overhead than playing an audio file through a MediaElement control since it does not at all depend on Windows Media Player for playback.

To load a WAV file into a SoundPlayer instance, you first need to set the SoundLocation property to the location of the WAV file and then call the Load method to start the loading process. Once the file is fully loaded you can play it using the SoundPlayer.Play method. The file won't start playing until it is fully loaded into memory.

This lightweight audio playback is also available in XAML using the SoundPlayerAction control, which lets you play a WAV file inside of an EventTrigger. This is very useful for things like interface sounds for button rollovers, for example. To use it, you simply place it into the EventTrigger.Actions block of the EventTrigger. Following is a code snippet of SoundPlayerAction being called in response to a button's MouseEnter event:

<EventTrigger RoutedEvent="Button.MouseEnter" SourceName="myButton">
<EventTrigger.Actions>
        <SoundPlayerAction Source="media/overSound.wav"/>
    </EventTrigger.Actions>
</EventTrigger>

Summary

I've explored here the available methods for integrating audio and video into Windows Presentation Foundation applications. If Windows Presentation Foundation does not provide the level of control needed for your project, you can always revert to using the DirectShow API for media integration. But the ability to add audio and video using XAML allows designers to handle simple media integration so that developers can focus on more complex coding. I am no longer the sceptic I once was. The amazing visual richness available in Windows Presentation Foundation makes media integration not only possible, but incredibly exciting!

Lee Brimelow, a Senior Design Technologist at frog design, is an award-winning interactive designer who specializes in Flash and WPF. He also runs the popular WPF technology blog at www.thewpfblog.com.


WiX Tricks: Automate Releases With MSBuild And Windows Installer XML

WiX Tricks

Automate Releases With MSBuild And Windows Installer XML

Sayed Ibrahim Hashimi
This article discusses:
  • An overview of Windows Installer XML
  • Creating WiX packaging instructions
  • Integrating WiX and MSBuild
  • Automating builds and packaging
This article uses the following technologies:
Visual Studio, Windows Installer XML (WiX), MSBuild
Code download available at: WiX2007_03.exe (498KB)
Contents
Introduction to WiX
Creating WiX Files
WiX and MSBuild
MSBuild Batching
Automating Builds and Packaging
Anatomy of the MSBuild Script
Putting It All Together
Customizing the Process
Conclusion

D
uring the course of development, it's important to have an automated build process. Equally important is having an automated means of creating releases. Unfortunately, in many organizations-especially smaller ones-this doesn't happen. Typically, you'll find that the release is simply cobbled together at the last minute. However, if you take the time to set up an automated build and release plan, you'll save countless hours that could be better spent on tasks other than building and releasing your project.

In this article, I'll show you how to achieve an automated and repeatable build and release process in your organization using the Microsoft® Build Engine (MSBuild) and the Windows® Installer XML (WiX) toolset. This article covers WiX v2 (note that when WiX v3 is released, some of the syntax examples won't translate directly). The techniques described here will be relevant whether or not you are using WiX to create your releases, though WiX does simplify the process. They can also be applied, with some modifications, to applications that are being developed without using the Microsoft .NET Framework 2.0.

I'll assume you're familiar with MSBuild (if you need to refresh, look at my June 2006 MSDN®Magazine article "Inside MSBuild: Compile Apps Your Way with Custom Tasks for the Microsoft Build Engine"). I will provide an overview of the WiX toolset for those who are not familiar with it. See the "MSBuild and WiX Resources" sidebar for references to more related articles and tools. In this article I'll use my Sedodream MSBuild project for demonstration purposes. You can get the latest sources at www.codeplex.com/Sedodream.

Introduction to WiX

When creating an application, the end result is usually to install and run it on production machines. The WiX toolset can help you do this. In this section, I will describe WiX and give an introduction to creating installers using WiX.

WiX describes what the installation looks like on the target machine. Windows Installer is used in many Microsoft applications, from Visual Studio® to Microsoft Office. You might be surprised to learn that WiX is actually an open source project and is hosted at sourceforge.net. You can download the latest binaries and sources at wix.sourceforge.net. When you download and install WiX, you'll find a number of executables installed as summarized in Figure 1. I will focus on using the Candle.exe and Light.exe tools.

Figure 1 WiX Components (Close Figure 1)

Name | Description
Candle.exe | Transforms the WiX source file into an intermediate representation. This is actually another XML file, but you should never manually change those generated files.
Dark.exe | Converts an MSI file into an appropriate WiX source file-can be thought of as "decompiling" the installer.
Light.exe | Generates the Windows installer from the intermediate representation of the WiX source file(s).
Lit.exe | Generates WiX libraries that can be used to build other installer packages.
Tallow.exe | Used to create the WiX source XML to replicate its files and folders in an installation directory or file.
WixCop.exe | Checks a WiX source file for potential problem areas, similar to FxCop.

WiX uses a declarative language, not a procedural one, which means you describe what your installation will look like, not what steps need to be followed in order to achieve it. This may be different from what you are accustomed to, but it is surprisingly easy to learn. Typically, your WiX source files will be populated by describing files that will be installed on the target machine. I will focus on those components here.

In a WiX source file, there are three main elements with respect to the files you intend to install: File, Component, and Feature. A File element is a reference to an individual file. Files must be contained in a Component element, which is the smallest unit to be installed. That is, if you have a component containing 100 files and you install that component, all of its contained files are installed. Conversely, if that component is not installed, none of the files are installed. Creating components that contain a large number of files is not recommended.

Components are always contained in a Feature element and can be contained in more than one feature. A feature is a set of components and potentially of sub-features. If your installer has a graphical interface that allows the user to select which items to install, they are actually selecting features.

Creating WiX Files

You can author WiX source files in any text or XML editor you choose. You can also use Visual Studio to author WiX source files with IntelliSense®. If IntelliSense is not enabled when you are editing a WiX source file using Visual Studio, all you have to do is copy the wix.xsd file into the Visual Studio schemas directory. Typically, this directory resides at %Program Files%\Microsoft Visual Studio 8\Xml\Schemas. As you create WiX source files you will also need to generate many GUIDs. Visual Studio has a tool that you can use for this purpose, and there are Visual Studio macros available so you can assign a shortcut to directly insert a new GUID as well.

Now let's start by creating a new WiX source file. The first element is always going to be a Wix element. Child elements for the Wix element include Product, Fragment, Module, and PatchCreation. The type of desired output will drive which of these you will use. In this case, I want the end result to be an installer database (MSI) for my project so I'll use the Product element. The beginning of the WiX v2 source file, Sedodream.wxs, for the sample project is shown here:

<?xml version='1.0' encoding='UTF-8'?>
<Wix xmlns='http://schemas.microsoft.com/wix/2003/01/wi'>
   <Product Name='Sedodream MSBuild Project'
        Id='C9D25926-FCE0-4EB6-8FF5-4686EE5AB089'
        Language='1033' Codepage='1252' Version='1.0.0'
        Manufacturer='SedoTech'
        UpgradeCode='9C5E4073-EFDE-419B-935D-CE2632BC560E'>

      <Package Id='????????-????-????-????-????????????'
         Keywords='Installer'
         Description='Sedodream MSBuild Project Installer'
         InstallerVersion='100' Languages='1031'
         Compressed='yes' SummaryCodepage='1252' />
...

As you can see ,the Product element has many attributes, the most important of which are Id and Name. The Id is used to uniquely identify the product. You must change this for a new major release. You should also make a note of its value for later reference. The value of the Name attribute is what will be displayed in the Add/Remove programs panel, and if your installer has a UI it will be shown on the introduction page.

The first child of the Product element, the Package element, is shown here as well. Notice that the Id attribute contains a series of ? characters in the format of a GUID. By using this syntax, the code specifies that the Id should be generated at build time. When you are creating installers, you want the Package Id to be different for each installer created, even from one build to the next. This is the only GUID you can have auto-generated; all other GUIDs need to stay the same and should be recorded for future reference.

Almost every installer will place at least one file onto the target machine, and this installer is no different. Since WiX uses a declarative approach, I'll declare what the directory structure is and that will be recreated on the target machine. I do this using a series of Directory elements:

<Media Id='1' Cabinet='Sedodream.cab' EmbedCab='yes'/>

<Directory Id='TARGETDIR' Name='SourceDir'>
   <Directory Id='ProgramFilesFolder' Name='PFiles'>
      <Directory Id='MSBuildDir' Name='MSBuild'>
         <Directory Id='INSTALLDIR' Name='Sedodrea'
          LongName='Sedodream'>

The Media element in this sample specifies that a cabinet file will be generated and placed into the installer database. This is a required element, and if you have installations spanning more than one medium you can have more than one. For example, if you are distributing your application on a CD, but it needs to span over multiple discs. In most cases, you won't have to worry about this issue.

Following Media is the series of Directory elements. The first Directory element will be a virtual element that simply encapsulates the other entries. The entry under the TARGETDIR directory element has the Id value of ProgramFilesFolder. As you may have guessed, this is a well-known location and Windows Installer will set its value when launched. There are other system folders you can use; a complete list is available from msdn.microsoft.com/library/en-us/msi/setup/system_folder_properties.asp. These system properties always resolve to their full paths.

The next elements provide names of custom directories. The directory will be named using the LongName attribute if possible; otherwise the Name attribute is used. If those directories don't exist, Windows Installer will create the appropriate one at install time. The complete tree of what is declared in the previous code fragment ends up being C:\Program Files\MSBuild\Sedodream.

After you declare your directory structure, it's time to start creating Component declarations. As noted earlier, the smallest unit that can be installed is the component, which can consist of many different items including files, shortcuts, registry keys, and certificates. Design your components so they are independent of each other. That is, when you install or uninstall a component there should be no adverse effect on other components. Inherently this means that items should only be contained in a single component. If that's not the case, you may want to reconsider how your components are organized. It's not unusual for a component to contain only a single file.

Each Component declaration must have an Id and a GUID attribute. The Id is the name that you use to refer to it, and the GUID is a unique identifier for Windows Installer to use. It is very important to ensure that none of your component GUIDs repeat.

Figure 2 shows a component from the sample WiX source file. You can see that it contains four File elements and two XmlFile elements. The File element is simply a reference to a file that needs to be placed on the target device. The File element can have many different attributes, but the ones you'll mostly use are Id, Name, LongName, Source, and DiskId, described in Figure 3.

Figure 3 File Attributes (Close Figure 3)

Attribute Name | Description
Id | An identifier that can be used to refer to the file.
Name | The short name of the file in 8.3 format.
LongName | The long name of the file; this is the name that will be used on the target machine if it supports long file names. Otherwise the Name value will be used.
Source | The relative path to the file on the machine that is creating the installer database.
DiskId | The identifier of the media item that this file will be contained in.
Figure 2 Defining Components (Close Figure 2)
<Component Id='TaskBinFiles' Guid='38736E2E-BEB7-48A9-A2B3-138A57A69D45'>
   <File Id='MSBCommonDLL' Name='CommoDLL'
      LongName='Sedodream.MSBuild.Common.dll'
      Source='Sedodream.MSBuild.Common.dll' Vital='yes' DiskId='1'/>
   <File Id='SedodreamLoggersDLL' Name='Logger'
      LongName='Sedodream.MSBuild.Loggers.dll'
      Source='Sedodream.MSBuild.Loggers.dll' Vital='yes' DiskId='1'/>
   <File Id='SedodreamTasksDLL' Name='Tasks'
      LongName='Sedodream.MSBuild.Tasks.dll'
      Source='Sedodream.MSBuild.Tasks.dll' Vital='yes' DiskId='1'/>

   <File Id='SedodreamTASKS' Name='SeTasks' LongName='Sedodream.tasks'
      Source='Sedodream.tasks' Vital='yes' DiskId='1'/>
   <!-- Merge in the Xsd for the custom tasks we have created to have
      IntelliSense -->
   <XmlFile Id='SedoSche'
      File='C:\Program Files\Microsoft Visual Studio 8\
               Xml\Schemas\1033\Microsoft.Build.xsd'
      Action='createElement' ElementPath='//xs:schema'
      Name='xs:include' Permanent='no' Sequence='0'/>
   <XmlFile Id='SedoSch2'
      File='C:\Program Files\Microsoft Visual Studio 8\
               Xml\Schemas\1033\Microsoft.Build.xsd'
      Action='setValue'
      ElementPath='//xs:schema/xs:include[\[]not(@schemaLocation)[\]]'
      Name='schemaLocation' Value='MSBuild\Sedodream.MSBuild.xsd'
      Permanent='no' Sequence='1' />
</Component>

The XmlFile element actually modifies an existing XML file on the target machine. It inserts a new element into the Microsoft.Build.xsd file, which allows IntelliSense to be enabled for the custom MSBuild tasks contained in the sample project.

Features are the items that the user will select for installation if your installer shows a UI. Here are the Feature declarations from the sample project:

<Feature Id='Complete'
   Title='Sedodrem Core' Description='MSBuild libraries'
   Display='expand' Level='1' ConfigurableDirectory='INSTALLDIR'>
   <ComponentRef Id='TaskBinFiles'/>
   <ComponentRef Id='NUnitFiles'/>

   <Feature Id='Samples' Title='Samples'
      Description='Contains samples of Sedodream usage'
      Display='expand' Level='100' ConfigurableDirectory='INSTALLDIR'>
      <ComponentRef Id='Samples'/>
   </Feature>
</Feature>

Like many other WiX elements, the Feature element has many possible attributes, some of the more commonly used ones are described in Figure 4.

Figure 4 Feature Attributes (Close Figure 4)

Attribute Name | Description
Id | An identifier that can be used to refer to the feature.
ConfigurableDirectory | Allows you to set the installation location from a property, which can be determined by a user with a UI.
Description | A description for the feature, which will be displayed to a user with a UI.
Level | Specifies the install level for the feature. A value of zero means the feature will not be installed. Non-zero values that are less than or equal to the INSTALLLEVEL mean the feature is installed.
Title | A title for the feature whose value will be displayed in the tree of features if a UI is being used.

In this WiX snippet the main feature, Complete, contains a reference to two components and a sub-feature. The sub-feature references are achieved by a child Feature element. The WiX source file can contain as many of these as necessary. The component reference uses a ComponentRef element. When using the ComponentRef element you have to use the same ID value that is used in the Component element you are trying to reference.

WiX and MSBuild

In the latest releases for WiX v2 you'll find a wix.targets file, which contains definitions that will assist you in creating WiX releases from MSBuild. The WiX team has created a set of MSBuild tasks that are referenced by the wix.targets file. Those tasks are all contained in the WiXTasks.dll assembly and summarized in Figure 5.

Figure 5 MSBuild Tasks for WiX (Close Figure 5)

Name | Description
Candle | Creates the intermediate representation of the WiX source files by calling the WiX compiler.
Lit | Creates a WiX library from the intermediate representation.
Light | Creates the final installer from the intermediate representation by calling the WiX linker.

The way in which you'll build installers is very similar to how managed projects are built now. For example, when creating a C# project, your project file will contain all the necessary properties and items to build the project, but all the steps to build the project are contained in the Microsoft.CSharp.targets file. This file is included into the project file with the MSBuild Import element. For WiX, your project file will define all necessary properties and items and then include the wix.targets file to define the logistics of how to build the final packages. Figure 6 provides a list of the minimum declarations required to successfully build an installer using wix.targets.

Figure 6 Declarations Needed to Build an Installer (Close Figure 6)

Name | Description
ToolPath | Contains the path to the WiX installation directory. This is where the wix.targets file resides.
OutputName | Path and name of the output that will be created. The name should not include any extension because it will be used for both the intermediate representation and the final package file. This property is analogous to the OutputFile argument for Candle.exe and Light.exe.
OutputType | Represents what the final file should be. The possible values are package, module, library, and object. This will determine whether the Light or Lit task will be invoked and the extension of the final file created. Since we want to create an installer, we will provide the value package for this property.
Compile | MSBuild item that contains the WiX source files to be compiled.
BaseInputPath | Serves as the input path for source files. Although technically not required, most times this should be employed. Otherwise you'll have to make sure to invoke msbuild.exe from the correct directory.

To create the installer you need to create an MSBuild file to define the required properties and items. In this case, I'll name the file SedodreamMSI.wproj. This file is shown in Figure 7.

Figure 7 MSBuild File SedodreamsMSI.wproj (Close Figure 7)
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Build">
  <!--===============================================================
    These must be declared BEFORE the statement
    that imports the wix.targets file
   ==================================================================-->
  <PropertyGroup>
    <!-- The location pointing where WiX is installed -->
    <ToolPath>C:\Data\Development\WiX\</ToolPath>
    <!-- Required Property by WiX -->
    <OutputName Condition="$(OutputName)==''" >
      $(Configuration)Sedodream</OutputName>
    <!-- Required property by WiX -->
    <OutputType
      Condition="$(OutputType)==''" >package</OutputType>
    <!-- Input path to source files  -->
    <BaseInputPath  Condition="$(BaseInputPath)==''">
      $(PackageRoot)$(Configuration)</BaseInputPath>
  </PropertyGroup>

  <ItemGroup>
    <!-- Required WiX item.
      Files in this item are sent to the Candle tool.
    -->
    <Compile Include="$(BaseInputPath)\Sedodream.wxs"/>
  </ItemGroup>

  <Import Project="$(ToolPath)wix.targets"/>
</Project>

As you can see, there's not much to this MSBuild file, just a few properties and a lone item defined. The file assumes that a WiX source file, Sedodream.wxs, has been created and is residing in the BaseInputPath directory. The WiX file also makes some assumptions about where files are placed. For now let's assume that all the files are where they need to be.

To build the installer I will need to invoke MSBuild on the project file. I open the Visual Studio 2005 command prompt, navigate to where the file resides, and execute the following command:

msbuild Sedodream.wproj

The output is shown in the Figure 8.

Since I didn't define a target to execute, the DefaultTargets were executed, and this was defined to be Build in the Sedodream.wproj file. From the output you can see that a file named ReleaseSedodream.msi was created in the bin\Release folder. At this point you can execute the installer to ensure that it works as expected. Before continuing with our main goal, I will cover an advanced MSBuild topic, batching, in the next section. This is necessary because batching is used throughout the process that will build the product and its release.

Figure 8 WiX Installer Creation Output from MSBuild
Figure 8 WiX Installer Creation Output from MSBuild(Click the image for a smaller view)
Figure 8 WiX Installer Creation Output from MSBuild
Figure 8 WiX Installer Creation Output from MSBuild (Click the image for a larger view)

MSBuild Batching

When using MSBuild you'll discover that there is no looping construct to be found. In place of a loop you can employ batching, which uses item metadata to break items into different categories called batches or buckets that consist of one or more items. Once the items are separated you can iterate through each batch. This is used throughout the build process for managed projects. There are two broad categories of batching: task batching and target batching. In task batching, when you execute a task, the MSBuild engine will determine what buckets need to be created and execute the task over those buckets. To use task batching, you provide an item's metadata to the task.

To demonstrate batching, imagine you are faced with the following scenario. You have to copy a set of files from one location to one or more other locations and you have two options: you can copy the files using individual Copy tasks to each location, or you can use batching to achieve this in one copy element. The first approach, although simple, is not a good solution because it is difficult to maintain. To illustrate the second approach, have a look at this very simple MSBuild file, batching01.proj:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
         DefaultTargets="CopyFiles">
  <ItemGroup>
    <SourceFiles Include="*.txt"/>
    <Dest Include="One;Two;Three;Four;Five"/>
  </ItemGroup>

  <Target Name="CopyFiles">
    <Copy SourceFiles ="@(SourceFiles)"
         DestinationFolder="%(Dest.FullPath)"/>
    <Message Text="Fullpath: %(Dest.FullPath)"/>
  </Target>
</Project>

In this file, I declare an item SourceFiles to include any file that ends with .txt, and I also declare a Dest item that includes five locations. In the target CopyFiles, I use the Copy task to copy the SourceFiles to each destination. Since I pass the FullPath metadata into the Copy task, the MSBuild engine will create a batch of all the Dest items that have distinct FullPath values, then the Copy task is invoked for each batch. This is task batching. To execute this I can invoke the following command from the directory that contains the batching01.proj file:

msbuild.exe batching01.proj

Now I'll demonstrate the same result with target batching. For target batching, the batches are created based on the Inputs and Outputs of the target so I have to pass an item's metadata for these values:

<Project xmlns="http://schemas.
  microsoft.com/developer/msbuild/2003"
         DefaultTargets="CopyFiles">
  <ItemGroup>
    <SourceFiles Include="*.txt"/>
    <Dest Include="One;Two;Three;Four;
                   Five"/>
  </ItemGroup>

  <!-- These targets demonstrate target
       batching -->
  <Target Name="CopyFiles"
          Inputs="@(SourceFiles)"
          Outputs="%(Dest.FullPath)">
    <Copy SourceFiles="@(SourceFiles)"
          DestinationFolder=
          "%(Dest.FullPath)"/>
  </Target>
</Project>

Target batching is driven off of the targets Inputs and Outputs. Since Outputs contains the Dest.FullPath declaration the target will be invoked once per bucket. This is similar to task batching, except that the contents of the entire target will be repeated once per batch, as shown in Figure 9.

Figure 9 MSBuild Output for Target Batching
Figure 9 MSBuild Output for Target Batching(Click the image for a smaller view)
Figure 9 MSBuild Output for Target Batching
Figure 9 MSBuild Output for Target Batching (Click the image for a larger view)

From this output you can see that the target, CopyFiles, was invoked once per batch for the Dest item. In the example of automating the build/package process, I'll use batching to create an installer for each combination of configuration platform and flavor.

Automating Builds and Packaging

For each organization the steps involved to build and package are going to be slightly different, in some cases due to the technologies used, and in others because of organizational requirements. I have put together a set of steps that I think will serve many organizations well, though it may require some fine tuning based on your particular scenario. The goal of an automated build-and-package process is twofold: to create a repeatable public build process and to create an automated repeatable package process. There are three core steps: get latest, build, and package, but this is broken down into many smaller steps as shown in Figure 10.

Figure 10 Build Process
Figure 10 Build Process(Click the image for a smaller view)
Figure 10 Build Process
Figure 10 Build Process (Click the image for a larger view)

A very important step is labeling the sources. This is a requirement because without this step you would not be able to repeat the process. When you create a build that will be sent for production, you need to be able to recreate that exact same build in order to maintain the deployed code. This step will be different based on the tool you are using for your source control provider. There are many different MSBuild tasks available for download to achieve this for various source control providers. See the "MSBuild and WiX Resources" sidebar for more detail.

Anatomy of the MSBuild Script

In this section, I'll introduce the MSBuild script that can be used to build and package the sample application. The overall structure is very similar to how managed projects are built. That is, you'll define a project file that contains what should be processed, and you import another project file, Sedodream.Package.targets, that defines the process flow. Think of these processes as sentences. Your packaging project file defines the nouns and the imported file defines the verbs. I will build the project by using the MSBuild task on the Solution file. To build the installer, the previous project file, SedodreamMSI.wproj, will be used.

When you have a process that has segregated elements, they somehow need to be brought together. This is done by defining a common set of properties, items, and targets (see Figure 11).

Figure 11 Properties and Items Used to Connect Elements (Close Figure 11)

Name | Description
SolutionFilePath | Property that defines the location of the solution file that will be used to build the product.
PackageRoot | Property that defines where the projects are built and serves as the temporary build location for WiX.
WiXSourceFiles | Item that contains all of the WiX files to be sent to the Candle tool for compiling.
AllConfigurations Item that contains all the configurations you want this process to be built for. Each configuration should contain metadata values for both FlavorToBuild and PlatformToBuild. An example declaration is:
  <AllConfigurations Include="Release|Any CPU">
     <FlavorToBuild>Release</FlavorToBuild>
     <PlatformToBuild>Any CPU</PlatformToBuild>
  </AllConfigurations>
OtherFiles Optional item that contains other files that should be copied to the package directory. These items should contain Destination metadata. This metadata defines the relative path of the destination to the package root. Here is a sample declaration:
  <OtherFiles Include=
     "..\Sedodream.MSBuild.Tasks\SampleTargets\**\*">
     <Destination>Samples\</Destination>
  </OtherFiles>

In this process, the solution will be built and the WiX source files will be copied into that same directory. You should define your WiX source files keeping this path in mind. Regardless of where your WiX files are contained in source control, they will be built from the same directory that the product is built to. Other required files are also copied to this same directory by the CopyFilesForPackaging target.

Also in this process, the PackageRoot property will define where the projects are built and where WiX will build the installers. In the sample, the directory is at the same level as the solution file, but it could be anywhere on the build machine-though I recommend not using a network share for this location.

The main targets declared in the Sedodream.Package.targets file are shown in Figure 12. All of these targets (and many others in the targets file) have their dependencies defined in properties so you can completely change the sequence of events in the process. For example, if you needed to inject a step before the Build target executed your project file, you could simply insert the following snippet after the Import statement for the Sedodream.Package.targets file:

<BuildDependsOn>
    CustomBeforeBuild;
    $(BuildDependsOn);
</BuildDependsOn>

Figure 12 Build Targets in the Sample Project (Close Figure 12)

Name | Description
Package | Only target in the DefaultTargets list; will be called to execute the entire process.
Build | Builds the solution file. The output of this build is routed to the correct directory by overriding the OutputPath property when invoking the MSBuild task.
CopyFilesForPackaging | Copies all necessary files that are not built to the build directories.
DeployPackage | Called near the end of the process. In the sample this target is empty, but in your implementation you may decide to override the target, to send your packages to the QA gate or to your deployment team.
Clean | Cleans up the mess made by the other targets.

Because the BuildDependsOn property was redefined, you effectively inject the CustomBeforeBuild step into the current process without modifying the existing targets file. For a more detailed discussion of this issue, you should take a look at my June 2006 article (referenced previously).

Before creating the installer, let's have a quick look at the CoreBuild target that builds the solution (see Figure 13). This will demonstrate how target batching is used to build the solution for each defined configuration.

Figure 13 CoreBuild Target (Close Figure 13)
<Target Name="CoreBuild"
   Inputs="%(AllConfigurations.PlatformToBuild);
           %(AllConfigurations.FlavorToBuild)"
   Outputs="%(AllConfigurations.PlatformToBuild);
            %(AllConfigurations.FlavorToBuild)" >
   <Message Text="Building for Flavor/Platform:
       %(AllConfigurations.FlavorToBuild)/
       %(AllConfigurations.PlatformToBuild)"/>
   <MSBuild Projects="@(SolutionFile)"
      Targets="Build"
      Properties="OutputPath=$(PackageRoot)
         %(AllConfigurations.FlavorToBuild);
      Configuration=%(AllConfigurations.FlavorToBuild);
      Platform=%(AllConfigurations.PlatformToBuild)">
      <Output ItemName="OutputFiles" TaskParameter="TargetOutputs"/>
   </MSBuild>
</Target>

Specifically, the first thing to notice is the Inputs and Outputs; these are the metadata values for the AllConfigurations item that will cause the target to be executed once per platform/flavor pair. Then, the MSBuild task is called to build the solution file with all the appropriate properties. The OutputPath is provided because I want to redirect the output to another directory.

Putting It All Together

At this point, I have discussed how to create an installer using WiX, some advanced MSBuild concepts, and the structure of the MSBuild packing script. The only remaining issue with regard to creating the installer for the sample project is creating the MSBuild project file to define the required properties. This is surprisingly simple. Finally, I'll show you how to customize the process to suit your needs.

Figure 14 shows the MSBuild file, Sedodream.Package.dproj, that will drive the process for the sample project. As discussed earlier, the main purpose of this project file is to describe what needs to be built. The imported file, Sedodream.Package.targets, knows all the details about how to create the final product, and you can also customize the process by injecting steps into the build process.

Figure 14 Sedodream.Package.dproj MSBuild Project (Close Figure 14)
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
         DefaultTargets="Package">

   <!-- Required properties by the deployment.targets file -->
   <PropertyGroup>
      <SolutionFilePath>..\Sedodream.MSBuild.sln</SolutionFilePath>
      <PackageRoot>..\Package\</PackageRoot>
   </PropertyGroup>

   <ItemGroup>
      <WiXSourceFiles Include="Sedodream.wxs"/>
      <OtherFiles Include="..\Sedodream.MSBuild.Tasks\
         SampleTargets\**\*">
         <Destination>Samples\</Destination>
      </OtherFiles>
      <!-- Copy the license to the BaseSearchPath so it can
         be included into the installer -->
      <OtherFiles Include="License.rtf">
         <Destination></Destination>
      </OtherFiles>
      <!-- Copy the custom installer bitmaps -->
      <OtherFiles Include="bitmaps\dlgbmp.bmp">
         <Destination>bitmaps\</Destination>
      </OtherFiles>
   </ItemGroup>

   <!-- Define all the configurations that you want to build here -->
   <ItemGroup>
      <AllConfigurations Include="Debug|x86">
         <FlavorToBuild>Debug</FlavorToBuild>
         <PlatformToBuild>Any CPU</PlatformToBuild>
      </AllConfigurations>
      <AllConfigurations Include="Release|Any CPU">
         <FlavorToBuild>Release</FlavorToBuild>
         <PlatformToBuild>Any CPU</PlatformToBuild>
      </AllConfigurations>
   </ItemGroup>

   <PropertyGroup>
      <DropLocation>C:\Data\Drops\MSBuild-Wix\</DropLocation>
   </PropertyGroup>

   <!-- Import the deployment target to do all the work for us -->
   <Import Project="Sedodream.Package.targets"/>
</Project>

In this project file, you see the required SolutionFilePath property defined. This is the path to the solution file that will be used to build the product. Also, you see the declaration of the WixSourceFiles item, which includes the set of files that will be used to compile your installer. These are the files that will be sent to the Candle.exe tool for compiling.

Along with this is the declaration of the OtherFiles item, which contains files that should be copied to the location that will be used to create the installer. These files will be copied to the BaseInputPath that will be used to create the installer. You can have them copied to any location underneath that by using the Destination metadata value. In this example, I am copying custom bitmaps to be used to generate the installer into the bitmaps folder. When the Light.exe tool is invoked to create the MSI file, it will pick up files that are in this directory before any default files that are located in the local directory.

Note the declaration of the DropLocation property. By declaring this property, all the files in the PackageRoot will be copied to a directory in the DropLocation. It's helpful to have the binaries and installer files in one location, in case you need to make some quick modifications.

To see the build and package executed for the sample project, open a Visual Studio 2005 command prompt and navigate to the projects deployment folder. Then execute the following command:

msbuild SedodreamPackage.dproj

Lots of output is generated to the command prompt, and you will find two directories, Debug and Release, under the Package directory. Each of these directories will contain the generated installer and its associated files.

Customizing the Process

When you set up this process for your product, you will need to create a file similar to the SedodreamPackage.dproj file. You can place your customizations directly inside that file. It shouldn't be necessary to change the Sedodream.Package.targets file itself. If you want to inject steps, you can follow the process described previously. If you want to redefine targets, you can simply override them by redeclaring the target after the import statement. For example, the Sedodream.Package.targets file defines a target, GetLatest, but it is empty. In this target you should place the necessary tasks to get the latest sources from your repository. Here is an outline of what it might look like in your project:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003"
       DefaultTargets="Package">
   <!-- Your properties and items defined here -->

   <!-- Import the deployment target to do all the work for us -->
   <Import Project="Sedodream.Package.targets"/>

   <!-- Place overriding customizations after this point -->

   <Target Name="GetLatest">
      <Message Text="Getting latest sources"/>
      <!-- Insert tasks to get latest from source control -->
   </Target>
</Project>

Figure 15 provides a list of important targets that are defined in the Sedodream.Package.targets file. You can customize these targets to suit your needs.

The remaining outstanding topic is how to automate this process, which is now extremely simple. All you have to do is automate the execution of MSBuild.exe on the deployment file, in this case SedodreamPackage.dproj. How this is accomplished will depend on how you are currently generating public builds. If you are using Visual Studio Team Foundation Server, you can use Team Build to perform this for you. You can use the MSBuild task in your TFSBuild.proj file. If you are using some other technologies, it is likely that they have native support for MSBuild. If you are not employing any of these types of technologies, you can use the Windows Scheduler to execute this on a timed basis.

Conclusion

In this article, I've shown how to create an automated build and package process based on WiX toolset. Once you have completed this integration you can build and package your products in a reliable and repeatable manner. This is very important for applications that are being sent into deployment.

If WiX is not your installer technology, you can still use the ideas and files, provided here with some modifications. All installer technologies that I know of support some form of a command-line execution. You would employ this in order to create your installers using these other technologies. Some third-party installer technologies have also shipped with MSBuild tasks and targets, which may assist you in recreating this process.

MSBuild and WiX Resources
  • Inside MSBuild: Compile Apps Your Way with Custom Tasks for the Microsoft Build Engine
  • Using the WiX Toolset to Integrate Setup into Your Development Process
  • WiX Project Home
  • Windows Installer SDK
  • rRob Mensching’s Blog (WiX Project Lead)
  • Justin Rockwood’s Blog (WiX Developer)
  • Sayed Hashimi’s Blog
  • MSDN MSBuild Forum
  • MSBuild Team Blog
  • MSBuild MSDN Documentation
  • MSBuild MSDN Reference
  • MSBuild WiKi

Sayed Ibrahim Hashimi has a computer engineering degree from the University of Florida. He is a developer and architect in Jacksonville, Florida. He is an expert in the financial, education, and collection industries. He works with .NET technologies at Latitude Software.


ASP.NET 2.0: Manage Web Users With Custom Profile Providers

ASP.NET 2.0

Manage Web Users With Custom Profile Providers

Jason N. Gaylord
This article discusses:
  • ASP.NET 2.0 profile providers
  • Building a custom provider
  • Authentication and authorization
This article uses the following technologies:
ASP.NET, XML
Code download available at: ProfileProviders2007_03.exe (268KB)
Contents
Using a Custom Profile Provider
Customizing the CreateUserWizard Control
Creating a Registration Procedure
Checking the Profile Version
Stepping through the Application
Where to Go from Here

W
ith ASP.NET 2.0, you can add authentication, authorization, and profiles to your Web site without writing a single line of code. That's quite a step forward from the way things used to be. Like most of the new features in ASP.NET 2.0, authentication, authorization, and profiles each have their own built-in providers. You can think of providers as modules that contain the methods for a particular task. These providers are quite flexible-they can be customized by specifying attributes that pass information into the provider to allow it to execute different behavior. For extreme customization, though, it may be necessary to replace a built-in version with a custom or third-party provider.

The Profile feature in ASP.NET allows developers to specify per-user settings or data. All this data can be stored in an anonymous profile so that settings can be retrieved without requiring the user to log into the site. However, if the user creates an account, these settings can be migrated to the logged in profile. The anonymous profile feature can also be turned off.

To store and retrieve a user's profile properties, simply name each profile property and add them to the profile element in the web.config file, as shown in Figure 1. This puts the profile properties into the profile's SettingsPropertyValueCollection. You may also specify the data type and a default value for each property.

Figure 1 Profile Section in web.config (Close Figure 1)
<profile>
  <properties>
    <add name="FirstName" />
    <add name="LastName" />
    <add name="Address1" />
    <add name="Address2" />
    <add name="City" />
    <add name="State" />
    <add name="Zip" />
    <add name="Phone" />
    <add name="ProfileVersion" type="int" defaultValue="0" />
  </properties>
</profile>

ASP.NET reads the web.config file in search of the profile section group. If one is found, ASP.NET then looks for a type or connection string attributes (not included in Figure 1). If neither is found, the machine.config file is examined. In most scenarios, the machine.config file would not have been altered since ASP.NET was installed. By default, the machine.config file has the System.Web.Profile.SqlProfileProvider registered as the default provider for the profile feature. The SqlProfileProvider is defined to use the LocalSqlServer connection string name to connect to the target database for profile information (all connection string names and attributes are stored in a group called connectionStrings, found in the web.config file). Again by default, the LocalSqlServer connection string is defined as a SQL Server™ 2005 Express database that can be found at |DataDirectory|aspnetdb.mdf (translated in ASP.NET as \App_Data\aspnetdb.mdb).

The database used by the SqlProfileProvider to store profile information must be configured using a known schema. If a SQL Server 2005 Express database is being used, ASP.NET can dynamically create the database file and create the necessary tables, views, stored procedures, and so forth on demand; otherwise, the aspnet_regsql tool from the Microsoft® .NET Framework SDK can be used. One of the tables, the aspnet_Profile table, is used for storing the profile information for each user. When a user assigns a profile property to his profile, a new row is added that contains the user's unique UserID, the names of each of the profile properties in a single column, and the data that relates to each of the profile names in another column. In Figure 2, you can see that the data is stored as a BLOB. The property names and values can be in a variety of data formats including a continuous string, an XML dataset, or a binary serialization of the data. As a result, storing profile data using the built-in profile provider can make it difficult for a developer or database administrator to query or modify the data using SQL. If the default provider's functionality doesn't completely meet your needs, a custom provider can be used.

Figure 2 Property Names and Values
Figure 2 Property Names and Values(Click the image for a smaller view)
Figure 2 Property Names and Values
Figure 2 Property Names and Values (Click the image for a larger view)

Using a Custom Profile Provider

Even while developing a new version of ASP.NET, the ASP.NET team is hard at work enhancing the existing versions. The useful resources they produce are called sandbox projects and can be downloaded from the official ASP.NET Web site at sandbox.asp.net. One of the cool things about many of these projects is that they can be used in production environments. Some of the best add-ons for ASP.NET can be found here, including the Web Deployment Project and the Table Profile Provider Samples. In this project, I'll use the Table Profile Provider Samples, which you'll find at asp.net/sandbox/samp_profiles.aspx?tabid=62.

Unlike the built-in ASP.NET Profile provider, the Table Profile Provider Samples allow profile data to be stored in an easily readable format. The download contains two sample providers and instructions. The SqlTableProfileProvider allows data to be stored in individual columns within a profile table. The table must include the UserID unique identifier as the primary key and the LastUpdatedDate date time field. The UserID field will allow each profile record to be associated with the User account contained in the aspnet_Membership and aspnet_Users tables. You'll notice that neither of these fields need to be specified in the web.config file. The other profile sample, the SqlStoredProcedureProfileProvider, passes each profile property value into a separate parameter in a SQL stored procedure. This option will allow a stored procedure to store the profile properties in a location up to the stored procedure and to use business logic to transform the data before storing it.

Before using these samples in an ASP.NET project, you must copy the code for them to the App_Code folder. The code is written in C#, so if the application is being developed in Visual Basic®, the samples will need to be in a separate folder under the App_Code folder. The folder must be added as a code subdirectory in the web.config file as shown here:

<compilation debug="true" strict="false" explicit="true">
  <codeSubDirectories>
    <add directoryName="ProfileProvider" />
  </codeSubDirectories>
</compilation>

By adding the folder as a code subdirectory, all C# files in the folder will be compiled separately from the rest of the Web application project. At run time, two DLLs will be generated in the ASP.NET temporary folder for the application.

For this application to be used with the SqlTableProfileProvider, I'll create a new table called CustomProfile that will contain columns to store the user's first and last name, full address, and phone number, along with a version number used to determine the current version of the user's profile. The table will also include the user's unique identifier as its primary key and the last updated date in the last column. For example, Figure 3 shows how the provider will be added to the profile section. Also note how each property will be added to the properties section group. In addition to a .NET type and default value, a custom provider data attribute will also need to be specified on each property. This attribute will include the column name from the SQL table and the SQL type of the column.

Figure 3 Profile Section (Close Figure 3)
<profile defaultProvider="MyCustomProfileProvider">
  <providers>
    <add name="MyCustomProfileProvider"
        type="Microsoft.SqlTableProfileProvider"
        connectionStringName="LocalSqlServer"
        table="CustomProfile"
        applicationName="/"
        profileVersion="1"
        />
  </providers>
  <properties>
    <add name="FirstName" type="string" defaultValue="[null]"
        customProviderData="FirstName;nvarchar" />
    <add name="LastName" type="string" defaultValue="[null]"
        customProviderData="LastName;nvarchar" />
    <add name="Address1" type="string" defaultValue="[null]"
        customProviderData="Address1;nvarchar" />
    <add name="Address2" type="string" defaultValue="[null]"
        customProviderData="Address2;nvarchar" />
    <add name="City" type="string" defaultValue="[null]"
        customProviderData="City;nvarchar" />
    <add name="State" type="string" defaultValue="[null]"
        customProviderData="State;nvarchar" />
    <add name="Zip" type="string" defaultValue="[null]"
        customProviderData="Zip;nvarchar" />
    <add name="Phone" type="string" defaultValue="[null]"
        customProviderData="Phone;nvarchar" />
    <add name="ProfileVersion" type="int" defaultValue="[null]"
        customProviderData="ProfileVersion;int" />
  </properties>
</profile>

Customizing the CreateUserWizard Control

Not only does ASP.NET 2.0 have built-in providers to support its authentication features, it also has built-in Web controls for dealing with authentication and authorization. These include Login, LoginView, PasswordRecovery, LoginStatus, LoginName, CreateUserWizard, and ChangePassword. The CreateUserWizard control allows users to create an account within an ASP.NET application. Much like the providers, the controls require little if any coding. To use the CreateUserWizard control, or any of these controls for that matter, simply drag it from the Toolbox onto any ASP.NET page or user control.

CreateUserWizard is a two-step wizard control. The first step, called "Sign Up for Your New Account," asks the user for information such as username, password, e-mail address, and a security question and answer. In some instances, the Membership provider defined in the web.config may be configured where the requiresQuestionAndAnswer property is set to false. In such cases, the security question and security answer fields are hidden at run time. The second step, called "Complete," confirms that the user account was created and displays a Finish button. Clicking the button redirects the user to the URL specified in the ContinueDestinationPageURL property or runs an event handler for the button's click event if one was provided. Usually, a developer will redirect the user to the homepage or to the referring page using the ContinueDestinationPageURL property. For more detailed information on the CreateUserWizard control, see msdn.microsoft.com/msdnmag/issues/05/10/ExtremeASPNET and msdn.microsoft.com/msdnmag/issues/04/11/CuttingEdge.

Creating a Registration Procedure

There are a few ways to create a custom user registration process. First, you can use the built-in CreateUserWizard control and modify it with an additional action in front of the Sign Up for Your New Account step. Second, you can convert each step to a navigation template and customize each control step as you see fit. This lets you move the user information and profile information to different steps or to leave all of the fields in the same step. Finally, you can create your own registration control using the Membership API's CreateUser method. The solution you choose depends on the design of the application and the level of customization you'd like to add. In this case, you'll add a step in front of Sign Up for Your New Account as this will let you add your profile form with minimal coding.

The new step will ask the user to submit values for each of the profile properties mentioned earlier, with the exception of the ProfileVersion property. As a reminder, the UserID and LastUpdatedDate will be populated by the provider and don't need values supplied by the user. The ProfileVersion property will be populated when the form is submitted by retrieving the value from an application setting or from another section in web.config. You can find out more about creating your own configuration sections at msdn.microsoft.com/msdnmag/issues/06/06/ConfigureThis.

Figure 4 shows the property being added to the SqlTableProfileProvider that has been provided by Microsoft. We will use the Profile sectionGroup to specify the value of the profile version. The Profile will be updated immediately after the user's account has been created. An object defined as the ProfileCommon will load the user's current profile properties and values. Since the profile and user account are new to the system, all of the values will be null. All of the profile properties will be updated to contain non-null values. Finally, the ProfileCommon will be saved so the data can be written back to the database using the provider. If the ProfileCommon isn't saved, there is no way for ASP.NET to know that a data write is needed.

Figure 4 SqlTableProfileProvider Class (Close Figure 4)
public class SqlTableProfileProvider : ProfileProvider
{
    ...
    private string _profileVersion;

    public override void Initialize(
        string name, NameValueCollection config)
    {
        ...

        // Check to see that a profile version exists
        _profileVersion = config["profileVersion"];
        if (string.IsNullOrEmpty(_profileVersion))
        {
            throw new ProviderException("No profile version specified");
        }

        // Validate the profile version
        int testProfileVersion;
        if (!Int32.TryParse(_profileVersion, out testProfileVersion))
        {
            throw new ProviderException(
                "Profile version must be an Integer");
        }

        ...

        config.Remove("profileVersion");
    }

    // Public property to obtain the ProfileVersion from the web.config
    static public Int32 ProfileVersion
    {
        get
        {
            ProfileSection profileConfig = (ProfileSection)
                WebConfigurationManager.GetSection("system.web/profile");
            return Int32.Parse(profileConfig.Providers[1].
                ElementInformation.Properties["profileVersion"].
                    Value.ToString());
        }
    }

Checking the Profile Version

One of the benefits of using ASP.NET is that it's modular. Since ASP.NET 1.0, developers could create an HTTP handler or module for ASP.NET. You can write an HTTPModule to see if a profile exists for a user and if it is the correct version. If a profile does not exist or if it's the wrong version, you could force the user to go to a centralized update-profile page.

To determine the profile version that a user is currently registered with, you need to handle the application's AuthorizeRequest method. In doing so, your own custom method will run when the user is authenticated and before the user is granted access to the page he wants (see Figure 5). Since you won't be able to pull the profile on users not logged in, you'll need to exclude those requests from your examination. You'll also want to exclude users who are currently on your editprofile.aspx page so they are not caught in a cycle of continuous redirection. You'll then obtain the value for the profileVersion property on the second registered profile provider, which will be the SqlTableProfileProvider. You don't want to pull the first provider in the provider collection because that will always be the built-in profile provider unless you explicitly removed it from the providers collection. Once you've pulled the current profile version on the application, load the user account's profile and check the profile version in the profile. To do this, create a new ProfileCommon object and load the user's profile based on his name. If the user's profile version is less than the application's profile version, the user will be redirected to the editprofile.aspx page. If it is the same, the page the user was attempting to access will load.

Figure 5 ProfileHttpModule Class (Close Figure 5)
Public Class ProfileHttpModule
    Implements IHttpModule

    Public Sub Dispose() Implements System.Web.IHttpModule.Dispose
    End Sub

    Public Sub Init(ByVal context As System.Web.HttpApplication) _
            Implements System.Web.IHttpModule.Init
        AddHandler context.AuthorizeRequest, AddressOf Me.CheckProfile
    End Sub

    Public Sub CheckProfile(ByVal s As Object, ByVal e As EventArgs)
        Dim objApp As HttpApplication = CType(s, HttpApplication)
        Dim objContext As HttpContext = objApp.Context

        'User is authenticated, check profile
        If objApp.User.Identity.IsAuthenticated And _
                objContext.Request.Path <> "editprofile.aspx" Then
            Dim ProfileVersion As Integer = _
                SqlTableProfileProvider.ProfileVersion

            Dim UserProfile As New ProfileCommon
            If CInt(UserProfile.GetProfile(objApp.User.Identity.Name). _
                ProfileVersion.ToString) < ProfileVersion And _
                !objContext.Request.Path.EndsWith("editprofile.aspx") _
                Then
                    objContext.Response.Redirect("~/editprofile.aspx")
            End If
        End If
    End Sub
End Class

Before you can build the application, make sure that your web.config file is configured for Forms authentication. If you don't, the application will assume that you are using Windows® authentication as defined in the machine.config file. The web.config file will also need a location section group to deny any users that are not authenticated access to the edit profile page. You must also add a login control so that your users can log into the site. You can add the control to your main page or to a separate login page. It may be a good idea to add a register link to the login control so that your users can find the registration page.

Stepping through the Application

To review, when a user visits the site and clicks on the link asking him to create a user account, he'll be redirected to the registration page and asked to fill in his profile information. When he clicks Next, he'll be asked to enter a username and password and will then be redirected to the default page. During each page load, the HTTPModule will check to see if the profile version specified in the web.config file is greater than the user's profile version. If it is, the user will be redirected to the edit profile page. If not, he'll continue browsing.

To ensure that the module is working properly, add a column to the custom profile table. The column must allow null values or you'll receive an error message when a user attempts to browse a page. This is because that property is not established yet for all users and has not been added to the web.config file. Add it and change the profile from version 1 to 2. When you save the web.config file and log onto the site, you can't browse to any page without updating the profile first, as expected.

The one catch with using the web.config file to store the profile version is that if you modify it when users are on the Web site, the application will reset and the users will be kicked off. Then they'll be redirected to the edit profile page on their next login attempt. With this in mind, you may want to create a simple text file or XML file to read the version number of the profile.

Where to Go from Here

If you need some help getting started, you can download a Visual Studio® Installer (VSI) project that I've put together. It includes full source code written in Visual Basic and ASP.NET and provides a great foundation for beginning your application. It can be downloaded from the MSDN®Magazine Web site. I've explored only a very small portion of the customization possibilities with the ASP.NET 2.0 Profile provider; I'm sure you'll discover lots more if you look. That's what's great about .NET-it's like having your own personal sandbox.

Jason N. Gaylord is the Project Manager and Senior Application Developer at a leading risk management information services company in Wilkes-Barre, Pennsylvania. He also runs the .NET Valley User Group. For more information, visit www.jasongaylord.com or contact him at jason@jasongaylord.com.


Editor's Note: Bring On the Swag

Editor's Note
Bring On the Swag
Joshua Trupin

Just because we're heads-down here at our editorial offices, laboring to bring you the finest in content for Windows-based development, doesn't mean that we don't pay attention to news, gossip, and rumors.

Suffice it to say that a recent story caught our interest. A few commentators were incensed over stories that a certain large software company based in the Pacific Northwest supplied laptops to some bloggers so they could evaluate Windows Vista. This was evidently the biggest transgression of all time and called into question the journalistic ethics of the entire blogosphere.

Supplying product to reviewers is a time-honored tradition in the world of publishing. How do you think books get reviewed? And how do you think magazines rate new MP3 players? They don't go out and buy one of each. You think we're going to spend our hard-earned money on 10 different version-control programs just to see which one has the most features?

We'd like to state for the record that our staff never gets anything good to review. We'd love to get nice new laptops with all the trimmings, but many of us have had to struggle by tirelessly on models that are almost six months old ("oh, the humanity"). Would it help if we made a list of products we may be "reviewing" over the next six months? A plasma TV would be nice. A big one, not one of those puny 42" jobs. We'd also consider reviewing 30 Zunes. Wait, make that 32. And maybe a pony.

The best part about the whole product review flap is the idea that bloggers need to live up to the ethics of professional journalists. Have you ever seen a professional journalist at a conference? They're the ones shoving giveaway pens into one pocket and complementary shrimp into the other. Present company excluded of course-we prefer bouncy balls to pens.

As blogging increases exponentially and becomes part of the culture at many companies, it has also changed the way information gets disseminated to customers-perhaps as much as the Internet itself did when everyday citizens and companies started to make greater use of it over a decade ago. This growth has been, as a whole, a great thing for everyone. Microsoft employees discuss the products they're working on, and loyal readers come away with better insight into the technology they use in their day-to-day work. MSDN Magazine has a regular blog now, which you can read at blogs.msdn.com/msdnmagazine. None of this takes the place of the in-depth coverage our magazine brings you, but blogs can be used to fill in gaps in your knowledge quite well.

Of course, as with all advances in communications, there are some abuses: blog spam and flogs. Spammers and frauds continually find ways to crack in and dump their crud onto us. You know the blog comments we're talking about. "Hey, this is a great blog about crocheting! Come visit mine at musclemaxsupplement.contoso.com!" So you click on the link, hoping to learn more about the needle arts, and the destination has almost nothing to do with crocheting at all.

Flogs, or fake blogs, are the other danger zone. Companies, or PR agencies working for them, will set up a blog that purports to be a grass-roots discussion of a great new technology. You're reading about how exciting the new color of crocheting yarn is, and all the ways that neon-green handmade ponchos will be in vogue this year, but you don't know that it was really written by the company that's marketing the neon-green acrylic. Anonymous comments on "real" blogs are occasionally created like this too. You can usually tell-the commenter has never said anything else, and all his messages are in defense of a particular technology, candidate, or company. But these gotchas are minor parts of the blog world, and you'll see the same tricks on the opinion pages of any newspaper.

Meanwhile, we'll just sit here and wait for all those Zunes. You've got the address, right?-J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Moustafa Ahmed, Bashar Badawi, Joe Duffy, Shawn Farkas, Matt Gibbs, Jayu Katti, Joe Klug, Mark Lawrence, Rob Mensching, Tom Mulcahy, Gruia Pitigoi-Aron, Stefan Schackow, Greg Schechter, Bob Schmidt, Tim Sneath, Aaron Stebner, and Robert Wlodarczyk.

Active Accessibility, Active Directory, ActiveX, Authenticode, BizTalk, DirectShow, IntelliSense, Internet Explorer, JScript, Microsoft, Microsoft Press, MSDN, Virtual Earth, Visio, Visual Basic, Visual Studio, Windows, Win32, Windows Media, Windows NT, Windows PowerShell, Windows Server, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners. MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. Msdn magazine disclaims any liability whatsoever for any use of such code or other information. The recommendations and technical guidelines in MSDN Magazine are based on specific environments and configurations. These recommendations or guidelines may not apply to dissimilar configurations. MSDN and Microsoft are registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries. MSDN Magazine, MSDN, and Microsoft logos are used by CMP under license from owner. MSDN Magazine is published by CMP Media LLC. CMP Media LLC is an independent company not affiliated with Microsoft Corporation. Microsoft Corporation is solely responsible for the editorial contents of this magazine.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: Share Your Desktop, Lightweight Code Editing, and More

Toolbox
Share Your Desktop, Lightweight Code Editing, and More
Scott Mitchell

Contents
Share Your Desktop
Lightweight Code Editing
Modeling with Graphs
The Bookshelf

Share Your Desktop

As an independent contractor working from home with clients scattered around the globe, being able to communicate quickly and effectively with my customers and coworkers is of paramount importance. Communication tools like e-mail and instant messaging are usually sufficient, but there are times when I need to show a client my desktop or view his desktop as I help troubleshoot a problem. Video conferencing is also invaluable when helping a coworker debug or when giving a demo or sales pitch to a prospective client.

The variety of technologies available for video conferencing and desktop sharing vary in cost, ease of setup and use, and features. Many of the free choices require opening specific firewall ports and installing software, steps that can be off-putting for less tech savvy coworkers and customers. Fortunately, there are a number of easy-to-use conferencing solutions that won't strain your wallet.

One such product is GoToMeeting from Citrix Online, which for $49 per month allows you to host an unlimited number of video conferences with up to 10 attendees per meeting. (Check out GoToMeeting Corporate if you need to host up to 25 attendees or GoToWebinar to support hosting up to 1,000 attendees.) To start a meeting, simply double-click the GoToMeeting icon in the taskbar. This initiates a meeting and creates a URL you can send to the other participants to invite them to your desktop. To join your meeting, invitees simply paste the URL into their browser's address bar—no need to install software or tinker with firewall settings.

Once they arrive, all participants can view your desktop and watch as you interact with your system. Additionally, you can give control of your desktop to any of the attendees. GoToMeeting is much more than just a simple desktop sharing program, though. With the click of a button you can switch presenters and have a selected attendee's desktop becoming the one all others see.

GoToMeeting also includes free audio conferencing. Participants simply call a phone number to join the conference call and, once connected, can listen and talk to all those who've joined the conference.

In many situations, being able to show your desktop to a remote client or coworker is a great timesaver. If you telecommute or regularly work with remote clients or coworkers, investing in a video conferencing service like GoToMeeting can save you both time and money.

Price: $49 per month for unlimited meetings with up to 10 attendees.

www.GoToMeeting.com

Lightweight Code Editing

While the rich design-time support in Visual Studio® makes it an ideal choice for developing UI-centric applications, its array of features and functionality often makes Visual Studio overkill for code-only applications, such as command-line programs, class libraries, and Web services. This is especially true for developers working on projects across multiple platforms, since having to master Visual Studio for Windows®-based development and some other code editor for another environment can be time consuming. SlickEdit version 11, by SlickEdit Inc., offers a happy medium between the no-frills user experience of a text editor and the complex feature set found in Visual Studio.

(Click the image for a smaller view)

SlickEdit can be used as a simple text editor, editing one file at a time, or you can categorize files into Projects and Workspaces, like the Projects and Solutions in Visual Studio. Workspaces can be created manually or imported from a Visual Studio project or solution file. The code editing user experience is quite similar to that of Visual Studio: the code is color-coded and auto-indented; the Auto-Completion feature mimics the IntelliSense® found in Visual Studio; and SlickEdit can be configured to support the same keyboard commands and shortcuts as Visual Studio (as well as those of a number of other popular editors). SlickEdit also offers a variety of refactoring options and pre-built code templates for quickly implementing common design patterns.

SlickEdit also contains a variety of features not found in Visual Studio. There's a built-in regular expression tester and menu options for merging files and highlighting differences in files. A highly configurable "Beautify" option lets you specify formatting rules—indentation, the style for the beginning and end of code blocks, comment style, and so on—and then, at the click of a button, all of the files can be beautified.

But what's most refreshing about SlickEdit is what isn't there. You won't find database editing tools or designers for building GUIs. By staying focused on its core task—code editing—SlickEdit offers a responsive and lightweight code editing environment without skimping on essential features.

SlickEdit is an ideal choice for developers or development teams that work in multiple environments. SlickEdit provides support for several OSs, including Windows, Linux, Mac OS X, Solaris, AIX, and other UNIX variants.

Price: $284 per single-user license.

www.slickedit.com

Modeling with Graphs

Graphs and flowcharts are excellent user interface elements for graphically depicting, creating, and modifying relationships among entities, for depicting workflow processes or networks, and for showing XML or other hierarchical data.

While adding such rich modeling interfaces to your own applications may seem daunting, it's actually a fairly straightforward process when using a third-party control designed to provide such functionality. One such control is FlowChart.NET version 4.2 from MindFusion, which can generate flowcharts, lane diagrams, hierarchies, and other kinds of classification graphs.

Figure flowchartnet Add Professional Graphs and Charts with FlowChart.NET
Figure flowchartnet Add Professional Graphs and Charts with FlowChart.NET(Click the image for a smaller view)
Figure flowchartnet Add Professional Graphs and Charts with FlowChart.NET
Figure flowchartnet Add Professional Graphs and Charts with FlowChart.NET (Click the image for a larger view)

To create one of these graphs, simply drag the FlowChart control from the Toolbox onto your form. A variety of objects can be programmatically added to the FlowChart, including boxes, tables, arrows, and any built-in or custom .NET controls.

End users can manipulate the flowchart, performing such actions as adding new elements, resizing and repositioning them, selecting items, and modifying them. The FlowChart control includes a number of events that fire as the user manipulates the graph in various ways, making it easy to detect and to respond to a user's interactions as needed. The flowcharts can be saved to a stream or a binary, text, or XML file, and loaded back with just a couple of lines of code. The Pro edition includes support for exporting the flowcharts to PDF, Visio®, and SVG formats.

The FlowChart.NET download includes a series of demos that showcase some of the possibilities of this component—including using graphs and flowcharts to build a Web site map, manipulating database schemas, and viewing class diagrams loaded from an assembly. The UIs for these demos are very powerful, allowing the end user to graphically construct the site map, database schema, or view class structure, and virtually all of this functionality is built directly into the component. There is very little code backing each demo, and it is terse and easy to follow.

Price: $250 for a single-user license for the standard version; $500 for the professional edition.

www.mindfusion.org

The Bookshelf

While .NET programs can be created with any .NET programming language, over the last few years C# has become the language of choice for a number of developers. Part of what has fueled C#'s popularity is its straightforward semantics and ease of use.

One book I found particularly helpful for mastering C# is Trey Nash's Accelerated C# 2005 (Apress, 2006). It is geared toward intermediate developers with past experience in an object-oriented programming language, with the goal of helping them become C# experts.

The first half of the book focuses on C# syntax and semantics, examining core concepts like types, methods, events, object initialization and destruction, interfaces, overloading, inheritance, and so on. It provides a solid overview of the essentials of the C# programming language and will be invaluable to anyone making the switch to C#. Even those who already use C# on a daily basis are certain to learn more than a thing or two from these first chapters.

The second half of the book looks at implementing common design patterns in C#. There are chapters on how best to handle exceptions, how to efficiently work with arrays and collections, using delegates and events, and creating multithreaded applications. There is plenty of content on new C# 2005 features as well, including generics, iterators, and anonymous functions.

Slimmer than most programming texts at 401 pages, this book will appeal to readers who prefer concise writing. There are few sidebars or added examples—just distilled information with no wasted words.

Price: $39.99.

www.apress.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.

All prices were confirmed at press time and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


CLR Inside Out: .NET Application Extensibility, Part 2

CLR Inside Out
.NET Application Extensibility, Part 2
Jack Gudenkauf and Jesse Kaplan

Code download available at: AddIns2007_03.exe (177KB)
Contents
Pipeline Architecture
Pipeline Components
Pipeline Implementation
Changing the Host and Its Views
Host and Add-In Compatibility
Summing Up

In our first column, where we introduced the new System.AddIn namespace in the Base Class Libraries (BCLs), we walked through the typical process customers undertake in order to provide extensibility through their host applications. We discussed the functionality needed to offer a comprehensive add-in solution. We also looked at some sample host code, as well as the new System.AddIn APIs used by the host. And, finally, we showed some sample add-in code that clearly demonstrated how the add-in developer can remain blissfully ignorant of the underlying mechanics of consuming the object model provided by the host.

We postponed the discussion of the System.AddIn pipeline and we promised to show the utilization of the model to enable compatibility as your host and add-in version independently. This time we'll first go into detail about the architecture of the model. Then we'll look at version 2.0 of the application we reviewed last month, using it to get an in-depth view of how our pipeline works and to demonstrate how you can use our system to keep add-ins working even as the host changes.

Before you read on, it may be helpful to look at the "Terminology" sidebar. Note also that the System.AddIn namespace and associated classes will be in the version of Visual Studio® expected to follow Visual Studio 2005, code-named "Orcas." You may download the new libraries prior to the full release of Visual Studio "Orcas" via the Community Technology Preview. You can find additional information at the CLR Add-in Team's blog (blogs.msdn.com/clraddins).

Terminology

Here are some terms that are used throughout this column.

Add-ins (also known as add ins, add ons, extensions, plug-ins, plug ins, snap-ins, and snap ins) are custom code, usually written by a third party that is dynamically loaded and activated by a host application, based upon some context (such as host startup or document loading).

Backward compatibility is the ability of the current version of an application or component to consume or use components targeted at an earlier version of the application or component. For example, the common language runtime (CLR) 2.0 is backward compatible with version 1.1 because it can run 1.1-targeted applications or components.

From a host and add-in perspective, a host application is said to be backward-compatible with its previous versions if add-ins written against those previous versions run without modification against the new version.

Forward compatibility represents the ability of the current version of an application or component to consume or use components targeted at a future version of the application or component. Forward compatibility is therefore mostly theoretical since it relies on the future. Full forward compatibility is only possible if nothing ever changes in the future version. For example, an application that was written against the CLR 2.0 features would be forward compatible if it also ran against the CLR 1.1, but this is not supported by the CLR.

From a host and add-in perspective, a host application is forward compatible if add-ins written against later versions of it can run unmodified on the previous version.

Pipelines are the communication machinery that enables the host and add-in to communicate over a version-resilient protocol.

Pipeline Architecture

Figure 1 shows pipeline components that enable a host to communicate with an add-in. The pipeline components are used in activating and communicating with the add-in.

Each component in the diagram is actually an assembly with many different types: for every type of complex object exchanged between the host and the add-in there are corresponding view, contract, and adapter types on both sides of the boundary. It is important to note that the diagram is completely symmetrical and is mirrored across the isolation boundary, with each side following the model shown in Figure 2. The host and add-in are entities that provide the real implementation and direct value to the user. One of the high-level goals of this model, and a primary driver of these design patterns, is that they can communicate and interact with each other even as one or both sides change.

Figure 1 Communication Pipeline between Host and Add-in
Figure 1 Communication Pipeline between Host and Add-in(Click the image for a smaller view)
Figure 1 Communication Pipeline between Host and Add-in
Figure 1 Communication Pipeline between Host and Add-in (Click the image for a larger view)

Figure 1 illustrates a number of important facts. First, the host and add-in are segregated from each other via an isolation boundary. Each one only references its view and thus gains a layer of abstraction over the contracts used for communication. Furthermore, there is no direct reference to the host/add-in itself, thus allowing relative isolation from all other components communicating with it.

Figure 2 Host/Add-In Model
Figure 2 Host/Add-In Model(Click the image for a smaller view)
Figure 2 Host/Add-In Model
Figure 2 Host/Add-In Model (Click the image for a larger view)

Pipeline Components

Contracts The contract assembly contains the non-versioning types (contracts) that are loaded in both sides of the isolation boundary as the communication mechanism over that boundary. By restricting to contracts all communication over the boundary, we prevent the host's and add-in's implementation types from leaking across the boundary and causing versioning issues. In our system, all contracts inherit from the base IContract which serves as the "object" type. Note that the contract assembly is not allowed to change, but that hosts and add-ins can use different contracts for communication without either entity being aware of it. If you are familiar with COM, you'll notice in the following code that IContract has similar functionality to IUnknown (QueryInterface, AddRef, and Release):

public interface IContract
{
    int AcquireLifetimeToken();
    int GetRemoteHashCode();
    IContract QueryContract(string contractIdentifier);
    bool RemoteEquals(IContract contract);
    string RemoteToString();
     void RevokeLifetimeToken(int token);
}

Contracts must inherit from IContract and must be interfaces; thus they do not allow a specific implementation. They provide a separation of integration and implementation types. Contract signatures are constrained to serializable types defined in mscorlib (such as Int32, String, and DateTime), other contracts, and enumerations defined in the contract assembly. Contract interfaces also reconcile with the contract model of Windows® Communication Foundation, to ensure that there is no implementation in these types and to allow multiple inheritance.

Views Views are assemblies containing abstract classes that represent the host's and add-in's view of each other and the types that flow between them. The purpose of views is to give the host and add-ins a stable view of the world that is not dependent on the contracts used to communicate between them. Views also separate the add-in and host from each other's implementation on either side of the boundary. The diagram shows that the host/add-in only depend on their view and that the view itself has no dependencies; this allows the adapters and contracts to be swapped out without impacting the host or add-in.

Adapters Adapters assemblies contain adapter classes that are used to convert to and from views and the contracts. We use the term "side" loosely in our diagram in order to discuss the side of the boundary an adapter resides. The reality is that depending on the direction of a call, the adapter is either converting from a view to a contract or a contract to a view. Therefore, if you have calls going in both directions, that is, the host calling into the add-in and the add-in calling into the host, you will have two adapters on each "side" of the boundary.

A class in the adapter assembly that converts a view to a contract is a view-to-contract adapter. This class implements the contract by calling into the view passed into its constructor and is marshaled across the boundary as a contract. The contract-to-view-adapter is a class in the adapter assembly that converts a contract to a view. This class inherits from the view and implements its abstract methods by calling into the contract passed into its constructor.

The diagram shows that the adapter assemblies reference both the view and the contract and thus provides the connection between the two. The adapters provide the glue that lets you keep hosts and add-ins connected and working even as one or both sides change dramatically. We'll have more details on this later.

Pipeline Implementation

Now let's take a look at the sample code and see how the implementation plays out in practice. We'll use version 2.0 of the sample application we discussed last time and explore how the host changed and how the pipeline hooks the add-in and host together. Then we'll see how to use our model to make sure the add-ins we built last time against the first version of the host can be made to work unchanged against the new version.

The system is involved only in activating the add-in and passing it back to the host. Once that occurs, it gets out of the way and lets the pipeline handle the flow of types. Since the activation path is the first step in host/add-in interaction, let's look at it in detail.

Activation of an add-in occurs after the host has discovered the available add-ins and has chosen which one to activate (for details see Program.cs in the download or last month's column). In our sample, activation occurs with this simple code:

//Activate the selected AddInToken in a new AppDomain
Calculator calculator =
    calcToken.Activate<Calculator>(AddInSecurityLevel.Internet);

From the host's point of view, it simply calls activate and receives the add-in back to it typed as its view "Calculator". Behind the scenes, however, the system is starting up the add-in and hooking up the pipeline.

To do so, it first creates the AppDomain that the add-in is going to run in and then loads the add-in, add-in view, add-in side adapter, and contract assembly into the newly created domain. The add-in view is shown in Figure 3 as add-in base, which is simply an attribute the pipeline uses to designate an add-in view class that is used in activation. Next, the system instantiates the add-in with its default constructor and the add-in adapter and passes the add-in, typed as the add-in base, to the adapter's constructor. Then it passes the add-in adapter across the AppDomain boundary typed as its contract (in this case ICalculatorContract2), loads the host-side-adapter assembly, and instantiates an instance of the host adapter, passing the add-in adapter, typed as the contract, to its constructor. Finally, it returns the host-side-adapter to the host, typed as the host add-in view (in this case, Calculator).

Figure 3 Activation Path
Figure 3 Activation Path(Click the image for a smaller view)
Figure 3 Activation Path
Figure 3 Activation Path (Click the image for a larger view)

This seems like a lot of overhead, but in reality the performance cost of using this model to communicate across the boundary, versus communicating across the boundary directly over an interface, is minimal.

If you look at the add-in activation path you'll notice that it is really just an example of an object being passed from the add-in to the host. The only thing different about it is that the system itself instantiates the add-in and connects the components; in order for the system to identify these components and instantiate them correctly, we require that each component involved in activation have a custom-attribute applied in order to identify it. Typically, one type in each pipeline component is involved in activation and will have one of these attributes applied to it with the only exception being the host-view. The host-view is provided directly to the system via our APIs and thus doesn't need to be attributed.

In our sample, the add-in is providing itself as a service to the host, and the only data types passed between them are doubles and strings. Because of this, the sample only includes types involved in the activation pathway. Please see our team blog for examples of hosts and add-ins passing complex data in both directions.

Changing the Host and Its Views

The host application we discussed previously needed the add-ins to provide it a service: specifically it needed them to supply basic calculator functionality in the form of the Add, Subtract, Multiply, and Divide methods.

The original host's view of its add-ins looked like this:

public abstract class Calculator {
    public abstract double Add(double a, double b);
    public abstract double Subtract(double a, double b);
    public abstract double Multiply(double a, double b);
    public abstract double Divide(double a, double b);
}

In our new version, however, we've decided that the host needs more functionality from the add-ins and more flexibility in what calculator services they should provide. Thus, it's defined this new view of its add-ins:

public abstract class Calculator {
    public abstract string Operations { get; }
    public abstract double Operate(
        string operation, double a, double b);
}

Rather than relying on add-ins for four specific operations, the host now wants to know what operations the calculator supports and then have the ability to run those operations. This type of change is typical of hosts that rely on add-ins for a service: either they discover they need additional functionality from them over time, or they decide to make the extension point more flexible. With our model, the host will only program against this updated view and will rely on the pipeline to connect the legacy add-ins to this new view.

You can find the definition of the new view in CalculatorView2.cs in the HostCalculatorView project in the sample. For details on how the host utilizes this new view, see Program.cs in SampleExtensibleCalculator.csproj.

Our custom contract (ICalculatorContract2), is the type used to communicate across an isolation boundary. The original contract looked like this:

[AddInContract]
public interface ICalculatorContract : IContract
{
    double Add(double a, double b);
    double Subtract(double a, double b);
    double Multiply(double a, double b);
    double Divide(double a, double b);
}

Now that the host has changed its view of add-ins, it is moving to a different contract as well and now uses the following:

[AddInContract]
public interface ICalculatorContract2 : IContract
{
    string GetAvailableOperations();
    double Operate(String operation, double a, double b);
}

You can see that the host view and the changes made to it are typically mirrored in its contracts.

The add-in view typically mirrors the host view and it changes in the same way the contracts do. In this sample, the only thing differentiating the add-in view from the host view is the assembly it is contained in and the attribute marking the add-in Base (the type used in activation):

[AddInBase]
public abstract class Calculator2
{
    public abstract string Operations{ get; }
    public abstract double Operate(
    string operation, double a, double b);
}

Add-ins must inherit from the add-in base classes and apply an attribute to themselves identifying them for activation. You can see the sample code for the full implementation in the download, but the add-in's definition looks like this:

[AddIn("Basic V2 Add-in", Version="2.0.0.0")]
public class SampleV2AddIn : Calculator2
{
    public override string Operations
    {
        get { ... }
    }

    public override double Operate(
        string operation, double a, double b)
    {
        ...
    }
}

In our sample code, the add-in is providing a service to the host application and therefore the host needs to call into the add-in for functionality that extends the host. In this scenario, the add-in side adapter is providing the role of View To Contract Adapter.

The sample code shown in Figure 4 is also in CalculatorView2ToContract2AddInAdapter.cs and may be found in the AddInSideAdaptersV2.csproj project in the download. The AddInAdapterAttribute is an attribute used by System.AddIn.Hosting APIs to find and activate the add-in adapter class.

Figure 4 CalculatorViewToContractAddInAdapter (Close Figure 4)
[AddInAdapter]
public class CalculatorViewToContractAddInAdapter : ContractBase,
    ICalculatorContract2
{
    private Calculator_view;

    public CalculatorViewToContractAddInAdapter(
            Calculator2 calculator) {
        view = calculator;
    }

    public string GetAvailableOperations(){
        return _view.Operations;
    }

    public double Operate(string operation, double a, double b){
        return _view.Operate(operation, a, b);
    }
}

The add-in adapter derives from ContractBase, which is a base class provided by System.AddIn.Pipeline. This class derives from MarshalByRefObject, IContract, and ISponsor. We will discuss ISponsor and lifetime management in the host adapter section below. The add-in adapter also derives from ICalculatorContract which is the custom contract. This contract is used to proffer up the new add-in calculator's GetAvailableOperations and Operate methods. Both of these methods are implemented in version 2.0 of our add-in (SampleV2AddIn.cs). The add-in adapter constructor takes in the add-in (SampleV2AddIn) as the add-in view (CalculatorContractsBase.Calculator). It implements the required functionality of ICalculatorContract2 by calling into the add-in (typed as the add-in Base Calculator 2).

In our sample code, the host is consuming a service provided by the add-in and therefore the host needs to call into the add-in for functionality that extends the host. In this scenario, the host side adapter is providing the role of contract-to-view adapter. The host will call through the functionality provided by the add-in adapter. The code in Figure 5 (from CalculatorContract2ToView2HostAdapter.cs in the HostSideAdapters.csproj project) illustrates.

Figure 5 CalculatorContractToViewHostAdapter (Close Figure 5)
[HostAdapter]
public class CalculatorContractToViewHostAdapter : Calculator
{
    private ICalculatorContract2 _contract;

    private System.AddIn.Pipeline.LifetimeTokenHandle _handle;

    public CalculatorContractToViewHostAdapter(
            ICalculatorContract2 contract)
    {
        _contract = contract;
        _handle = new LifetimeTokenHandle(contract);
    }

    public override string Operations {
        get {  return _contract.GetAvailableOperations(); }
    }

    public override double Operate(string operation, double a, double b) {
        return _contract.Operate(operation, a, b);
    }
}

In this example, HostAdapterAttribute is used by the System.AddIn.Hosting APIs to find and activate the host adapter class (CalculatorContractToViewHostAdapter). The host adapter derives from the host view. As the label implies, the host view is the host's view of the add-in. The host adapter constructor takes in the add-in adapter typed as the custom contract (in this case ICalculatorContract2).

You'll notice that the host adapter code looks very similar to the add-in adapter code. The main difference is that it is adapting from a contract to a view, rather than from a view to a contract. The host adapter takes the contract into its constructor and inherits from the view while the add-in adapter does the opposite. The only other interesting piece of code shown here is the LifetimeTokenHandle, which we'll address below.

The host is calling through to the add-in for functionality, therefore it needs to hold a reference to the remote object, which resides in the add-in's AppDomain. As a result, the lifetime of the object needs to be determined by the host, not by the add-in. We provide a simple mechanism, built on remoting, which ensures the add-in-provided object in the add-in's AppDomain remains alive as long as the host holds a reference to the object. When the host releases this reference and the Garbage Collector (GC) collects the host-referenced object, the remote object lease will also expire. This mechanism of lifetime management hides the complexities of the underlying mechanics from the host and add-in. Both the add-in developer and host developer may program as though the objects (views) they consume and provide are local resources with the memory for these resources managed by the GC.

This reference mechanism works for the types that are implemented by either the host or the add-in. The same mechanism is also used in determining the lifetime of the add-in. When the host releases the reference to the add-in (it is an object after all), the add-in is unloaded. If the add-in is the last object running in the AppDomain that the host owns, the pipeline will then unload the AppDomain.

The following sample code in CalculatorContract2ToView2HostAdapter.cs shows you the incredible ease of implementing lifetime management:

private System.AddIn.Pipeline.LifetimeTokenHandle _handle;
_handle = new System.AddIn.Pipeline.LifetimeTokenHandle(contract);

The LifetimeTokenHandle class works with the implementer of IContract, in most cases the system-provided ContractBase class, to manage the lifetime of objects passed across the boundary.

When the LifetimeTokenHandle class is instantiated in the ContractToViewAdapter (which is the side receiving the contract), it acquires a LifetimeToken from the contract. The GC finalizes the object once it has determined that no more references to that adapter exist, and the GC then revokes the LifetimeToken. On the other side, in the ViewToContractAdapter (which is the side sending the contract), the adapter keeps track of all of the outstanding lifetime tokens; when the count drops to zero, the adapter allows itself to be garbage collected. Additionally, if this is the add-in's AppDomain and the host has no other references into the domain, it unloads the AppDomain.

Host and Add-In Compatibility

If the host (host view) or its data types change from one version to the next, in order to get the old add-ins working on the new host view, the host creates a second add-in side adapter, which inherits from the new contract and understands the new data types and converts them to the original add-in view. The host or pipeline developer will need to provide default functionality for new methods (see Figure 6).

You can find the sample code for such an adapter in AddInSideV1toV2Adapter.csproj in the download. For full details on how to implement this, please see the sample, but the important part is the definition of the class and its constructor.

Figure 6 New Host, Old Add-Ins for Backward Compatibility
Figure 6 New Host, Old Add-Ins for Backward Compatibility(Click the image for a smaller view)
Figure 6 New Host, Old Add-Ins for Backward Compatibility
Figure 6 New Host, Old Add-Ins for Backward Compatibility (Click the image for a larger view)

The key takeaway from Figure 7 is that to adapt the V1 add-ins to the V2 contract, you just need to build an add-in adapter that inherits from the V2 contracts (ICalculatorContracts2) and takes in the V1 add-in base in its constructor (Calculator).

Figure 7 CalculatorV1ViewToV2ContractAddInAdapter (Close Figure 7)
[AddInAdapter]
public class CalculatorV1ViewToV2ContractAddInAdapter : ContractBase,
    ICalculatorContract2
{
    CalculatorContractsBase.Calculator _view;

    public CalculatorV1ViewToV2ContractAddInAdapter(Calculator calc)
    {
        _view = calc;
    }

    public string GetAvailableOperations()
    {
        return  "+, -, *, /" ;
    }

    public double Operate(string operation, double a, double b)
    {
        switch (operation)
        {
            case "+": return _view.Add(a, b);
            case "-": return _view.Subtract(a, b);
            case "*": return _view.Multiply(a, b);
            case "/": return _view.Divide(a, b);
            default:
                throw new InvalidOperationException(
                    "This add-in does not support: " + operation);
        }
    }
}

The benefit of this approach is that old add-ins will continue to work even with the changes and that new add-ins can build and take advantage of the new object model. The application itself doesn't need to keep track of different versions of the add-ins and the host only deals with a single host view that the different add-in side adapters convert into the different add-in versions. This greatly simplifies the host's code base and lets a host drop support for legacy versions by simply not shipping an adapter for that version.

Summing Up

Our goal in designing this model was to enable dynamic composition of version-resilient, isolatable components. Our primary concern was to ensure the add-in developer experience would not be complicated by the mechanics involved in meeting this goal.

The System.AddIn model consists of the BCLs found in the System.AddIn namespace, as well as guidance on the design patterns you apply in communicating between components. This column discussed the pipeline architecture, an implementation of the pipeline, lifetime management, and utilization of the model to enable compatibility as your host and add-in version independently.

Send your questions and comments for James to clrinout@microsoft.com..

Jack Gudenkauf is a Software Architect on the CLR team at Microsoft. He is currently focused on engaging with ISV/VAR/SI customers to ascertain .NET architectural adoption issues. Jack has held positions including Development Manager, MIS Director, software engineering, and consulting services. Before he joined Microsoft, he ran his own consulting company, The Logical Choice.

Jesse Kaplan is the Program Manager for Extensibility and add-ins for the CLR team at Microsoft. His past responsibilities include compatibility and managed/native interoperability.


Test Run: Testing Custom Transform Streams

Test Run
Testing Custom Transform Streams
Dr. James McCaffrey

Code download available at: TestRun2007_03.exe (156KB)
Contents
A RotateStream Class
Testing the RotateStream Class
Two More Scenarios
Preserving Size
Conclusion

One of the first things that developers new to the Microsoft .NET Framework learn is how to read from and write to text files using methods from the FileStream class of the System.IO namespace. However, the .NET I/O stream model also provides developers with a powerful way to create custom stream classes. In this month's column, I'll explain key techniques you can use to test such custom stream classes-and specifically those that deal with transforming data read from or written to another stream.

To get a feel for where I'm headed in this column, take a look at the screenshot in Figure 1. The complete source code for the program that generated the output shown in the figure is available in the code download for this column.

A FileStream object is a programming abstraction that allows you to read from and write to a file easily without having to deal with the many low-level details that actually occur behind the scenes. The first few lines of the output in Figure 1 show a typical example of using the FileStream class to read data from a file:

FileStream fs = File.OpenRead("..\\..\\TextFile1.txt);
byte [] bytes = new byte[fs.Length];
fs.Read(bytes, 0, bytes.Length);
Console.WriteLine("TextFile1.txt contains: " +
    Encoding.UTF8.GetString(buffer));
Console.WriteLine("File in hexadecimal   : " +
    BitConverter.ToString(bytes));

The next few lines of output demonstrate the use of that same FileStream in conjunction with a custom stream I implemented:

fs.Position = 0;
Console.WriteLine("Filtering TextFile1.txt through " +
                   "custom RotateStream");
RotateStream rs = new RotateStream(fs, 2);
int b = rs.ReadByte();
Console.WriteLine(
    "First byte through the transform is " + b.ToString("X") + "h");

After resetting the current position of the FileStream object to the beginning of the stream using the Position property, I create an instance of a program-defined, custom stream class named RotateStream. This performs a bit-wise rotation transformation, which I'll explain shortly. The RotateStream constructor accepts two input arguments. The first argument is a stream (in this case, the FileStream associated with file TextFile1.txt). The second is the number of bit positions the transform will rotate (in this case, 2). When calling the ReadByte method of my custom RotateStream class, the first byte of input will be read and transformed by a 2-bit rotation to the left. So, because the first byte of input is 'H' = 0x48, (01001000 in binary), when read through the RotateStream filter, the result will be 0x21 (00100001 in binary).

One reason why many application developers and testers are not familiar with custom stream classes is that custom stream classes are not absolutely essential. By this I mean you can achieve the result of a custom stream implementation without actually creating a custom stream class. For example, the output in Figure 1 could have been produced by code along the lines of:

create a FileStream object, fs
associate fs with file TextFile1.txt
read a byte with fs.ReadByte()
rotate the byte 2 positions to the left
display the transformed byte

Figure 1 Custom I/O Stream Demo
Figure 1 Custom I/O Stream Demo(Click the image for a smaller view)
Figure 1 Custom I/O Stream Demo
Figure 1 Custom I/O Stream Demo (Click the image for a larger view)

In other words, you can achieve the same effect produced by a custom stream class by using built-in .NET stream classes combined with byte-level programming techniques.

So why write a custom stream class? There are two main reasons that make it worth considering. First, because a .NET stream is an abstraction of a sequence of bytes-such as a file or a TCP/IP socket-and .NET stream objects can be easily composed with each other (typically through their constructors), a custom stream class can provide a generic view of all types of input and output, isolating you from the specific details of the underlying devices, transports, or algorithms. Second, by wrapping byte-level input, output, and transformation routines into a class, you modularize your code, making it easier to test and maintain.

In this column, I will first present the implementation of a custom RotateStream class so you can understand exactly what it is we want to test. Then I will describe the details of how to test the custom stream class. I'll conclude with a brief discussion of how you can adapt and extend the code and techniques presented here to meet your own needs. I believe that you'll find this a useful addition to your personal skill set regardless of your job role.

A RotateStream Class

A common type of custom stream class is one that performs some sort of byte-level transformation. For example, the CryptoStream class of the System.Security.Cryptography namespace can encode and decode bytes using several techniques, including the Advanced Encryption Standard (AES). Stream classes usually implement both read and write methods. Typically, transformation stream classes use the write method to encode bytes and the read method to decode bytes. (I'm using the terms "encode" and "decode" rather loosely here, but they're easier to understand than slightly more accurate phrases such as "the byte transformation performed by the write operation" and "the byte transformation which reverses the transformation operation performed by the write operation.")

In this section, I will implement a custom RotateStream stream class that performs a bit-level rotate-right transformation ("encode") in the class Write method and a bit-level rotate-left transformation ("decode") in the class Read method. The overall structure of my RotateStream class is shown in Figure 2.

Figure 2 Custom Stream Class Structure (Close Figure 2)
public class RotateStream : Stream
{
  private Stream stream;
  private int n;

  public RotateStream(Stream stream, int n) { ... }

  public override bool CanRead { ... }
  public override bool CanWrite { ... }
  public override bool CanSeek { ... }
  public override long Length { ... }
  public override long Position { ... }

  public override void Flush() { ... }
  public override void Close(){ ... }
  public override long Seek(long offset, SeekOrigin origin) { ... }
  public override void SetLength(long value) { ... }

  public override int Read(byte[] array, int offset, int count) { ... }
  public override void Write(byte[] buffer, int offset, int count) { ... }

  private static byte ROR(byte b, int n) { ... }
  private static byte ROL(byte b, int n) { ... }
}

Custom stream classes are derived directly or indirectly from the System.IO.Stream class. The Stream class is an abstract base class that forms the foundation for all .NET stream classes.

I need two private field members for my RotateStream class. The Stream object named stream represents a generic sequence of bytes that will act as a source stream for Read operations and a destination stream for Write operations; the field named n holds the number of bit positions the class Read and Write methods will rotate. The custom RotateStream constructor accepts an existing stream that will be associated with my byte transform and an integer that instructs the Read and Write methods how many bit positions to rotate. In some programming languages, notably Java, stream classes are typically either read-classes or write-classes, but not both. However, .NET-based custom stream classes may be both read and write classes.

When creating a custom stream class, you must implement the get accessors on the CanRead and CanWrite properties so calling programs can query for these capabilities. In addition, you must implement a CanSeek property so calling code can determine whether your custom stream class supports seeking to random positions within the stream. Length and Position are additional abstract base properties you must implement-though if your stream is not "seekable," these will typically be implemented simply to throw an exception.

When creating a custom stream, you must implement a total of five methods. The Read and Write methods are, of course, the primary methods of a custom stream class. You must also implement the Flush, Seek, and SetLength methods. I have also chosen to implement a Close method, even though it's not strictly necessary. Finally, I've included two private helper methods, ROR (rotate right) and ROL (rotate left), for use in my Write and Read methods, respectively.

From this discussion, it seems like having to override so many routines is a lot of work, but as you'll see shortly, most of these overrides are just single lines of code.

My custom RotateStream constructor simply copies its input arguments to the corresponding class fields:

public RotateStream(Stream stream, int n)
{
  if (stream == null) throw new ArgumentNullException("stream");
  if (n <= 0 || n >= 8) throw new ArgumentOutOfRangeException("n");
  this.stream = stream;
  this.n = n;
}

Let's begin the custom RotateStream implementation by coding the Write and Read methods. My Write method places bytes into the output stream, performing a rotate-right bit transformation:

public override void Write(byte[] buffer, int offset, int count)
{
  byte[] temp = new byte[count];
  for (int i = offset, j = 0; i < offset + count; ++i, ++j)
    temp[j] = ROR(buffer[i], this.n);
  stream.Write(temp, 0, count);
}

Because I'm overriding the Write method in the abstract base Stream class, my Write method's signature corresponds to other Write signatures that you are probably familiar with, such as FileStream.Write. The buffer parameter holds an array of untransformed bytes to place into the output stream. The offset parameter is a zero-based index, which points into the buffer array telling the Write method where to begin fetching bytes. The count parameter tells the Write method how many bytes from array buffer to place into the output stream. In the body of my Write method, I first allocate a local byte array of the correct size.

(Note that because this column is primarily instructional, I have removed most error-checking code to keep the main ideas of custom streams as clear as possible. In a production situation, you would need to add all the usual error-checking code, such as verifying that the buffer is not empty.)

After allocating my scratch byte array, I iterate through the input byte array, starting at position offset, and store each rotate-right transformed byte into the scratch array. Once the scratch array has been filled, it is placed into the output stream-this is done by calling the Write method of the stream that is connected to the RotateStream (recall that the RotateStream constructor associates an existing stream to RotateStream). The ROR helper method uses classical bit manipulation operations:

private static byte[] _mask =
    { 0, 0x01, 0x03, 0x07, 0x0F, 0x1F, 0x3F, 0x7F };

private static byte ROR(byte b, int n)
{
    int val = b;
    return (byte)
        (((val &  _mask[n]) << (8 - n)) |
         ((val & ~_mask[n]) >> n));
}

Let's quickly review the difference between a shift-right operation and a rotate-right operation. Shift-right moves bits to the right, from the high-order bit to the low-order bit, and fills the missing high-order bits with 0s. (This would be modified slightly if dealing with signed bytes, but the System.Byte type in .NET is unsigned). For example, if you start with byte 11011101 and right-shift two positions, the result is 00110111. Notice the low-order bits fall off the right side and are lost. A rotate-right operation, on the other hand, checks the value of the low-order bits and then performs a right-shift. Rather than filling the missing high-order bits with 0s, a rotate-right operation fills the high-order bits with the old low-order bits (effectively rotating the bits). For example, if you start with 11011101 and rotate-right one position, the result is 11101110 and if you rotate that result once more to the right, you get 01110111. Figure 3 illustrates the concept of rotate-right.

Figure 3 Rotate-Right One Position
Figure 3 Rotate-Right One Position(Click the image for a smaller view)
Figure 3 Rotate-Right One Position
Figure 3 Rotate-Right One Position (Click the image for a larger view)

There are several ways to implement this in code. A simple solution is to make a copy of the lowest bit, shift the whole value right by one bit, and then put the lowest bit back as the highest bit. This can be done n times, where n is the number of bit positions I want to rotate. But this approach is quite slow. Since this operation is executed for every byte read from the stream, it exists on a "hot" path and therefore should be streamlined.

A faster solution, as shown in the previous code snippet, uses a lookup table to quickly swap the two sides of the byte (where the division between sides is guided by the amount of rotation required). For example, with the bit sequence 11011101, and with n=5, the two sides are 110 and 11101. I want to swap those in order to perform the rotation, resulting in the value 11101110.

To achieve this, my _mask array stores a bit mask with eight values, each representing an additional bit on the right side of the number set to 1 (00000000, 00000001, 00000011, 00000111, 00001111, and so on). The array is used as a lookup table to retrieve the appropriate mask, based on the value of n.

The Read method of my custom RotateStream class implements a decode transformation by performing a left-rotate as it extracts bytes from the input stream (undoing the result of RotateStream.Write calls that perform rotate-right "encode" operations):

public override int Read(byte[] buffer, int offset, int count)
{
  int numRead = stream.Read(buffer, offset, count);
  for (int i = offset; i < numRead + offset; ++i)
    buffer[i] = ROL(buffer[i], this.n);
  return numRead;
}

As with the Write method, my Read method has a standard signature that you're probably familiar with. The buffer parameter holds the transformed bytes, which are read from the input stream; the offset parameter tells Read where to begin placing input into the buffer parameter; and the count parameter tells Read how many bytes to attempt to read from input. The return value is the actual number of bytes read from input. My Read implementation reads from the associated input stream into array and then iterates through buffer, performing a rotate-left transformation using the ROL helper method. The ROL helper method is almost identical to the ROR helper, except with different shift directions and a different mask array (bits are filled in from the left rather than the right):

private static byte[] _leftMask =
    { 0, 0x80, 0xC0, 0xE0, 0xF0, 0xF8, 0xFC, 0xFE };

private static byte ROL(byte b, int n)
{
    int val = b;
    return (byte)
        (((val &  _leftMask[n]) >> (8 - n)) |
         ((val & ~_leftMask[n]) << n));
}

When writing a custom stream class that wraps another stream, some methods will simply delegate to the same method on the contained stream. For example, the Seek method in my class is implemented as follows:

public override long Seek(long offset, SeekOrigin origin)
{
  return stream.Seek(offset, origin);
}

All I do is call the Seek method of the underlying stream, which is associated with my custom RotateStream stream in the constructor. However, not all streams support a Seek operation. Seek is primarily intended for use with files. Streams like Console I/O streams, streams derived from pipes, and NetworkStream objects do not support a Seek method. If I associate my RotateStream with one of these types of streams and attempt to call Seek, a NotSupportedExeception exception will be thrown. Thus, any tests should be aware of the seeking capabilities of a stream. An alternative for custom streams that don't support seeking is to provide a placeholder implementation:

public override long Seek(long offset, SeekOrigin origin)
{
  throw new NotSupportedException("Seek not supported.");
}

The same approach is possible with derived Read and Write methods, too. Because derived Read, Write, and Seek methods may or may not be supported, you must implement CanRead, CanWrite, and CanSeek properties when writing a custom stream class so that calling programs (and test frameworks) can query your custom stream. My three read/write/seek query properties are:

public override bool CanRead { get { return stream.CanRead; } }
public override bool CanWrite { get { return stream.CanWrite; } }
public override bool CanSeek { get { return stream.CanSeek; } }

Since my custom RotateStream is associated with some existing source stream (for Read) or destination stream (for Write), I can just check to see if the associated stream supports each operation. This is usually a good approach for common transformation-type streams, such as RotateStream. But if you are writing a stream class that is primarily intended to act as connector between a source and destination (such as the FileStream and NetworkStream classes), then you will likely return an explicit true or false value. Say I definitely do not want to support a Seek operation. I can write:

public override bool CanSeek { get { return false; } }

Implementing the Length and Position properties in a custom stream class is similar to the CanRead, CanWrite, and CanSeek properties. If your custom stream class is a transform and associated with an existing source or destination stream, you can simply return the underlying Length and Position properties. Alternatively, you can decide not to support these operations by throwing an exception. Note that if you decide not to support the Length property then for consistency you should not support the SetLength method either.

The final required override is the Flush method:

public override void Flush() { stream.Flush(); }

As before, I simply invoke the associated stream's Flush method. This clears all buffers for the stream and forces any buffered data to be written to output.

And finally, in order to ensure that the underlying stream is closed when my stream is closed, I override the Close method:

public override void Close()
{
  stream.Close();
  base.Close();
}

Testing the RotateStream Class

Now that you know how to create a custom transform-type stream class, let's look at how you can test its functionality. Testing a custom stream class is simple in principle: you test the properties and methods that you've implemented. In this regard, testing a custom stream class is similar to API/unit testing. The most fundamental test is a Read test, as shown in the top part of Figure 4. Then Figure 5 shows the code that produced the Read test output.

Figure 5 Read Test (Close Figure 5)
Console.WriteLine(“\n==== RotateStream.Read test”);
byte[] inputBytes = new byte[] { 0x48, 0x65, 0x6C, 0x6C, 0x6F };
byte[] resultBytes = new byte[inputBytes.Length];

using(MemoryStream ms = new MemoryStream(inputBytes))
using(RotateStream rs = new RotateStream(ms, 1))
{
  int nr = rs.Read(resultBytes, 0, resultBytes.Length);
  Console.WriteLine(
    “\nReading “ + nr + “ bytes from memory through RotateStream”);
}

string input = BitConverter.ToString(inputBytes);
string expected = “90-CA-D8-D8-DE”;
string actual = BitConverter.ToString(resultBytes);
Console.WriteLine(“\nTest input      = “ + input);
Console.WriteLine(“Expected result = “ + expected);
Console.WriteLine(“Actual result   = “ + actual + “\n”);
Console.WriteLine(actual == expected ? “Pass” : “Fail”);
Figure 4 Read and Write Tests
Figure 4 Read and Write Tests(Click the image for a smaller view)
Figure 4 Read and Write Tests
Figure 4 Read and Write Tests (Click the image for a larger view)

As with all test cases, this requires an input and an expected result. I create my input as a sequence of hardcoded bytes (0x48, 0x65, 0x6C, 0x6C, and 0x6F). However, I could have created my input from a string like this:

string s = "Hello";
byte[] inputBytes = Encoding.ASCII.GetBytes(s);

This is usually not a good idea, though, since strings can't generate most byte patterns.

Before I can instantiate an instance of my RotateStream class and call Read, I need to establish an underlying source stream. You can do this by creating a MemoryStream object. (This is a good approach because the MemoryStream constructor can accept a byte array.) Then I instantiate a RotateStream object, associating it with the MemoryStream and specifying a 1-bit left-rotation.

With the preparation in place, I call RotateStream.Read and capture the result into array resultBytes. In order to compare my actual result with an expected result, I use the BitConverter class to convert bytes into a formatted string and then do a string comparison. This approach is not very efficient, but it is somewhat more readable than the alternative of comparing raw byte values. Using this test case as a guide, I can create additional test cases that examine different bit-rotation values and different inputs.

Testing the Write method of a custom stream class is similar to testing the Read method. When run, the code in Figure 6 produces the output shown in the lower part of Figure 4. Notice that for this test case, I use the expected output from the Read test case example as test input. Because Read performs a rotate-left operation and Write performs a rotate-right operation, my expected result will be the original 0x48, 0x65, 0x6C, 0x6C, 0x6F byte sequence. I use a MemoryStream object to act as a destination stream for the bytes transformed by my RotateStream object, and I do a simple string comparison to check my actual result with my expected result. This serves as a model for the many other test cases I would go on to create (including specifying different values of byte input and different rotate arguments, writing just part of input, attempting to write more output than available bytes, and so on).

Figure 6 Write Test (Close Figure 6)
Console.WriteLine("\n\n==== RotateStream.Write test");
byte[] inputBytes2 = new byte[] { 0x90, 0xCA, 0xD8, 0xD8, 0xDE };
byte[] resultBytes2 = new byte[inputBytes2.Length];
using(MemoryStream ms2 = new MemoryStream(resultBytes2))
using(RotateStream rs2 = new RotateStream(ms2, 1))
{
  rs2.Write(inputBytes2, 0, inputBytes2.Length);
  Console.WriteLine("\nWriting " + (inputBytes2.Length - 0) +
    " bytes to memory through RotateStream");
}

string input2 = BitConverter.ToString(inputBytes2);
string expected2 = "48-65-6C-6C-6F";
string actual2 = BitConverter.ToString(resultBytes2);
Console.WriteLine("\nTest input      = " + input2);
Console.WriteLine("Expected result = " + expected2);
Console.WriteLine("Actual result   = " + actual2 + "\n");
Console.WriteLine(actual2 == expected2 ? "Pass" : "Fail");

In order to test my Seek method, I need to associate a RotateStream instance with a source stream that supports Seek. The FileStream class makes a good choice here (see Figure 7). I create a text file, named TextFile1.txt, that contains "Hello", and I instantiate a FileStream object that references the file to act as a source stream. Next, I instantiate an instance of my RotateStream class under test with a bit-rotation argument of 2. I specify a seek offset of 3, which points at the second lower-case l character in TextFile1.txt. Then I call RotateStream.Seek, checking the return value to make sure Seek did not fail.

Figure 7 Using the FileStream Class (Close Figure 7)
Console.WriteLine("\n==== Seek test");
using(FileStream fs = File.OpenRead("..\\..\\TextFile1.txt")
using(RotateStream rs = new RotateStream(fs, 2))
{
  int inputSeek = 3;
  long x = rs.Seek(inputSeek, SeekOrigin.Begin);
  if (x == -1) throw new Exception("Seek operation error");
  int actualByte = rs.ReadByte();
  rs.Close();

  int expectedByte = 0xB1;
  Console.WriteLine("\nInput Seek    = " + inputSeek);
  Console.WriteLine("Expected byte = " + expectedByte);
  Console.WriteLine("Actual byte   = " + actualByte + "\n");
  Console.WriteLine(actualByte == expectedByte ? "Pass" : "Fail");
}

At this point, I could simply do a FileStream.Read to fetch the current byte and check to see if it is 0x6C, or 'l'. Here, however, I instead use my inherited RotateStream.ReadByte method. The ReadByte method essentially calls its Read method with a count value of 1 and returns the resulting byte. If my input is 0x6C, then my expected result will be that input after a 2-bit rotate-right transformation, or 0xB1. As a general rule of thumb, you shouldn't test two different things at the same time-in this case, the Seek method and the ReadByte method-because if your test case fails, the cause of the failure isn't revealed. But as with any rule of thumb, this is just a guideline. A double test like this can be more efficient than separate tests, especially when performing regression tests where you expect all test cases to pass the majority of the time.

Of course, you'll need a bunch of additional tests; this is only a start. For example, for the Seek method, if the expected result is an exception (due to testing an error condition or to Seek not being supported), you should catch any returned exceptions and report success accordingly.

Two More Scenarios

This sample RotateStream custom stream class and the techniques used to test it are representative of the kind of simple byte-transform custom stream classes and tests that are common in application programming scenarios. The information presented here should allow you to understand and test your own custom transform stream. However, there are two major scenarios I did not cover. In addition to custom transform streams, which generally assume the existence of source and destination streams, you can create what I call custom end-point streams. These are intended to act as conduits between a custom byte source and destination. In fact, most of the .NET stream classes you are familiar with, including FileStream and NetworkStream, are end-point streams. But creating these types of streams is more common in a systems programming environment than in an application programming environment, and the topic is outside the scope of this column.

The second major custom stream topic I haven't covered here is the creation and testing of asynchronous Read and Write methods. However, asynchronous methods are generally implemented in end-point custom stream classes, but not in transform custom stream classes.

Preserving Size

The custom RotateStream class presented in this column is what I call a size-preserving transform stream. By this I mean that the transformation performed by the Read and Write methods preserves the size of input and output. But not all transform streams preserve size. Consider a hypothetical PaddingStream class, which is similar to the RotateStream class in the sense that the PaddingStream constructor associates the PaddingStream object with an existing source stream like a FileStream. Now suppose the PaddingStream.Write method encodes by adding 0 bytes to every other byte of raw input, and that the PaddingStream.Read method decodes by stripping away every other 0-byte. If some raw byte source consists of, say, 0x11223344 then the Write method will place 0x0011002200330044 into the output stream. And the Read method would undo this padding.

Custom streams that do not preserve size can be very, very tricky. Fortunately, in application development scenarios, the need to create custom streams that do not preserve size is rare.

Conclusion

To summarize, the .NET environment lets you easily create custom stream classes. In an application development environment, the most common type of custom stream classes are those that perform a byte-level transformation with a Write method acting as an encoder and a Read method acting as a decoder.

Because the effect of a custom transform stream can be created using existing .NET stream classes combined with bit manipulation techniques, custom stream classes are not widely used or understood by application developers and testers. However, creating a custom stream class can modularize transform code, making it easier to test, modify, and maintain.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products, including Internet Explorer and MSN Search. James is the author of .NET Test Automation Recipes (Apress, 2006). He can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Cutting Edge: Validating ASP.NET Query Strings

Cutting Edge
Validating ASP.NET Query Strings
Dino Esposito

Code download available at: CuttingEdge2007_03.exe (168KB)
Contents
The Problem
Defining a Strategy
Declarative Query Strings
Coding the QueryString HTTP Module
Query String Validation
Considerations and Alternatives
Summary

For years, classic ASP developers implemented page authentication by inserting some generic code at the top of each page that would grab user credentials, attach a cookie, and redirect. All that repetitive code was swept away by the ASP.NET HTTP modules for authentication. As a result, ASP.NET applications don't have to link each and every page being secured to the authentication module of choice. Everything can be done declaratively through the web.config file and a bunch of external resources, such as the login page and the membership database.

ASP.NET also introduced other system modules and programming techniques that minimize repetitive code and rationalize the implementation of a Web application's common features. For example, site maps, anonymous users, and profiles are now built-in features and you no longer have to write or copy the code for them over and over.

The focus on security also led to the inclusion of a fair number of native barriers in the ASP.NET runtime-this saves the developer from the burden of cross-checking input data against at least some possible forms of attack. Of course, this doesn't mean that an ASP.NET application is secure by design, but it does mean that the security bar starts higher than in the past. However, it is still up to developers to raise it even higher.

ASP.NET pages are designed to post data to themselves and group input parameters in the body of an HTTP POST packet. Most ASP.NET applications don't use the query string to pass input data as frequently as classic ASP applications did. Nonetheless, the query string is still a legitimate way to import external data into an ASP.NET page. But who validates this data?

Recent statistics show that cross-site scripting (XSS) attacks are gaining momentum and they claim the lion's share of discovered attacks. Successful XSS attacks are always due to unvalidated or improperly validated input data, and more often than not, this data comes through the query string.

Starting with version 1.1, ASP.NET preprocesses any posted data (forms and query string), looking for suspicious combinations of characters that may be exploited by XSS attackers. But this barrier is not a silver bullet and, as Michael Howard says in his November 2006 MSDN®Magazine article, "Secure Habits: 8 Simple Rules For Developing More Secure Code" (available at msdn.microsoft.com/msdnmag/issues/06/11/SecureHabits), you have to take responsibility. If your pages use query string parameters, you need to ensure that they are properly validated before use. How do you do that?

In this column, I build an HTTP module that reads an XML file where you have hardcoded the expected structure of the query string. The module then validates the query string of any requested page against the given schema. And you don't need to touch the code of any page. (For more on preventing XSS attacks, see the Microsoft Anti-Cross Site Scripting Library v1.5)

The Problem

Developers can't afford to just leave pages that take input from the query string unattended. Values have to be validated and the format of the query string carefully checked. Such a validation process contains two distinct steps: static validation (which checks the type and existence of required parameters) and dynamic validation (which verifies whether specified values are coherent with the expectations of the rest of the code). Dynamic validation is specific to each page and can't be delegated to an external, page-agnostic component. In contrast, static validation, relies on a list of general-purpose checks (parameter required, type, length) that can be executed without instantiating the page.

As with classic ASP where you had to include authentication generic code in each and every secured page, in ASP.NET you have to include query string validation code in each page. ASP.NET moved the authentication standard code to a small group of system-provided HTTP modules, but didn't take care of the query string. On the other hand, the growth of XSS and SQL injection attacks lately poses the problem of cross-checking any possible source of input. An external component linked to the application that implements strict static validation for the query string parameters is a great help because it automatically ensures that no ASP.NET page requests are ever executed when the query string doesn't comply with the declared schema.

More importantly, with an external component, no changes are required to the page's source code. All that you have to do is register the component with the application through the configuration file and add an XML file that describes the query string syntax for each interested page. Let's detail the strategy in a bit more depth.

Defining a Strategy

ASP.NET offers HTTP modules as a tool to inject your own code into the runtime pipeline before the requested page class is instantiated and processed. From a syntax perspective, an HTTP module is merely a class that implements a given interface. From a broader architectural perspective, an HTTP module is a kind of observer with the same lifetime as the application. The module observes the request processing activity and registers to listen for a few specific events, such as BeginRequest, EndRequest, or PostMapRequestHandler. The full list of application events for an ASP.NET request can be found in the documentation of the System.Web.HttpApplication class (msdn2.microsoft.com/0dbhtdck.aspx).

Once installed, an HTTP module kicks in each time a request processed by the ASP.NET runtime reaches the stage when the observed event is fired. Note that the ASP.NET runtime doesn't necessarily process requests for all resources hosted by the ASP.NET application. By default, static resources, such as cascading style sheets (CSS) and JPG files, are served directly by the Web server without even bothering the ASP.NET application, unless IIS has been configured to allow ASP.NET to handle these resources.

My query string HTTP module will listen for the begin-request event and validate the contents of the query string against a previously loaded schema. If the number of parameters match and the provided value is compatible with the expected type, the module lets the request reach the next stage. Otherwise, the request is terminated with an appropriate HTTP status code or an ASP.NET exception is thrown.

I mentioned an XML file where the syntax of the query string would be stored. This doesn't really have to be an XML file. (If it is an XML file, the schema is entirely up to you.) You just need a data source that declaratively persists information about the expected structure of a page's query string. It could be a simple XML file, or it could be a sophisticated provider-based service. My June 2006 column provides a good example of a custom application service designed to use providers (msdn.microsoft.com/msdnmag/issues/06/06/CuttingEdge).

Declarative Query Strings

Figure 1 shows a sample XML file and schema that the query string HTTP module will recognize. Under the root node <querystring>, there are as many <page> nodes as there are pages in the application that may process values from the query string. In the companion code of this column, the file shown in Figure 1 is named web.querystring. The name is arbitrary, of course, as is the schema.

Figure 1 Sample web.querystring Configuration File (Close Figure 1)
<!--
<page url="..." abortOnError="TRUE|false">
  <param name="..."
         type="Int|Text|Bool"
         optional="FALSE|true"
         length="number (for Text type only)"
         casesensitive="false|true" />
</page>
-->

<querystring>

  <page url="/source/test.aspx" abortOnError="true">
    <param name="id" type="Int" optional="true" casesensitive="true" />
    <param name="code" type="Text" length="5" optional="false" />
    <param name="detailed" type="Bool" optional="false" />
  </page>

  <page url="/source/Test1.aspx" abortOnError="false">
    <param name="guid" type="Int" />
  </page>

</querystring>

(From a security standpoint, the main issue is not that a page receives values through the query string but that the page may use those values. If some code in the page processes input sent through the query string, then as a developer you must ensure that the input is safe and not evil. For this reason, you might want to add into the XML file a <page> node only for the pages in the application that actually consume data via the query string.)

In the sample schema, the <page> element has two attributes: url and abortOnError. The former indicates the relative URL to the page, while the latter is an optional Boolean attribute that indicates whether the page request should be aborted in case of bad input. If you choose to abort the page, then the user either receives an HTTP error or an ASP.NET exception, based on what you decide to do after unacceptable data is found in the query string. Regardless of how you manifest the outcome, there's no need to edit the code of the involved ASP.NET page. The possible termination of the request occurs in the HTTP module, before the page class is identified and instantiated.

There is an alternative approach. In it, the HTTP module lets the request go, but adds detailed information to the HTTP context to notify the page class of what was detected. The page then has the responsibility of taking proper countermeasures, such as showing an ad hoc error page. In this situation, the page author must integrate any query string anomalies in the context of the error-handling strategy set for the application. The downside of this approach is that it requires changes to the code of each page involved with the query string. (I'll return to this point later.)

The abortOnError attribute is set to true by default, meaning that any anomaly in the query string will abort the page request. Under each <page> node, there is a list of <param> nodes-one for each supported query-string parameter. In the sample code, a parameter can be defined using the attributes in Figure 2.

Figure 2 Attributes Supported by the <param> Node (Close Figure 2)

Attribute | Description
Name | Indicates the name of the query parameter.
Type | Indicates the type of the query string parameter. Feasible values are: Text, Int, Bool.
Optional | A Boolean attribute, this indicates whether the parameter is optional or not. It is set to false by default.
Length | Indicates the maximum length of a parameter of type Text.
CaseSensitive | A Boolean attribute, this indicates whether the name of the parameter is case sensitive. It is set to false by default, meaning the parameter can be specified on a query string with any combination of lowercase and uppercase letters.

All values passed on the query string are received by ASP.NET as strings. The QueryString property defined on the HttpRequest object is, therefore, a NameValueCollection object where keys and values are strings. The string format, though, is a mere serialization format. Each query string parameter can surely represent not only a string but also a Boolean or numeric value, plus special string subtypes like URL, GUID, and file names. Therefore, in the web.querystring file, you specify the expected type of the parameter using the values of a custom enum type, QueryStringParamTypes:

Friend Enum QueryStringParamTypes As Integer
    Text = 1
    Int = 2
    Bool = 3
End Enum

The list of supported types can be extended to add, for example, various numeric types. Parameters of type Text can also specify a maximum length through the attribute Length. A page that can accept, say, a 5-char customer ID from the query string has no reason for not limiting the length of that parameter. In addition, the web.querystring may be used to enable checks on the case sensitivity of the parameter's name and may designate a parameter as optional. The contents of the web.querystring file are parsed by the query string HTTP module and transformed into an in-memory object.

Coding the QueryString HTTP Module

The source code of the QueryString HTTP module is shown in Figure 3. As mentioned, an HTTP module class implements IHttpModule, which consists of Init and Dispose methods. These methods are invoked when the module is loaded and unloaded in the context of the application. In the Init method, an HTTP module typically registers a listener for the application events it wants to observe. In this case, it registers a handler for the BeginRequest event. In addition, the module processes the web.querystring file and creates an in-memory representation of its contents. The Init method is invoked only once per application-the contents of the configuration file are read the first time and cached, and changes to the web.querystring file are not detected until the Web application is restarted. This is not necessarily a problem as it is fairly unlikely you will need to enter changes to the web.querystring file in production without stopping and restarting the application. However, you could also extend the code in Figure 3 to use a file watcher object that detects any changes to the web.querystring file and reload it in a timely fashion.

Figure 3 QueryStringModule Class (Close Figure 3)
Imports System
Imports System.Web
Imports System.IO

Public Class QueryStringModule : Implements IHttpModule

    Private _app As HttpApplication
    Private _queryStringData As Hashtable

    Public Sub Init(ByVal context As System.Web.HttpApplication) _
            Implements System.Web.IHttpModule.Init
        _app = context
        AddHandler _app.BeginRequest, AddressOf OnEnter

        ' Load and cache the XML querystring file
        Dim fileName As String = _
            HttpContext.Current.Server.MapPath("web.querystring")
        _queryStringData = QueryStringHelper.LoadFromFile(fileName)
    End Sub

    Public Sub Dispose() Implements System.Web.IHttpModule.Dispose
    End Sub

    Private Sub OnEnter(ByVal source As Object, ByVal e As EventArgs)
        ' Retrieve the query string data structure for the current page
        Dim currentPage As String = _
            HttpContext.Current.Request.Path.ToLower()
        Dim qsDesc As QueryStringDescriptor = _
            _queryStringData.Item(currentPage)

        ' Validate the query string
        Dim isValid As Boolean
        isValid = QueryStringHelper.Validate( _
           HttpContext.Current.Request.QueryString, qsDesc)

        ' Abort the request if validation fails
        If Not isValid Then
            If qsDesc.AbortOnError Then
                HttpContext.Current.Response.StatusCode = 500
                HttpContext.Current.Response.[End]()
            Else
                ' Add information about the error to Context.Items
                HttpContext.Current.Items( _
                    QueryStringHelper.QueryStringValidationStatus) = _
                        QueryStringHelper.GetErrorCode()
            End If
        Else
            ' Add information for the page to the Context.Items
            HttpContext.Current.Items( _
                QueryStringHelper.QueryStringValidationStatus) = _
                    QueryStringHelper.GetErrorCode()

            ' Add typed values
            HttpContext.Current.Items( _
                QueryStringHelper.QueryStringValues) = _
                    QueryStringHelper.GetTypedValues( _
                        HttpContext.Current.Request.QueryString, qsDesc)
        End If
    End Sub
End Class

The contents of the web.querystring file are mapped to an object of type QueryStringDescriptor, as shown in Figure 4. The descriptor contains the URL of the page, a flag to indicate what to do in case of failed validation, and the list of supported query string parameters. Each parameter is described through an instance of the QueryStringParamInfo class. QueryStringParamCollection is the related collection class. It is a typical generic collection class enriched with a pair of Find methods: one to verify whether a parameter with a given name can be found in the collection and one to return the parameter descriptor instance.

Figure 4 Helper Classes for the QueryString Module (Close Figure 4)
Friend Class QueryStringDescriptor
    Public Url As String
    Public AbortOnError As Boolean
    Public Parameters As QueryStringParamCollection
End Class

Friend Class QueryStringParamInfo
    Public Name As String
    Public [Type] As QueryStringParamTypes
    Public Length As Integer
    Public [Optional] As Boolean
    Public CaseSensitive As Boolean
End Class

Friend Class QueryStringParamCollection : Inherits Collection( _
        Of QueryStringParamInfo)

    Public Overloads Function Contains(ByVal name As String) As Boolean
        For i As Integer = 0 To Count - 1
            Dim comparison As StringComparison = _
                StringComparison.OrdinalIgnoreCase
            Dim currentItem As QueryStringParamInfo = Item(i)
            If currentItem.CaseSensitive Then
                comparison = StringComparison.Ordinal
            End If

            If String.Equals(currentItem.Name, name, comparison) Then
                Return True
            End If
        Next

        Return False
    End Function

    Public Function Find(ByVal name As String) As QueryStringParamInfo
        For i As Integer = 0 To Count - 1
            Dim currentItem As QueryStringParamInfo = Item(i)
            If String.Equals(currentItem.Name, name, _
                    StringComparison.OrdinalIgnoreCase) Then
                Return currentItem
            End If
        Next

        Return Nothing
    End Function
End Class

<Flags()> _
Public Enum QueryStringErrorCodes
    NoError = 0
    TooManyParameters = 1
    InvalidQueryParameter = 2
    MissingRequiredParameter = 4
    InvalidContent = 8
End Enum

The query string descriptor caches information about the query string of a given page. The web.querystring file, though, can reference multiple pages. For this reason, all descriptors for all pages referenced by web.querystring are grouped in a hash table using the page URL as the key. The following code snippet shows how the BeginRequest handler of the HTTP module retrieves the descriptor for the currently requested page:

Dim currentPage As String
currentPage = HttpContext.Current.Request.Path.ToLower()
Dim qsDesc As QueryStringDescriptor = _
    _queryStringData.Item(currentPage)

The query string descriptor is an in-memory representation of the correct syntax for the page's query string. The next step is to validate the posted query string against this schema.

Query String Validation

The validation process consists of three steps. First, the module counts the number of parameters in the posted query string. If the posted query string has more parameters than expected, the validation fails. Next, the module iterates on posted query string parameters and ensures that each parameter matches an entry in the declared schema. If additional, unknown parameters are found, the validation fails. Finally, the module iterates on all parameters defined in the schema and verifies that all required parameters are specified and that each specified parameter has a value of the proper type.

The data validation step attempts to parse the value of a given parameter to its declared type. Here's the code snippet used to validate numeric values:

If paramType = QueryStringParamTypes.Int Then
    Dim result As Integer
    Dim success As Boolean = Int32.TryParse(paramValue, result)
    If Not success Then Return False
End If

By design, Boolean values are parsed only from strings like true and false. The validation subsystem of the querystring HTTP module also accepts strings such as yes and no.

In the end, the contents of the query string are parsed and type-validated as the first step in the request's pipeline. If everything is OK, the request is processed. Otherwise, the request is immediately terminated with a proper HTTP status code. Here's an example:

HttpContext.Current.Response.StatusCode = 500
HttpContext.Current.Response.[End]()

The user is served a page like the one in Figure 5. You might complain that nothing indicates the real reason for the IIS error, but the HTTP status code, as well as the generic description, is clear about the origin of the error-an internal server-side error during the processing of the request. As the aforementioned Michael Howard article explains, the least possible amount of information should always be disclosed in error pages to avoid the risk of gently handing out details to potential hackers. In this respect, an HTTP 500 error is generic enough about what really occurred. Anyway, as the previous code snippet shows, the HTTP status code can be set at will.

Figure 5 Result of a Bad Query String
Figure 5 Result of a Bad Query String(Click the image for a smaller view)
Figure 5 Result of a Bad Query String
Figure 5 Result of a Bad Query String (Click the image for a larger view)

Considerations and Alternatives

Should you really abort the request in case of badly formatted data or are you better off caching somewhere the result of the validation and letting the page code make the final decision about the user? Furthermore, should you really catch and process the query string so early in the request lifecycle? Let's tackle this latter point first.

Figure 6 lists the application-wide events that characterize the request processing. If not at the beginning of the request, where else should you check the query string? A good place is immediately after authorization. If the request proceeds beyond the authorization stage, you can be relatively sure that the page HTTP handler will be invoked.

Figure 6 Global Application Events (Close Figure 6)

Event | Description
BeginRequest | Indicates the beginning of the request processing.
AuthenticateRequest PostAuthenticateRequest | Wraps the request authentication process.
AuthorizeRequest PostAuthorizeRequestPostAuthorizeRequest | Wraps the request authorization process.
ResolveRequestCache PostResolveRequestCache>PostResolveRequestCache | Wraps the process that checks whether the request can be served with previously cached output pages.
PostMapRequestHandler | Indicates that the HTTP handler to serve the request has been found.
AcquireRequestState PostAcquireRequestState | Wraps the retrieval of session state for the request.
PostRequestHandlerExecute | Indicates that the HTTP handler to serve the request has been executed.
ReleaseRequestState PostReleaseRequestState | Wraps the release of session state for the request.
UpdateRequestCache PostUpdateRequestCache | Wraps the process that checks whether the output of the requested resource should be cached for further reuse.
EndRequest | Indicates the end of the request processing.

But can you do it later? In general, any event handler up to and including PostAcquireRequestState would work. The user code-the code page authors write in the codebehind or inline in the ASPX file-is executed only past the PostAcquireRequestState event. Subsequently, there's no way the page can consume the query string up until the global PostAcquireRequestState event is fired. However, you shouldn't wait this long. Checking the query string after authorization but before page execution can save you a couple of extra operations-referring to retrieval of session state and checking the output cache. If you're going to kill the page due to a bad query string, there's no reason to first load the session state, especially if it's coming from an out-of-process source like SQL Server™.

In the end, there are only two application events where the query string check should be placed: BeginRequest or PostAuthorizeRequest. You should opt for the latter if user information is required to process the query string, for example if some users are allowed to specify certain parameters based on their role. In this case, you can also add a roles attribute to the schema of Figure 1. In any other scenario, by placing the interception in BeginRequest, you can kill the page at a very early stage in the pipeline, preventing further processing.

Things are quite different if you still want the page code to handle the bad query string and try to gracefully degrade or recover. For this, I think any event before the page is executed will work fine. I would opt for PostAcquireRequestState, which is the latest point in the pipeline where you can check the query string before the page code executes. You also have session state available at this point. I haven't noted it yet, even though the context makes it patently clear: the query string information is available from the beginning in the QueryString collection of the Request intrinsic object.

So let's assume that you want the HTTP module to check the query string and pass its findings down the pipeline and up to the page code. There are a couple of possible approaches you could take. Before I discuss them, I should note that any such approach is code intrusive and requires changes to the source code of each page with a query string.

The simplest way for an HTTP module to communicate with the handler in charge of a given request is by stuffing data into the Items collection of the HttpContext object. The property Items is a hash table for HTTP modules and handlers to write and read information. Any data stored in the Items table has the same lifetime as the request.

The HTTP module gains access to the context object of the current request using the static Current property on the HttpContext class, like so:

HttpContext.Current.Items("QueryStringStatus") = errorCode

With Items being a System.Collections.Hashtable, both key and value can be any .NET type. The query-string module uses a public enum type to list all possible error codes:

<Flags()> _
Public Enum QueryStringErrorCodes
    NoError = 0
    TooManyParameters = 1
    InvalidQueryParameter = 2
    MissingRequiredParameter = 4
    InvalidContent = 8
End Enum

The combination of these codes, which better describes what's wrong with the query string, is stuffed into a conventionally named slot in the hash table. The HTTP module and pages must agree on a naming convention so the page can retrieve and use this information. The HTTP module defines a public constant, which represents the name of the slot:

Public Const QueryStringValidationStatus As String = _
       "QueryStringValidationStatus"

The page can use the following code to retrieve the message from the HTTP module and decide what to do with the info:

Dim result As QueryStringErrorCodes = _
    DirectCast(Context.Items( _
        QueryStringHelper.QueryStringValidationStatus), _
        QueryStringErrorCodes)

Say you also want the module to provide the page with typed values taken from a valid query string. Consider the following URL and assume the query string is correct:

http://www.yourserver.com/page.aspx?detailed=true

The page should incorporate code to parse the query string value and transform it into a Boolean value. Such a conversion is already done in the HTTP module during the validation step. Nothing is easier than sharing these typed values with the target page by placing a hash table of them in another Items slot (see the source code for details).

A neater approach is to add a new read-only property to each page with a query string. Assuming you call it IsValidQueryString, it will look like this:

Public Property IsValidQueryString As Boolean
   Get
      Dim result As QueryStringErrorCodes = DirectCast( _
        Context.Items(QueryStringHelper.QueryStringValidationStatus), _
            QueryStringErrorCodes)
      Return (result = QueryStringErrorCodes.NoError)
   End Get
End Property

Even better, you could define such a property on a base class and derive all query-string enabled pages from this class.

Summary

Not all ASP.NET pages use the query string. However, the query string can be used as input for Web pages. As such, it is a potential vehicle for attacks on pages with security holes. If your pages need a query string barrier, be ready to write the same code over and over again in all of the pages where you use the query string.

The QueryString module presented in this column requires no coding in source pages and automatically checks the posted query string against a given schema saved in a separate XML file. This means there's zero impact on existing code, while offering one more built-in barrier against attackers. But remember that this is no silver bullet.

Send your questions and comments for Dino to cutting@microsoft.com cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Wicked Code: Scalable Apps with Asynchronous Programming in ASP.NET

Wicked Code
Scalable Apps with Asynchronous Programming in ASP.NET
Jeff Prosise

Code download available at: WickedCode2007_03.exe (202KB)
Contents
Asynchronous Pages
Asynchronous HTTP Handlers
Asynchronous HTTP Modules
Conclusion

Do you want to know a secret? A deep, dark, dirty secret? One that, if revealed, would cause great angst in the ASP.NET community and prompt shouts of "Aha!" from the anti-Microsoft crowd?

Most Web sites I've seen built with ASP.NET aren't very scalable, not because of a flaw in ASP.NET, but because of how the technology is used. They suffer a self-imposed glass ceiling that limits the number of requests they can process per second. These sites scale just fine until traffic rises to the level of this invisible ceiling. Then throughput begins to degrade. Soon after, requests start to fail, usually returning "Server unavailable" errors.

The underlying reason has been discussed many times in MSDN®Magazine. ASP.NET uses threads from a common language runtime (CLR) thread pool to process requests. As long as there are threads available in the thread pool, ASP.NET has no trouble dispatching incoming requests. But once the thread pool becomes saturated-that is, all the threads inside it are busy processing requests and no free threads remain-new requests have to wait for threads to become free. If the logjam becomes severe enough and the queue fills to capacity, ASP.NET throws up its hands and responds with a "Heck no!" to new requests.

One solution is to increase the maximum size of the thread pool, allowing more threads to be created. That's the course developers often take when their customers report repeated "Server unavailable" errors. Another common course of action is to throw hardware at the problem, adding more servers to the Web farm. But increasing the thread count-or the server count-doesn't solve the issue. It just provides temporary relief to what is in reality a design problem-not in ASP.NET itself, but in the implementation of the actual site. The real problem for apps that don't scale isn't lack of threads. It's inefficient use of the threads that are already there.

A truly scalable ASP.NET Web site makes optimum use of the thread pool. That means making sure request-processing threads are executing code instead of waiting for I/O to complete. If the thread pool becomes saturated due to all the threads grinding away on the CPU, there's little you can do but add servers.

However, most Web apps talk to databases, Web services, or other external entities, and limit scalability by forcing thread-pool threads to wait for database queries, Web service calls, and other I/O operations to complete. A request targeting a data-driven Web page might spend a few thousandths of a second executing code and several seconds waiting for a database query to return. While the query is outstanding, the thread assigned to the request is unable to service other requests. That's the glass ceiling. And that's a situation you must avoid if you care about building highly scalable Web sites. Remember: when it comes to throughput, I/O is evil unless handled properly.

I/O isn't evil, though, if it doesn't gum up the thread pool. And ASP.NET supports three asynchronous programming models that act as anti-gumming agents. These models are largely unknown to the community, in part due to scant documentation. Yet knowing how-and when-to use them is absolutely essential to building cutting-edge Web sites.

Asynchronous Pages

The first, and generally most useful, of the three asynchronous programming models ASP.NET supports is the asynchronous page. Of the three models, this is the only one that's specific to ASP.NET 2.0. The others are supported all the way back to version 1.0.

I won't go into detail about asynchronous pages here because I did that in the October 2005 issue (msdn.microsoft.com/msdnmag/issues/05/10/WickedCode). The upshot is that if you have pages that perform relatively lengthy I/O operations, they're candidates to become asynchronous pages. If a page queries a database and the query takes, say, 5 seconds to return because it either returns a large amount of data or targets a remote database over a heavily loaded connection, that's 5 seconds that the thread assigned to the request can't be used for other requests. If every request behaved like this, the application would quickly get bogged down.

Figure 1 illustrates how an asynchronous page neatly solves this problem. When the request arrives, it's assigned a thread by ASP.NET. The request begins processing on that thread, but when the time comes to hit the database, the request launches an asynchronous ADO.NET query and returns the thread to the thread pool. When the query completes, ADO.NET calls back to ASP.NET, and ASP.NET grabs another thread from the thread pool and resumes processing the request.

Figure 1 Asynchronous Pages at Work
Figure 1 Asynchronous Pages at Work(Click the image for a smaller view)
Figure 1 Asynchronous Pages at Work
Figure 1 Asynchronous Pages at Work (Click the image for a larger view)

While the query is outstanding, zero thread pool threads are consumed, leaving all of the threads free to service incoming requests. A request that's processed asynchronously doesn't execute any faster. But other requests execute faster because they don't have to wait for threads to become free. Requests incur less delay in entering the pipeline, and overall throughout goes up.

Figure 2 shows the codebehind class for an asynchronous page that performs data binding against a SQL Server™ database. The Page_Load method calls AddOnPreRenderCompleteAsync to register begin and end handlers. Later in the request's lifetime, ASP.NET calls the begin method, which launches an asynchronous ADO.NET query and returns immediately, whereupon the thread assigned to the request goes back into the thread pool. When ADO.NET signals that the query has completed, ASP.NET retrieves a thread from the thread pool (not necessarily the same one it used before) and calls the end method. The end method grabs the query results and the remainder of the request executes as normal on the thread that executed the end method.

Figure 2 Asynchronous Page (Close Figure 2)
using System;
using System.Data;
using System.Data.SqlClient;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Web.Configuration;

public partial class AsyncDataBind : System.Web.UI.Page
{
    private SqlConnection _connection;
    private SqlCommand _command;
    private SqlDataReader _reader;

    protected void Page_Load(object sender, EventArgs e)
    {
        if (!IsPostBack)
        {
            AddOnPreRenderCompleteAsync(
                new BeginEventHandler(BeginAsyncOperation),
                new EndEventHandler(EndAsyncOperation)
            );
        }
    }

    IAsyncResult BeginAsyncOperation (object sender, EventArgs e,
        AsyncCallback cb, object state)
    {
        string connect = WebConfigurationManager.ConnectionStrings
            ["AsyncPubs"].ConnectionString;
        _connection = new SqlConnection(connect);
        _connection.Open();
        _command = new SqlCommand(
            "SELECT title_id, title, price FROM titles", _connection);
        return _command.BeginExecuteReader (cb, state);
    }

    void EndAsyncOperation(IAsyncResult ar)
    {
        _reader = _command.EndExecuteReader(ar);
    }

    protected void Page_PreRenderComplete(object sender, EventArgs e)
    {
        Output.DataSource = _reader;
        Output.DataBind();
    }

    public override void Dispose()
    {
        if (_connection != null) _connection.Close();
        base.Dispose();
    }
}

Not shown in Figure 2 is the Async="true" attribute in the ASPX's Page directive. This is required for an asynchronous page: it signals ASP.NET to implement the IHttpAsyncHandler interface in the page (more on this in a moment). Also not shown in Figure 2 is the database connection string, which includes an Async="true" attribute of its own so that ADO.NET knows to perform an asynchronous query.

AddOnPreRenderCompleteAsync is one way to structure an asynchronous page. Another way is to call RegisterAsyncTask. This has a couple of advantages over AddOnPreRenderCompleteAsync-the most important being that it simplifies the task of performing multiple asynchronous I/O operations in one request. For details on this, see the October 2005 installment of Wicked Code.

Asynchronous HTTP Handlers

The second asynchronous programming model featured in ASP.NET is the asynchronous HTTP handler. An HTTP handler is an object that serves as an endpoint for requests. Requests for ASPX files, for example, are processed by an HTTP handler for ASPX files. Likewise, requests for ASMX files are handled by an HTTP handler that knows how to deal with ASMX services. In fact, ASP.NET comes with HTTP handlers for a variety of file types. You can see these file types-and the corresponding HTTP handlers-in the <httpHandlers> section of the master web.config file (in ASP.NET 1.x, it's in machine.config).

You can extend ASP.NET to support additional file types by writing custom HTTP handlers. But even more interesting is the fact that you can deploy custom HTTP handlers in ASHX files and use them as targets of HTTP requests. This is the proper way to build Web endpoints that generate images on the fly or retrieve images from databases. You simply include an <img> tag (or Image control) in the page and point it to an ASHX that creates or fetches the image. Targeting an ASHX file with requests is more efficient than targeting an ASPX file because an ASHX file incurs much less overhead at processing time.

By definition, HTTP handlers implement the IHttpHandler interface. Handlers that implement that interface do their processing synchronously. The ASHX file in Figure 3 contains one such HTTP handler. At run time, TerraServiceImageGrabber makes multiple calls out to the Microsoft® TerraServer Web service to convert a city and state into latitude and longitude, retrieve satellite images ("tiles"), and stitch the images together to form a composite image of the specified location.

Figure 3 Synchronous HTTP Handler (Close Figure 3)
<%@ WebHandler Language="C#" Class="TerraServiceImageGrabber" %>

using System;
using System.Web;
using System.Drawing;
using System.Drawing.Imaging;
using System.IO;

public class TerraServiceImageGrabber : IHttpHandler
{
    public void ProcessRequest (HttpContext context)
    {
        // Extract user input from the query string
        string city = context.Request["City"];
        string state = context.Request["State"];
        string scale = context.Request["Scale"];

        // If city or state wasn't specified, throw an exception
        if (String.IsNullOrEmpty(city) || String.IsNullOrEmpty(state))
            throw new ArgumentException(
                "City and state must be specified via query string");

        // Determine the scale
        Scale res = Scale.Scale8m;
        if (!String.IsNullOrEmpty (scale))
        {
            switch (scale)
            {
                case "1":
                    res = Scale.Scale1m;
                    break;

                case "2":
                    res = Scale.Scale2m;
                    break;

                case "4":
                    res = Scale.Scale4m;
                    break;

                case "8":
                    res = Scale.Scale8m;
                    break;

                case "16":
                    res = Scale.Scale16m;
                    break;

                case "32":
                    res = Scale.Scale32m;
                    break;
            }
        }

        // Generate the requested image
        using(Bitmap bitmap = GetTiledImage(city, state, res, 900, 600))
        {
            // Set the response's content type
            context.Response.ContentType = "image/jpeg";

            // Write the image to the HTTP response
            bitmap.Save(context.Response.OutputStream, ImageFormat.Jpeg);
        }
    }

    public bool IsReusable { get { return true; } }

    private Bitmap GetTiledImage (string city, string state,
        Scale scale, int cx, int cy)
    {
        // Instantiate the TerraService proxy
        TerraService ts = new TerraService ();

        // Get the latitude and longitude of the requested city
        Place place = new Place ();
        place.City = city;
        place.State = state;
        place.Country = "USA";
        LonLatPt point = ts.ConvertPlaceToLonLatPt (place);

        // Compute the parameters for a bounding box
        AreaBoundingBox abb = ts.GetAreaFromPt (point, Theme.Photo,
            scale, cx, cy);

        // Create an image to fit the bounding box
        Bitmap bitmap = new Bitmap (cx, cy, PixelFormat.Format32bppRgb);
        using(Graphics g = Graphics.FromImage(bitmap))
        {
            int x1 = abb.NorthWest.TileMeta.Id.X;
            int y1 = abb.NorthWest.TileMeta.Id.Y;
            int x2 = abb.NorthEast.TileMeta.Id.X;
            int y2 = abb.SouthWest.TileMeta.Id.Y;

            for (int x=x1; x<=x2; x++) {
                for (int y=y1; y>=y2; y--) {
                    TileId tid = abb.NorthWest.TileMeta.Id;
                    tid.X = x;
                    tid.Y = y;
                    using(Image tile = Image.FromStream(
                        new MemoryStream(ts.GetTile(tid))))
                    {
                        g.DrawImage(tile,
                            (x - x1) * tile.Width  -
                                (int) abb.NorthWest.Offset.XOffset,
                            (y1 - y) * tile.Height -
                                (int) abb.NorthWest.Offset.YOffset,
                            tile.Width, tile.Height);
                    }
                }
            }
        }

        // Return the image
        return bitmap;
    }
}

The results are shown in Figure 4. The page shown contains an Image control whose ImageUrl property targets the ASHX file in Figure 3. Once the user selects a city and state and clicks a button, the HTTP handler converts the input into a satellite image.

The results are impressive. But there's a catch. TerraServiceImageGrabber is a perfect example of how not to write HTTP handlers. Think about it. TerraServiceImageGrabber requires several seconds (at least) to make all its Web service calls and process the results. The vast majority of that time is spent simply waiting for Web service calls to complete. Repeated requests for the ASHX file can deplete the ASP.NET thread pool in a fraction of a second, preventing other pages in the application from being served up (or at least forcing them to be queued until a thread becomes available). You can't build a scalable application in this way unless you scale out the hardware. But why spend tens of thousands of dollars on a Web farm when one server might handle the load with properly written software?

Figure 4 TerraServiceImageGrabber at Work
Figure 4 TerraServiceImageGrabber at Work(Click the image for a smaller view)
Figure 4 TerraServiceImageGrabber at Work
Figure 4 TerraServiceImageGrabber at Work (Click the image for a larger view)

HTTP handlers don't have to be synchronous. By implementing the IHttpAsyncHandler interface, which itself derives from IHttpHandler, an HTTP handler can be asynchronous. When used correctly, an asynchronous handler utilizes ASP.NET threads more efficiently. This is done in the same manner as an asynchronous page. In fact, asynchronous pages leverage the asynchronous handler support that predated asynchronous pages in ASP.NET.

Figure 5 contains an asynchronous version of the handler shown in Figure 3. Async-TerraServiceImageGrabber is slightly more complex, but it is also far more scalable.

Figure 5 Asynchronous HTTP Handler (Close Figure 5)

<%@ WebHandler Language="C#" Class="AsyncTerraServiceImageGrabber" %>

using System;
using System.Web;
using System.Drawing;
using System.Drawing.Imaging;
using System.Threading;
using System.IO;

public class AsyncTerraServiceImageGrabber : IHttpAsyncHandler
{
    private TerraService _ts;
    private TerraServiceAsyncResult _ar;
    private Scale _scale = Scale.Scale8m;
    private AreaBoundingBox _abb;
    private Bitmap _bitmap;
    private int _count = 0;
    private int _max;
    private HttpContext _context;
    private Exception _ex;
    private int _cx = 900, _cy = 600; // Width and height of bitmap

    public void ProcessRequest (HttpContext context)
    {
        // Never called
    }

    public bool IsReusable { get { return false; } }

    public IAsyncResult BeginProcessRequest(HttpContext context,
        AsyncCallback cb, object state)
    {
        _context = context;

        // Extract user input from the query string
        string city = context.Request["City"];
        string region = context.Request["State"];
        string scale = context.Request["Scale"];

        // If city or state wasn’t specified, throw an exception
        if (String.IsNullOrEmpty(city) || String.IsNullOrEmpty(region))
            throw new ArgumentException(
                "City and state must be specified via query string");

        // Determine the scale
        if (!String.IsNullOrEmpty (scale))
        {
            switch (scale)
            {
                case "1":
                    _scale = Scale.Scale1m;
                    break;

                case "2":
                    _scale = Scale.Scale2m;
                    break;

                case "4":
                    _scale = Scale.Scale4m;
                    break;

                case "8":
                    _scale = Scale.Scale8m;
                    break;

                case "16":
                    _scale = Scale.Scale16m;
                    break;

                case "32":
                    _scale = Scale.Scale32m;
                    break;
            }
        }

        // Instantiate a TerraServiceAsyncResult
        _ar = new TerraServiceAsyncResult(cb, state);

        // Instantiate the TerraService proxy
        _ts = new TerraService();

        // Make an async call to get the latitude and longitude
        // of the requested city
        Place place = new Place();
        place.City = city;
        place.State = region;
        place.Country = "USA";

        _ts.BeginConvertPlaceToLonLatPt(place,
            new AsyncCallback(ConvertPlaceToLonLatCompleted), null);

        // Return an IAsyncResult that delays EndProcessRequest until
        // the final asynchronous Web service call has completed
        return _ar;
    }

    private void ConvertPlaceToLonLatCompleted(IAsyncResult ar)
    {
        try
        {
            // Complete the async call
            LonLatPt point = _ts.EndConvertPlaceToLonLatPt(ar);

            // Make an async call to compute the parameters
            // for a bounding box
            _ts.BeginGetAreaFromPt(point, Theme.Photo, _scale, _cx,
              _cy, new AsyncCallback(GetAreaFromPointCompleted), null);
        }
        catch (Exception ex)
        {
            _ex = ex;
            _ar.CompleteCall();
        }
    }

    private void GetAreaFromPointCompleted(IAsyncResult ar)
    {
        try
        {
            // Complete the async call
            _abb = _ts.EndGetAreaFromPt(ar);

            // Create an image to fit the bounding box
            _bitmap = new Bitmap(_cx, _cy, PixelFormat.Format32bppRgb);

            int x1 = _abb.NorthWest.TileMeta.Id.X;
            int y1 = _abb.NorthWest.TileMeta.Id.Y;
            int x2 = _abb.NorthEast.TileMeta.Id.X;
            int y2 = _abb.SouthWest.TileMeta.Id.Y;

            _max = (x2 - x1 + 1) * (y1 - y2 + 1);

            // Place concurrent async calls to TerraService to
            // fetch image tiles
            for (int x = x1; x <= x2; x++)
            {
                for (int y = y1; y >= y2; y--)
                {
                    TileId tid = new TileId();
                    tid.Theme = _abb.NorthWest.TileMeta.Id.Theme;
                    tid.Scale = _abb.NorthWest.TileMeta.Id.Scale;
                    tid.Scene = _abb.NorthWest.TileMeta.Id.Scene;
                    tid.X = x;
                    tid.Y = y;
                    _ts.BeginGetTile(tid,
                        new AsyncCallback(GetTileCompleted),
                        new Point(x - x1, y1 - y));
                }
            }
        }
        catch (Exception ex)
        {
            _ex = ex;
            _ar.CompleteCall();
        }
    }

    private void GetTileCompleted(IAsyncResult ar)
    {
        try
        {
            // Complete the async call
            using(Image tile = Image.FromStream(
                new MemoryStream(_ts.EndGetTile(ar))))
            {
                // Draw the tile onto the bitmap
                Point point = (Point)ar.AsyncState;
                int dx = point.X;
                int dy = point.Y;

                lock (_bitmap)
                {
                    using(Graphics g = Graphics.FromImage(_bitmap))
                    {
                        g.DrawImage(tile,
                            dx * tile.Width –
                                (int)_abb.NorthWest.Offset.XOffset,
                            dy * tile.Height –
                                (int)_abb.NorthWest.Offset.YOffset,
                            tile.Width, tile.Height);
                }
            }

            // Increment the tile count and complete the request if all
            // tiles have been fetched
            int count = Interlocked.Increment(ref _count);
            if (count == _max) _ar.CompleteCall();
        }
        catch (Exception ex)
        {
            _ex = ex;
            _ar.CompleteCall();
        }
    }

    public void EndProcessRequest(IAsyncResult ar)
    {
        if (_ex != null)
        {
            // If an exception was thrown, rethrow it
            throw _ex;
        }
        else
        {
            // Otherwise return the generated image
            _context.Response.ContentType = "image/jpeg";
            _bitmap.Save(_context.Response.OutputStream,
                ImageFormat.Jpeg);
            _bitmap.Dispose();
        }
    }
}

class TerraServiceAsyncResult : IAsyncResult
{
    private AsyncCallback _cb;
    private object _state;
    private ManualResetEvent _event;
    private bool _completed = false;
    private object _lock = new object();

    public TerraServiceAsyncResult(AsyncCallback cb, object state)
    {
        _cb = cb;
        _state = state;
    }

    public Object AsyncState { get { return _state; } }

    public bool CompletedSynchronously { get { return false; } }

    public bool IsCompleted { get { return _completed; } }

    public WaitHandle AsyncWaitHandle
    {
        get
        {
            lock (_lock)
            {
                if (_event == null)
                    _event = new ManualResetEvent(IsCompleted);
                return _event;
            }
        }
    }

    public void CompleteCall()
    {
        lock (_lock)
        {
            _completed = true;
            if (_event != null) _event.Set();
        }

        if (_cb != null) _cb(this);
    }
}

Asynchronous processing begins when ASP.NET calls the handler's BeginProcessRequest method. BeginProcessRequest makes an async call to TerraService via the TerraService proxy's BeginConvertPlaceToLonLatPt method. The thread assigned to the request then goes back into the thread pool. When the async call completes, another thread is borrowed from the thread pool to execute the ConvertPlaceToLonLatCompleted method. That thread retrieves the results of the last call, makes an async call of its own, and then goes back into the thread pool. This pattern repeats until all the asynchronous calls have completed, at which time the handler's EndProcessRequest method is called and the resulting Bitmap is returned to the requestor.

To forestall EndProcessRequest until the final Web service call has completed, AsyncTerraServiceImageGrabber returns its own implementation of IAsyncResult from BeginProcessRequest. If it returned the IAsyncResult returned by BeginConvertPlaceToLonLatPt instead, EndProcessRequest would be called (and the request terminated) the moment the first Web service call completed.

The class that implements IAsyncResult, TerraServiceAsyncResult, features a public CompleteCall method that can be called at any time to finish the request. Normally, AsyncTerraServiceImageGrabber calls CompleteCall only after the final Web service call has completed. However, if an exception is thrown in one of the methods that executes between BeginProcessRequest and EndProcessRequest, the handler caches the exception in a private field (_ex), calls CompleteCall to terminate the request, and then rethrows the exception from EndProcessRequest. Otherwise, the exception would be lost and the request would never complete.

AsyncTerraServiceImageGrabber is more scalable than its synchronous counterpart since it consumes ASP.NET threads for only a fraction of the total time required to process a request. The vast majority of the time, it's simply waiting for an asynchronous Web service call to complete.

In theory, AsyncTerraServiceImageGrabber can also outperform TerraServiceImageGrabber because, rather than making repeated calls to TerraService's GetTile method in series, it makes the calls in parallel. In reality, however, only two outbound calls targeting a given IP address can be pending at a time unless you increase the runtime's default maxconnection setting:

<system.net>
  <connectionManagement>
    <add address="*" maxconnection="20" />
  </connectionManagement>
</system.net>

Other configuration settings can also affect concurrency. For more information, refer to the Knowledge Base article "Contention, Poor Performance, and Deadlocks when You Make Web Service Requests from ASP.NET Applications" (support.microsoft.com/kb/821268).

Even if only one callout executes at a time, AsyncTerraServiceImageGrabber should perform no worse than TerraServiceImageGrabber. And its design is far superior because it uses ASP.NET threads as efficiently as possible.

Asynchronous HTTP Modules

The third asynchronous programming model you can leverage in ASP.NET is the asynchronous HTTP module. An HTTP module is an object that sits in the ASP.NET pipeline, where it can see-and even modify-incoming requests and outgoing responses. Many of the key services in ASP.NET are implemented in the form of HTTP modules, including authentication, authorization, and output caching. You can extend ASP.NET by writing custom HTTP modules and plugging them into the pipeline. And when you do, you should carefully consider whether those HTTP modules should be asynchronous.

Figure 6 contains the source code for a simple, synchronous HTTP module called RequestLogModule, which logs incoming requests in a text file named RequestLog.txt. It creates the file in the site's App_Data directory so users can't browse to it. (Note that the security principal that ASP.NET runs as-ASPNET or Network Service, for example-must have write permission to App_Data for this to work.) The module implements the IHttpModule interface, which is the one and only requirement for an HTTP module. When the module loads, its Init method registers a handler for HttpApplication.PreRequestHandlerExecute events, which fire from the pipeline in each and every request. The event handler opens RequestLog.txt (or creates it if it doesn't already exist) and writes into it a line of text containing pertinent information about the current request, including the time and date that the request arrived, the requestor's user name if the request is authenticated (or the requestor's IP address if authentication is turned off), and the requested URL. The module is registered in the <httpModules> section of web.config, prompting ASP.NET to load it each time the application starts up.

Figure 6 Synchronous HTTP Module (Close Figure 6)
using System;
using System.Web;
using System.IO;

public class RequestLogModule : IHttpModule
{
    public void Init (HttpApplication application)
    {
        application.PreRequestHandlerExecute +=
            new EventHandler(OnPreRequestHandlerExecute);
    }

    public void Dispose () {}

    void OnPreRequestHandlerExecute(Object sender, EventArgs e)
    {
        HttpApplication app = (HttpApplication)sender;
        DateTime time = DateTime.Now;

        using(StreamWriter writer = new StreamWriter(
            app.Server.MapPath("~/App_Data/RequestLog.txt"), true))
        {
            string line = String.Format(
                "{0,10:d}    {1,11:T}    {2, 32}   {3}",
                time, time,
                app.User.Identity.IsAuthenticated ?
                app.User.Identity.Name :
                app.Request.UserHostAddress,
                app.Request.Url);
            writer.WriteLine (line);
        }
    }
}

The problem with RequestLogModule is twofold. First, it performs file I/O in each and every request. Second, it uses a request-processing thread to perform the I/O-a thread that could otherwise be used to service additional incoming requests. This module, simple as it is, imposes a penalty on throughput. While you could mitigate the delay by batching the file I/O operations, a better approach is to make the module asynchronous (or better yet, batch the file I/O and make the module asynchronous).

Figure 7 shows the asynchronous version of RequestLogModule. Called AsyncRequestLogModule, it performs exactly the same work, but returns the thread assigned to the request to the thread pool before writing to the file. When the write completes, a new thread is borrowed from the thread pool and used to finish the request.

Figure 7 Asynchronous HTTP Module (Close Figure 7)
using System;
using System.Web;
using System.IO;
using System.Threading;
using System.Text;

public class AsyncRequestLogModule : IHttpModule
{
    private FileStream _file;
    private static long _position = 0;
    private static object _lock = new object();

    public void Init (HttpApplication application)
    {
        application.AddOnPreRequestHandlerExecuteAsync (
            new BeginEventHandler (BeginPreRequestHandlerExecute),
            new EndEventHandler (EndPreRequestHandlerExecute)
        );
    }

    IAsyncResult BeginPreRequestHandlerExecute (Object source,
        EventArgs e, AsyncCallback cb, Object state)
    {
        HttpApplication app = (HttpApplication)source;
        DateTime time = DateTime.Now;

        string line = String.Format(
            "{0,10:d}    {1,11:T}    {2, 32}   {3}\r\n",
            time, time,
            app.User.Identity.IsAuthenticated ?
            app.User.Identity.Name :
            app.Request.UserHostAddress,
            app.Request.Url);

        byte[] output = Encoding.ASCII.GetBytes(line);

        lock (_lock)
        {
            _file = new FileStream(
                HttpContext.Current.Server.MapPath(
                    "~/App_Data/RequestLog.txt"),
                FileMode.OpenOrCreate, FileAccess.Write,
                FileShare.Write, 1024, true);

            _file.Seek(_position, SeekOrigin.Begin);
            _position += output.Length;
            return _file.BeginWrite(output, 0, output.Length, cb, state);
        }
    }

    void EndPreRequestHandlerExecute (IAsyncResult ar)
    {
        _file.EndWrite(ar);
        _file.Close();
    }

    public void Dispose () {}
}

What makes AsyncRequestLogModule asynchronous? Its Init method calls HttpApplication.AddOnPreRequestHandlerExecuteAsync in order to register begin and end methods for PreRequestHandlerExecute events. The HttpApplication class features other AddOn methods for other per-request events. For example, an HTTP module can call AddOnBeginRequestAsync to register async handlers for BeginRequest events. AsyncRequestLogModule's BeginPreRequestHandlerExecute method uses the framework's FileStream.BeginWrite method to begin an asynchronous write. The moment BeginPreRequestHandlerExecute returns, the thread goes back into the thread pool.

AsyncRequestLogModule contains some thread synchronization logic that merits special mention. It's possible-likely even-that multiple requests running on multiple threads will write to the log file at the same time. To ensure that concurrent writes don't overwrite each other, AsyncRequestLogModule stores the position in the file for the next write in a private field shared by all module instances (_position). Before each call to BeginWrite, the module reads the position from the field and updates the field to point to the first byte following what's about to be written to the file. Logic that reads and updates _position is wrapped in a lock statement so that no more than one thread can execute it at a time. That prevents one thread from reading the position before another gets the chance to update it.

Now the bad news. For BeginWrite not to use another thread from the pool, the isAsync parameter to FileStream's constructor must be set to true, as I've done in my example. However, a little-known consequence of using FileStream.BeginWrite to initiate an asynchronous write is that there's no guarantee the write will actually be asynchronous, even if you've requested asynchronous operation. Windows® reserves the right to perform asynchronous file I/O synchronously if it believes that synchronous I/O will be faster. For more information, read the Knowledge Base article at support.microsoft.com/kb/156932. The good news is that if Windows writes to the request log synchronously, the writes are, in theory, performed so quickly that they have minimal impact on the scalability of the host application.

Conclusion

Asynchronous programming is an excellent way to build more scalable applications by using the ASP.NET thread pool as efficiently as possible. In my travels, I rarely see ASP.NET developers utilize asynchronous programming models, in part because they simply don't know that these models exist. Don't let sparse documentation stand in your way; start thinking asynchronously today, and you'll be building better applications tomorrow.

Be aware that the downloadable sample code accompanying this article comes in both C# and Visual Basic® versions. I frequently get e-mail asking for Visual Basic versions of the samples. This time, you don't have to ask; they're already there!

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (Microsoft Press, 2002). He's also a cofounder of Wintellect, a software consulting and education firm that specializes in .NET.


Service Station: BizTalk Server 2006 Web Services

Service Station
BizTalk Server 2006 Web Services
Aaron Skonnard

Contents
Web Services Adapters
Using the SOAP Adapter
SOAP Transport Properties
Using the SOAP Adapter in Orchestrations
The WSE 2.0 Adapter
The WCF Adapter
Summary

This month I'll introduce you to the exciting world of BizTalk Server 2006 and its support for today's Web services technologies. You'll learn how to use the SOAP and Web Services Enhancements (WSE) adapters available right now, and I'll tell you about the forthcoming Windows® Communication Foundation (WCF) adapter included in the BizTalk® Server 2006 R2 release.

BizTalk Server 2006 is built on a flexible messaging subsystem that promotes loosely coupled messaging interactions across heterogeneous applications. The messaging layer provides numerous integration benefits such as message routing, schema transformations, and format translations. At the heart of the messaging layer is a SQL Server™ database referred to as the message box. All messages processed by the messaging layer travel through the message box for routing, tracking, and error-handling purposes. The message box operates by a "leave no message behind" mantra, which is a key feature when reliability is paramount. However, to be fair, this feature can also become a liability when performance is more important.

You control how messages are routed between applications by defining message subscriptions (also known as "filters") in the message box. You can define filters in the administration console (on send ports) or you can define them indirectly within orchestrations using logical port definitions. As illustrated in Figure 1, when a message is published, the message box evaluates the incoming message against subscriptions and routes the message to all matching subscribers (send ports or orchestrations). This publish/subscribe architecture completely decouples senders from receivers.

Figure 1 BizTalk Messaging Architecture
Figure 1 BizTalk Messaging Architecture(Click the image for a smaller view)
Figure 1 BizTalk Messaging Architecture
Figure 1 BizTalk Messaging Architecture (Click the image for a larger view)

BizTalk interacts with the outside world through adapters. An adapter receives bytes over a particular transport and creates a new BizTalk message that will be published to the message box after the receive pipeline and a map (an XSLT transform) have had a chance to operate on the incoming message (see Figure 2). The reverse is true for transmitting outgoing messages. After the map and send pipeline have had a chance to operate on the outgoing message, the adapter transmits the resulting bytes to the transport. You configure these details through send and receive ports in BizTalk.

Figure 2 BizTalk Port Internal
Figure 2 BizTalk Port Internal(Click the image for a smaller view)
Figure 2 BizTalk Port Internal
Figure 2 BizTalk Port Internal (Click the image for a larger view)

BizTalk supports one-way and two-way ports. On the receive side, you can define one-way receive ports that simply receive messages but don't return anything. You can also define request-response receive ports that return response messages to the caller. Similarly, on the send side, you can define one-way or solicit-response ports.

BizTalk Server 2006 comes with a variety of adapters that support a wide range of transports and protocols, including several designed specifically for SOAP and WS-*. The fact that BizTalk supports numerous communication mechanisms in addition to SOAP and WS-* is one of its most compelling characteristics. This makes BizTalk a central figure in systems that must straddle legacy applications and newer service-oriented investments.

Web Services Adapters

BizTalk Server 2006 integrates with SOAP and WS-* through the different Web service adapters listed in Figure 3. The SOAP adapter, which ships with BizTalk Server 2004 and later versions, supports WS-I Basic Profile 1.1 (BP 1.1) messages. If you need support for WS-* protocols, you'll have to turn to one of the WSE adapters available today or wait for the Windows Communication Foundation adapter coming in the BizTalk Server 2006 R2 release.

Figure 3 BizTalk Web Service Adapters from Microsoft (Close Figure 3)

Adapter | BizTalk Server 2004 | BizTalk Server 2006 | BizTalk Server 2006 R2
SOAP | Included | Included | Included
WSE 2.0 | Download available | Download available (requires SP1) | Download available (requires SP1)
WCF | Not included | Not included | Included

Microsoft provides a WSE 2.0 adapter that you can download from microsoft.com/biztalk/evaluation/adapter/adapters/wse. The WSE 2.0 adapter runs on both BizTalk Server 2004 and BizTalk Server 2006 (although the latter requires the SP1 release). Microsoft chose not to implement an adapter for WSE 3.0 because the forthcoming Windows Communication Foundation adapter will provide equivalent functionality. However, there are third-party WSE 3.0 adapters available.

Figure 4 Using a Web Services Adapter
Figure 4 Using a Web Services Adapter(Click the image for a smaller view)
Figure 4 Using a Web Services Adapter
Figure 4 Using a Web Services Adapter (Click the image for a larger view)

You can use any of these adapters to send and receive SOAP messages in BizTalk. You can use them in conjunction with one-way or two-way ports. For example, you can use the SOAP adapter to send and receive simple SOAP messages that conform to BP 1.1, or you can use the WSE 2.0 adapter to send and receive SOAP messages that are secured according to the WS-Security, WS-Trust, and WS-SecureConversation specifications.

On the receive side, it's the adapter's job to perform the SOAP and WS-* processing and to publish the body of the message to the message box. In Figure 4, the ASMX endpoint is essentially the adapter. At run time, the ASMX code communicates with BizTalk to retrieve the receive location details from the configuration database so that it can use the configured receive pipeline and map set during publishing. The fact that you can preprocess incoming SOAP messages with traditional BizTalk pipelines and maps is a powerful combination that will increase your integration options.

BizTalk requires that all HTTP-based receive locations run in an isolated host process such as IIS instead of a BizTalk-managed process. This means BizTalk does not fully control the configuration of SOAP/HTTP receive locations. Instead, it assumes the isolated host will control the various SOAP/HTTP processing details. You'll also have to write (or better yet, automatically generate) the isolated host application code.

You typically use a wizard to produce the adapter code that will run in the isolated host. However, you don't have to use a wizard; it's possible to write a generic SOAP endpoint capable of receiving and publishing any SOAP message into BizTalk.

You can also use any of the Web service adapters listed in Figure 3 to send BizTalk messages from the message box to external Web services. You simply define a send port that uses one of the adapters and then you configure it with any of the various settings it provides for transmitting the outgoing SOAP message.

It's important to note that you can do all of this directly within the BizTalk messaging layer without using orchestrations. Of course, you can also consume Web services within orchestrations. This, is an attractive option because the orchestration designer greatly simplifies the developer experience while offering advanced support for long-running processes, correlation, and error handling. It's also possible to publish an orchestration as a Web service to automatically expose its public receive ports.

Now let's take a closer look at performing some of the tasks I've just described using the SOAP and WSE adapters available today.

Using the SOAP Adapter

Let's assume you already have a BizTalk solution in place for processing orders. You already have several receive locations configured for receiving orders over different transports, say MSMQ and FTP, and you already have subscriptions in place for processing the published order messages. And now you simply want to add support for receiving orders via SOAP in addition to everything else. In this case, you already have the order message schema-you just need to generate an ASMX service around it.

You can generate an ASMX service from existing schema types by running the BizTalk Web Services Publishing Wizard. The wizard lets you choose between publishing schemas or an orchestration. In this case, you want to publish schemas.

Next, you'll be asked to describe the Web service contract that you wish to expose. Since you're just publishing a schema, you must manually define the name of the Web service and the name of each WebMethod operation that you want to create. Then, for each WebMethod, you can map the request/response messages to a specific schema type found within a BizTalk assembly (see Figure 5). The wizard asks for additional details about the ASMX code that it's about to generate, such as the target namespace, the ASMX parameter style, and whether you'd like to make BP 1.1 conformance claims. It also asks if you would like to support additional or unknown SOAP headers to allow for future processing.

Figure 5 Publishing a Schema
Figure 5 Publishing a Schema(Click the image for a smaller view)
Figure 5 Publishing a Schema
Figure 5 Publishing a Schema (Click the image for a larger view)

Finally, it asks for the virtual directory where it should deploy the generated ASMX code. It will also create a corresponding receive location in an existing BizTalk application for you. When you finish the wizard, it generates the ASMX code, publishes it to the specified directory, and creates a BizTalk receive location configured for the service.

Before you can browse to the generated ASMX endpoint, however, you'll need to verify that the virtual directory is configured to run within an application pool that has permission to access the BizTalk databases (the identity of the application pool needs to be a member of IIS_WPG and BizTalk Isolated Hosts Users). At this point, you can also configure other aspects of HTTP and ASP.NET processing using traditional configuration techniques in IIS and the Microsoft® .NET Framework.

The other half of the configuration is found within the BizTalk receive location. If you open the BizTalk administration console, you'll find the SOAP receive location created by the wizard. The receive location properties will look similar to Figure 6. Notice it's configured to run in the isolated host instance. Also, if you click the Configure button to configure the SOAP transport, you'll notice it only contains the address of the service (this is the key used by the isolated host to look up other aspects of the BizTalk configuration at run time, such as the receive pipeline and maps).

Figure 6 SOAP Receive Location Configuration
Figure 6 SOAP Receive Location Configuration(Click the image for a smaller view)
Figure 6 SOAP Receive Location Configuration
Figure 6 SOAP Receive Location Configuration (Click the image for a larger view)

After the virtual directory is configured properly, you can browse to the generated ASMX endpoint and you'll see the traditional ASMX documentation page. If you inspect the operation details, you'll notice that the body of each SOAP message is simply the schema type you specified in the wizard. Now clients can access the ASMX endpoint to retrieve the service's Web Services Description Language (WSDL) definition for generating client code. WSDL code generation makes it a lot easier for SOAP-savvy clients to integrate with the BizTalk solution.

Now let's suppose that there's an external billing service that you need to send messages to in response to an incoming order message. And let's also assume that you already have a Web services proxy class developed for the purpose of communicating with the external Web service. (You would have created this using wsdl.exe or Add Web Reference.)

You can create a send port capable of sending messages to the external billing service. Simply choose the SOAP adapter and specify the address of the external Web service (see Figure 7). Then, on the Web Service tab, select a Web service proxy class and method to use (see Figure 8). BizTalk will use this proxy class when sending the message. The assembly needs to be in the global assembly cache (GAC) at run time.

Figure 7 SOAP Send Port Configuration
Figure 7 SOAP Send Port Configuration(Click the image for a smaller view)
Figure 7 SOAP Send Port Configuration
Figure 7 SOAP Send Port Configuration (Click the image for a larger view)
Figure 8 Specifying a Proxy Class
Figure 8 Specifying a Proxy Class(Click the image for a smaller view)
Figure 8 Specifying a Proxy Class
Figure 8 Specifying a Proxy Class (Click the image for a larger view)

The hardest part about making this work is supplying the correct message to the SOAP adapter at run time. If the specified proxy class method takes more than one parameter, BizTalk expects you to supply a multi-part message where each part corresponds to a parameter in the signature. BizTalk will deserialize each message part into the corresponding .NET Framework type before invoking the Web service proxy method, which then serializes everything back into a SOAP message.

Note that when you need to implement an existing WSDL definition, you can use the BizTalk Business Process Execution Language (BPEL) import wizard to generate an orchestration skeleton from the WSDL. You can also use the traditional wsdl.exe /server in conjunction with some manual intervention to produce the correct adapter code. For more details on these various contract-first techniques, see msdn.microsoft.com/library/en-us/bts_2004wp/html/7f82d724-4f6f-4440-9b01-921d0cb51fc0.asp.

SOAP Transport Properties

When using the SOAP adapter, there are numerous SOAP-specific message context properties available to you. They are all found within the SOAP namespace and you can access them from within pipeline components and orchestration shapes.

For example, SOAP.AssemblyName and SOAP.MethodName identify the Web service proxy class and method to use when sending messages through a SOAP send port. You may also want to configure client credentials dynamically through SOAP.Username and SOAP.Password. Or you may need to increase the response timeout value by setting SOAP.ClientConnectionTimeout. You can use these properties to configure a variety of dynamic send port properties at run time. Refer to the SOAP adapter documentation for a complete list of SOAP adapter properties and how they work.

On the receive side, you may need to process various SOAP headers found in the incoming SOAP message. There are two ways to accomplish this task. First, if you defined any known SOAP headers when using the publishing wizard, these headers will be made available to you as custom properties in a special SOAPHeader namespace. Otherwise, all unknown SOAP headers are stored in the SOAP.UnknownHeaders property. In addition, if you need to add more SOAP headers to outgoing SOAP messages, you can populate the SOAP header properties before sending the message.

One last thing I should mention before continuing is the new failed message routing feature in BizTalk Server 2006. When enabled on send/receive ports, this feature causes failed messages to be republished to the message box with additional error report properties (found in the ErrorReport namespace) in the message context. This makes it possible to subscribe to the various ErrorReport properties in order to be notified when certain errors occur within your SOAP send/receive ports.

Using the SOAP Adapter in Orchestrations

So far I've covered the fundamentals of using the SOAP adapter within the messaging layer without using BizTalk orchestrations. Of course, you can also use the SOAP adapter in conjunction with orchestrations, which is typically an easier proposition.

You can publish an orchestration as a Web service by running the same BizTalk Web Services Publishing wizard, only this time you'll choose to publish an orchestration. The wizard asks you to choose an orchestration and the public ports that you wish to expose. The service contract will be derived from the selected ports, so you won't have to define the contract manually as you did in the past. The rest of the steps in the wizard will be the same as before: it asks for the various IIS and ASP.NET settings and it generates an ASMX service. The main difference is that the wizard is able to automatically derive the contract from the orchestration.

You can also consume Web services from within orchestrations. You start by using the Add Web Reference command in Visual Studio®. This brings in the required schema and port types that you'll need to invoke the external service from an orchestration. Then you define a new orchestration port and select the Web port type added by the Add Web Reference command. Using the generated Web port type ensures that you'll be sending and receiving the correct XML messages to or from the Web service. Finally, you simply connect Send and Receive shapes to the Web port to model the Web service invocation.

It's also possible to invoke Web services from within orchestration expression shapes. In this case, the messages do not travel through the BizTalk messaging layer and have the low-latency characteristics of a traditional Web services call. However, heavy use of this technique can also make your orchestrations more difficult to manage after deployment.

You can handle SOAP faults returned by the service within the orchestration by using a Scope shape and an exception handling block. If you want to process the details of the SOAP fault, the exception handling block should be configured to catch exceptions of type System.Web.Services.Protocols.SoapException. Likewise, your orchestrations can return SOAP faults to callers when you've published the orchestration as a service.

The WSE 2.0 Adapter

The process of using the WSE 2.0 adapter is similar to that of using the SOAP adapter described in the previous sections. Let's focus on the differences now.

The first thing you'll notice is that the WSE 2.0 adapter comes with its own publishing wizard. When you run the wizard, it looks almost identical to the one for the SOAP adapter, until you get to the step shown in Figure 9, where it asks you to configure the security policy for the request, response, and fault messages. You can specify the token type you want to use, and even some token details within the wizard.

If you inspect the code generated by the WSE 2.0 wizard, the first thing you'll see is that the application is configured with a WSE 2.0 policy file found in the virtual directory. The wizard generates this policy based on the settings you provide. In some cases, you may need to run the WSE 2.0 configuration tool to complete the policy configuration after using the wizard, and there's no problem with doing so.

You may also notice that the generated code is no longer using the ASMX programming model, but rather the WSE 2.0 messaging API. The generated BizTalk receive location also looks similar to the SOAP version, but the WSE transport configuration dialog looks a bit different because it exposes a few more security-related configuration options.

On the send side, things are a little simpler than with the SOAP adapter. With WSE 2.0 send ports, you don't need to specify a Web service proxy class anymore. You simply specify the Web service URL and the appropriate policy configuration; the adapter takes care of producing and sending the appropriate message automatically.

When using the WSE 2.0 adapter to consume Web services within orchestrations, you no longer use the Add Web Reference command either. Instead, you use the Add Generated Items command and select the Add Adapter Metadata option. When the Add Adapter Wizard appears, select WSE from the list of registered adapters and click Next. Then the BizTalk Adapter for WSE Schema Generation Wizard appears.

The BizTalk Adapter for WSE Schema Generation Wizard asks you for the address to the WSDL definition, which it downloads, and to retrieve a list of possible services and operations. From this list, it asks you to choose a service and a set of methods. Then it asks you to configure the WSE 2.0 policy file needed to communicate with the service. You can either download the policy from a URL (if available) or you can configure it manually using the policy configuration dialog shown in Figure 9.

As with Add Web Reference, the wizard adds the schemas and port types you'll need to talk to the service. It also generates a WSE 2.0 policy file named PolicyCache.xml and saves it to the Policy directory within the WSE 2.0 adapter installation directory (see %PROGRAMFILES%\Microsoft BizTalk Server Adapter for WSE 2.0\Policy). This is where BizTalk looks for all WSE 2.0 policies. When you install the WSE 2.0 adapter, it actually updates BTSNTSvc.exe.config with a <policy> element referencing this global PolicyCache.xml file.

The WSE 2.0 adapter also introduces a variety of new message properties that provide access to WSE-specific information such as WSE.SoapAction, WSE.SoapHeaders, WSE.AddressingHeaders, WSE.ReferralHeaders, WSE.SecurityHeaders, and WSE.ClientCertificate. You can take a look at the WSE 2.0 adapter documentation for further information.

To summarize, the main new features offered by the WSE 2.0 adapter are support for message-based security policies and support for content-based routing in messaging-only solutions where you're not using orchestrations.

BizTalk Resource
  • Understanding BizTalk Server 2006
  • BizTalk Web Services FAQ
  • BizTalk Server 2004 and Web Services
  • Implementing Service Contracts with BizTalk Server 2004
  • Walkthrough for BizTalk Adapter for WSE 2.0

The WCF Adapter

The BizTalk Server 2006 R2 release will ship with several Windows Communication Foundation adapters that will collectively provide full support for WS-* and transport-neutral messaging. There will actually be a separate adapter for many of the built-in Windows Communication Foundation bindings such as BasicHttpBinding, WSHttpBinding, NetTcpBinding, NetNamedPipeBinding, and NetMsmqBinding. There will also be a custom Windows Communication Foundation adapter that will allow you to fully configure its binding. Although this custom adapter will be the only one you really need, the other bindings are much easier to use when targeting a common messaging scenario.

The custom adapter will make it possible to do what you can do today with the SOAP and WSE adapters plus much more. It will provide full support for the WS-* stack including the various security, reliable messaging, and transaction specifications. The support for these specifications will allow you to handle more sophisticated security and routing scenarios, and you'll even be able to include the message box in Web service transactions.

Moreover, it will also offer simplified hosting, communication, and configuration options. For example, with the Windows Communication Foundation adapter, you'll be able to host non-HTTP services within BTSNtSvc.exe by simply configuring a receive location and enabling it within the BizTalk administration console (no publishing necessary).

But perhaps the most compelling feature is unification. Specifically, the Windows Communication Foundation adapter will unify the features provided by the SOAP and WSE adapters, thereby removing the need to use them. In the future, you should only need the various Windows Communication Foundation adapters when integrating with a variety of different service configurations. For more information on Windows Communication Foundation and its interoperability with ASP.NET and WSE Web services, see the Windows SDK at msdn2.microsoft.com/ms735119.aspx.

Summary

BizTalk Server 2006 provides support for today's Web services technologies through the built-in SOAP adapter and the WSE adapters discussed previously. (For more information on BizTalk, see the "BizTalk Resources" sidebar.) The SOAP adapter offers BP 1.1 communications while the WSE adapters offer more advanced WS-* support. However, the promise of transport-neutral WS-* messaging will not be fully realized until the Windows Communication Foundation adapter ships with BizTalk Server 2006, which is just around the corner. These are just the Windows Communication Foundation adapter highlights. I promise to provide in-depth coverage in a future column, closer to the first public beta release.

Send your questions and comments for Aaron to sstation@microsoft.com.

Aaron Skonnard is a cofounder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight's Applied Web Services 2.0, Applied BizTalk Server 2006, and Introducing Windows Communication Foundation courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at pluralsight.com/aaron.


Bugslayer: GUI Control to Major Tom

Bugslayer
GUI Control to Major Tom
John Robbins

Code download available at: Bugslayer2007_03.exe (199KB)
Contents
Microsoft UI Automation
UISpy
Better UI Automation Testing
The UIWindow Class
The UIMenuBar Class
Wrap-Up
Tip

In my 14 years of Windows development, the absolute best demo I've ever seen was at the Longhorn Professional Developer's Conference (PDC) in 2003. While sessions on Windows® Presentation Foundation and the Visual Studio® 2005 betas excited many attendees, the program manager for Active Accessibility® got the only standing ovation that I've ever seen at a technical conference. He demonstrated a build of "Longhorn," which is now Windows Vista™, that allowed separate input queues for a GUI application. His demo was playing Solitaire in the foreground while a Microsoft® .NET-based application using the underlying automation APIs from Active Accessibility added and removed hardware devices from Device Manager in the background! He even clicked around in the Device Manager and all input was ignored because the controller program had complete control of that input queue.

The problem in Windows development is that when you attempt to automate a GUI application with a mouse and keyboard playback utility, such as the Tester application I developed in previous Bugslayer columns, simply bumping the mouse at the wrong time can break your automation. What was so exciting about the PDC demo was the realization of the dream of being able to automate the GUI portions of your application plus the guarantee that the playback would be exactly what you expected. If you're writing any sort of Windows-based application, you've got some part of it that has a GUI, be it the whole application, the install, or the latest in AJAX buzzword compliance. Being able to test the GUI portion of the application consistently and correctly is a key to great code.

As I was working on this column, Windows Vista was released to manufacturing. Sadly, the multiple input queues feature that garnered such great applause didn't make it to the final release. However, a good portion of the GUI automation did make it into Active Accessibility. Additionally, the new GUI automation portions are available through the .NET Framework 3.0, so it not only works on Windows Vista, but also Windows XP and Windows Server® 2003. Moreover, the Active Accessibility code works seamlessly with both Windows Presentation Foundation UIs and HWND-based applications. Perhaps the next operating system release will reach the ultimate goal of separate input queues, but with the new Microsoft UI Automation, getting the GUI portions of your applications automated is now quite a bit easier-and you can use it today.

My goal for this column is to show you how to get started with GUI automation using the UI Automation tools and APIs. As with most Bugslayer columns, there's a large set of code that I provide to make your GUI testing far easier than using the API directly. I'll assume that you've read over the documentation from msdn2.microsoft.com/ms747327.aspx; if you haven't, you should do it now or the rest of the column probably won't make much sense. As a reminder, you'll need the .NET Framework 3.0 installed as well as the Windows Vista and .NET Framework 3.0 Software Development Kit.

Microsoft UI Automation

As you read through the documentation for Microsoft UI Automation, you'll see the roots of life are truly in Active Accessibility. Much of the discussion centers around finding all the particular child controls in a window. As screen readers and other assistance tools need that information quickly, it's nice to have an API to easily get those trees for both HWND-based as well as Windows Presentation Foundation applications without having to write two separate sets of code on your own.

There's also an excellent discussion on how to write your own providers, which serve up your custom controls through the Microsoft UI Automation API in a uniform manner. If you're writing your own unique custom controls, you'll definitely want to read the UI Automation Providers Overview (msdn2.microsoft.com/ms750446.aspx) so anyone using assistive technology can use your application. The work necessary to create a provider is not too onerous, but it's outside the scope of what I want to cover for automated testing in this column.

One part of the Microsoft UI Automation API that's very interesting is its excellent event support. While you can install a journal hook to record keyboard and mouse usage on a machine, there's no clean API to get notifications about which control has focus, when windows open and close, and so on. If you look back at the code for my Tester application, you can see that I was using Computer Based Training (CBT) hooks and a good bit of guessing to get those notifications for HWND-based applications. What I find most impressive about the new eventing system is that you can isolate the event notifications so that you can get everything from a single control, to a window with child controls, to the whole desktop. For an example of using the eventing controls to record an application's usage, the Test Script Generator Sample is a big help (msdn2.microsoft.com/ms771275.aspx).

A key point of the whole Microsoft UI Automation API is that everything is based around a control type. Instances of the AutomationElement class wrap each control, which allows you to quickly identify what a particular control on the screen is and what it can do. In the past, it took a commercial application to provide that level of detail, and if you've ever had to deal with any of those expensive and custom automated testing tools in the past, you'll realize how nice it is to have everything in a free download.

There are 38 defined control types, covering everything from a button to a data grid (msdn2.microsoft.com/ms749005.aspx). What makes the provider model interesting from an automation testing standpoint is that these control types show exactly what properties and control patterns each supports. As you can guess, the properties are the subset of properties on the base AutomationElement class that are required or optional for that type of control. For example, a button control should support the AcceleratorKeyProperty to return the accelerator key used for that button.

You'll notice that the control patterns are a bit more interesting. The UI Automation Control Patterns Overview documentation (msdn2.microsoft.com/ms752362.aspx) sums it up best: "Control patterns provide a way to categorize and expose a control's functionality independent of the control type or the appearance of the control." As you'll see later in the column, we'll work a lot with control patterns in automating a GUI. What makes the control patterns so nice is that once you get the particular pattern for the control, the pattern will do the work of performing that action. For example, if you're dealing with a button, the IInvokePattern interface exposes an Invoke method that will do the button click. That way you don't have to wrestle with Windows messages or the SendInput API.

While there's not much in the documentation on using the API for automation testing, between the samples and a bit of reading, it's not too hard to see what's necessary. I do have to give the team who designed the UI Automation API some major credit for their work on internationalization. With previous tools and APIs for automating, you were forced to embed strings like "File" and "Open" to look for menu items. This meant that you had to provide all the internationalization inside your GUI automation scripts, which complicated your life and reduced the reusability of your testing scripts. Today, when you want to access a particular control in your application, you can use the AutomationId property. This name is unique and guaranteed to be the same on any machine no matter the locale. Now you can reuse those scripts and help your translators immensely because they can test your applications just as you did.

One UI testing feature missing from the UI Automation API is mouse support. There's none at all. For most applications, that's not much of a problem because the control patterns provide the equivalent functionality. However, if you're working on a painting application, you won't find any help to draw those smiley faces with a mouse. In that case, you can still use something like Tester, which does offer mouse support, in conjunction with a UI Automation API-using program to get you to the point where you need to rely on the mouse.

UISpy

Before I jump into the cool code I wrote to make your UI testing quite easy, I need to talk about a tool that you're going to use extensively to assist you in writing your tests: UISpy (shown in Figure 1). You can find UISpy in the Windows Vista and .NET Framework 3.0 Software Development Kit. Essentially, UISpy is a tool to walk and show in the Microsoft Automation UI format all the controls from the desktop on down so you can start seeing what the properties are for your particular controls. Additionally, as you play around with the Microsoft Automation UI API, you can execute patterns on controls as well as query for particular properties.

Figure 1 UISpy User Interface
Figure 1 UISpy User Interface(Click the image for a smaller view)
Figure 1 UISpy User Interface
Figure 1 UISpy User Interface (Click the image for a larger view)

Simply using the tool and clicking through the UISpy menus will teach you how everything works. However, I've spent a huge amount of quality time with UISpy getting the code that accompanies this month's column working correctly, and to save you a ton of hassles, I want to discuss a few of the snags I ran into.

The tree controls in UISpy show you the hierarchy for a particular view: Raw, Control, or Content. The Raw View shows the hierarchy most closely related to the particular native programmatic structure, and is one you'll rarely use. The Control View shows the hierarchy as it maps to what the user perceives when looking at the user interface. The Content View, which is a subset of the Control View, shows just the controls that have true information in a user interface. When using the Microsoft Automation UI API, you'll access the controls in your application using one of these hierarchies.

In UISpy, there's appears to be a bug in the Control View tree control: UISpy does not always update the selection when you right-click on a node. As the right-click menu is extremely important for refreshing the view, setting focus, or accessing control patterns, you'll find yourself executing the menu options on the previous selected node. Whenever you want to access the right-click menu, always select the item first with the mouse, and then right-click on it. I don't know how many times I was looking at the control patterns for the wrong control!

The documentation for the Microsoft Automation UI API says in several places that using the root element, the Desktop Window, as the start of your searches is extremely slow. Trust the documentation on this. Whenever you're using UISpy to look at your application, immediately select and right-click the child node for your application and select Scope to Element. That will isolate all focus tracking and highlighting to only the application you're interested in using.

One great feature of UISpy is the Hovering mode, which allows you to move the mouse over a control on the screen and, when you press the Ctrl key, jump to that node in all the open tree controls. The Hovering mode is especially important when you're looking at menu items because until a menu is expanded you won't see the child nodes. The problem is that UISpy doesn't always realize that the Ctrl key has been pressed, so get in the habit of pressing the Ctrl key multiple times.

The last, and worst, issue with UISpy isn't UISpy's fault, but it's something that will drive you bonkers when trying to figure out how to automate portions of the user interface you don't own. A perfect example is the new common Open dialog on Windows Vista. Looking at the dialog, most of the controls look like typical HWND-based edit boxes, buttons, list controls, and so on. But don't let your eyes deceive you. What looks like the Open button is sometimes reported in UISpy as a button control and sometimes as a pane control, which, according to the documentation, "represents a level of grouping lower than windows or documents, but above individual controls."

If you don't own the user interface you're trying to automate you'll need to inspect it with UISpy numerous times to see exactly what the control type really is. Between using the Hovering mode and the full Focus Tracking mode, which jumps to the control with focus in the UISpy tree views, you'll eventually find the exact type of those mystery controls.

Better UI Automation Testing

Once I got a handle on using UISpy and started getting a feel for the basic API, I began using the Microsoft UI Automation API to drive everyone's favorite application, Notepad. My plan was to learn about the API so I could show how to integrate automation testing into the wonderful unit-testing tool in Visual Studio Team System. You can use NUnit (nunit.org) to do essentially the same thing.

Using the straight Microsoft UI Automation API, I wrote an application that did the following:

  • Ensured the process was DPI aware
  • Started Notepad and executed the File | Open menu
  • In the Open dialog, opened the source code for the program
  • Executed the Help | About menu and clicked the OK button
  • Executed the File | Exit menu

You can see part of this program in Figure 2, and I strongly suggest you read down all the code and comments as it will give you an excellent idea how to use the Microsoft UI Automation API. Please note that there is no error handling in the code to focus strictly on the core API usage.

Figure 2 Raw Microsoft UI Automation Usage (Close Figure 2)
#region Using Statements
using System;
using System.Diagnostics;
using System.Collections.Generic;
using System.Text;
using System.Windows.Automation;
using System.Windows.Automation.Provider;
using System.Threading;
using System.ComponentModel;
using System.IO;
using System.Globalization;
using System.Windows.Forms;
using System.Runtime.InteropServices;
#endregion

namespace RawUsageExample
{
    internal static class NativeMethods
    {
        // The Windows Vista DPI aware function.
        [DllImport ( "user32.dll" , EntryPoint = "SetProcessDPIAware" )]
        [return: MarshalAs ( UnmanagedType.Bool )]
        private static extern bool RealSetProcessDPIAware ( );

        internal static bool SetProcessDPIAware ( )
        {
            Boolean returnValue = false;
            try
            {
                returnValue = RealSetProcessDPIAware ( );
            }
            catch ( EntryPointNotFoundException )
            {
                // Not running on Windows Vista.
            }
            return ( returnValue );
        }
    }

    class Program
    {
        [STAThread]
        static void Main ( )
        {
            // Set this process to DPI aware if running on Windows Vista.
            NativeMethods.SetProcessDPIAware ( );

            // Start Notepad and connect up to it.
            // Wait for the process to get going enough to have a UI.
            Process proc = Process.Start ( "notepad" );
            Thread.Sleep ( 1000 );

            // Attach to the main window.
            AutomationElement rootElement =
                AutomationElement.FromHandle ( proc.MainWindowHandle );

            // Create the AndCondition to find the menuBar.
            AndCondition menuBarFind = new AndCondition (
                new PropertyCondition (
                    AutomationElement.ControlTypeProperty,
                    ControlType.MenuBar ) ,
                new PropertyCondition (
                    AutomationElement.AutomationIdProperty, "MenuBar" ) ,
                Automation.ControlViewCondition );

            // Find the menuBar, which is a child of the main window.
            AutomationElement menuBarElement =
                rootElement.FindFirst ( TreeScope.Children, menuBarFind );

            // Create the AndCondition to find the File menu.
            AndCondition fileMenuFind = new AndCondition (
                 new PropertyCondition (
                     AutomationElement.ControlTypeProperty ,
                     ControlType.MenuItem ) ,
                 new PropertyCondition (
                     AutomationElement.AutomationIdProperty , "Item 1" ) ,
                 Automation.ControlViewCondition );

            // Find the File menu.
            AutomationElement fileMenuElement =
                menuBarElement.FindFirst ( TreeScope.Children ,
                fileMenuFind );

            // Get the control pattern for ExpandCollapse and do the
            // Expand to get the children.
            ExpandCollapsePattern fileExpandPattern = fileMenuElement.
                GetCurrentPattern ( ExpandCollapsePattern.Pattern )
                    as ExpandCollapsePattern;
            fileExpandPattern.Expand ( );
            Thread.Sleep ( 1000 );

            // Create the AndCondition to find the Open menu.
            AndCondition openMenuFind = new AndCondition (
                 new PropertyCondition (
                     AutomationElement.ControlTypeProperty ,
                     ControlType.MenuItem ) ,
                 new PropertyCondition (
                     AutomationElement.AutomationIdProperty , "Item 2" ) ,
                     Automation.ControlViewCondition );

            // Find the Open menu.
            AutomationElement openMenuElement =
                fileMenuElement.FindFirst ( TreeScope.Descendants,
                    openMenuFind );

            // Get the InvokePattern off the openMenu.
            InvokePattern openInvokePattern =
                openMenuElement.GetCurrentPattern (
                    InvokePattern.Pattern ) as InvokePattern;

            // Invoke the menu to get the Open dialog.
            openInvokePattern.Invoke ( );
            Thread.Sleep ( 1000 );

            ...

As you read through the code in Figure 2, your eyes will probably glaze over because, while the Microsoft UI Automation API gives you a great deal of power, it has to be one of the most intensively fiddly APIs ever developed. Most of the code in Figure 2 simply opens the File | Open menu. Using the Microsoft UI Automation API directly means you have to do the slightly different 5 to 10 lines of code over and over (and over and over!). In fact, the full code takes 250 lines to perform the steps in the five bullet points I discussed earlier. The problem with APIs like this is that a developer will resort to "editor inheritance" (also known as cut and paste) to build up their unit tests, resulting in all sorts of bugs in the code. In turn, this will lead to the discontinued use of the API.

While I can see what the API designers were trying to accomplish, the big mistake, is that it's a flat API; all you're working with is a single class, AutomationElement, for every single control type. This leads to seven lines of code to find the File menu, find the ExpandCollapsePattern, perform the actual expand, and give the GUI time to show the menu. Sometimes when you attempt infinite flexibility, you produce APIs that are harder to use than they really need to be.

Right after my second attempt to learn the Microsoft UI Automation API, I simply had to work at encapsulating each of the 38 control types in their own classes so you would be able to figure out the exact type of control you are working with. Plus I wanted to hide as much of the continual grunt work in the raw API as possible, but still provide access to the underlying API for any advance work you need to perform. The code in Figure 3 shows the same program in Figure 2, but using the Bugslayer.TestTools.GuiAutomation namespace code. Again, I don't have any error checking to keep the focus on the code.

Figure 3 Bugslayer.TestTools.GuiAutomation Wrappers (Close Figure 3)
#region Using Statements
using System;
using System.Diagnostics;
using System.Collections.Generic;
using System.Text;
using System.Windows.Automation;
using System.Windows.Automation.Provider;
using System.Threading;
using System.ComponentModel;
using System.IO;
using System.Globalization;
using System.Windows.Forms;
using Bugslayer.TestTools.GuiAutomation;
#endregion

namespace UIUsageExample
{
    class Program
    {
        [STAThread]
        static void Main ( )
        {
            // Start Notepad and connect up to it.
            // Wait for the process to get going enough to have a UI.
            Process proc = Process.Start ( "notepad" );
            Thread.Sleep ( 1000 );
            UIWindow notepadWindow = new UIWindow ( proc );

            // Now we're on the main window so execute the
            // File Open command.
            UIMenuBar menuBar = notepadWindow.FindChild (
                "MenuBar" , ControlType.MenuBar ) as UIMenuBar ;

            // This wraps all the goo necessary to invoke the Open menu.
            menuBar.InvokeMenus ( "Item 1" , "Item 2" );

            ...

The code in Figure 3, as you might notice, is much shorter than Figure 2. In fact, the full program is only 81 lines long, which is 32 percent of the original code! I'm also willing to bet that you'll find that the code using Bugslayer.TestTools.GuiAutomation is far easier to read and understand. And, of course, if the code is easy to use, you're more likely to use it.

The UIWindow Class

At this point, you may want to download the code for this month's Bugslayer to get a good look. You can follow along because I want to show you some of the implementation details. If you have Visual Studio Team Developer Edition or Visual Studio Team Suite, open the Bugslayer.TestTools.GuiAutomation\Bugslayer.TestTools.GuiAutomation.SLN file to see both the code and the unit tests. If you only have the Visual Studio Professional Edition, you'll want to open UIUsageExample\UIUsageExample.SLN so you won't get errors about missing projects and references. Both solution files include the Bugslayer.TestTools.GuiAutomation.CSPROJ file, which contains all the core code.

The class you'll want to examine is UIWindow, as that's the basis of all life for Bugslayer.TestTools.GuiAutomation. It wraps the AutomationElement class, which is exposed as the RootElement property. Nearly all the actual functionality for the derived classes is found in UIWindow. For example, all the most common properties you'll access on AutomationElements such as AutomationId, Name, IsEnabled, and so on, are there.

The child control finding methods are also on UIWindow. In thinking about how developers would use my API, I felt that the two main searches would be for a direct child control and occasionally for descendent controls. Consequently, I have numerous FindChild/FindDescendent methods that take the AutomationId to look up. Additional overloads let you also specify the exact ControlType so you can be sure you're finding the exact control you expect.

The AutomationId values are excellent for internationalization, but you do have to be very careful when searching down descendent trees because the values get reused. For example, the File menu on a menu bar has a value of "Item 1," but expanding the File menu, the Open menu item will also have a value of "Item 1."

In addition to searching by AutomationId values, I also support searching by the localized name of the control with the FindChildByName and FindDescendentByName methods. No matter what you search on, my Find* methods are wrappers around AutomationElement.FindFirst method so they only return the first item matching the specific condition. I chose to do that because when driving a GUI application, you know in advance which controls are in the user interface so you're generally not wondering what controls are on the screen. If you do need to call the AutomationElement.FindAll method to return all the matching controls for validation purposes, grab the UIWindow.RootElement property to search and sort all you want.

If you read about finding controls in the documentation, you saw there was a big discussion of searching in RawViews, ControlViews, and ContentViews. The UIWindow. Find* methods default to searching on the Automation.ControlViewCondition. If you want to change the searching for a particular UIWindow, set the TreeSearchCondition to Automation.ContentViewCondition, Automation.ControlViewCondition, or Automation.RawViewCondition.

The final point I want to mention about the Find* methods is that you may want to read through the UIWindow code, because it will show how I refactored the code down to some very small methods and how to think about finding controls in the Microsoft UI Automation API.

The big work in the UIWindow class is to provide protected wrappers around the control patterns. The 38 different control types use a number of the control patterns and I wanted to bury all the details of accessing and using the patterns in simple wrapper properties and methods. For example, a menu item control type may implement the ExpandCollapsePattern. My code has a UIMenuItem class that has a SupportsExpandCollapsePattern that calls the protected static UIWindow.ImplementsExpandCollapse method. The UIMenuItem class has public methods and properties for the rest of the ExpandCollapsePattern, ExpandCollapseState, Expand, and Collapse that you can call if SupportsExpandCollapsePattern returns true.

By implementing the control pattern code all in UIWindow, other control types, such as UIComboBoxControl (which also may support the ExpandCollapsePattern), just need to implement the public methods for the pattern. My idea is to expose only the methods and properties those classes support so it's much easier to use the API.

The big challenge that I faced when designing Bugslayer.TestTools.GuiAutomation was figuring out how to have the UIWindow.Find* methods return the exact type. If you look at UIWindow.FindDescendant for example, it returns a UIWindow. I wanted my actual automation test code to find a menu item and get back the UIMenuItem class. That meant I was going to have to give the UIWindow class a priori knowledge of the derived classes. While this idea was not exactly pure in the object-oriented sense of the word, it definitely works and makes Bugslayer.TestTools.GuiAutomation far easier to use. If I didn't do this, you'd have to write code like this:

UIWindow rawMenuBarWindow = notepadWindow.FindChild (
    "MenuBar" , ConwwtrolType.MenuBar );
UIMenuBar menuBarWindow = new MenuBar ( rawMenuBarWindow.RootElement );

Whenever the UIWindow methods need to create a new UIWindow, it calls the UIWindow.AllocateExactType method, which takes an AutomationElement as the only parameter. Inside AutomationElement, AllocateExactType looks at the ControlType property and allocates the appropriate derived class. By doing the exact class type allocation, you can write code like the following using the as operator in all its glory:

UIMenuBar menuBar = notepadWindow.FindChild (
    "MenuBar" ,  ControlType.MenuBar ) as UIMenuBar ;

If you add additional control types to Bugslayer.TestTools.GuiAutomation, you must make sure to add the appropriate allocation in AllocateExactType. I'll trade API ease of use over object-oriented purity any day.

One slightly interesting point concerns entering text. If you read through the control types documentation, you'll see that only a few control types support the TextPattern and ValuePattern necessary for getting and setting text. What I decided for those particular controls was to make the UITextControl class, which models a static text control, serve as the base class for all controls that support TextPattern and ValuePattern. That way the functionality is shared and I can build up a hierarchy where UIEditControl and UIToolTopControl are both derived from UITextControl. In addition, UIDocumentControl is derived from UIEditControl. This supports code reuse and also ensures that the correct ControlType is being used with the appropriate class.

In the Microsoft UI Automation API, any control that allows for multiple line input is considered a ControlType.Document. The documentation says that document controls are never supposed to support the ValuePattern, which means there's no way to set the value of those controls. (The TextProvider will give you the data from a document control.) This makes sense, as every program that supports data more complicated than plain text would be extremely hard to model. However, as sometimes just pumping text into a document is useful, I wrapped the SendKeys class from System.Windows.Forms into UIWindow. I also put the smarts in the class to ensure the focus is set to the modeled window before performing the actual keyboard input.

The UIMenuBar Class

The last interesting helper I want to discuss is the UIMenuBar class. When using the Microsoft UI Automation API, executing normal dropdown menus involves finding the menu, expanding it, finding the appropriate child, and invoking it. You can see how tedious the work back in Figure 2 was simply to execute the File | Open menu. Seeing that working with menus is so painful, the UIMenuBar.InvokeMenus command can take variable-length string parameters and perform the common work in a single call. The first string is assumed to be the initial menu that's a child of the menu bar and all subsequent parameters, except for the last, are menus to be expanded. The last menu is generally the one with the InvokePattern so UIMenuBar.InvokeMenus will invoke it. As you can see from Figure 3, it certainly cuts down on a huge amount of code you would normally have to write.

Wrap-Up

Now that you have what I think is an easy-to-use library in Bugslayer.TestTools.GuiAutomation, I hope you'll look hard at getting your user interface automation tied into your unit tests. Like anything else involving testing code, you'll need to treat it as a deliverable like the product, but the payoff is huge because code quality goes through the roof!

If you're looking for other ideas on how to use Bugslayer.TestTools.GuiAutomation, make sure to check out the unit tests in the Bugslayer.TestTools.GuiAutomation\Tests\Bugslayer.TestTools.GuiAutomation.Tests directory. Those tests use all aspects of the library in order to test it. As a huge believer in code coverage, I'm proud to report the existing unit tests hit over 90 percent of the code (on Windows XP)! If you have the appropriate version of Visual Studio, run the tests because it's wild to see five or six different instances of Notepad being automated at the same time (the tests run in random order and can thus be interleaved).

I want to leave you with a couple of ideas on how to extend Bugslayer.TestTools.GuiAutomation. As you look at the code, you'll see that I haven't modeled all the control types. Add the rest of the control types following the development patterns I've established. If you do add additional control types, make sure to add the appropriate unit tests.

The lack of mouse support in the Microsoft UI Automation API is a problem for many applications. Consider adding a PlayMouse method to UIWindow that takes a string in the SendKeys format and performs the operations using the SendInput API. You can look at how I implemented the PlayKeys command in Tester for help with building and parsing and the SendInput array.

If you were really motivated, you could build a recorder that would watch your interaction with an application using the excellent eventing support in the Microsoft UI Automation API. From the interaction, you can create a C# or Visual Basic® source file ready for compilation and testing.

If you do any of the above, please let me know, as I'll be happy to coordinate the work on Bugslayer.TestTools.GuiAutomation.

Tip

Tip 77 I love Team Foundation System, but was faced with a major problem the other day. I had to move work items from one project to another. Just as I was about to start crying, thinking I was going to have to do each item manually, Eric Lee came to the rescue with his Hemi tool, which moves work items between projects via a few mouse clicks. See how to use Hemi and download this wonderful free tool at blogs.msdn.com/ericlee/archive/2006/11/20/work-item-moving-tool-is-back.aspx.

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in both .NET and Windows. His latest book is Debugging Microsoft .NET Applications (Microsoft Press, 2006). You can contact John at www.wintellect.com.


Foundations: Tracking Services in Windows Workflow Foundation

Foundations
Tracking Services in Windows Workflow Foundation
Matt Milner

Code download available at: Foundations2007_03.exe (178KB)
Contents
Tracking Architecture
Tracking Profiles
SQL Tracking Service
Querying Tracking Data
Custom Tracking Services

One of the most powerful features in Windows Workflow Foundation is tracking. It allows you to monitor events, activity properties, and custom data in your workflows. In this column, I will examine the tracking infrastructure, showing you how to use the built-in SQL Server™-based tracking service and how to create custom tracking services for a variety of purposes. Along the way, I'll demonstrate how to consume information that is being tracked and how to solve some common requirements of applications through the use of tracking.

Many applications require insight into the execution of program logic and processing steps. This could be for regulatory compliance, managing task lists based on execution milestones, or a variety of other reasons. Windows® Workflow Foundation provides a flexible infrastructure where you can overlay your custom implementations, rather than creating a different tracking system for each application. This simplifies the development model, allowing you to focus on the business requirements for tracking.

Tracking Architecture

When hosting a workflow in Windows Workflow Foundation, you host both a workflow runtime and a set of runtime services. Many runtime services, such as the persistence service, are required to have at most one instance registered in the runtime. Tracking services are the exception, allowing for multiple tracking services to be added to the runtime in order to support different business requirements. This allows you to apply several different tracking services to one application, with each tracking service responsible for a specific implementation.

Tracking in Windows Workflow Foundation is based on a central interceptor, which monitors activity and workflow events and sends the appropriate data to each of the configured tracking services. (This architecture is illustrated in Figure 1.) The tracking interceptor receives notification about all events, but in most cases only a subset of these events will be useful. Moreover, if you are using multiple tracking services, it is likely that each service will be interested in a unique subset of information. Thus, the tracking interceptor queries each configured tracking service for a tracking profile for each workflow type.

Figure 1 Tracking Architecture
Figure 1 Tracking Architecture(Click the image for a smaller view)
Figure 1 Tracking Architecture
Figure 1 Tracking Architecture (Click the image for a larger view)

Tracking profiles provide information to the interceptor about the types of events the tracking service wants to receive. These events can fall into three categories: workflow, activity, and user.

Workflow events allow you to track the various states that a workflow can be in and the time at which they occurred. For events that include custom event arguments, such as with WorkflowTerminated or WorkflowCompleted, the event arguments themselves are tracked and can later be queried.

Activity events allow you to track when an activity entered a particular state and the specific values of properties on the activity when it entered that state. For example, if you have an activity that sends e-mail, you can track the To address property when the activity enters the Closed state.

Finally, user events let you track data manually in the code of your workflow or activities.

Tracking Profiles

Each tracking service must provide tracking profiles to the tracking interceptor. A service can provide the same profile for all workflows or different profiles for different workflow types. It is up to the service author to determine how to manage and produce profiles.

Tracking profiles can be created with an object model found in the System.Workflow.Runtime.Tracking namespace or directly in XML. They are usually created using the object model, but this is not the only way to represent the profile, nor is it always the best way. For this reason, the tracking profile can be serialized to and deserialized from XML. You can use the TrackingProfileSerializer class to convert a TrackingProfile object into XML text and vice versa. A useful example of this, and a good tool for basic tracking profile editing, is the Tracking Profile Designer sample in the Windows SDK. This app allows you to visually define the tracking profile using the workflow design surface. While the tool has some limitations, such as lack of support for user tracking points, it does let you view the serialized form of the tracking profile. It can also speed the development and deployment of tracking profiles in many scenarios.

A tracking profile is essentially a collection of track points. Each track point correlates to the types of events that can be tracked, providing information to the runtime about when to extract data and what data to extract. To indicate that workflow events should be tracked, for example, a WorkflowTrackPoint must to be added to the tracking profile. A track point contains one or more tracking locations, which actually define where and when to track the data. For instance, on a WorkflowTrackPoint, the MatchingLocation property identifies the various workflow events to track. The code in Figure 2 shows a tracking profile being created. It has a single WorkflowTrackPoint, which indicates that the service wants to track failure conditions, such as exceptions, terminations, and abortions of workflows.

Figure 2 Workflow TrackPoint (Close Figure 2)
//create the profile object and set the version
TrackingProfile profile = new TrackingProfile();
profile.Version = new Version(1, 0, 0);

//define a tracking point
WorkflowTrackPoint wfTrack = new WorkflowTrackPoint();

//create a list of events we are interested in
List<TrackingWorkflowEvent> events = new List<TrackingWorkflowEvent>();
events.Add(TrackingWorkflowEvent.Aborted);
events.Add(TrackingWorkflowEvent.Terminated);
events.Add(TrackingWorkflowEvent.Exception);

//create a tracking location to indicate which events should be tracked
WorkflowTrackingLocation wfLocation =
    new WorkflowTrackingLocation(events);

//set the matching location on the track point
wfTrack.MatchingLocation = wfLocation;

//add the track point to the profile
profile.WorkflowTrackPoints.Add(wfTrack);

ActivityTrackPoint supports much richer tracking. This is logical since activities make up the heart of workflows. The ActivityTrackPoint contains both MatchingLocations and ExcludedLocations collections, allowing you to closely control when data is collected. For both collections, you define an ActivityTrackingLocation that represents the set of conditions you want to match.

The ActivityTrackingLocation class allows you to define the activities you want to track based on their type. You can also indicate whether you want to match derived activities. For example, if you want to match all activities, you would use the type System.ComponentModel.Activity and set MatchDerivedTypes to True. This would match the base activity class and all activities that derive from it, meaning everything. You then add this information into the MatchingLocations collection of the ActivityTrackPoint to indicate that you want to track these events or into the ExcludedLocations collection to indicate that this data should not be collected.

For more granular control, you can specify conditions, in the form of property equality, that must be met in order to track the data. For example, you might choose to track e-mail messages only when the priority property on the SendEmail activity is set to High. Figure 3 shows an example of this.

Figure 3 ActivityTrackPoint (Close Figure 3)
ActivityTrackPoint actTrack = new ActivityTrackPoint();

ActivityTrackingLocation actMatch = new ActivityTrackingLocation(
    typeof(SendEmail), false, new ActivityExecutionStatus[] {
        ActivityExecutionStatus.Closed });

ActivityTrackingCondition condition =
    new ActivityTrackingCondition("Priority", "High");
actMatch.Conditions.Add(condition);

actTrack.MatchingLocations.Add(actMatch);

profile.ActivityTrackPoints.Add(actTrack);

You can also extract data from the activity or workflow in the form of property values. For example, you can track the To and Subject properties on the SendEmail activity when your matching locations are reached in the workflow. You do this by adding ActivityDataTrackingExtract or WorkflowDataTrackingExtract items into the Extracts collection on the ActivityTrackPoint. Each ActivityDataTrackingExtract identifies the member to be extracted and sent to the tracking service. Adding data extracts to the activity track point so that the To and Subject properties tracked look like this:

actTrack.Extracts.Add(new ActivityDataTrackingExtract("To"));
actTrack.Extracts.Add(new ActivityDataTrackingExtract("Subject"));

UserTrackPoints, which identify the final point of tracking, are the most flexible of the three types. Sometimes you need to track data from your workflow but the data is not available as a property on an activity. Or you may need to use logic beyond the simple condition support in an ActivityTrackingLocation. In such cases, you can call the TrackData method on the ActivityExecutionContext or the Activity class. This lets you send data to the tracking interceptor along with an optional key or name for the data item. For a tracking service to receive this data, it must define a UserTrackPoint with the correct matching locations. You can also define workflow and activity data extracts to have properties on the workflow or the current activity tracked when the UserTrackPoint is matched.

SQL Tracking Service

Windows Workflow Foundation includes a SQL Server tracking service, which allows you to define tracking profiles for a workflow type and deploy them to SQL Server. The tracking service then uses the profile at run time to configure the tracking interceptor. The SQL Server tracking service includes several compelling features, including profile versioning, profile update notification, and data partitioning.

Tracking profiles are deployed to the SQL tracking service by calling the UpdateTrackingProfile stored procedure and passing in the serialized tracking profile along with type information for the workflow and a version number. You can use this stored procedure to insert the tracking profile specific to your workflow type, but you must use a unique version number each time you update the profile. TrackingProfileSerializer, which I mentioned earlier, lets you get the XML text of your profile so you can pass it to the stored procedure.

If you do not specify a tracking profile for your workflow type, SQL tracking service uses the UseDefaultProfile setting to determine whether it should supply a default profile to the tracking interceptor. The default profile tracks all status events for the workflow and all activities. It also includes matching locations to track any user tracking events that are submitted. If you do not want to have this default behavior, you need to set the UseDefaultProfile property to False when configuring the SQL tracking service. Then only those workflow types with profiles specified will be tracked.

To ensure it is current, the tracking interceptor periodically asks each tracking service for profiles that it has already received. To avoid the repeated loading or creation of tracking profiles, a tracking service can implement the IProfileNotification interface. This defines two events: ProfileUpdated and ProfileRemoved. When a tracking service implements this interface, the tracking interceptor does not query for updated profile information unless one of these events is raised. To support this model, the SQL Server tracking service periodically queries the database (every 60 seconds by default) to see if the tracking profile for a workflow type has been updated or removed. You can configure how often the service queries for updates by setting the ProfileChangeCheckInterval property to a value that represents the milliseconds between polling.

Since tracking data can accumulate quickly in a production environment, the SQL tracking service supports data partitioning. You can configure data to be segmented by day, week, month, or year; the service will create tracking tables for each time period. In addition, you can control whether data is moved on demand (so it can be scheduled for downtime) or automatically (when the workflow completes). To configure this setting, you set the PartitionOnCompletion property on the SQL Server tracking service to true to indicate that each workflow should be moved to the appropriate tracking partition when it completes. To manually partition the data, you call the PartitionCompletedWorkflowInstances stored procedure in the tracking database.

Querying Tracking Data

When you collect data using the SQL tracking service, the data is stored in SQL Server, which means you can use traditional reporting tools to view the data and do analysis. But in some scenarios, you'll want to query the workflow data in your application. Rather than forcing you to write SQL queries and determine the correct relationships in the database, Windows Workflow Foundation includes a tracking query programming interface that allows you to search for tracking information for workflow instances. The information is returned as a class, which provides access to all of the tracked events and data that was specified by your tracking profile.

The SqlTrackingQuery class allows you either to try to load a particular workflow instance's tracking data using the instance ID of the workflow or to query for a list of workflows using a discrete set of parameters. When loading a particular instance, you use the TryGetWorkflow method, which returns a Boolean value indicating whether the request was successful. To query for multiple workflows, you use the GetWorkflows method, which requires a SqlTrackingQueryOptions instance.

The query options allow you to define the parameters to use when searching for workflows in the tracking database. The simplest options include a minimum and maximum date value and the type and status of the workflow. For example, you can configure it to search for all workflows matching your workflow type which have a status of Completed and fall between today and a week ago. This lets you retrieve targeted information about a workflow type and how it turned out.

I already showed you a tracking profile that lets you indicate interest in failure conditions. Using the code shown in Figure 4, you can query the database for all instances of workflows that terminated and output the exception details. Note that the tracking information contains the exception details as the event arguments of the terminated event that were tracked.

Figure 4 Querying for Terminated Workflow Instance (Close Figure 4)
SqlTrackingQueryOptions options = new SqlTrackingQueryOptions();
options.WorkflowStatus = WorkflowStatus.Terminated;

SqlTrackingQuery query = new SqlTrackingQuery(connectionString);
IList<SqlTrackingWorkflowInstance> workflows =
    query.GetWorkflows(options);

foreach (SqlTrackingWorkflowInstance workflow in workflows)
{
    foreach(WorkflowTrackingRecord workflowEvent in
        workflow.WorkflowEvents)
    {
        TrackingWorkflowTerminatedEventArgs args =
            workflowEvent.EventArgs as
                TrackingWorkflowTerminatedEventArgs;
        if (args != null)
        {
            Console.WriteLine(
                "workflow {0} terminated with exception: {1}",
            workflow.WorkflowInstanceId, args.Exception.Message);
        }
    }
}

You can also search based on extracted values. For example, if you have a tracking profile configured to extract the To address of a SendEmail activity, then you can write a query to return the workflow instances where the To address matches a certain value, such as a particular employee or partner. The following code is an example of using TrackingDataItemValue items in the SqlTrackingQueryOptions to refine the search to particular items based on extracted values:

SqlTrackingQueryOptions options = new SqlTrackingQueryOptions();
TrackingDataItemValue item = new TrackingDataItemValue(
    "sendEmail1", "To", "highpriority@example.com");
options.TrackingDataItems.Add(item);

SqlTrackingQuery query = new SqlTrackingQuery(connectionString);
IList<SqlTrackingWorkflowInstance> workflows =
    query.GetWorkflows(options);

If you choose to directly query the SQL Server database for reporting or other needs, you should use the views that are defined in the table rather than using the tables directly. This is a good practice in general, but it is very important here as the views will contain data from all of the tracking partitions that have been created. By using the views, you remove the burden of having to union the data from all of those tables together yourself.

Custom Tracking Services

While tracking data into a SQL Server database is an important use case, and likely the most common, it is not the only scenario for using tracking information. Because it is such a common requirement, Microsoft decided to include this rich service with Windows Workflow Foundation. However, a service such as this, with such wide use, must be as general as possible. This means that it may not be appropriate in many situations. In such cases, you can create a custom tracking service.

There are many requirements in workflow solutions that can be met using a custom tracking service. Here I'll discuss the basics of creating your own tracking service, as well as provide a few examples of custom services that meet common requirements. The important thing to remember is that tracking is your view into the workflow. Once you understand how to configure a tracking service with tracking profiles so you can get the results you need, it is then just a matter of how your custom service consumes or represents that tracked information.

In order to create a custom tracking service, you need to do two things. First, you have to determine how your tracking profiles for your service will be created. In some cases, the profile may be static and unchanging across all workflow types. In that case, you will be able to create your profile in code and simply return it to the runtime when the profile is requested. However, if you need your tracking service to support multiple profiles or versions of profiles, then you need a way to store that information and retrieve it from your tracking service.

Next, you have to implement two classes that provide the runtime behavior of your service: a tracking service and a tracking channel. The tracking service class acts as the workflow runtime service, and it is the direct interface for the host application and the runtime to interact with your tracking service. You will create an instance of this class and add it to the workflow runtime just like any other service, using code or configuration.

The tracking service class has two primary functions: to provide tracking profiles to the tracking interceptor and to provide tracking channels to which the listener can write data. The tracking channel is the class you implement that has the responsibility of receiving and handling the tracking data. It can write that data out to a store, such as SQL Server or the Windows event log; it can raise events, such as Windows Management Instrumentation (WMI) events for operations; or it can send the data back to the tracking service to hold it in memory and make it available to the host application. There are certainly other options, and what you do with the data will depend on your needs and requirements. Your custom tracking services will be differentiated by the tracking profiles they use and how they handle the data that they receive.

The tracking channel is the easiest place to start as it has two simple methods: Send and InstanceCompletedOrTerminated. Your tracking service is asked to supply a tracking channel for each workflow instance and it is common practice to supply a unique instance of your tracking channel for each request. You generally want to provide a constructor that takes TrackingParameters as a parameter-you may need this information in your implementation of the other methods. The TrackingParameters class has information about the workflow instance being tracked (such as the instance ID), access to the root activity definition, and information about parent activities and calling workflows (if this workflow was invoked from another workflow).

The InstanceCompletedOrTerminated method allows your tracking service to get notification when a workflow is complete so that it can clean up any resources related to tracking the workflow. This is a common place to need access to the workflow instance ID as supplied by the TrackingParameters because the InstanceCompletedOrTerminated method does not take any parameters.

The Send method on your tracking channel is called for every tracking record that was requested in the tracking profile. Mimicking the profile, there are workflow, activity, and user tracking records which all derive from the base class TrackingRecord. In the Send method, it is your responsibility to check what type of tracking record has been submitted and then handle it accordingly. If you know that your tracking profile does not indicate an interest in user tracking events, for example, then you do not need to handle user tracking records in your tracking channel.

The tracking service class itself is responsible for managing the tracking profiles. The tracking service base class provides several methods for retrieving the profile that allow you to manage profiles based on workflow type and workflow instance. This means that your service can return profiles that range from being the same for all workflows, to unique for each workflow instance.

There are two overloads for the GetProfile method. One takes an instance ID and the other takes the type and profile version number. Also, TryGetProfile takes the type information. In all of these cases, the job of your custom tracking service is to return a tracking profile that is appropriate for the type or instance.

There is also a method called TryReloadProfile, which the runtime calls to get an updated profile if one exists. The tracking interceptor, wanting to make sure it is up to date, will periodically ask your tracking service for an updated profile. Rather than having to create a new profile each time this request is made, or to look one up, you can implement the IProfileNotification interface in your custom tracking service. The IProfileNotifcation interface includes two events: ProfileRemoved and ProfileUpdated. When you implement this interface in your tracking service, the tracking interceptor no longer polls for updates. Instead, it waits for notification of any updates through the events on this interface.

Included in the code download for this column are several sample tracking services. The current state sample uses tracking to get the current state of a state machine workflow and make it available to the host application. The tracking profile for this service contains an activity track point for the executing status of State activities. The tracking channel then updates a dictionary in the tracking service class with the current state name when an activity tracking record is received. This is an example of a service that has the same profile for all workflows and writes data back to the service so it can be queried by the host application.

The other sample tracking service included in the code download is geared toward solving a common developer problem in Windows Workflow Foundation. Because of the nature of how workflows are executed and separated from the host, it is not a simple procedure to get a reference to the running workflow and extract property values. This is a common requirement, especially in ASP.NET scenarios. Once a ManualWorkflowScheduler has completed executing the steps in the workflow, it may be necessary to update the user interface with property values from the workflow.

While the information in the workflow can be passed out using the CallExternalMethod activity, this is not always the desired solution and can prove to be overwhelming in a large project. The workflow property tracking service sample uses tracking to solve this problem. In this tracking service, the profile returned is different for each type of workflow in order to reflect the different properties on the workflow. Reflection is used to build a collection of workflow data tracking extracts and add them to an activity track point. The location of the tracking is on the closed event of each activity. Therefore, after each activity closes, the property values from the workflow instance are extracted and sent to the tracking channel. The channel then updates a dictionary in the service making those values available to the host application.

These are two fairly simple examples of custom tracking services, but both address common developer scenarios and provide simple, reusable implementations taking advantage of the existing tracking infrastructure. Without the base to build on, each of these challenges would have been much larger, requiring more custom code, and the solutions likely not consistent.

As you can see, the tracking services in Windows Workflow Foundation are extremely flexible and can help in solving a number of different business requirements. Tracking is the key to workflow visibility and custom tracking services are your tool for tapping into that visibility in your applications.

Send your questions and comments to mmnet30@microsoft.com.

Matt Milner is an independent software consultant and instructor for PluralSight specializing in Microsoft technologies including .NET, Web services, Windows Workflow Foundation, Windows Communication Foundation, and BizTalk Server. Matt lives in Minnesota with his wife Kristen and his two sons. Contact Matt via his blog at pluralsight.com/blogs/matt.


Concurrent Affairs: Implementing the CLR Asynchronous Programming Model

Concurrent Affairs
Implementing the CLR Asynchronous Programming Model
Jeffrey Richter

Code download available at: ConcurrentAffairs2007_03.exe (157KB)
Contents
The Heart of the APM: IAsyncResult
Implementing the APM
Testing and Performance
Conclusion

Slow and unpredictable are words that typically characterize I/O operations. When an application performs a synchronous I/O operation, the application is basically giving up control to the device that is doing the actual work. For example, if an application calls the StreamRead method to read some bytes from a FileStream or NetworkStream, there is no telling how much time will pass before that method returns. If the file being read is on a local hard drive, then Read may return almost immediately. If the remote server housing the file is offline, then the Read method may wait several minutes before timing out and throwing an exception. During this time, the thread making the synchronous request is tied up. If that thread is the UI thread, the application is frozen and stops responding to user input.

A thread waiting for synchronous I/O to complete is blocked, which means that thread is idle but is not allowed to perform useful work. To improve scalability, many application developers create more threads. Unfortunately, each thread introduces significant overhead such as its kernel object, user-mode and kernel-mode stacks, increased context switching, the calling of DllMain methods with thread attach/detach notifications, and so on. The result is actually reduced scalability.

An application that wishes to remain responsive to the user, improve scalability and throughput, and increase reliability should not perform I/O operations synchronously. Instead, the application should use the common language runtime (CLR) Asynchronous Programming Model (APM) to perform asynchronous I/O operations. Much has been written about how to use the CLR APM, including Chapter 23 of my book CLR via C#, 2nd Edition (Microsoft Press®, 2006). However, I'm not aware of anything that has been written explaining how to define a class with methods that implement the APM. So I decided to focus on this topic for this column.

There are basically four reasons why a developer would want to implement the APM itself. First, you might be building a class that communicates directly with hardware (such as a file system on a hard disk, a network, a serial port, or a parallel port). As already mentioned, device I/O is unpredictable and therefore an application should perform asynchronous I/O operations when communicating with hardware in order for the application to remain responsive, scalable, and reliable.

Fortunately, the Microsoft® .NET Framework already includes classes that communicate with many hardware devices. Thus, you would not need to implement the APM yourself unless you were defining a class that communicated with a hardware device not already supported by the Framework Class Library (FCL), such as a parallel port. However, there are devices that are supported by the FCL, yet certain sub-features are not supported. In this situation, you might need to implement the APM if you wanted to perform I/O operations. For example, the FCL does offer a FileStream class that lets your application communicate with a disk, but the FileStream doesn't let you access opportunistic locks (microsoft.com/msj/0100/win32/win320100.aspx), sparse file streams (microsoft.com/msj/1198/ntfs/ntfs.aspx), or other cool features offered by the NTFS file system. If you wanted to write P/Invoke wrappers to call the Win32® APIs that exposes these features, you would want the wrappers to support the APM, thus allowing the operations to be performed asynchronously.

Second, you might be building an abstraction layer over an already defined class that communicates directly with hardware. Several examples of this already exist in the .NET Framework. For instance, let's say you're sending data to a Web service method. On the Web service client proxy class, there is a method that accepts your arguments, which could be a set of complex data structures. Internally, the method serializes these complex data structures into an array of bytes. The byte array is then sent over the wire by using a NetworkStream class (which already has the ability to communicate with the hardware using asynchronous I/O). Another example occurs when accessing a database. The ADO.NET SqlCommand type offers BeginExecuteNonQuery, BeginExecuteReader, and other BeginXxx methods that parse arguments to send data over a network to a database. When the I/O completes, the corresponding EndExecuteNonQuery, EndExecuteReader, and other EndXxx methods are called. Internally, these EndXxx methods parse the resulted data and return rich data objects back to the caller.

Third, your class might offer a method that could potentially take a long time to execute. In this case, you might want to offer BeginXxx and EndXxx methods, allowing the callers the convenience of the APM. Unlike the previous examples, which ended up being I/O-bound operations, this time your method is performing a compute-bound operation. Because the operation is compute-bound, a thread must be used to perform the work; you are just defining BeginXxx and EndXxx methods as a convenience to users of your class.

Finally, your class might wrap a Win32 method that performs synchronous I/O. Unfortunately, there are a number of Win32 methods that perform I/O but for which Win32 offers no way to perform these I/O operations asynchronously. For example, the Win32 registry and event logging functions could communicate with a local or remote registry/event log. Microsoft could create asynchronous versions of these Win32 functions allowing a thread not to block on them. However, as of today, asynchronous versions of these Win32 functions do not exist. When I wrap these kinds of methods with managed code, I always offer BeginXxx and EndXxx methods so that I can do the right thing in managed code even though my application is not as efficient as it could be because my methods must have a thread block while Windows® performs the synchronous I/O operation. However, if Microsoft ever adds asynchronous versions of these methods to Win32, I could change my wrappers to take advantage of the new methods, thereby gaining efficiency, without having to change my client code at all.

The Heart of the APM: IAsyncResult

At the heart of the CLR APM is the IAsyncResult interface, which is defined as shown in Figure 1.

Figure 1 IAsyncResult (Close Figure 1)
public interface IAsyncResult {
   WaitHandle AsyncWaitHandle { get; } // For Wait-Until-Done technique
   Boolean    IsCompleted     { get; } // For Polling technique
   Object     AsyncState      { get; } // For Callback technique
   Boolean    CompletedSynchronously { get; } // Almost never used
}

When any BeginXxx method is called, this method must internally construct an object whose type implements IAsyncResult and its four read-only properties. This object identifies the status of the asynchronous operation that has just been started. After the BeginXxx method returns to application code, the application can query these properties to determine whether the operation has completed. This object also contains the state of the completed operation: a result value if the operation completed successfully, or an exception if the operation failed to complete. An application passes the IAsyncResult object to an EndXxx method, which waits for the operation to complete (assuming it's not finished yet). The EndXxx method either returns the result or throws the exception letting the caller know the operation's result or error.

Figure 2 defines an AsyncResultNoResult class that implements the IAsyncResult interface. This simple class can be used for asynchronous operations that have no return value-specifically, the operation either succeeds or fails. The Stream's BeginWrite and EndWrite methods are an example of this. When you start an asynchronous write operation to a stream, the result is either success or failure-the EndWrite method is prototyped as returning void.

Figure 2 AsyncResultNoResult Class (Close Figure 2)
internal class AsyncResultNoResult : IAsyncResult
{
   // Fields set at construction which never change while
   // operation is pending
   private readonly AsyncCallback m_AsyncCallback;
   private readonly Object m_AsyncState;

   // Fields set at construction which do change after
   // operation completes
   private const Int32 c_StatePending = 0;
   private const Int32 c_StateCompletedSynchronously = 1;
   private const Int32 c_StateCompletedAsynchronously = 2;
   private Int32 m_CompletedState = c_StatePending;

   // Field that may or may not get set depending on usage
   private ManualResetEvent m_AsyncWaitHandle;

   // Fields set when operation completes
   private Exception m_exception;

   public AsyncResultNoResult(AsyncCallback asyncCallback, Object state)
   {
      m_AsyncCallback = asyncCallback;
      m_AsyncState = state;
   }

   public void SetAsCompleted(
      Exception exception, Boolean completedSynchronously)
   {
      // Passing null for exception means no error occurred.
      // This is the common case
      m_exception = exception;

      // The m_CompletedState field MUST be set prior calling the callback
      Int32 prevState = Interlocked.Exchange(ref m_CompletedState,
         completedSynchronously ? c_StateCompletedSynchronously :
         c_StateCompletedAsynchronously);
      if (prevState != c_StatePending)
         throw new InvalidOperationException(
             "You can set a result only once");

      // If the event exists, set it
      if (m_AsyncWaitHandle != null) m_AsyncWaitHandle.Set();

      // If a callback method was set, call it
      if (m_AsyncCallback != null) m_AsyncCallback(this);
   }

   public void EndInvoke()
   {
      // This method assumes that only 1 thread calls EndInvoke
      // for this object
      if (!IsCompleted)
      {
         // If the operation isn't done, wait for it
         AsyncWaitHandle.WaitOne();
         AsyncWaitHandle.Close();
         m_AsyncWaitHandle = null;  // Allow early GC
      }

      // Operation is done: if an exception occured, throw it
      if (m_exception != null) throw m_exception;
   }

   #region Implementation of IAsyncResult
   public Object AsyncState { get { return m_AsyncState; } }

   public Boolean CompletedSynchronously {
      get { return Thread.VolatileRead(ref m_CompletedState) ==
                c_StateCompletedSynchronously; }
   }

   public WaitHandle AsyncWaitHandle
   {
      get
      {
         if (m_AsyncWaitHandle == null)
         {
            Boolean done = IsCompleted;
            ManualResetEvent mre = new ManualResetEvent(done);
            if (Interlocked.CompareExchange(ref m_AsyncWaitHandle,
               mre, null) != null)
            {
               // Another thread created this object's event; dispose
               // the event we just created
               mre.Close();
            }
            else
            {
               if (!done && IsCompleted)
               {
                  // If the operation wasn't done when we created
                  // the event but now it is done, set the event
                  m_AsyncWaitHandle.Set();
               }
            }
         }
         return m_AsyncWaitHandle;
      }
   }

   public Boolean IsCompleted {
      get { return Thread.VolatileRead(ref m_CompletedState) !=
                c_StatePending; }
   }
   #endregion
}

As you can see, the AsyncResultNoResult class has a constructor that accepts AsyncCallback and Object arguments that are used to start all asynchronous operations. The constructor just saves these arguments in private fields. IAsyncResult's AsyncState property returns the Object field to the caller. The class defines an m_CompletedState field used to implement IAsyncResult's IsCompleted and CompletedSynchronously properties. It also defines an m_AsyncWaitHandle field to implement IAsyncResult's AsyncWaitHandle property. Finally, the class defines an m_exception field. This field is set when the operation completes. If the operation completes successfully, then the field is set to null (same as its initialized value); if the operation fails, then the field is set to the Exception-derived object indicating the reason for failure.

If you analyze the AsyncResultNoResult class, you'll notice that the code as a whole is very straightforward-except for the section that deals with the m_AsyncWaitHandle field. This field, a reference to a ManualResetEvent object, is only needed if the code starting the asynchronous operation queries the AsyncWaitHandle property or if the code calls the EndInvoke method before the operation has actually completed executing. The most common and recommended way to use the APM is to specify an AsyncCallback method that should automatically be invoked when the operation completes. For this common usage, the ManualResetEvent object is not necessary at all. As a result, I've gone to great lengths to avoid creating and using this object unless the code using an AsyncResultNoResult object absolutely needs it.

The reason I go to such great lengths is because creating and using a kernel object (such as a ManualResetEvent) is relatively expensive. For more information about the performance hit of using kernel objects, please see my October 2005 Concurrent Affairs column (msdn.microsoft.com/msdnmag/issues/05/10/ConcurrentAffairs).

When the asynchronous operation completes, some piece of code must call AsyncResultNoResult's SetAsCompleted method, passing in null if the operation completed successfully or a reference to an Exception-derived object if the operation failed. The code also indicates whether the operation completed synchronously (almost never) or asynchronously (almost always). This information is returned from IAsyncResult's CompletedSynchronously property, but very few applications actually care about it.

Internally, SetAsCompleted saves the exception in the m_exception field and changes the state of the m_completedSynchronously field. Then, if the manual reset event object was created, it is set. Finally, if an AsyncCallback method was specified when the AsyncResultNoResult object was constructed, this method is called back, letting the application code know that the asynchronous operation completed so that it can process the result (or failure).

To get the results of the operation, the application code will call some EndXxx method that will, in turn, call AsyncResultNoResult's EndInvoke method to determine whether the operation succeeded. If EndInvoke is called prior to the operation completing, then EndInvoke uses the manual reset event to suspend the calling thread until the operation has completed. If the operation completes, EndInvoke either returns or throws the exception that was saved off earlier when SetAsCompleted was called.

Since many asynchronous operations do have a return value, I also defined a class to support this: AsyncResult<TResult> (see Figure 3). This generic class is derived from AsyncResultNoResult and really just adds support for a return value of type TResult. This support comes in the form of a private field to hold the result (m_result), an overload of the SetAsCompleted method that accepts a TResult value, and a new EndInvoke method that waits for the operation to complete and then returns the result if the operation succeeded or throws an exception if the operation failed.

Figure 3 AsyncResult with a Return Value (Close Figure 3)
internal class AsyncResult<TResult> : AsyncResultNoResult
{
   // Field set when operation completes
   private TResult m_result = default(TResult);

   public AsyncResult(AsyncCallback asyncCallback, Object state) :
      base(asyncCallback, state) { }

   public void SetAsCompleted(TResult result,
      Boolean completedSynchronously)
   {
      // Save the asynchronous operation's result
      m_result = result;

      // Tell the base class that the operation completed
      // sucessfully (no exception)
      base.SetAsCompleted(null, completedSynchronously);
   }

   new public TResult EndInvoke()
   {
      base.EndInvoke(); // Wait until operation has completed
      return m_result;  // Return the result (if above didn't throw)
   }
}

Also, many BeginXxx methods accept arguments in addition to the AsyncCallback and the Object. For example, the Socket class has a BeginAccept method that takes IPAddress (address) and Int32 (port) arguments. If you want to use the AsyncResultNoResult or AsyncResult<TResult> class with a BeginXxx method that takes additional arguments, you'll want to define your own type derived from either of these two base classes (depending on whether your EndXxx method returns void). In your class, define additional fields-one for each argument-and set them in your class's constructor. Then the method that does the actual work can extract these argument values from your class's fields when the time is right.

Implementing the APM

Now that you see how to define a type that implements the IAsyncResult interface, I'll show how to use my AsyncResult<TResult> and AsyncResultNoResult classes. I defined a LongTask class (see Figure 4) that offers a synchronous DoTask method that takes a long time to execute and returns a DateTime instance indicating when the operation completed.

Figure 4 LongTask Simulates Asynchronous I/O (Close Figure 4)
internal sealed class LongTask
{
   private Int32 m_ms;  // Milliseconds;

   public LongTask(Int32 seconds)
   {
      m_ms = seconds * 1000;
   }

   // Synchronous version of time-consuming method
   public DateTime DoTask()
   {
      Thread.Sleep(m_ms);  // Simulate time-consuming task
      return DateTime.Now; // Indicate when task completed
   }

   // Asynchronous version of time-consuming method (Begin part)
   public IAsyncResult BeginDoTask(AsyncCallback callback, Object state)
   {
      // Create IAsyncResult object identifying the
      // asynchronous operation
      AsyncResult<DateTime> ar = new AsyncResult<DateTime>(
         callback, state);

      // Use a thread pool thread to perform the operation
      ThreadPool.QueueUserWorkItem(DoTaskHelper, ar);

      return ar;  // Return the IAsyncResult to the caller
   }

   // Asynchronous version of time-consuming method (End part)
   public DateTime EndDoTask(IAsyncResult asyncResult)
   {
      // We know that the IAsyncResult is really an
      // AsyncResult<DateTime> object
      AsyncResult<DateTime> ar = (AsyncResult<DateTime>)asyncResult;

      // Wait for operation to complete, then return result or
      // throw exception
      return ar.EndInvoke();
   }

   // Asynchronous version of time-consuming method (private part
   // to set completion result/exception)
   private void DoTaskHelper(Object asyncResult)
   {
      // We know that it's really an AsyncResult<DateTime> object
      AsyncResult<DateTime> ar = (AsyncResult<DateTime>)asyncResult;
      try
      {
         // Perform the operation; if sucessful set the result
         DateTime dt = DoTask();
         ar.SetAsCompleted(dt, false);
      }
      catch (Exception e)
      {
         // If operation fails, set the exception
         ar.SetAsCompleted(e, false);
      }
   }
}

As a convenience, I also offer BeginDoTask and EndDoTask methods that follow the CLR APM, allowing users to execute the DoTask method asynchronously. When a user calls my BeginDoTask method, I construct an AsyncResult<DateTime> object. Then I have a thread pool thread call a small helper method, DoTaskHelper, which wraps a call to the synchronous DoTask method.

The DoTaskHelper method simply calls the synchronous version of the method with a try block. If the DoTask method runs to completion without failing (throwing an exception), then I call SetAsCompleted to set the operation's return value. If the DoTask method throws an exception, then DoTaskHelper's catch block will catch the exception and indicate that the operation has completed by calling SetAsCompleted, passing in the reference to the Exception-derived object.

The application code calls LongTask's EndDoTask method to get the results of the operation. All EndXxx methods are passed an IAsyncResult. Internally, the EndDoTask method knows that the IAsyncResult object passed to it is really an AsyncResult<DateTime> object, casts it, and calls EndInvoke on it. As discussed earlier, AsyncResult<TResult>'s EndInvoke method waits for the operation to complete (if necessary) and then returns the result or throws an exception indicating back to the caller the outcome of the asynchronous operation.

Testing and Performance

The FunctionalTest method (see Figure 5) shows some code that uses my implementation of the APM. It tests the three rendezvous techniques offered by the APM: wait until done, polling, and callback method. If you examine the code, you'll see that it looks identical to any other usage of the APM that you've ever seen. Of course, this is the point of the whole exercise.

Figure 5 Using LongTask (Close Figure 5)
private static void FunctionalTest()
{
  IAsyncResult ar;
  LongTask lt = new LongTask(5);

  // Prove that the Wait-until-done technique works
  ar = lt.BeginDoTask(null, null);
  Console.WriteLine("Task completed at: {0}", lt.EndDoTask(ar));

  // Prove that the Polling technique works
  ar = lt.BeginDoTask(null, null);
  while (!ar.IsCompleted)
  {
     Console.WriteLine("Not completed yet.");
     Thread.Sleep(1000);
  }
  Console.WriteLine("Task completed at: {0}", lt.EndDoTask(ar));

  // Prove that the Callback technique works
  lt.BeginDoTask(TaskCompleted, lt);
  Console.ReadLine();
}

private static void TaskCompleted(IAsyncResult ar)
{
  LongTask lt = (LongTask)ar.AsyncState;
  Console.WriteLine("Task completed at: {0}", lt.EndDoTask(ar));
  Console.WriteLine("All done, hit Enter to exit app.");
}

The PerformanceTest method (see Figure 6) compares the performance of my IAsyncResult implementation to the implementation that the CLR provides when using a delegate's BeginInvoke and EndInvoke methods. My implementation seems to perform better than the FCL's current implementation, apparently due to it always constructing a ManualResetEvent whenever it creates its IAsyncResult object regardless of whether this event is needed by the application.

Figure 6 Testing IAsyncResult Performance (Close Figure 6)
private const Int32 c_iterations = 100 * 1000; // 100 thousand
private static Int32 s_numDone;
private delegate DateTime DoTaskDelegate();

private static void PerformanceTest()
{
   AutoResetEvent are = new AutoResetEvent(false);
   LongTask lt = new LongTask(0);

   Stopwatch sw;

   s_numDone = 0;
   sw = Stopwatch.StartNew();
   for (Int32 n = 0; n < c_iterations; n++)
   {
      lt.BeginDoTask(delegate(IAsyncResult ar)
      {
         if (Interlocked.Increment(ref s_numDone) == c_iterations)
            are.Set();
      }, null);
   }
   are.WaitOne();
   Console.WriteLine("AsyncResult Time: {0}", sw.Elapsed);
   s_numDone = 0;
   DoTaskDelegate doTaskDelegate = lt.DoTask;

   sw = Stopwatch.StartNew();
   for (Int32 n = 0; n < c_iterations; n++)
   {
      doTaskDelegate.BeginInvoke(delegate(IAsyncResult ar)
      {
         if (Interlocked.Increment(ref s_numDone) == c_iterations)
            are.Set();
      }, null);
   }
   are.WaitOne();
   Console.WriteLine("Delegate    Time: {0}", sw.Elapsed);
}

Conclusion

I think it is interesting to understand what is going on inside the CLR when we use mechanisms such as the APM. After examining my implementation here, you can get a sense of the size of IAsyncResult objects, what their state is and how they manage their state. This understanding can lead to improved ways of architecting your own applications and to better performance.

In this column, I used my IAsyncResult implementation to perform compute-bound tasks using thread pool threads. In a future column, I'll show how to use my IAsyncResult implementation with I/O-bound operations.

Send your questions and comments for Jeffrey to mmsync@microsoft.com.

Jeffrey Richter is a cofounder of Wintellect (www.Wintellect.com), an architecture review, consulting and training firm. He is the author of several books, including CLR via C# (Microsoft Press, 2006). Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.


{ End Bracket }: Developing the Virtual Earth 3D Control

{End Bracket}
Developing the Virtual Earth 3D Control
Duncan Lawler

The Virtual Earth 3D control is a new tool released by Microsoft for visualizing the entire world in three dimensions. Rather than a standalone application, Virtual Earth™ 3D is implemented as a managed control, which allows seamless integration with Web sites or standalone applications. The Microsoft® .NET Framework security model gives users the capability of dynamically loading plug-ins to supply data, but restricting them to execution in a partial-trust environment.

One of the primary goals of the Virtual Earth 3D team's project was to enable developers to build their own applications. The beta release integrates seamlessly with the JScript® API available in the Virtual Earth SDK. Web developers who use the SDK to integrate mapping functions into their Web sites will now be able to show a 3D view with no additional work. For those not doing Web development, there will soon be SDK documentation available on how to embed the control directly in your own Windows® Forms or native application. We will also be documenting and providing samples of how to write and use your own plug-in extensions to change the control model, add your own data to the scene, and more.

To create the Virtual Earth 3D control, the team developed techniques for automatically generating 3D textured models of structures in the real world based on aerial photography with resolutions better than 12 inches. This control allows application developers to create mapping or visualization apps that show real-world locations with a level of detail that gives the user the experience of actually "being there."

Figure 1 (Click the image for a smaller view)

Figure 1 (Click the image for a larger view)

As you might expect, the data required to visualize the world at the resolution we wanted was many tens of terabytes in size. So it was immediately obvious that we would not be able to host all this data locally on the user's PC. The data would have to be hosted on a large server and transmitted to the client as needed. Even so, we needed to reduce the file sizes. We started by using standard terrain-rendering techniques to create the surface mesh with high detail in the foreground of the scene and lower detail as you move away from the camera. The fully textured buildings presented a real challenge because of the amount of data that can be visible in a single scene (each of the surface textures generated by the Virtual Earth automated process is unique).

To improve performance, first we created multiple levels of detail for the building models. The source data has a full detail geometry model and a texture map for each building face. Initially, we created a single texture atlas from the source textures for the building. We then took nearby buildings and created a single composite mesh that included all the buildings and a single associated texture atlas. Smaller buildings were removed since they generally won't be visible when the lower-detail level is displayed. We repeated this process with nearby composite meshes collapsed into larger and larger composite meshes. As a result, when the frame is rendered, the lower-detail composite meshes are displayed in the background of the scene instead of the full-detail meshes. Collapsing the textures and models allows Virtual Earth to render them with a single set of calls and greatly improves performance.

Bandwidth required for transmitting this data was the next challenge. To get the best quality for a given texture size, we used the new image formats available in Windows Imaging Components to compress textures for transmission.

Since users often want to visit more than one city, the control continuously monitors the current view and tries to guess where the user will go next. It asynchronously issues priority-ordered requests for the areas that are currently visible or will soon be visible. Also, the control reduces the level of detail it requests based on the connection speed so the user gets a more interactive experience.

Most users will tend to view their own city most often, so caching is a very effective way to improve performance on subsequent visits. Once the data is downloaded, Virtual Earth keeps the textures in system memory to render them to the screen. When the hardware supports it, the textures are converted to a compressed format in video memory. However, this conversion is CPU-intensive, so the team had to carefully throttle this on background threads to avoid causing stutters in the frame rate.

For more information and SDKs, visit the Virtual Earth forums at http://dev.live.com.

Duncan Lawler is Development Manager for the Virtual Earth 3D team. He has been with Microsoft for 10 years, all of it with the Virtual Earth team, and has contributed to MapPoint, Atlas, Streets & Trips, MSN Maps, and Pocket Streets. Contact him at duncal@microsoft.com.