 MSDN Magazine July 2004

MSDN Magazine July 2004

Visual Studio 2005: Bridge the Gap Between Development and Operations with Whitehorse

Microsoft is introducing a new suite of tools (code-named "Whitehorse") that will make it easier for you to design and implement systems that conform to a service-oriented architecture. Two of these tools -- the SOA Design Suite and the Class Designer -- support the graphical design of systems and components with support for code generation and support for bi-directional synchronization which lets you ensure that your diagram always represents your system design. This article introduces these tools and shows you how they'll improve your design and development efforts.

Brian A. Randell and Rockford Lhotka SharePoint: Use Windows SharePoint Services as a Platform for Building Collaborative Applications

In this article the authors provide an in-depth examination of the architecture of SharePoint products and technologies: WSS and SPS. WSS provides the foundation for creating collaborative Web sites that support customization and personalization and SPS complements WSS by playing the role of a content aggregator. This article explains how.

Jason Masterman and Ted Pattison .NET Tools: Ten Must-Have Tools Every Developer Should Download Now

This article introduces 10 development tools that can increase your productivity, give you a better understanding of .NET, and maybe even change the way that you develop applications. The tools covered include NUnit to write unit tests, Reflector to examine assemblies, FxCop to police your code, Regulator to build regular expressions, NDoc to create code documentation and five more.

James Avery User Preferences: Manage User Settings in Your .NET App with a Custom Preferences API

There are plenty of options out there for managing user preferences including custom configuration files, the Windows registry, and the isolated storage. But each option has its pros and cons -- and a bad choice can make life difficult. In this article, the author evaluates various options and identifies the characteristics of a good preferences API. Based on those characteristics, he introduces an API that is specifically designed for preferences management, one that offers the best of all the options.

Ray Djajadinata Web Services: Capturing and Analyzing Client Transaction Metrics for .NET-Based Web Services

This article presents a general-purpose client quality reporting mechanism that can be used in any .NET-based transaction system that employs HTTP/SOAP. The design uses client response time and quality recording, upload of logs as SOAP headers attached to new transaction requests, and server handoff of these headers to a low priority queue for logging and analysis. This technique gives an enterprise near real-time information on actual end-user response times. These response times reflect network delays, client application overhead and server delays. By using this technique, enterprises can avoid the need to develop custom software to mine HTTP logs.

Brian Connolly Editor's Note: Topic Coverage Update

Over the past couple of months, we've received a lot of mail about our recent issues. Although opinions are mixed, several people have let us know that they feel we've covered too much forward-looking content recently.

New Stuff: Resources for Your Developer Toolbox

FlexiSoft Solutions Inc. has just released the latest version of their coding standards enforcement tool, Standards Master 2004, an add-in for Visual Studio® . NET. Standards Master 2004 brings many of the best features of Standards Master 2.

Nancy Michell Web Q&A: Pop-Ups, Encrypting an ADO.NET Data Stream, and More

Edited by Nancy Michell Data Points: Migrating from ADO to ADO.NET

In recent years ADO has taken the lead as the preferred method for implementing data access in Windows®-based applications. Huge numbers of ADO applications are in use today and many developers are well versed in ADO development.

John Papa Cutting Edge: Implement Custom Cache Dependencies in ASP.NET 1.x

One of the most compelling improvements that ASP. NET brought to ASP programming was the Cache object. The Cache has some similarities to the Application object and is a container of global data (as opposed to session-specific data) that features a fair number of innovative characteristics.

Dino Esposito The XML Files: Messages vs. Methods

Aaron Skonnard Test Run: Using Combinations to Improve Your Software Test Case Generation

James McCaffrey Security Briefs: Mind Those Passwords!

Keith Brown Advanced Basics: Data Binding Radio Buttons to a List

Duncan Mackenzie .NET Matters: How Microsoft Uses Reflection

In this fourth installment of . NET Matters, I'm taking a breather from the Q&A format. Instead, this month I bring you stories from the trenches; developers in product groups within Microsoft describe how they use .

Edited by Stephen Toub C++ Q&A: Message Values, Managed String Literals, Obfuscating Code, and More

Paul DiLascia { End Bracket }: What Makes Good Code Good?

When MSDN Magazine asked me to write a page on something I care about, I said, "You mean, like abolishing taxes, car phones, and SUVs?" Alas, they meant something to do with programming. Well! After pondering, I realized that something I care about is writing good code.

Paul DiLascia


Visual Studio 2005: Bridge the Gap Between Development and Operations with Whitehorse

Visual Studio 2005

Bridge the Gap Between Development and Operations with Whitehorse

Brian A. Randell and Rockford Lhotka
This article is based on a prerelease version of Visual Studio 2005. All information contained herein is subject to change.
This article discusses:
  • The Whitehorse SOA designer
  • The Visual Studio 2005 Class Designer
  • Generating diagrams from code
  • Generating code from diagrams
This article uses the following technologies:
Visual Basic and C#
Contents
Application Design
Deployment Cycle
The Class Designer
Conclusion

W
henever you want to use a modeling tool to describe software systems, you're frequently faced with a serious investment of time and money. Often the diagrams that result look great, but are quickly forgotten as development gets into full swing. While there are tools on the market that provide for synchronization between diagrams and code, their cost and the time required to learn them has put off many if not most users (unless their full-time job is to produce such diagrams).

An additional problem has been the mismatch between the design of the system and the characteristics of the environment into which it will be deployed. Many times a solution looks great in a diagram, gets coded, and then when deployment time comes, there's a rush to solve problems that emerge because the model does not map correctly. The competing priorities of architects, designers, developers, and infrastructure personnel can lead to conflict. Often there are conflicting constraints that must be reconciled in the beginning. These constraints, such as physical server security, become obstacles to the successful deployment and implementation of new systems. Added to the mix is the increasing complexity of today's environments which have evolved from central mainframe-only systems to the client/server model, and on to distributed systems.

Today, one very popular environment is the service-oriented architecture (SOA), which is built from loosely coupled, autonomous applications that offer services, communicate using messages, and often cross trust boundaries. Right now XML-based Web services built with ASP.NET provide a platform on which to build these kinds of systems. As technologies evolve, Microsoft will be shipping a variety of technologies to make it easier for you to build and manage these systems and to extend the capabilities of the underlying platform.

In the time frame of Visual Studio® 2005, Microsoft is introducing a new suite of graphical design tools, code-named "Whitehorse," to make it easier to design and implement systems using SOA. These include the new Class Designer as well as a group of four tightly integrated tools referred to as the Distributed System Designers. One of these four, the Application Connection Designer, and the Class Designer share a key characteristic. Both support the graphical design of systems and components and include support for code generation. What's more, the designers support bidirectional synchronization so you no longer have to be concerned that a diagram doesn't represent the system and its components. Whitehorse represents a renaissance in modeling in that it goes beyond the typical documentation-centric view of modeling systems. The designers are tightly integrated into the Visual Studio shell and each diagram is a first-class artifact in a solution, just like other program sources. This subjects the diagrams to source code control to support team scenarios.

The new Logical Datacenter Designer tool allows operations architects to provide a description of the types of servers in the target environment and record the configurations of the host software packages loaded on those servers. With this detailed knowledge of the target environment, and detailed information about the logical software model, Whitehorse allows the software developer to bind services to servers and provide a validation process to make sure that the design will actually deploy successfully in the datacenter. Otherwise, it will highlight the need to make changes in the datacenter configuration itself. In this way, organizations can avoid costly downstream deployment problems and can apply fixes early in the development process.

This ability to design a service-oriented system and immediately validate it against the operations infrastructure is called Design for Operations. Applications are often designed in one environment but are meant to be deployed in another or even multiple environments. With Whitehorse, the ability to validate a design against a model of a target datacenter makes deployment smoother. Whitehorse is a key part of the new Microsoft® Dynamic Systems Initiative (DSI), a long-term industry initiative to simplify and automate the development, deployment, and operation of connected systems. With it the operations team can map their view of the world to the architect's view. Constraints can be enforced in both worlds, forcing communication through the validation mechanism from the start of the project to help ensure successful system deployments.

The best way to understand the tools is to walk through building a simple system using the Distributed System Designers.

Application Design

The Whitehorse tools introduce a new Distributed System solution type. When you create a new solution, the first designer you'll encounter is the Application Connection Designer. This designer provides a design surface for defining Web service applications and their client Web or Windows-based applications. These applications can be created directly on the design surface or, if an Application Connection Diagram is added to an existing solution, it will analyze the solution to create the diagram. The designer enables you to connect applications, defining the communication pathways that must be supported in the code. It also enables you to experiment freely with the application settings and design before generating the skeleton implementations for developers to implement. Basically, it supports "whiteboarding" your design before committing to code, thus allowing you to explore and experiment with your system design.

The Visual Studio Toolbox provides a set of components that represent Windows-based applications, Web services, and databases (see Figure 1). Using drag and drop, it's easy to lay out a system. For example, to specify that a database connection is required, you drag and drop an ExternalDatabase component onto the design surface. Then using the standard Properties windows in Visual Studio, you set the connection string for the database. With your data source defined, the next step is to add a data access layer in the form of a Web service. After dragging a Web service onto the design service, it can be bound to the previously defined data source by dragging a connection between the two.

Figure 1 Toolbox
Figure 1 Toolbox

The next step is to expose the data via a Web service interface. To do this, you click the Web service's endpoint, created by default, and using the Endpoint Details window, define the public entry points. To generate the application skeleton, right-clicking on the Web service provides an Implement command. Running this command adds a new Web service project to the solution, configures the local instance of IIS, and generates a set of language-specific source and configuration files. Opening the Code folder in the Solution Explorer exposes two class files, one for the implementation of the Web service, the other a useful utility class for retrieving the database connection string generated into the web.config file. Filling the stubs of the Web service class with some standard data access code completes the process. At this point, if a correction or adjustment was made to the public entry points in code, these changes would automatically be synchronized with the system diagram.

Testing the initial design works just as it did in earlier versions of Visual Studio, so you simply press F5 and it runs. Assuming that everything works fine, it's now time to add a front end to the Web service. Back in the Application Connection Designer, drag and drop an ASP.NET Web Application onto the designer surface. Just like before, you need to bind the components together. This is done by clicking the exposed endpoint of the Web service and dragging a connection link between the Web service and the Web Application. Next, right-click over the Web Application and select Implement. Visual Studio will then add a new Web Application project to the solution and create a Web reference to the Web service. You then add a default Web page just as in previous versions by using the Add New Item command.

Figure 2 App in Distributed Solution
Figure 2 App in Distributed Solution

Using the Web Forms designer, we add an ObjectDataSource component to link the Web service to our UI, which is a GridView control on the Web page. The ObjectDataSource takes care of invoking the exposed methods from the Web service. In this example, we don't have to write any code. We have to make just a few adjustments to the GridView control's properties and then press F5 to show data from our Web service in the Web browser. Figure 2 shows the completed application in the Application Connection Designer.

Deployment Cycle

The next phase requires that a logical model of the network and actual server types in a deployment environment be defined. This step could have already been completed by the operations team. For this example we'll use the Logical Datacenter Designer, which is used to create diagrams that represent a logical model of the application hosting capabilities of a datacenter. The designer allows the infrastructure architect to specify the datacenter topology by presenting system entities like database servers, Web servers, logical security zones, and communication endpoints. In addition, constraints can be applied against the application configuration based upon the needs of the datacenter.

Figure 3 Types
Figure 3 Types

In the existing solution, we'll add a new Logical Datacenter Diagram. Just like before, we'll use elements, exposed on the Toolbox, specific to this designer. Figure 3 shows a list of the current logical server and endpoint types supported. We'll add two zones to the diagram. The first represents the demilitarized zone (DMZ), and the second represents a secure inner zone of the network. Next, we'll define a hardened IIS Web server in the DMZ. In the inner zone, we'll add an IIS Web server called "AppServer," and a machine running SQL Server™. In the case of the IIS Web servers, predefined input and output endpoints exist. Using the Connection tool, we connect the various input and output channels, including the connection between the AppServer and the SQL Server machine. Figure 4 shows the completed logical datacenter diagram.

Figure 4 Completed Logical System Architecture
Figure 4 Completed Logical System Architecture

With the diagram defined, we can now add constraints to the logical servers. For example, we could require that the AppServer use impersonation and SSL. We simply select the AppServer and open the Settings and Constraints editor as shown in Figure 5.

Figure 5 Settings and Contraints Editor
Figure 5 Settings and Contraints Editor

There are several types of constraints supported in Whitehorse: Constraint dialogs, user-defined constraints, and implicit (or built-in) constraints. The Constraint dialogs (shown in Figure 5) logically group together settings of a particular type, for example ASP.NET membership or Web site Configuration. These dialogs have editing rules built into them to allow specific combinations of settings to define the constraint. User-defined constraints allow a user to have full control over desired values and ranges for any setting available on the logical server or application. Implicit constraints are supplied in the model and are applied without the user having to perform an explicit action (other than validate). For example, if the .soap and .asmx script maps were removed from IIS, the Web server wouldn't be able to support Web services. Note that settings may also be imported from existing servers such that an accurate model of the Web server in the actual deployment environment is used for constraint evaluation.

The last step in the process is to map the applications we defined earlier to our newly defined logical system architecture. We can either make an assumption that the configuration used in the development environment is the configuration needed for deployment and define what is referred to as a Trial Deployment, or we can explicitly create a new system in which we can configure settings that are specific to the deployment we are planning. In most cases, people will start with a Trial Deployment to weed out the most serious issues and then create a system as the final step to locking down the configuration. Systems can be composed into more complex definitions, allowing reuse in the design of large-scale architectures.

We are going to create a Trial Deployment. We select Define Trial Deployment from the Application Connection Diagram and then, in a pop-up dialog, select the Logical Datacenter design we created earlier. The Deployment Designer is opened with a new deployment diagram depicting the logical datacenter as a backdrop. The Deployment Designer then allows us to bind each application to a logical server, indicating the type of server to which it will be deployed. This is done by dragging and dropping the application either from a simple outline control or directly from the Application Connection Diagram. If the server is incompatible with the application to which we're trying to bind, the no-drop symbol is displayed. The process is repeated with each application, creating a virtual deployment of the system. Once we've mapped all of our components to the appropriate hosts, we can validate our mapping, as shown in Figure 6.

Figure 6 Subsystem Mapper
Figure 6 Subsystem Mapper

Validation checks all of the bindings, confirming the constraints defined in both diagrams and verifying that there are communication pathways in a logical datacenter that support the pathways and protocols required by the application. If there are any errors, a task item is added to the Visual Studio task list. Double-clicking on the items in the error list takes us to the offending diagram and component. In this example, we set a constraint in the Logical Datacenter Designer to require our AppServer to use ASP.NET Impersonation. When we ran validation, we got an error. By double-clicking on the task, we're taken to the Application Connection Designer with the invalid endpoint selected. After adjusting the correct property in the Settings editor, the value is automatically written to the web.config file, we revalidate, and everything is done. The advantage here is that we find deployment problems before we deploy (possibly even before core development has started), not during the actual stress-filled hours before going live.

At this point, the design can be handed off to developers who can start working with the Class Designer in order to map out the heart and soul of the system.

The Class Designer

Most object-oriented developers like to create diagrams of their application designs and often these take the form of Unified Modeling Language (UML) documents. UML defines several types of diagrams you can use to model your application. The most commonly used diagram type is the class diagram, which shows each class in your application, along with their relationships to other classes. Typically, each class is displayed in a box with its fields and methods listed.

Unfortunately, UML has fallen behind current technologies such as the .NET Framework. For instance, there is no provision in UML for the Friend and internal scope in Visual Basic® .NET and C#. Nor is there any elegant way to list property methods or events, which are key parts of the interface for .NET classes.

It is difficult to keep code and diagrams in sync. Some UML diagramming tools allow code generation from a diagram and diagram generation from code, but there are limitations. Therefore, most developers avoid the use of code generation, which means they must manually keep the diagrams and code in sync over time.

To help address these problems, Microsoft is developing a Class Designer tool which is scheduled to ship as part of Visual Studio 2005. The Class Designer produces diagrams similar to a UML class diagram, with the addition of features to support all the design concepts available to programmers using .NET. Additionally, the Class Designer is fully integrated into Visual Studio 2005, so the diagrams it creates are part of your project along with your code.

The integration with Visual Studio 2005 is not only convenient, it means that the Class Designer can do real-time code generation and generating diagrams from code. Put simply, the diagram is instantly updated as you change the code, and your code is instantly updated as you change the diagram.

Figure 7 shows the Class Diagram tool in Visual Studio 2005, along with the matching code that was generated for the diagram. Changing either the diagram or the code is fine. Both remain in sync automatically.

Figure 7 Basic Class Diagram (Close Figure 7)

Using the Class Designer requires that you add a Class Diagram to your project in Visual Studio 2005. You do this by adding a new item to the project and choosing the Class Diagram template.

If you already have classes in the project, they can be added to the diagram by dragging them from the Class View window onto the diagram's designer surface. This will cause the Class Designer to add the class to the diagram, and list all the methods, fields, properties, and events in the class. An example of such a class is shown in Figure 8. Note that not only are the details of the class shown in the diagram itself, but when the class is selected in the diagram, the details also appear in the Class Details window at the bottom of the display.

Figure 8 Dragging a Class from Class View to Class Diagram
Figure 8 Dragging a Class from Class View to Class Diagram

Using the Class Details window you can view and change the interface of the class. This includes adding and removing methods, fields, properties, and events. You can also change the scope, parameters, and return types of these elements. Any change made in the Class Details window is immediately applied to the code.

Notice the Summary column in the Class Details window. This column displays the summary information about the interface element from the XML documentation in the code. You can also change the XML documentation in the code by editing this column. Better still, when you select this column, a button is available to bring up a dialog where you can edit more of the XML documentation.

The resulting Visual Basic .NET comments are automatically added to the code, as shown in the following code snippet.

''' <summary>
''' Returns the ID value of the product.
''' </summary>
Public ReadOnly Property ID() As Guid
  Get
    Return mID
  End Get
End Property

The same is true for C#, C++, or J# code (though there are some restrictions for C++). Changes to the diagram are immediately reflected in the code of your project; likewise, changes to your code are reflected in the diagram. Suppose, for example, you start with the C# code in Figure 9. Dragging this class from the Class View to the Class Diagram produces the illustration shown in Figure 10.

Figure 9 A Simple C# Class (Close Figure 9)
using System;

namespace Sales
{
  public class Customer
  {
    string _name = string.Empty;

    public string Name
    {
      get
      {
        return _name;
      }
      set
      {
        _name = value;
      }
    }
  }
}
Figure 10 MTS
Figure 10 MTS

Not only can you drag and drop your classes, but you can drag and drop classes from other assemblies referenced by your assembly. When you do this, the external classes are shown in the diagram in a read-only mode since they aren't directly part of the current assembly. This allows you to provide a more comprehensive diagram that illustrates the relationships between your classes and the classes from referenced assemblies.

You will also be able to drag and drop an entire project from Solution Explorer onto the diagram if you want all the items in the assembly displayed in the diagram. Initially there won't be any auto-formatting provided for the newly added elements in the diagram, but hopefully that feature will be added prior to the release of the tool.

To illustrate how the diagram is updated when the code changes, you can add the following property to the class:

int _id;

public int ID
{
  get { return _id; }
  set { _id = value; }
}
The diagram is immediately updated to display the newly added ID property and field.

It is also possible to start with the diagram, rather than with code. When the Class Diagram Designer is open, you can drag and drop elements from the Toolbox onto the diagram. This automatically creates the associated code file and adds the item to the diagram. Using the Toolbox, you can add most common coding constructs, including classes (both abstract and concrete), enumerations, interfaces, structures, delegate definitions, and Visual Basic .NET modules.

You can also use the Toolbox to add inheritance and association relationships between elements on the diagram. These relationships are immediately reflected in the code. For instance, if you click the Association tool, then click and drag from a Product class to an InventoryQOH class, the following code is generated in the Product class file:

Public Property InventoryQOH() As Inventory.InventoryQOH
  Get

  End Get
  Set(ByVal Value As Inventory.InventoryQOH)

  End Set
End Property

This new InventoryQOH property does not appear as a property in the Product class because it is displayed graphically as an association line between the Product and InventoryQOH classes in the diagram. Likewise, if you add an Inheritance relationship between two classes, the appropriate inheritance relationship is added to the code.

From the Toolbox you can also add arbitrary comments to the diagram itself. These comments are only part of the diagram and have no impact on the code in the project. These are different from the XML documentation associated with each code element in the diagram, which are part of both the diagram and the code.

The Class Diagram file in a project has a .cd file extension. It is a simple XML file that contains details about each element in the diagram. Since most of the data in the diagram flows directly from your code, there is not a lot of information stored in the XML file itself. For instance, the Product class shown earlier has the XML shown in Figure 11.

Figure 11 XML File for the Class Diagram (Close Figure 11)
<Class>
  <AbsoluteBounds X="0.5" Y="0.5"
   Height="1.794798172314962" Width="1.5" />
  <ShowAsAssociation>
    <Property Name="InventoryQOH" />
  </ShowAsAssociation>
  <TypeIdentifier>
    <FullName>Inventory.Product</FullName>
    <FileName>
         C:\Documents and Settings\Administrator\My Documents\Visual
         Studio\Projects\Inventory\Inventory\Product.vb</FileName>
    <SiblingOrder>1</SiblingOrder>
    <HashCode>AAAAEAAAAAAAgAAAAIAAAAQAAAAAAAAAAAgAAAAAAAA=</HashCode>
  </TypeIdentifier>
</Class>

The core of the element is the <TypeIdentifier> element, which contains the name of the class and a pointer to the actual class file. This allows the designer to read the class file to derive most of the details about the class itself. The <AbsoluteBounds> element indicates the position of the item in the diagram.

There is also a <ShowAsAssociation> element. This exists because our diagram, shown in Figure 7, has an association between Product and InventoryQOH. Because this relationship is reflected in the diagram, the XML includes a reference to the specific property in the Product class that creates the association.

If we return to the Class Diagram, we can right-click the association line and choose the Show as Property option. When you select this option, the association line between the Product and InventoryQOH classes is removed from the diagram, and the InventoryQOH property is displayed as a property within the Product class itself. To reverse the process, you can right-click on the InventoryQOH property in the Product class and choose the "Show as Association" menu option.

By turning off the "Show as Association" option, we have changed the diagram, but not the code. In other words, the association between Product and InventoryQOH is unchanged and only the display is affected. This is reflected in the XML for the diagram because the <ShowAsAssociation> element has been removed from the Product entry in the XML.

Conclusion

The Class Designer should be a very nice tool for object-oriented designers and developers using Visual Studio 2005. It provides a UML-like class diagram that is capable of reflecting all the language features of Visual Basic .NET and C# in the Microsoft .NET environment. Not only does it allow you to create accurate diagrams, but it keeps the diagram and code in sync in real time. You can now look forward to a time when your class diagrams truly reflect your code. The ability to move from diagram to code and back will ensure that your pretty pictures are more than just a memory of something that could have been but, in fact, are accurate maps of your functioning system.

Brian A. Randell is a senior consultant with MCW Technologies, a Microsoft Certified Partner specializing in custom application development with .NET, SQL Server, and the Microsoft Office family of products. Contact him at http://www.mcwtech.com.

Rockford Lhotka is the author of Expert One-on-One Visual Basic .NET and C# Business Objects (APress) and is a columnist for MSDN online. He is the Principal Technology Evangelist for Magenic Technologies, a Microsoft Gold Certified Partner.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


SharePoint: Use Windows SharePoint Services as a Platform for Building Collaborative Applications

SharePoint

Use Windows SharePoint Services as a Platform for Building Collaborative Applications

Jason Masterman and Ted Pattison
This article discusses:
  • SharePoint architecture
  • Configuration and content databases
  • SharePoint integration with IIS 6.0 and ASP.NET
  • Site collections and virtual servers
  • Administration of WSS and SPS
This article uses the following technologies:
Windows Server 2003, IIS 6.0, ASP.NET 1.1
Contents
Windows SharePoint Services
Databases
IIS Web Sites and Virtual Servers
Site Collections, Sites, and Workspaces
SPS Architecture
Conclusion

O
ver the last few years, Microsoft has made a significant investment in collaborative technologies. Windows® SharePoint® Services (WSS) and SharePoint Portal Server (SPS) are two flagship products positioned in this space. This article is written in two parts. In this month's installment we will introduce the fundamental architecture of SharePoint and discuss the opportunities that SharePoint provides the developer using the Microsoft® .NET Framework. In the second installment, we will discuss programming with the WSS and SPS object models as well as the creation of custom Web Parts for a SharePoint site.

Windows SharePoint Services is an add-on product that is considered an official part of the Windows Server™ 2003 operating system. WSS supplies a framework for building collaborative Web sites that makes it possible for a company to share information and documents across teams, departments, and large organizations easily and reliably. Any user can access a WSS Web site using a Web browser or through the new collaboration features built into Microsoft Office 2003 products such as Word and Excel.

WSS also provides the infrastructure for generating UIs by including Web Part Pages and Web Parts. Web Part Pages and Web Parts are a powerful aspect of SharePoint because each WSS site provides a browser-based UI that is fully customizable. Web Parts can also be used to track each users' personalization information. We will discuss Web Parts in greater detail later in this article.

SharePoint Portal Server 2003 is used to build enterprise-level portal sites that are part of the Microsoft Office 2003 System. It is important to note that SPS is built on top of WSS. SPS complements WSS by adding manageability features that have been designed to assist users in navigating through vast amounts of information and documents. SPS also supplies additional functionality to enhance portal sites using indexing, searching, audience targeting, and single sign-on.

There is a fundamental difference in the roles played by WSS and SPS, as shown in Figure 1. WSS is based on a collaboration theme in the sense that it's designed to store and share list-based data and documents. SPS, on the other hand, is based on an aggregation theme. An SPS portal site is useful for aggregating information and documents from many different places. SPS adds value because it provides users with a quick and easy way to find information and documents that are spread out around a private network or scattered throughout the Internet.

Figure 1 The Roles of WSS and SPS (Close Figure 1)

Windows SharePoint Services 2.0 Collaboration Theme
Site creation and management
Membership and authorization
Shared lists and document libraries
Generation of user interface with
Web Part Pages and Web Parts
Personalization
Alerts and Notifications
Site-level search
SharePoint Portal Server 2003 Aggregation Theme
Enterprise-level search
Topic Areas (aggregate views)
Active Directory integration
Audiences and profiles
Shared services
My Site

In essence, WSS gives you a place to put all your content while SPS provides the means to navigate and search through your content when you need it. These two roles are quite complementary to one another. WSS allows an enterprise-level company to create and maintain tens of thousands of collaborative Web sites, while one or more SPS portal sites allow users to find what they're looking for by searching through all this content.

You should understand that SPS depends on WSS to provide many essential services. For example, WSS provides SPS with the capacity to track members and share lists and documents. Furthermore, SPS doesn't need to reinvent its own rendering engine for generating the HTML for a portal site's user interface. Instead, SPS takes advantage of the WSS Web Part Page and Web Part infrastructure to construct the user interface for an SPS portal site.

Windows SharePoint Services

WSS is free to any individual or company that has purchased the Windows Server 2003 operating system. The WSS installation files can be downloaded using the Windows Update service or from the Microsoft.com SharePoint site.

The WSS framework is built on top of Windows Server 2003, IIS 6.0, and ASP.NET. Figure 2 shows how the fundamental pieces of the WSS framework fit together. Note that before you can successfully install WSS on Windows Server 2003, you must first configure the host computer as an Application Server by enabling IIS 6.0 and ASP.NET 1.1.

Figure 2 WSS Architecture
Figure 2 WSS Architecture

Learning about WSS is easier once you know a few facts about its history. WSS is part of the second generation of SharePoint products and technologies. The first generation was built on an earlier IIS-based framework named SharePoint Team Services (STS). The STS framework is similar to WSS in that it provides a collaborative framework for sharing list-based data and documents. STS was not, however, built on the .NET Framework or ASP.NET. Instead, it was built using a proprietary ISAPI extension.

Customizing and extending STS Web sites has always been relatively difficult due to limited tool support. Site customization is much easier with WSS and the second generation of SharePoint because of WSS-compatible Web designers such as FrontPage® 2003. WSS also offers a much better extensibility model because you can write custom applications and Web Parts for WSS sites and SPS portal sites with Visual Studio® .NET using either C# or Visual Basic® .NET.

STS and the first generation of SharePoint products and technologies have also suffered from scalability problems. An underlying architecture that runs a stateful design on front-end Web servers is responsible. This flaw makes it impossible to scale out STS Web sites using a Web farm environment. When Microsoft engineers started to design WSS and the second generation of SharePoint products and technologies, they made it a primary design goal to fix the scaling problems of STS. Consequently, they designed their architecture to support WSS and SPS deployments with tens of thousands of users and tens of thousands of Web sites.

Figure 3 Web Farm Configuration
Figure 3 Web Farm Configuration

The WSS architecture is based on stateless front-end Web servers. It is founded on an integrated storage strategy where all the list-based data and documents associated with a Web site are stored in a SQL Server™ database, as shown in Figure 3. A key objective of this integrated storage strategy is that it allows a deployment of WSS Web servers to effectively scale out in a Web farm environment.

Databases

WSS relies on two different kinds of SQL Server databases: configuration databases and content databases. As you might suspect, the configuration database holds deployment-specific configuration information for each physical Web server, IIS virtual server, and WSS Web site. Content databases, on the other hand, hold data associated with WSS Web sites.

WSS requires exactly one configuration database for each deployment of WSS Web servers. A simple deployment can involve a single host computer running both the WSS front-end Web server components and SQL Server with the configuration database and a single content database. A more scalable WSS deployment involves a Web farm scenario with multiple front-end Web servers. A WSS deployment can also involve multiple back-end database servers. For example, in certain deployments scaling advantages can be gained by spreading several content databases across multiple host computers running SQL Server. However, there is only one configuration database for each WSS deployment. The configuration database provides a central repository of information to coordinate all front-end and back-end servers.

Each content database stores the data for one or more WSS Web sites. WSS data that is stored on a site-by-site basis includes lists and documents as well as all information pertaining to site customization and personalization. This integrated storage strategy provides a definite improvement over STS where site-specific data was stored in the file system and in the registry in addition to the database. The fact that everything pertaining to a site is stored within a single SQL Server database makes it significantly easier to back up and restore Web sites under WSS.

If you're just getting started with WSS, you can install everything you need on a single computer running Windows Server 2003. Once you have installed Windows Server 2003, configure the computer as an Application Server by enabling IIS 6.0 and ASP.NET 1.1. When you configure IIS and ASP.NET, make sure that you do not install the FrontPage Server Extensions because they are incompatible with WSS. After that you can install WSS by running the installation program STSV2.EXE (see Microsoft Windows SharePoint Services Is Now Available).

For single-machine deployments, WSS doesn't require you to use the full-blown version of SQL Server. The WSS installation program can optionally install a special version of MSDE called Windows MSDE (WMSDE). WMSDE is unlike the standard version of MSDE because it doesn't limit you to 2GB of storage and 10 database connections. However, WMSDE is more restrictive than MSDE because it only supports table schemas signed by Microsoft. This means you cannot use WMSDE to create your own custom databases and tables.

If you would like to install SPS on a development workstation, the installation process is more complicated. First of all, there are several SPS features that require Active Directory. Therefore, you should install SPS on servers that are part of an Active Directory® domain. Before installing SPS on a host computer running Windows Server 2003, first add that machine to an existing Active Directory domain or configure it to be a domain controller. Once the computer running Windows Server 2003 is in a domain, configure it to be an Application Server by enabling IIS 6.0 and ASP.NET 1.1. With SPS, we also recommend that you install SQL Server instead of relying on WMSDE to store the database for portal sites.

Once you have installed Windows Server 2003 on a computer with IIS 6.0, ASP.NET 1.1, and SQL Server, and added it to an Active Directory domain, you are ready to install SharePoint Portal Server 2003. The first thing the SPS installation program does is install WSS. Next, SPS installs itself on WSS. Finally, the SPS installer displays a wizard that steps you through the creation of the configuration database and the initial portal site.

Note that SPS creates additional databases to track data for SPS services, user profiles, audiences, and security credentials for the single sign-on service. SPS also extends the standard WSS database schemas for the configuration database as well as for content databases that contain portal sites.

Before installing either WSS or SPS in a deployment with SQL Server (as opposed to WMSDE), you might be required to add a SQL login for the Windows account that serves as the identity for the IIS application pools used by SharePoint. By default, WSS and SPS configure IIS application pools to run under the identity of the NT AUTHORITY\NETWORK SERVICE account. If you don't like this default setting, you can reconfigure any IIS application pool to use a different Windows account. Whichever Windows account you decide to use, however, you will need to have a SQL login to log on to SQL Server.

Once you have added the SQL login for the appropriate account within SQL Server, you must add the account into the Server Roles of Security Administrators and Database Creators. If you don't follow these steps, you will experience errors when installing or configuring either WSS or SPS. That's because WSS and SPS by default use IIS application pools that run under the identity of the NT AUTHORITY\NETWORK SERVICE account, which has no permissions to log onto SQL Server. You must configure SQL Server to grant this account the permissions to create databases and configure their security settings.

IIS Web Sites and Virtual Servers

WSS Web site configuration starts at the level of the IIS Web site. A default installation of IIS creates an IIS Web site named "Default Web Site" that is configured to listen for incoming HTTP requests across port 80. You can use the IIS administrative tools to create additional IIS Web sites that listen for incoming HTTP requests on a different port or with a different host header. Figure 4 shows an example of an IIS Web server with two virtual servers that have been configured for user access.

Figure 4 Two Virtual Servers for User Access
Figure 4 Two Virtual Servers for User Access

Note that the installation of WSS adds its own special IIS Web site named SharePoint Central Administration. WSS uses this IIS Web site to run the SharePoint Central Administration HTML pages. The SharePoint Central Administration Web site is configured to use a random port number during installation.

In SharePoint terminology, an IIS Web site is known as a virtual server. A virtual server must be extended with WSS in order to run WSS Web sites. When you install WSS with the default settings, it automatically extends the virtual server that is listening on port 80. You can extend WSS onto another virtual server by using the SharePoint Central Administration HTML pages or a WSS command-line administration tool named STSADM.EXE.

WSS is unlike ASP.NET in that it doesn't configure each Web site using an IIS virtual directory. Instead, WSS tracks all configuration information for WSS Web sites inside the configuration database and content databases. This means that once WSS has extended a virtual server and you start creating WSS Web sites, they will not appear in the IIS metabase. Instead an entry will be created in the configuration database and in the appropriate content database. In reality, IIS doesn't know whether a WSS-extended virtual server contains a single WSS Web site or 10,000 WSS Web sites. Because WSS doesn't need to configure an IIS virtual directory for each new WSS Web site, you get better scalability and improved maintenance.

When WSS extends itself onto a virtual server, it installs a custom ISAPI filter called the WSS filter (STSFLTR.DLL) which intercepts each request routed to that virtual server and determines whether the request should be handled by WSS or IIS. To make this determination, the WSS filter inspects the URL of the incoming request and consults the configuration database to determine who should process it. Keep in mind that you may have a virtual server in which you plan to run WSS sites as well as other ASP and ASP.NET applications.

When WSS extends itself onto a virtual server, it adds a web.config file to the root directory of the hosting virtual server. This web.config file provides initial configuration settings for WSS and for all ASP.NET code that runs from inside that virtual server. By default, this web.config file contains fairly restrictive security settings. You may be required to modify sections of this web.config file when you want your code to run with elevated permissions or when you want to test or deploy a new custom Web Part.

In SharePoint terminology, a URL space is the set of all possible URLs that target a virtual server. WSS divides the URL space of an extended virtual server into managed paths. Paths that are managed by WSS are considered to be included paths, while those that are not managed by WSS are considered to be excluded paths. When the WSS filter sees an incoming request with a URL that is part of an excluded path, it knows to route the request back to IIS for standard ASP or ASP.NET processing.

What should you do when you want to host a standard ASP.NET application within a virtual server that has been extended with WSS? For example, imagine you want to deploy an ASP.NET Web application at http://AcmeServer/webapp1 so that it's available alongside WSS sites in the same virtual directory. At this point you will need to understand how to configure both included and excluded paths. In particular, you need to add an excluded path for the URL space used by the ASP.NET Web application. Figure 5 shows part of the user interface provided by the SharePoint Central Administration HTML pages for defining managed paths.

Figure 5 WSS Admin UI for Defining Managed Paths (Close Figure 5)

Take note that Figure 5 shows the virtual server's included path as well as its excluded paths. Some included paths such as sites are defined as "wildcard inclusions." The purpose of a wildcard inclusion is make URL parsing more efficient. WSS doesn't allow the nesting of excluded paths within the URL space of a wildcard inclusion. Once the WSS filter sees that the first part of the URL for an incoming request matches a wildcard inclusion such as http://AcmeServer/sites, it knows that the request should be directed to WSS for processing.

When the WSS filter determines that the URL of an incoming request is inside an included path, it consults the configuration database to locate the target WSS site. WSS tracks each of its sites in the configuration database using the site's URL and an identifying GUID. There is additional information in the configuration database linking each site to a target content database.

Within a WSS deployment, it is often useful to extend more than one virtual server. You might want to do this for flexibility. Each virtual server can be configured using an IIS Application Pool to run its WSS Web sites in an isolated process. IIS also makes it possible to configure the security authentication requirements differently for each virtual server. One virtual server can be configured to require Basic Authentication while a second can be configured to require Integrated Windows Authentication.

In some deployments, every WSS-extended virtual server will have its own separate content database. It's also possible to extend a virtual server so that it attaches to the same content database used by another virtual server. This allows a pair of virtual servers to provide two different access points to a common set of WSS sites. This is valuable because the security settings for each virtual server can be configured independently. For example, a virtual server used to provide public-facing Web sites is typically configured to require SSL and Basic Authentication. A second virtual server providing access to users behind a firewall can be configured to require Integrated Windows Authentication.

Site Collections, Sites, and Workspaces

Within a virtual server, WSS sites are partitioned using site collections—a set of one or more sites that constitutes a unit of ownership. Creating a new site collection requires you to supply a Windows login account and an e-mail address for its owner. WSS tracks information about site collection owners so it can grant them administrative permissions and send them e-mail notifications regarding site collection maintenance issues.

A site collection always includes a top-level site that has the same URL as the site collection itself. When you create a new site collection, the top-level site is then created automatically. In addition to its top-level site, a site collection can optionally contain other secondary sites that are related to the top-level site through parent-child relationships, as is shown in the diagram in Figure 6. Every site must be created within a specific site collection and all the sites in the same site collection are stored in the same content database.

Figure 6 Site Collections within a Virtual Server (Close Figure 6)

The site collection serves as a unit of backup and restore in WSS. The STSADM.EXE administrative tool provides easy-to-use commands for backing up and restoring a site collection. After backing up a site collection, you can later restore it on the same Web server or on a different one. You will find that WSS makes it a little more difficult to back up and restore individual sites that exist within a site collection below the top-level site. If you are creating a site that you would like to back up and restore independently, you should create the site as a top-level site within a new site collection.

When performing day-to-day maintenance in a large WSS deployment with hundreds of thousands of site collections, it is not necessary to back up site collections individually. Instead, you can perform backups at the level of the content database. Backing-up content databases will be easy for many companies because they can use the same maintenance procedures that they use to back up other SQL Server databases.

Now let's take a step back and ask a fundamental question: what is a WSS site? First, a site is a container for storing content. Site contents are primarily stored in the form of lists, document libraries, and child sites. Second, a site is a securable entity whose content is accessible to a configurable set of users. A site can either define its own set of users or it can inherit the users of its parent site. Each site user maps to a Windows account defined within an Active Directory domain or a local Windows security accounts database. A site also contains a configurable set of groups and permissions that define the level of accessibility that various users have on the site's lists and document libraries.

Third, a site also happens to be a Web application with its own extensible and fully-customizable user interface. A site owner or Web designer can customize the layout and appearance of a site's pages and modify the site's navigation structure using either the browser or FrontPage 2003.

Finally, a site is the foundation for using Web Part Page and Web Part technology. Site owners and Web designers can customize Web Part Pages by adding and configuring shared Web Parts. A user can further personalize a Web Part Page by modifying, adding, or removing Web Parts. All the customization and personalization data associated with the Web Parts on a Web Part Page are automatically stored in the content database.

WSS supports several different site templates for "workspaces." Technically speaking, a workspace is really just a standard WSS site. However, the purpose of a workspace is more narrowly defined than other types of WSS sites because it often focuses on a single document or meeting. Furthermore, sites that are workspaces contain predefined lists to provide better integration with Microsoft Office products such as Word, Excel, Access, and Outlook®.

Every site is initially created from a site template. A site template is a blueprint that defines an initial set of lists, document libraries, Web Part Pages, and Web Parts that exist within a new site. WSS ships with eight site templates including Team Site, Blank Site, Document Workspace, Basic Meeting Workspace, Blank Meeting Workspace, Decision Meeting Workspace, Social Meeting Workspace, and Multipage Meeting Workspace. When a site is first accessed after being created, WSS prompts the user to select one of the available site templates using the HTML page shown in Figure 7.

Figure 7 Selecting a Site Template
Figure 7 Selecting a Site Template

WSS supports the creation and use of custom site templates. You can customize a site to your heart's content with FrontPage 2003 and then save the site as a site template. You can save a site as a site template with a few clicks in the browser. Once the custom site template has been created, it is then available for creating new sites within the same site collection. With some additional effort, a custom site template can be exported to other site collections and can also be used to create top-level sites.

SPS Architecture

SPS builds on top of the architectural layer provided by WSS. When you create a new SPS portal site, you are really creating a new physical WSS site collection. However, there are a few noteworthy restrictions about how SPS uses WSS. There can be only one portal site per virtual server. Furthermore, the site collection containing a portal site must always be created at the root URL of the hosting virtual server.

Once you have created a portal site at the root of a virtual server, you can then create additional standard WSS site collections within the same virtual server. If fact, that's exactly what SPS does when you execute the Create Site command from a portal site. That is, SPS creates a top-level site within a new site collection. When a portal site user creates a personal site by clicking the "My Site" link, SPS also creates this personal site as a top-level site within a new site collection. This strategy makes it easier to back up and restore shared and personal sites within an SPS deployment.

SPS extends the way in which WSS stores content by introducing areas and listings. Remember that SPS is a product designed around the theme of aggregation. Areas and listings are used by portal site content managers to aggregate content from other places. An area is a container of listings and nested sub-areas. Occasionally, a listing will contain HTML-based text. More often, a listing contains a link to non-portal content such as a document, Web page, or WSS list across the network. The key point is that listings allow SPS to link users to non-portal content such as shared folders, Microsoft Exchange public folders, WSS sites, public Web sites, and Lotus Notes, as shown in Figure 8.

Figure 8 SPS Listings
Figure 8 SPS Listings

The areas within a portal site are structured into a parent-child hierarchy. Figure 9 shows the HTML-based interface that SPS provides for working with areas. This user interface makes it relatively simple for portal site content managers to add and delete areas and listings from within the areas hierarchy of a portal site. SPS also makes it simple for anyone to use the browser to relocate an area or listing within the areas hierarchy.

Figure 9 SPS Areas
Figure 9 SPS Areas

Areas and listings facilitate aggregation because they make it easy to navigate through content both inside and outside a portal site. Areas and listings are also integrated with the SPS search service. When you run an SPS search within the scope of an area, SPS searches through the content and links of that area's listings as well as the content and links of listings within sub-areas nested below.

The SPS search service is complemented by the SPS indexing service. The indexing service is designed to build indices by crawling through the content links provided by areas and listings. The indexing service even knows how to crawl through Word documents and Excel spreadsheets and search for specific keywords. Once the set of indices has been built, the SPS search service is able to perform fast searches that allow users to find the content they need.

Conclusion

This article has provided an in-depth examination of the architecture of SharePoint products and technologies. WSS and SPS are two different products that have been designed to work well together. WSS provides the foundation for creating collaborative Web sites that support customization and personalization. SPS complements WSS by playing the role of a content aggregator. You have also seen that SPS is fully dependent on the underlying infrastructure of WSS. In fact, the more you know about the foundations of WSS, the easier it will be to get the most out of SPS.

In one sense, SharePoint products and technologies reduce the need for custom software development because WSS and SPS provide so much functionality right out of the box. WSS and SPS also provide valuable opportunities to write custom applications and Web Parts. This article has given you a roadmap to get started with SharePoint development. Next time we will continue where we left off and dive into the WSS and SPS object models as well as the development of custom Web Parts.

Jason Masterman and Ted Pattison have both been working in the software industry for over a decade writing software, consulting, and educating developers. In 2003, Jason and Ted founded Barracuda .NET, a leading-edge consulting and training company that focuses on Microsoft collaboration technologies. Go to http://Barracuda.net to find out more about their training classes and online events.


.NET Tools: Ten Must-Have Tools Every Developer Should Download Now

New information has been added to this article since publication.
Refer to the Editor's Update below.

.NET Tools

Ten Must-Have Tools Every Developer Should Download Now

James Avery
This article discusses:
  • NUnit to write unit tests
  • NDoc to create code documentation
  • NAnt to build your solutions
  • CodeSmith to generate code
  • FxCop to police your code
  • Snippet Compiler to compile small bits of code
  • Two different switcher tools, the ASP.NET Version Switcher and the Visual Studio .NET Project Converter
  • Regulator to build regular expressions
  • .NET Reflector to examine assemblies
This article uses the following technologies:
.NET, C# or Visual Basic .NET, Visual Studio .NET
Contents
Snippet Compiler
Regulator
CodeSmith
Building a Custom Template
NUnit
Writing an NUnit Test
FxCop
Lutz Roeder's .NET Reflector
NDoc
NAnt
NAnt in Action
Switch Tools
Conclusion

Y
ou cannot expect to build a first-class application unless you use the best available tools. Besides well-known tools such as Visual Studio® .NET, there are a multitude of small, lesser-known tools available from the .NET community. In this article, I'm going to introduce you to some of the best free tools available today that target .NET development. I'll walk you through a quick tutorial of how to use each of them, some of which will save you a minute here and there, while others may completely change the way that you write code. Because I am squeezing so many different tools into this single article, I will not be able to cover each of them extensively, but you should learn enough about each to decide which tools are useful for your projects.

Snippet Compiler

The Snippet Compiler is a small Windows®-based application that allows you to write, compile, and run code. This tool is useful if you have small pieces of code for which you don't want to create an entire Visual Studio .NET project (along with all the files that come with it).

As an example, let's say that I wanted to show you how to launch another application from the Microsoft® .NET Framework. In the Snippet Compiler I would start by creating a new file which creates a small console application. The snippet can be created inside the Main method of the console application, which is what I will do here. The following code snippet demonstrates how to create an instance of Notepad from the .NET Framework:

System.Diagnostics.Process proc = new System.Diagnostics.Process();
proc.StartInfo.FileName= "notepad.exe";
proc.Start();
proc.WaitForExit();
Of course this snippet would not compile by itself, but that is where Snippet Compiler comes into play. Figure 1 shows this code sample in Snippet Compiler.

Figure 1 Snippet Compiler
Figure 1 Snippet Compiler

To test this snippet, just press the play button (green triangle), and it will run in debug mode. The snippet will generate a console application popup, and Notepad will appear. When you close Notepad, the console application will close as well.

Personally, I have found Snippet Compiler to be invaluable when trying to create a small example for someone who has asked me for help, when normally I would have to create a project, make sure everything compiles, send them the code snippet, and then delete the project. Snippet Compiler makes this process much easier and much more pleasant.

Snippet Compiler was written by Jeff Key and can be downloaded from http://www.sliver.com/dotnet/SnippetCompiler.

Regulator

Regulator is the most recent addition to my top tools list. It is a full-featured tool that makes it easy to build and test regular expressions. There is a renewed interest in regular expressions because of the excellent support for them in the .NET Framework. Regular expressions are used to define patterns in strings based on characters, frequency, and character order. They are most commonly used as a means to validate user input or as a way to find a string of characters inside a larger string—for instance, when looking for a URL or e-mail address on a Web page.

Regulator allows you to enter a regular expression and some input against which you would be running this expression. This way you can see how the regular expression will act and what kind of matches it will return before implementing it in your application. Figure 2 shows Regulator with a simple regular expression.

Figure 2 Regulator (Close Figure 2)

The document contains the regular expression, in this example it is [0-9]* which should match any number of digits in a row. The box in the bottom-right contains the input for this regular expression, and the box on the bottom-left shows the matches that this regular expression finds in the input. The ability to write and test regular expressions in a separate application like this is much easier than trying to work with them in your app.

One of the best features in Regulator is the ability to search the online regular expressions library at regexlib.com. For example, if you enter the string "phone" in the search box, you will find more than 20 different regular expressions that will match various phone numbers, including expressions for UK, Australian, and many other phone numbers.

Regulator was written by Roy Osherove and can be downloaded at http://regex.osherove.com.

CodeSmith

CodeSmith is a template-based code-generation tool that uses a syntax similar to ASP.NET to generate any type of code or text. Unlike many other code-generation tools, CodeSmith does not require you to subscribe to a particular application design or architecture. Using CodeSmith, you can generate anything from a simple, strongly typed collection to an entire application.

When you are building an application, you will often find yourself repeating certain tasks, whether it's writing data access code or building custom collections. CodeSmith is particularly useful at such times because you can write templates to automate those tasks and not only improve your productivity but also automate the tasks that are the most tedious to perform.

CodeSmith ships with a number of templates, including ones for all the .NET collection types as well as ones to generate stored procedures, but the real power of this tool comes from being able to create custom templates. To get you started, I'll provide a quick introduction to building a custom template.

Building a Custom Template

CodeSmith templates are simply text files which you can create in any text editor. Their only requirement is that they be saved with the .cst file extension. The sample template that I'm going to build will accept a string and then build a class based on that string. The first step to creating a template is to add the template header, which declares the language of the template, the target language, and a brief description of the template:

<%@ CodeTemplate Language="C#"
   TargetLanguage="C#"
   Description="Car Template" %>

The next part of the template is the property declarations, where you declare the properties that will be specified each time the template is run. With this template, the single property that I'm going to use is just a string, so the property declaration looks like this:

<%@ Property Name="ClassName" Type="String" Category="Context"
    Description="Class Name" %>

This property declaration will make the ClassName property appear in the CodeSmith property window so that it can be specified when the template is run. The next step is to actually build the body of the template, which is very similar to coding with ASP.NET. You can see the body of the template in Figure 3. [ Editor's Update - 6/16/2004: The code in Figure 3 has been updated to be safe for multithreaded operations.]

Figure 3 Class Generation (Close Figure 3)

Custom Template
public sealed class <%= ClassName %>
{
      private static volatile <%= ClassName %> _instance;
      private <%= ClassName %>() {}
      private static readonly object _syncRoot = new object();

      public static <%= ClassName %> Value
      {
            get
            {
                  if (_instance == null)
                  {
                        lock(_syncRoot)
                        {
                              if (_instance == null)
                              {
                                    _instance = new <%= ClassName %>();
                              }
                        }
                  }
                  return _instance;
            }
      }
}
SingletonClass
public sealed class SingletonClass
{
      private static volatile SingletonClass _instance;
      private SingletonClass() {}
      private static readonly object _syncRoot = new object();

      public static SingletonClass Value
      {
            get
            {
                  if (_instance == null)
                  {
                        lock(_syncRoot)
                        {
                              if (_instance == null)
                              {
                                    _instance = new SingletonClass();
                              }
                        }
                  }
                  return _instance;
            }
      }
}

As you can see, the template will take the string input and generate a singleton class using that class name. In the template body, the same opening and closing tags are used as in ASP.NET. In this template, I am simply inserting the property value, but you can also use any type of .NET code inside these tags. Once the template is complete, you load it into CodeSmith by either double-clicking or opening it from the CodeSmith application. Figure 4 shows this template loaded into CodeSmith.

Figure 4 CodeSmith Template (Close Figure 4)

You can see that the property on the left is the one I declared in the template. If I enter "SingletonClass" as the class name and click the Generate button, the class shown in the bottom part of Figure 3 will be generated.

CodeSmith is relatively easy to use and can produce some incredible results if applied correctly. One of the most common sections of an application that is targeted for code generation is the data access layer. CodeSmith includes a special assembly called the SchemaExplorer which can be used to generate templates from tables, stored procedures, or almost any other SQL Server™ object.

CodeSmith was written by Eric J. Smith and is available for download at http://www.ericjsmith.net/codesmith.

NUnit

NUnit is an open source unit testing framework built for the .NET Framework. NUnit allows you to write tests in the language of your choice to test a specific function of your application. Unit tests are an excellent way to test the functionality of your code when you first write it, and also to provide a method for regression testing of your application. The NUnit application provides a framework for writing unit tests, as well as a graphical interface to run these tests and view the results.

Writing an NUnit Test

As an example, I'm going to test the functionality of the Hashtable class in the .NET Framework to determine if two objects can be added and then retrieved. My first step will be to add a reference to the NUnit.Framework assembly, which will give me access to the attributes and methods of the NUnit framework. Next I'll create a class and mark it with the TestFixture attribute. This attribute lets NUnit know that this class contains NUnit tests:

using System;
using System.Collections;
using NUnit.Framework;

namespace NUnitExample
{
    [TestFixture]
    public class HashtableTest {
        public HashtableTest() {

        }
    }
}

Next I'll create a method and mark it with the [Test] attribute so that NUnit knows that this method is a test. Then I'll set up a Hashtable and add two values to it, then use the Assert.AreEqual method to see if I can retrieve the same values that I added to the Hashtable, as shown in the following:

[Test]
public void HashtableAddTest()
{
    Hashtable ht = new Hashtable();

    ht.Add("Key1", "Value1");
    ht.Add("Key2", "Value2");

    Assert.AreEqual("Value1", ht["Key1"], "Wrong object returned!");
    Assert.AreEqual("Value2", ht["Key2"], "Wrong object returned!");
}

This will confirm that I can add and then retrieve values from the Hashtable—a simple test, but one that showcases the capabilities of NUnit. There are a number of test types, as well as various Assert methods, that can be used to test every part of your code.

To run this test, I'll need to build the project, open the generated assembly in the NUnit application, and then click the Run button. Figure 5 shows the results. I get a warm and fuzzy feeling when I see that big green bar because it lets me know that the test passed. This simple example shows how easy and powerful NUnit and unit testing can be. Being able to write a unit test that can be saved and rerun whenever you change code not only makes it easier for you to detect defects in your code, but the result is that you can deliver better applications.

Figure 5 NUnit
Figure 5 NUnit

NUnit is an open-source project that is available for download from http://www.nunit.org. There is also an excellent NUnit Visual Studio .NET add-in which allows you to run unit tests directly from Visual Studio. This can be found at http://sourceforge.net/projects/nunitaddin. For more information on NUnit and its place in test-driven development, see the article "Test-Driven C#: Improve the Design and Flexibility of Your Project with Extreme Programming Techniques" in the April 2004 issue of MSDN® Magazine.

FxCop

The .NET Framework is very powerful, which means there is great potential to create excellent applications, but there is equal opportunity to create poor programs. FxCop is one of the tools that can be used to help create better applications by enabling you to examine an assembly and check it for compliance using a number of different rules. FxCop comes with a set number of rules created by Microsoft, but you can also create and include your own rules. For instance, if you decided that all classes should have a default constructor that takes no arguments, you could write a rule that checks for a constructor on each class of an assembly. This way, no matter who writes the code, you will have a certain level of consistency. If you want more information on creating custom rules, see John Robbins' Bugslayer column on the subject in the June 2004 issue of MSDN Magazine.

So let's take a look at FxCop in action and see what it finds wrong with the NUnitExample assembly that I have been working with. When you open FxCop you first need to create an FxCop project and then add to it the assembly that you want to test. Once the assembly is added to the project, you can press Analyze, and FxCop will examine the assembly. The errors and warning found in this assembly are shown in Figure 6.

Figure 6 Errors and Warning Found by FxCop (Close Figure 6)

FxCop found a couple of problems with my assembly. You can double-click on an error to see the details, including a description of the rule and where you can find more information. (Something you can do for fun is run FxCop on the Framework assemblies and see what turns up.)

FxCop can help you create better, more consistent code, but it cannot make up for poor application design or just plain poor programming. FxCop is also not a replacement for peer code review, but because it can catch a lot of errors before code review, more time can be spent on serious issues rather than having to worry about naming conventions.

FxCop was developed by Microsoft and is available for download from http://www.gotdotnet.com/team/fxcop.

Lutz Roeder's .NET Reflector

The next essential tool is called .NET Reflector, which is a class browser and decompiler that can examine an assembly and show you just about all of its secrets. The .NET Framework introduced reflection which can be used to examine any .NET-based code, whether it is a single class or an entire assembly. Reflection can also be used to retrieve information about the various classes, methods, and properties included in a particular assembly. Using .NET Reflector, you can browse the classes and methods of an assembly, you can examine the Microsoft intermediate language (MSIL) generated by these classes and methods, and you can decompile the classes and methods and see the equivalent in C# or Visual Basic® .NET.

To demonstrate the workings of .NET Reflector, I am going to load and examine the NUnitExample assembly already shown. Figure 7 shows this assembly loaded in .NET Reflector.

Figure 7 NUnitExample Assembly
Figure 7 NUnitExample Assembly

Inside of .NET Reflector there are various tools that you can use to examine this assembly further. To view the MSIL that makes up a method, click on the method and select Disassembler from the menu.

In addition to being able to view the MSIL, you can also view the method as C# by selecting Decompiler under the Tools menu. You could also view this method decompiled to Visual Basic .NET or Delphi by changing your selection under the Languages menu. Here is the code that .NET Reflector generated:

public void HashtableAddTest()
{
    Hashtable hashtable1;
    hashtable1 = new Hashtable();
    hashtable1.Add("Key1", "Value1");
    hashtable1.Add("Key2", "Value2");
    Assert.AreEqual("Value1", hashtable1["Key1"],
       "Wrong object returned!");
    Assert.AreEqual("Value2", hashtable1["Key2"],
       "Wrong object returned!");
}
The previous code looks very much like the code I actually wrote for this method. Here is the actual code from this assembly:
public void HashtableAddTest()
{
    Hashtable ht = new Hashtable();

    ht.Add("Key1", "Value1");
    ht.Add("Key2", "Value2");

    Assert.AreEqual("Value1", ht["Key1"],
        "Wrong object returned!");
    Assert.AreEqual("Value2", ht["Key2"],
        "Wrong object returned!");
}
Although there are some minor differences with the code, they are functionally identical.

While this example was a good way to show actual code versus decompiled code, it does not represent what I consider to be the best use of .NET Reflector, which is to examine .NET Framework assemblies and methods. The .NET Framework offers many different ways to perform similar operations. For example, if you need to read a set of data from XML, there are a variety of different ways to do this using XmlDocument, XPathNavigator, or XmlReader. By using .NET Reflector, you can see what Microsoft used when writing the ReadXml method of the DataSet, or what they did when reading data from the configuration files. .NET Reflector is also an excellent way to see the best practices for creating objects like HttpHandlers or configuration handlers because you get to see how the team at Microsoft actually built those objects in the Framework.

.NET Reflector was written by Lutz Roeder and can be downloaded from http://www.aisto.com/roeder/dotnet.

NDoc

Code documentation is almost always a dreaded task. I am not talking about the early design documents, or even the more detailed design documents; I am talking about documenting individual methods and properties on classes. The NDoc tool will automatically generate documentation for your code using reflection to examine the assembly and using the XML generated from your C# XML comments. XML comments are only available for C#, but there is a Visual Studio .NET Power Toy called VBCommenter which will do something similar for Visual Basic .NET. In addition, the next release of Visual Studio will support XML comments for more languages.

With NDoc you are technically still documenting your code, but you are documenting as you write it (in the XML comments), which is much easier to swallow. The first step when using NDoc is to turn on XML comments generation for your assembly. Right-click the project and select Properties | Configuration Properties | Build, then enter a path in which to save the XML file in the XML Documentation File option. When the project is built, an XML file will be created with all of the XML comments included. Here is a look at a method from the NUnit example documented with XML:

/// <summary>
/// This test adds a number of values to the Hashtable collection
/// and then retrieves those values and checks if they match.
/// </summary>
[Test]
public void HashtableAddTest()
{
    //Method Body Here
}

The XML documentation on this method will be extracted and saved in the XML file, shown here:

<member name="M:NUnitExample.HashtableTest.HashtableAddTest">
  <summary>This test adds a number of values to the Hashtable collection
    and then retrieves those values and checks if they match.</summary>
</member>
NDoc uses reflection to look at your assembly, then reads the XML in this document, and matches them up. NDoc uses this data to create any number of different documentation formats, including HTML help files (CHMs). After generating the XML file, the next step is to load the assembly and the XML file into NDoc so they can be processed. This is done simply by opening NDoc and clicking the Add button.

Once the assembly and XML file are loaded into NDoc and after you customize the output using the range of properties available, clicking on the Generate button will start the process of generating the documentation. Using the default properties, NDoc generates some very attractive and functional .html and .chm files, thereby automating in a quick and efficient manner what would otherwise be a tedious task.

NDoc is an open source project and can be downloaded from http://ndoc.sourceforge.net.

NAnt

NAnt is a .NET-based build tool that, unlike the current version of Visual Studio .NET, makes it easy to create a build process for your project. When you have a large number of developers working on a single project, you can't rely on the build from a single user's box. You also do not want to have to build the project manually on a regular basis. Instead, you create an automated build process that runs every night. NAnt allows you to build your solution, copy files, run NUnit tests, send e-mail, and much more. Unfortunately, NAnt is lacking a nice looking graphical interface, but it does have a console application and XML files that specify which tasks should be completed during the build process. Note that MSBuild, the new build platform that's part of Visual Studio 2005, provides for very robust build scenarios and is similarly driven by XML-based project files.

NAnt in Action

In this example I am going to create an NAnt build file for the NUnitExample solution that I created earlier. First I need to create an XML file with the .build extension, place it in the root of my project, and then add an XML declaration to the top of the file. The first tag I need to add to the file is the project tag:

<?xml version="1.0"?>
<project name="NUnit Example" default="build" basedir=".">
    <description>The NUnit Example Project</description>
</project>
The project tag is also used to set the name of the project, the default target, and the base directory. The description tag is used to set a brief description of this project.

Next, I'll add the property tag, which can be used to store a setting in a single location that can then be accessed from anywhere in the file. In this case, I am going to create a property called debug, which I can then set to true or false, reflecting whether or not I want the project to be compiled in the debug configuration. (In the end, this particular property does not actually affect how the project will be built; it is simply a variable that you set and which will be read from later when you are actually determining how to build the project.)

Next, I need to create a target tag. A project can contain multiple targets which can be specified when NAnt is run. If no target is specified, the default is used, which I set in the project element. In this example, the default target is build. Let's take a look at the target element, which will contain the majority of the build info:

<target name="build" description="compiles the source code">
</target>
Inside the target element, I am going to set the name of the target to build and create a description of what this target will do. I'll also create a csc element, which is used to specify what should be passed to the csc C# compiler. Let's take a look at the csc element:
<csc target="library" output=".\bin\debug\NUnitExample.dll"
    debug="${debug}">
<references>
    <includes name="C:\program files\NUnit V2.1\bin\NUnit.Framework.dll"/>
</references>
    <sources>
       <includes name="HashtableTest.cs"/>
    </sources>
</csc>

First, I have to set the target of the csc element. In this case I will be creating a .dll file so I set the target to library. Next, I have to set the output of the csc element, which is where the .dll file will be created. Finally, I need to set the debug property, which determines whether the project will be compiled in debug. Since I created a property earlier to store this value, I can use the following string to access the value of that property: ${debug}. The csc element also contains a number of sub-elements. I need to create two elements: the references element will tell NAnt which assemblies I need to reference for this project, and the sources element will tell NAnt which files to include in the build. In this example, I reference the NUnit.Framework.dll assembly and include the HashtableTest.cs file. The complete build file is shown in Figure 8. (You would normally also create a clean target that would be used to delete the generated files, but I have omitted it for the sake of brevity.)

Figure 8 NAnt Build File (Close Figure 8)
<?xml version="1.0"?>
<project name="NUnit Example" default="build" basedir=".">
    <description>The NUnit Example Project</description>
    <property name="debug" value="true"/>
    <target name="build" description="compiles the source code">
        <csc target="library" output=".\bin\debug\NUnitExample.dll"
                debug="${debug}">
            <references>
                <includes name="C:\program files\NUnit
             V2.1\bin\NUnit.Framework.dll" />
            </references>
            <sources>
                <includes name="HashtableTest.cs"/>
            </sources>
        </csc>
    </target>
</project>

To build this file I need to go to the root directory of my project, where the build file is located, and execute nant.exe from that location. If the build is successful, you can find the .dll and .pdb file in the bin directory of this application. While using NAnt is definitely not as easy as clicking Build in Visual Studio, it is a very powerful tool for developing a build process that runs on an automated schedule. NAnt also includes helpful features such as the ability to run unit tests or copy additional files (features that are not supported by the current Visual Studio build process).

NAnt is an open source project and can be downloaded from http://sourceforge.net/projects/nant.

Switch Tools

I have lumped together two separate tools under the heading Switch Tools. These two tools are rather simple, but can be extremely useful. The first is the ASP.NET Version Switcher, which can be used to switch the version of ASP.NET that a virtual directory is running under. The second tool is the Visual Studio Converter, which can be used to switch a project file from Visual Studio .NET 2002 to Visual Studio .NET 2003.

When IIS handles a request, it looks at the extension of the file that is being requested, and then based on the extension mappings for that Web site or virtual directory, it either delegates the request to an ISAPI extension or handles it itself. This is how ASP.NET works; extension mappings are registered for all of the ASP.NET extensions and directs them to the aspnet_isapi.dll. This works flawlessly until you install ASP.NET 1.1, which upgrades the extension mapping to the new version of aspnet_isapi.dll. This causes errors when an application built on ASP.NET 1.0 tries to run with version 1.1. To fix this, you can switch all of the extension mappings back to the 1.0 version of aspnet_isapi.dll, but with 18 extension mappings it is not a lot of fun to do this by hand. This is where the ASP.NET Version Switcher becomes useful. This small utility can be used to switch the version of the .NET Framework that any single ASP.NET application is using.

Figure 9 ASP.NET Version Switcher
Figure 9 ASP.NET Version Switcher

Figure 9 shows the ASP.NET Version Switcher in action. Using it is as simple as selecting the application and then selecting the version of the .NET Framework that you would like the application to use. The tool then uses the aspnet_regiis.exe command-line tool to switch the application to the selected version of the Framework. This tool will become even more useful as future versions of ASP.NET and the .NET Framework are released.

ASP.NET Version Switcher was written by Denis Bauer and is available for download from http://www.denisbauer.com/NETTools/ASPNETVersionSwitcher.aspx.

The Visual Studio .NET Project Converter (see Figure 10) is very similar to the ASP.NET Version Switcher, except that it is used to switch the version of a Visual Studio project file. Even though there is only a small difference between versions 1.0 and 1.1 of the .NET Framework, once a project file from Visual Studio .NET 2002 is converted to Visual Studio .NET 2003, it cannot be converted back. While this might not be an issue most of the time (since there are few breaking changes between the .NET Framework versions 1.0 and 1.1), at some point you may need to switch a project back. This converter can convert any solution or project file from Visual Studio 7.1 (Visual Studio .NET 2003) to Visual Studio 7.0 (Visual Studio .NET 2002), and back if necessary.

Figure 10 Visual Studio .NET Project Converter
Figure 10 Visual Studio .NET Project Converter

The Visual Studio .NET Project Converter was written by Dacris Software. It is available for download from http://www.codeproject.com/macro/vsconvert.asp.

Conclusion

This has been a bit of a whirlwind tour of these tools, but I have tried to present you with at least enough information to whet your appetite. I trust that this article has provided you with some insight into a couple of free tools that you can start using right away to write better projects. I also urge you to make sure you have all the other right tools available to you, whether it is newest version of Visual Studio, a powerful computer, or a free utility. Having the right tools can make all the difference.

James Avery is a consultant working with .NET and other Microsoft technologies. He has written a number of books and articles, his most recent book being ASP.NET Setup and Configuration Pocket Reference (Microsoft Press, 2003). You can e-mail him at javery@infozerk.com and read his weblog at http://www.dotavery.com/blog.


User Preferences: Manage User Settings in Your .NET App with a Custom Preferences API

User Preferences

Manage User Settings in Your .NET App with a Custom Preferences API

Ray Djajadinata
This article discusses:
  • Preferences management
  • Isolated storage in Windows
  • A custom preferences API in .NET
This article uses the following technologies:
Win32, the .NET Framework, and C#
Code download available at: CustomPreferences.exe (158KB)
Contents
A Good Preferences API
Support for Hierarchical Organization of Preferences
Encapsulation of Data Location
Shielding from Backing Store Details
XmlSerializer and Preferences Management
Preference Change Notification
Data Isolation
Security
Application Preferences Shared by All Users
An Easier Way to Manage Preferences
Introducing NAP
Tree Nodes and Leaves
What's Happening Under the Tree?
Simulating Isolation by Assembly
Simulating Isolation by Application Domain
Conclusion

T
he Microsoft® .NET Framework boasts tons of useful APIs that make it simple to create applications for Windows®. Surprisingly though, the Framework could be stronger in the area of user preference management. If you're a developer for Windows using .NET, you only have two preferences APIs available, one of which is a legacy of 16-bit Windows: the initialization file API. This API is not easily accessible from .NET-compliant languages other than C++ and it is no longer recommended for use (though it is possible using interop). The second option, and the only viable one, is the registry API.

What about the types inside the System.IO.IsolatedStorage namespace? Rather than a preferences API, the isolated storage API is more of a file I/O API that gives you access to your own private (isolated) file system. Although it has some properties that make it suitable for storing user preference settings, using it is equivalent to performing I/O operations. It doesn't make preferences management any easier; it just gives you a storage mechanism in the form of files and directories.

This means that if you want to store your preferences settings in any backing store other than the registry, you'll have to actually write your own code. Whether you use the file system, isolated storage, or a database, there's no API available for preferences management in the .NET Framework. Fortunately, writing code for a custom preferences API is not particularly difficult, as you'll see.

A Good Preferences API

What makes a good preferences API? This is actually a question about two different things. The first concerns how the API is designed. Can you, for example, manipulate your preferences values easily without being burdened with storage-specific details? Can you easily organize your preferences data into logical groups? Can you easily move from one group to another? Can you feel reasonably safe that other applications and users won't trample your data? And so the list goes.

The second point to consider is the API's backing store. For instance, the registry API's backing store is the registry, the central configuration settings data depository for Windows and Windows-based applications. This makes the registry quite delicate: sloppy programming, careless editing, or malicious code can corrupt the registry and bring the whole machine down, preventing Windows from even starting. On the other hand, a database may not always be available (and database queries can be expensive), so a preferences API that requires a database may not be very useful.

The right API and a suitable backing store can go a long way to making your preferences management tasks simpler, so let's consider what to look for in choosing them.

Support for Hierarchical Organization of Preferences

Usually, the more complex your application, the more preferences your users, and you the developer, have to worry about. As such, it is important that the API you use supports organization and partitioning of the preferences data.

Consider a sophisticated development environment such as Visual Studio® .NET. There are a large number of user preferences in such an application, including editor preferences such as whether to convert tabs to spaces as well as debugger preferences such as whether to display values in hexadecimal. In such a situation, it makes sense to store your data in a tree-like structure because it allows for easy hierarchical organization. You can divide preferences into main branches (such as Text Editor or Debugging), and have subbranches when it needs to be broken down into smaller logical groups (such as Edit and Continue, Just-In-Time, or Native). If the preferences API you're using doesn't support organizing preferences like this, then you're stuck with writing your own code to impose a hierarchical structure on your data.

For instance, the initialization files API only offers two levels of organization: by file name and by section. If your preferences organization has a deeper hierarchy than that (for example, Plug-In Development | Compilers | Features), then you can either partition your data into more files or write some extra code that can manage those extra levels.

On the other hand, the registry API gives you a tree structure to start with, as shown in Figure 1. This makes it quite easy for you to partition the preference settings information according to its logical grouping as you do when you're using Visual Studio.

Figure 1 Visual Studio Registry Settings (Close Figure 1)

Encapsulation of Data Location

If you store your preferences data in files, you'll end up maintaining the preferences data plus the location where you store them. That is, at some point you have to hardcode the location (or at least the names) of the files in your source code or in your configuration files somewhere. This means that your preferences code will work correctly only if your files are exactly in the location you specified. If the files are missing for some reason, so is the backing store for your preferences data.

Compare this to APIs such as the registry or isolated storage which shield you from the location of the data store. Unlike files, they are always there. Barring the absence of the required permissions, the registry predefined keys are always available to you. With isolated storage, you can always do the following to get a data store that is isolated by user and by assembly:

IsolatedStorageFile isfAssem =
    IsolatedStorageFile.GetUserStoreForAssembly();
The following code will get a data store isolated by user, assembly, and application domain:
IsolatedStorageFile isfDomain =
    IsolatedStorageFile.GetUserStoreForDomain();
This way, there's no question of whether your data store is there or not. You don't need to worry about possible cases of missing registry or missing isolated storage, and the need to hardcode a storage-specific location in your code is reduced.

Shielding from Backing Store Details

A good API allows you to concentrate on the task at hand so that when you're dealing with preferences you don't have to worry about the details of the backing store such as database tables or files and directories. In this regard, the registry API is more preferences-friendly than the isolated storage API. To illustrate this, let's consider a person who's developing a .NET-based messenger application called the Imaginary Messenger. This developer may decide to organize the user preferences as shown in Figure 2.

Figure 2 Imaginary Messenger Options
Figure 2 Imaginary Messenger Options

Now how would you go about storing this preference data? Based on Figure 2, if you're using the registry, you may organize it as shown in Figure 3. Following is the code that created the registry keys and values in Figure 3:

RegistryKey imKey =
    Registry.CurrentUser.CreateSubKey(@"Software\ImaginaryMessenger");

RegistryKey optionsPersonalKey = imKey.CreateSubKey(@"Options\Personal");
optionsPersonalKey.Close();

RegistryKey optionsPrivacyKey = imKey.CreateSubKey(@"Options\Privacy");
optionsPrivacyKey.SetValue("AllowNonFriendsToSeeStatus", 0);
optionsPrivacyKey.SetValue("LoginAsInvisible", 1);
optionsPrivacyKey.SetValue("ShowIdleTimeToFriends", 1);
optionsPrivacyKey.Close();

Figure 3 Organized Preferences in the Registry (Close Figure 3)

Although this registry code is clean and straightforward, the equivalent code for isolated storage won't be because, unlike the registry, you won't find a Hashtable-like interface with which to set the values and their names. Instead, you'd want to store each value in a separate file, as shown in Figure 4. You could also put the values into a Hashtable and serialize it into a file like this:

Hashtable ht = new Hashtable();
ht.Add("AllowNonFriendsToSeeStatus", 0);
ht.Add("LoginAsInvisible", 1);
ht.Add("ShowIdleTimeToFriends", 1);
Stream streamToValues = new IsolatedStorageFileStream(
    @"Options\Privacy\values",
    FileMode.OpenOrCreate, isf);
IFormatter formatter = new SoapFormatter();
formatter.Serialize(streamToValues, ht);
streamToValues.Close();

Figure 4 Creating the Isolated Storage (Close Figure 4)
IsolatedStorageFile isf = IsolatedStorageFile.GetUserStoreForDomain();
isf.CreateDirectory(@"Options");
isf.CreateDirectory(@"Options\Personal");
isf.CreateDirectory(@"Options\Privacy");

// handle preferences under Privacy now...
TextWriter writer = new StreamWriter(new IsolatedStorageFileStream(
    @"Options\Privacy\AllowNonFriendsToSeeStatus",
    FileMode.OpenOrCreate, isf));
writer.Write(0);
writer.Close();

TextWriter writer2 = new StreamWriter(new IsolatedStorageFileStream(
    @"Options\Privacy\LoginAsInvisible",
    FileMode.OpenOrCreate, isf));
writer2.Write(1);
writer2.Close();

TextWriter writer3 = new StreamWriter(new IsolatedStorageFileStream(
    @"Options\Privacy\ShowIdleTimeToFriends",
    FileMode.OpenOrCreate, isf));
writer3.Write(1);
writer3.Close();
isf.Close();

Of course, there are a multitude of other ways you can store these values. The point remains, though, that since the isolated storage API is more like a file I/O API than a preferences API, eventually you'll end up writing code to make sense of the contents of the files inside the store. The registry presents a simpler, friendlier interface in this case.

XmlSerializer and Preferences Management

Some of you may be wondering why I don't simply create a class to hold those values and then use XmlSerializer to read and write the class as a whole into a file in the isolated store (see Figure 5). In fact, this approach offers many advantages. You get strong typing for your preferences, which is good. Also, thanks to the forgiving nature of XmlSerializer, adding or removing fields from your preferences class doesn't invalidate the previous serialized versions of the class. It just works as expected most of the time, without requiring you to do any extra work for custom serialization the way BinaryFormatter or SoapFormatter might.

Figure 5 XmlSerializer Stores and Loads Preferences (Close Figure 5)
public class OptionsSettings {
    private OptionsPersonalSettings personal;
    private OptionsPrivacySettings privacy;
    public OptionsSettings() {
        personal = new OptionsPersonalSettings();
        privacy = new OptionsPrivacySettings();
    }
    public OptionsPersonalSettings Personal {...}
    public OptionsPrivacySettings Privacy {...}
    public void Store(Stream stream) {
        XmlSerializer xs = new XmlSerializer(this.GetType());
        xs.Serialize(stream, this);
    }
    public static OptionsSettings Load(Stream stream) {
        XmlSerializer xs = new XmlSerializer(typeof(OptionsSettings));
        return (OptionsSettings)xs.Deserialize(stream);
    }

}

public class OptionsPersonalSettings {...}

public class OptionsPrivacySettings {
    private bool allowNonFriendsToSeeStatus;
    private bool loginAsInvisible;
    private bool showIdleTimeToFriends;
    public bool AllowNonFriendsToSeeStatus {...}
    public bool LoginAsInvisible {...}
    public bool ShowIdleTimeToFriends {...}
}

public class TestSettings {
    public static void RunTestSettings() {
        OptionsSettings os = new OptionsSettings();
        os.Privacy.AllowNonFriendsToSeeStatus = false;
        os.Privacy.LoginAsInvisible = true;
        os.Privacy.ShowIdleTimeToFriends = true;

        IsolatedStorageFile isf =
            IsolatedStorageFile.GetUserStoreForAssembly();
        isf.CreateDirectory(@"Options");

        Stream stream =
            new IsolatedStorageFileStream(@"Options\Settings",
                FileMode.OpenOrCreate, isf);
        os.Store(stream);
        stream.Close();

        Stream stream2 =
            new IsolatedStorageFileStream(@"Options\Settings",
                FileMode.Open, isf);
        OptionsSettings os2 = OptionsSettings.Load(stream2);
        stream2.Close();

        isf.Close();
    }
}

Additionally, the directory structures that were coded as paths are now expressed as the relationships between a class and its properties. This means you would use OptionSettings.Privacy.AllowNonFriendsToSeeStatus instead of getting it from the file Options\Privacy\AllowNonFriendsToSeeStatus. This method also enables your application to use IntelliSense® for preferences.

This doesn't obviate the need for a generic preferences API, though. You still need to partition your data, especially when your applications get bigger. Instead of having one huge class that contains every single preference value, you may need to break the preference data into several smaller classes, making each of them more manageable. Also, a preferences API can encapsulate these serialization details into something like this:

// get the node that corresponds to
// Options from the preferences tree
IPreferencesNode options = IPreferencesNode.GetNodeForPath("/Options");

OptionsSettings os = new OptionSettings();
// set the preferences values here...

// serialize the class into the preferences tree.
options["Settings"] = os;

Although the registry API feels more natural for managing preferences, it is tied to its backing store, namely, the registry. There is no way you can use the API with another storage mechanism such as the file system or the isolated storage. A change in the backing store will be disruptive to your code because the interface changes. A good preferences API shields you from these details, allowing a change of backing store with minimal code change.

Preference Change Notification

Depending on the nature of your application, sometimes you want to get a notification when some preference data has changed. If the API you're using doesn't support event notification, then you need to raise an event manually whenever you're changing the preference value. If the API does support event notification, then you can just update the preference value and be confident that interested parties will be notified properly.

The APIs available to you have varying degrees of support for events. If you're using the file system, it is possible to monitor changes using the System.IO.FileSystemWatcher class, although the notification may be too coarse because monitoring happens at the file level. If you're using the registry, the .NET version of its API (in the Microsoft.Win32 namespace) doesn't directly support events, so you have to resort to the Win32® function RegNotifyChangeKeyValue to achieve this. Isolated storage doesn't directly expose any events either, although it is theoretically possible to use the FileSystemWatcher to monitor the isolated storage files and raise events, since isolated storage uses the file system underneath anyway.

Data Isolation

So far I've covered things that make an API suitable for dealing with preferences. Other than the API, you also need a backing store to put the preferences data into. There really isn't any perfect backing store. For instance, unlike isolated storage, the registry and the file system offer no isolation from other applications, so you need to take precautions to minimize the likelihood of other applications corrupting your data. But on the other hand, there are scenarios in which you can't use isolated storage. Your choice here really depends on your needs.

The file system and the registry do provide some degree of isolation, but the isolation is by user only. In the file system, the Environment.SpecialFolder.LocalApplicationData special folder is tied to a specific Windows user. Likewise, every Windows user gets their own HKEY_CURRENT_USER key in the Registry, so whatever you put in there is isolated by user.

Beyond that, there's only what I'll call "isolation by convention." That is, you can expect an application from Example Inc. to behave nicely and store its data under HKEY_CURRENT_ USER\Software\Example\UselessApp, but it's always possible that it accidentally writes into the parent key, HKEY_CURRENT_USER\Software, messing with the data of other applications.

Isolated storage greatly reduces the likelihood that an application will write its data to the wrong registry key. Given the right isolation scope, there's no way another application can even touch your data through the isolated storage API. With sufficient permissions, an application can always go directly to the file system and corrupt data it finds there, but it has to be intentionally malicious to do that. Moreover, if a malicious application has already obtained access to the file system like that, all bets are off anyway.

For reasons like these it's critical to choose the right isolation scope. If you choose to isolate your data by user and by assembly only, applications that share assemblies can access one another's data, so you have the isolation by convention problem all over again. To prevent this, you should isolate by user, assembly, and application domain. On the other hand, isolation by user and assembly allows multiple applications to share common data in the assembly's store, which may be what you need.

Security

Isolated storage reduces the potential amount of damage that applications can do to a system. In the same way an application's isolated store keeps other applications out, it also prevents the application itself from damaging other application's data stores. Also, administrators can impose quotas on the amount of isolated storage that can be used, which further reduces the application's potential for damaging the system.

In contrast, poorly written or malicious applications that have access to the file system or the registry can do substantial damage to your system. This makes isolated storage a good choice for a wider range of applications. In some cases, it's the only choice. For instance, by default controls downloaded from the Internet are not allowed to access the file system directly, nor are they allowed to read from or write to the registry, but they can use isolated storage to store their data safely.

Application Preferences Shared by All Users

There are some situations in which the isolated storage is too isolated, such as when you need to store application-level preferences that are shared by all users. With the registry, you can store your app-wide preferences under the HKEY_LOCAL_MACHINE\Software tree. In the file system, you can store your application's shared preferences under the Environment.SpecialFolder.CommonApplicationData special folder, but there is no way to do this with isolated storage because an isolated store is always isolated by user.

Note that a .NET-based application can have an application configuration file which you can access programmatically through the classes in the System.Configuration namespace. However, they have read-only access; System.Configuration will allow you to read the values in, but not write the values out. It may not be practical and it may not even make sense to store preferences values that are shared by all users in application configuration files.

An Easier Way to Manage Preferences

By now it may seem like there is no good all-around solution for preferences management in the .NET Framework. Each of the backing stores has its own strengths and weaknesses, so you have to choose which one to use according to your needs.

Considering this, it seems that an API that is designed for preferences management, and that allows you to choose and switch the backing store according to your needs, would simplify some of the problems. Figure 6 depicts just such an API. With this API, applications never talk to the backing store directly for preferences management. They only interact with the API layer of the Preferences library, which in turn talks to the backing store through a set of well-defined interfaces that I've dubbed the Preferences Service Provider Interface (SPI).

Figure 6 Preferences API for .NET (Close Figure 6)

This design allows application developers to worry about preferences management instead of storage-specific details (which are hidden inside the backing store-specific implementations of the Preferences SPI). Also, since the API layer only interacts with the backing store through the interfaces defined in the SPI layer, you can easily switch to another backing store (or a more efficient implementation of the same back-ing store) with minimal changes to your preferences management code.

Introducing NAP

.NET Application Preferences (NAP) is my own implementation of the API shown in Figure 6. If you have a background in Java, you'll probably find this class library quite similar to the Java Preferences API. Indeed, I wrote this library because I missed the convenience of the Java Preferences API in .NET, yet I wanted to have control over which backing store to use in my applications (the Java Preferences API doesn't expose a standard way to do this). The following code snippet uses the registry to store the same set of preferences you saw in Figure 3:

IPreferencesNode imNode =
    PreferencesFactory.GetUserRootNode(RegistryStore.Instance);
IPreferencesNode optionsPersonalNode =
    imNode.GetNodeForPath("/Options/Personal");
IPreferencesNode optionsPrivacyNode =
    imNode.GetNodeForPath("/Options/Privacy");

optionsPrivacyNode["AllowNonFriendsToSeeStatus"] = 0;
optionsPrivacyNode["LoginAsInvisible"] = 1;
optionsPrivacyNode["ShowIdleTimeToFriends"] = 1;
The result of running this code is shown in Figure 7.

Figure 7 .NET Application Preferences (Close Figure 7)

Note the garbled key names below the Nap\Preferences key. NAP is smart enough to simulate the isolation features of System.IO.IsolatedStorage, so it's unlikely that applications will be able to see (and corrupt) each other's data. Also, NAP makes changing the backing store very easy. To use the file system, only one line needs to be changed, from

IPreferencesNode imNode =
    PreferencesFactory.GetUserRootNode(RegistryStore.Instance);
to
IPreferencesNode imNode =
    PreferencesFactory.GetUserRootNode(WindowsFileSystemStore.Instance);
or to use isolated storage:
IPreferencesNode imNode =
    PreferencesFactory.GetUserRootNode(ISUserStore.Instance);

Note the name ISUserStore, which was chosen because isolated storage doesn't support stores that can be shared by all users. The registry and the file system, however, don't have such a limitation. Again, you only need to change one line of code to get a system node (a preferences node shared by all users):

IPreferencesNode imNode =
    PreferencesFactory.GetSystemRootNode(RegistryStore.Instance);
Or you can do the following:
IPreferencesNode imNode =
    PreferencesFactory.GetSystemRootNode(WindowsFileSystemStore.Instance);
This way, you get preferences trees that can be shared by users, yet which are completely isolated by assembly and application domain to prevent applications from corrupting each other's data. (Calling GetSystemRootNode with ISUserStore will throw a NotSupportedException.)

Note that I'm passing an object, not an enumeration, into GetSystemRootNode and GetUserRootNode. This makes it possible for NAP to accept a new backing store provider without having to be recompiled. Simply create the required provider code and pass the backing store provider instance in, like this:

IPreferencesNode imNode =
    PreferencesFactory.GetSystemRootNode(SQLServerStore.Instance);

Tree Nodes and Leaves

Once you've got your root node, traversing the tree is easy. It's almost like using the command prompt in that you supply paths, which can be either relative or absolute. The best way of illustrating this would be through a code snippet, shown here in Visual Basic:

Dim userRoot As IPreferencesNode = _
  PreferencesFactory.GetUserRootNode(RegistryStore.Instance)
Dim one As IPreferencesNode = _
  userRoot.GetNodeForPath("one")
Dim one_a As IPreferencesNode = _
  one.GetNodeForPath("a")
Dim one_a_i As IPreferencesNode = _
  one_a.GetNodeForPath("i")

Dim two As IPreferencesNode = _
  userRoot.GetNodeForPath("/two")
Dim two_a As IPreferencesNode = _
  userRoot.GetNodeForPath(" two a")
Dim two_a_i As IPreferencesNode = _
  userRoot.GetNodeForPath("/two/a/i")

Just like in the file system, a path can be absolute or relative. If it's absolute, the tree is traversed starting from the root. If it's relative, the traversal starts from the node on which GetNodeForPath is called. The result of running the previous code snippet is shown in Figure 8.

Figure 8 Create Node with Absolute Path
Figure 8 Create Node with Absolute Path

Now that I've got a node, I need to do something with the leaves under that node. Again, NAP makes this very simple. Each leaf has a name and a value associated with it. Dealing with NAP leaves is just like dealing with key-value pairs in a Hashtable. To create a new leaf or modify an existing one, you use the indexer as follows:

one("someBoolValue") = True
one_a("singleValue") = 234.534
For the OptionsSettings class that was shown in Figure 5, you can do the following:
Dim os As New OptionsSettings
os.Privacy.AllowNonFriendsToSeeStatus = False
os.Privacy.LoginAsInvisible = False
os.Privacy.ShowIdleTimeToFriends = False

one_a_i("optionsSettings") = os
Under the covers, this instance of the OptionsSettings class is automatically serialized using XmlSerializer. Retrieving it is equally easy, as you can see here:
Dim os2 As OptionsSettings = _
    one_a_i.GetSerializedObject("optionsSettings", _
        GetType(OptionsSettings), Nothing)

Of course, just as you can create nodes and leaves, you can remove them. The methods to do this are pretty straightforward:

' remove all leaves/preferences under one
one.RemoveAllPreferences()

' remove a specific leaf
one_a_i.RemovePreference("optionsSettings")

' remove the whole branch from the tree, starting from this node
one.RemoveNode()

What's Happening Under the Tree?

Another useful feature of NAP is that it allows you to install event handlers to find out what's happening under the preferences tree. For instance, in a GUI application, you may want to find out when a UI-related preference is changed so that you can immediately update the application's appearance accordingly. It is also sometimes useful to know when a child node has been added under a node or removed from under one. The following code snippet illustrates this in C#:

   one["someBoolValue"] = true;
// we want to know when there's a change in the preference
one.PreferenceChanged += new
    Nap.Event.PreferenceChangeEventHandler(one_PreferenceChanged);
one["someBoolValue"] = false;
The handler method looks like this:
private static void one_PreferenceChanged(object sender,
    Nap.Event.PreferenceChangeEventArgs args) {
    Console.WriteLine(args.Key);
    Console.WriteLine(args.NewValue);
}

This will print "someBoolValue" and "false" to the console window if you run the code snippet that precedes it. The other two events are ChildNodeAdded and ChildNodeRemoved, which will be fired by a node when it has a new subnode added to it or removed from it. The following code snippet illustrates the handling of the event being raised when a subnode is added to a node:

IPreferencesNode two = userRoot.GetNodeForPath("/two");
two.ChildNodeAdded += new
    Nap.Event.ChildNodeEventHandler(two_ChildNodeAdded);
IPreferencesNode two_a = userRoot.GetNodeForPath("/two/a");

The handler method looks like this:

private static void two_ChildNodeAdded(object sender,
    Nap.Event.ChildNodeEventArgs args) {
    Console.WriteLine(args.ChildNode.Name);
}
This will print "a" to the console window in this case.

That's all there is to know about using NAP. There are other methods and properties, but they are mostly self-documenting (such as the Root property returning the root node, and the Parent property returning the parent node). Now all that's left is to explore how NAP implements isolation.

Simulating Isolation by Assembly

The isolated storage offers isolation by user, by assembly, and by application domain. User isolation is easy. For the file system, there's the special folder Environment.SpecialFolder.LocalApplicationData, and for the registry there's HKEY_CURRENT_USER. This still leaves you with simulating isolation by assembly and isolation by application domain.

The next question is, what constitutes an assembly's identity? As it turns out, the answers to these questions depend on what you're trying to achieve. For instance, the Hash class in the namespace System.Security.Policy provides an assembly identity that is not ambiguous, but for this purpose it is so unambiguous that it is not useful. If you fix a bug in your code, thereby changing the resulting Microsoft intermediate language (MSIL), according to the Hash class that's a different assembly. If you change a single character in a string literal, that's yet another assembly. Also, consider the nightmare you'll have during development when each revision or bug fix generates a different assembly, rendering all the previously stored data inaccessible.

How about using an assembly's full name as its identity? When an assembly is not strongly named, its full name looks like this:

test, Version=1.0.1474.24153, Culture=neutral, PublicKeyToken=null
By default, the build number and revision number parts of Version will change with every build, so identity based on full name will change also, but you can address this by controlling the version number manually, like so:
[assembly: AssemblyVersion("1.0.0.0")]
Again, this may not be the behavior you would expect. During development, you may want to raise the version number while retaining access to the stored data of the previous builds.

Isolated storage uses a kind of identity that turns out to be quite reliable and practical most of the time. As the .NET Framework documentation makes clear, assembly identity is the evidence of the assembly. This might come from a cryptographic digital signature, the software publisher of the assembly, or from its URL identity. If, for example, an assembly has both a strong name and a software publisher identity, the software publisher identity is the one that is used; if the assembly comes from the Internet and is unsigned, the URL identity is used instead.

During development, if the assembly is unsigned, as long as you're building to the same location (for example, C:\Acme\SpaceInvaders\bin\Debug\Start.exe), the assembly is always considered the same assembly, no matter how many times you perform bug fixes or add features. Now what if the assembly has a strong name? The following code is the result of calling the ToString method on a StrongName instance:

<StrongName version="1"
            Key="DB5F93741AD8DDC3C03E077E1D9B..."
            Name="ConsoleApplication1"
            Version="1.0.1474.39142"/>

The strong name still contains the assembly's version number, which may change with every build. This means that if the strong name is used as an identity, every build may generate a different assembly as well. Fortunately, isolated storage only considers a strong-named assembly different if there is a change in the major version number. To isolated storage, version 1.2.1453.23409 of MyApplication and version 1.9.1498.24890 of MyApplication are the same assembly, but versions 2.0.1453.23409 and 1.0.1453.23409 are different. With information from the .NET Framework Developer's Guide, it was quite easy to code my own isolation by assembly. The code to do this is shown in Figure 9.

Figure 9 Getting the Identity String of an Assembly (Close Figure 9)
public static string GetAssemblyIdentity(Assembly assembly) {
    string assemblyIdentityBase32String = null;
    foreach(object ogj in assembly.Evidence){
        // If an assembly has both a strong name and a software
        // publisher identity, then the software publisher identity
        // is used.
        else if(obj is Publisher) {
            Publisher pub = (Publisher)obj;
            X509Certificate cert = pub.Certificate;
            assemblyIdentityBase32String = "Publisher." +
                Base32.ToBase32String(cert.GetCertHash());
            break;
        }
        else if(obj is StrongName) {
            StrongName sn = (StrongName)obj;
            StringBuilder sb = new StringBuilder(sn.Name);
            sb.Append(sn.PublicKey.ToString());
            sb.Append(sn.Version.Major);
            assemblyIdentityBase32String = "StrongName." +
                Base32EncodedHash.GetFromString(sb.ToString());
            break;
        }
        // If the assembly comes from the Internet and is unsigned,
        // the URL identity is used.
        else if(obj is Url) {
            Url url = (Url)obj;
            assemblyIdentityBase32String = "Url." +
                Base32EncodedHash.GetFromString(url.Value);
        break;
        }
    }
    if(assemblyIdentityBase32String == null) {
        // fall back to assembly's code base
        assemblyIdentityBase32String =
            Base32EncodedHash.GetFromString(assembly.EscapedCodeBase);
    }
    return assemblyIdentityBase32String;
}

I chose the Base32 encoding used in Figure 9 because it is case insensitive, just like Windows directory names and registry keys. Also, the character set used doesn't contain any character that would be illegal for a Windows directory name or a registry key name (such as \). You can take a look at the Base32 encoder implementation in the NAP source code, which is available with the download for this article.

Simulating Isolation by Application Domain

How about application domain identity? According to the .NET Framework Developer's Guide, domain identity represents the evidence of the application, which might be the full URL in a Web application. For code that is hosted by the shell, the domain identity might be based on the application directory path. For example, if the executable runs from the path C:\Office\MyApp.exe, the domain identity would be C:\Office\MyApp.exe.

So, simulating isolation by app domain is even simpler than isolation by assembly. Figure 10 shows the code for generating a string that corresponds to app domain identity.

Figure 10 Getting the Identity String of an App Domain (Close Figure 10)
public static string GetDomainIdentity(AppDomain appDomain) {
    string domainIdentityBase32String = null;
    foreach(object obj in appDomain.Evidence){
        if(obj is Url) {
            Url url = (Url)obj;
            domainIdentityBase32String =
                "Url." + Base32EncodedHash.GetFromString(url.Value);
        break;
        }
    }
    if(domainIdentityBase32String == null) {
        // Unlikely, but fall back, just in case Url is not there
        domainIdentityBase32String =
            Base32EncodedHash.GetFromString(appDomain.ToString());
    }
    return domainIdentityBase32String;
}

Conclusion

In this article, I've discussed the features that a preferences-friendly API should have. I also reviewed the backing stores that are available to most .NET-based applications and showed how deciding on whether to use the file system, the registry, or isolated storage really depends on your needs.

As I hope I've made clear, my .NET Preferences library allows you to choose or switch to any backing store as necessary, without drastic changes in your code. I've shown how to use this library and how you can simulate the isolation features of .NET isolated storage. I am not saying that NAP is the perfect API for preferences management, but I've found it to be very useful in all of my preferences-related programming tasks so far, and I'm confident that you'll find it useful as well.

Ray Djajadinata is a Senior Software Engineer at Savi Technology. With the engineering team, he deals with supply chain and asset management applications for large enterprises. You can reach Ray at rayfd_2000@yahoo.com.


Web Services: Capturing and Analyzing Client Transaction Metrics for .NET-Based Web Services

Web Services

Capturing and Analyzing Client Transaction Metrics for .NET-Based Web Services

Brian Connolly
This article discusses:
  • The importance of Web services metrics
  • Why you need to measure on the client
  • Writing a metrics measuring app in C#
  • Analyzing the results
This article uses the following technologies:
SOAP, Web Services, C#, SQL, MSMQ
Code download available at: TransactionMetrics.exe (155KB)
Contents
Defining the Problem
Defining the Solution: Client Metrics Recording
Database Design and Structure
Server Work Queues
Data Access Layer
TransactionMetricsPayload
Web Services
ConsoleReader—the Application "Server"
MetricsReader—the Metrics Logger
Client Design and Coding
WSDL-Generated Service1 Class
PayloadManager—The Client Metrics Recording Library
ConsoleClient—the Client Simulator
The Sample Application
Running the Transaction System
Using the Metrics Database
Performing the Test
Analyzing the Results
Limitations
Further Work and Possible Extensions

T
he success or failure of IT departments can depend on their ability to deliver a high throughput of requests to the services they support. This is especially important for Web services. In order to ensure adequate throughput, you really need to be able to measure response times. To achieve that goal, this article presents a general-purpose reporting mechanism that can be used in any .NET system that employs HTTP and SOAP.

Throughput measurement and the expected peak number of operations influence enterprise capacity planning, allowing planners to determine how many servers are required. These are important metrics, yet typical Web service logging consists solely of raw HTTP logs. Logging and reporting tools available for Web logs analyze only low-level HTTP requests. The architecture proposed here gathers more accurate and useful metrics than those offered by systems based solely on Web server HTTP logs.

Defining the Problem

A typical enterprise transaction system built using Web services is shown in Figure 1. Application servers execute business logic and request database operations and are usually placed on an internal network separate from the Web servers for security purposes. A set of Web servers running IIS and ASP.NET accept incoming requests and use a queuing resource manager such as Microsoft® Message Queue (MSMQ) to deliver them to application servers for processing. Client applications use a Web Services Description Language (WSDL)-generated proxy in order to submit operations to the Web server.

Figure 1 Web Services Transaction Architecture
Figure 1 Web Services Transaction Architecture

Most client transactions will consist of multiple SOAP requests. For example, a financial market trade might consist of a SOAP request to submit an order and a subsequent request to check for its execution. Or a product purchase might consist of one request to submit an order, a second request to submit credit data, and a third request to obtain an order number for a completed order.

It can be very difficult to measure and report these operations from IIS logs. Suppose you need a report on the number of orders that were processed as well as their average completion time. You would need to do several things, such as developing a database representation of HTTP logs or purchase one as part of a Web log reporting system. You would also need to identify individual clients, either by distinct IP addresses, cookies, or through various other HTTP headers.

Finally, you would need to write custom SQL procedures to identify these clients and the user transactions they issued. This would be quite complex as a typical user transaction consists of multiple HTTP requests. The SQL procedure would need to identify the first HTTP request of a transaction and then use a cursor to look ahead in the log for the concluding status check request.

This can be a complicated and expensive project, and often you don't realize the need for it until a system is already in production and user support issues arise. Even if a Web server log reporting system is developed, it can't provide a clear picture of response times from the client's point of view. It cannot record client delays in connecting to a server, network transmission delays, or any client overhead involved in processing a transaction.

In addition, an HTTP request is logged and timestamped only after it is serviced, so delays receiving service from an overloaded Web server will not be recorded. These factors can add significantly to a client's perception of response time. An organization that relies solely on Web logs will lack a clear picture of the true quality of service they offer to their clients.

That said, architects must be aware of some issues in order to use this approach. While these client-based metrics will paint a picture of what the client actually sees, that data isn't necessarily useful to the service designers. Long delays due to network latency and bandwidth will affect these numbers and yet can't be easily changed for most systems. Issues also arise regarding how the client components are distributed to the users. The assumption I make is that these components are distributed with the software itself or are otherwise distributed to the clients in a robust manner. Regardless of these potential pitfalls, there are many valuable applications for the client-based performance data you obtain. This article explores one possible solution.

Defining the Solution: Client Metrics Recording

While there are many ways you can solve such a problem, my solution is based on one overriding principle. Client applications already implement the business logic that relates low-level requests to user transactions. Any reporting system based on server HTTP logs is an unnecessary reimplementation of a model that already exists in the client. The solution is to have the client application itself record metrics about the operations it generates. When a user initiates a set of these operations, the client can record the start time and the elapsed time before results are presented to the user. These records can be buffered in the client and then submitted to the server using SOAP headers. No extra requests or responses are sent between the client and server—these headers are sent as an additional payload with the next transaction initiated by the user.

Figure 2 shows the architecture used in this approach. New components are shown in red. A client Payload Manager is used to record metrics, such as transaction start and end times, number of items ordered, payment method, or any other business data that will aid product support and capacity planning. This transaction data is added in optional headers to the same SOAP messages that are used to complete the user operations. The ASP.NET Web services that process SOAP requests strip off any metrics headers and place them on a queue for low-priority processors. Metrics loggers dequeue these headers and place them into a database designed to allow general-purpose transaction reporting. IIS logs are no longer needed.

Figure 2 Web Services Transaction Architecture with Metrics
Figure 2 Web Services Transaction Architecture with Metrics

The solution I am presenting has the following features. First, the Payload Manager, the Metrics database schema, and the Metrics Loggers are written as general-purpose metrics-recording components that should be able to record data of interest to a wide variety of applications.

Client transaction metrics are implemented in a manner that doesn't lower the reliability of the system as a whole. Any raised exceptions are caught and handled in a manner that doesn't impede successful execution of a business transaction. Also, the actual client response times that are recorded reflect delays experienced by a user.

It is important to note that this is a sampling implementation—not all transactions will be recorded. In particular, the final operation of each application session will not show up. My intention was to record most operations in order to serve as a user support tool and to assist in capacity planning.

In writing this article I used Microsoft SQL Server 2000 and Visual Studio® .NET to develop a simple system to illustrate the design for gathering client metrics. The model transaction is simple: computing the prime factors of large numbers. Clients invoke one Web service to submit a number to be factored and invoke a second Web service to check if results are ready. Client response time is measured as the time difference between submitting a number to be factored and the final status check (the one that returns the prime factors).

The implementation I'm using here is based on Windows® XP Professional with MSMQ installed. All of the client, server, and database code I'll be discussing is available in the code download. The system consists of the following components:

  • Two databases: a simple application database called Factor, and a more complex client Metrics database.
  • A data access layer library that offers application server access to the Factor database. This is used by both the ASP.NET Web services and the ConsoleReader application.
  • Definition of a Header class that reflects the Metrics database schema. This will hold client metrics reports.
  • Two message queues: factorjob, to hold numbers to be factored, and transactionmetrics, to hold metrics Headers.
  • An application server called ConsoleReader that computes prime factors and saves them to the Factor database.
  • An application server similar to ConsoleReader called MetricsReader that obtains Headers from the transactionmetrics queue and logs them to the Metrics database.
  • An ASP.NET Web service that defines two WebMethods: Factor and CheckJob. Factor accepts a number to be factored, and CheckJob returns either its prime factors or an indication that factoring is still in progress.
  • A PayloadManager library, used by the client to record transaction statistics as Header objects for upload.
  • A ConsoleClient application that creates multiple threads to simulate many SOAP clients submitting factor requests of random sizes.

Database Design and Structure

Figure 3 shows the Factor database, which is used to service the application. When the Web service receives a Factor request, it creates an entry in tblTransaction, assigns a GUID as the TransactionID, and sets the ServerStartTime to the current time. The transaction Status is initially set to "In Progress." The transaction is queued, and the TransactionID is passed back to the client to be used as a handle to check for the completion of the operation.

Figure 3 The Factor Database
Figure 3 The Factor Database

ConsoleReader computes the prime factors and stores them as rows in tblResults. ConsoleReader updates the ServerEndTime in tblTransaction and also sets the status to "Finished."

Three stored procedures are shown in Figure 4. These support transaction creation and completion, as well as results storage. The procedure upNewTransaction uses NEWID to create a GUID and create a transaction entry with the current time and "In Progress" status. The procedure upUpdateStatus is used to mark a completed transaction as "Finished." The procedure upCheckTransaction is used to check if a transaction has been completed.

Figure 4 Stored Procedures (Close Figure 4)
CREATE PROCEDURE upNewTransaction
    @guid uniqueidentifier out
AS
declare
   @id uniqueidentifier
begin
   set @id = NEWID()
   insert tblTransaction(TransactionID, ServerStartTime,Status)
   values(@id, getdate(), 'In Progress')
   set @guid = @id
end

CREATE PROCEDURE upUpdateStatus
    @guid uniqueidentifier, @status varchar(255)
AS
declare
   @id uniqueidentifier
begin
   update tblTransaction
   set status = @status, ServerEndTime = getdate()

   where TransactionID = @guid
end

CREATE PROCEDURE upCheckTransaction
    @guid uniqueidentifier, @status varchar(20) out
AS
begin
   select @status = status
   from tblTransaction
   where TransactionID = @guid

   if @status is null
      select @status = 'In Progress'
end

The Metrics recording database, shown in Figure 5, is kept distinct from the application database because it is designed to be application independent. In this article, it shares the same data server with the Factor database, although in practice it would reside on a separate data server in order to reserve system resources for application processing. The only entity the two databases have in common is the notion of a unique TransactionID value.

Figure 5 Metrics Recording Database (Close Figure 5)

TblTransaction represents the client view of transactions. TransactionID is the same identifier as the TransactionID in the Factor database—it is passed to the client when the first service call of a transaction is invoked, then passed back to the server with the client metrics for the transaction. ClientID is a GUID that is generated by the client application that submitted the transaction, and so tblClient represents all the instances of clients that have submitted any transaction metrics.

The Transactions table has the commonly useful metrics for transactions: the client's name for the transaction type (Type), the client system time when it was initiated (ClientStartTime), and the corresponding ServerStartTime. The server time allows transactions metrics to be aligned with server resource utilization logs. The transactions table contains the most important metric of all: ClientElapsedMS, the difference between the time the first SOAP call of a transaction was initiated and the time a status check indicated that the server completed the transaction.

Both tblClientAVPair and tblTransactionAVPair are intended as general-purpose descriptors for their associated entities. Examples of client attributes might be software version or machine type. In general, transaction attributes would be any attribute that makes a transaction more or less difficult for the server. For example, a trading application might add an attribute ("All-or-None", "True") to all-or-none orders.

All-or-none orders require special processing by electronic exchanges since the entire order quantity must be filled in one trade. Since these types of orders take more time to execute, it is important to distinguish them when analyzing system performance. In the sample factoring application developed for this article, the clients report the number of prime factors obtained from the server, since fewer factors generally imply more CPU-intensive factoring.

Both of those tables were implemented in a fully denormalized manner, using additional space to gain the ability to add new client and transaction attributes without changing the schema. In fact, the design allows the client application to add arbitrary attributes without requiring changes to other design components.

Server Work Queues

Servers usually contain a pool of resources that can process user requests. One way of distributing work among these resources is to use queues. Work queues allow operations to be executed asynchronously. Front-end Web servers can simply deposit new requests into a queue and confirm receipt to the user. When a back-end resource becomes available, it can read the next request from the queue and process it.

The Windows XP Computer Management application can be used to create two work queues for the simple transaction system: factorjob holds numbers to be factored by the ConsoleReader application server, transactionmetrics holds the SOAP headers uploaded by the client.

Data Access Layer

The DataAccessLayer namespace source code is in DataAccessLayer.cs. This is used by both the Web services and the ConsoleReader application to access the Factor database. Since this is used by both components, the WorkItem (the object passed to the factorjob queue) is defined here. It contains the transaction GUID and the data to be passed to the application server—in this case, the decimal number to be factored.

The Factor Web service uses NewTransaction to create a new user transaction and to obtain a transactionID that will be returned to the client. The CheckJob Web service uses CheckStatus in order to check whether the transaction denoted by transactionID has, in fact, been completed.

The ConsoleReader application uses UpdateStatus to mark a transaction as "Finished," and it then uses SaveResults to save the prime factors for the problem it was given.

DataAccessLayer is a simplification of a typical database encapsulation layer. What makes it simple is that it only uses a single database connection, while a real data access layer would employ a pool of connections. Since multiple threads will service SOAP clients, it is necessary to single-thread the NewTransaction and CheckStatus methods. Note that for simplicity the database connection string that I'm using in the implementation has been hardcoded. This is an obvious exception to the standard practice of defining connection strings in an application configuration file rather than right in the code.

TransactionMetricsPayload

The TransactionMetricsPayload namespace (see Figure 6) defines the metrics Header class that will hold client metrics reports for eventual logging to the Metrics database. Since I want the client to upload data within SOAP headers, I have subclasssed from System.Web.Services.Protocols, as you can see. SoapHeader.Header is used by the Web services and the MetricsReader application, and its design parallels the Metrics database schema. ClientReport has the general attributes of [Metrics].tblTransaction, and also the application-specific arrays of AVPair objects that will be placed in [Metrics].tblClientAVPair and tblTransactionAVPair. The actual Header object has an array of ClientReport objects, so that a client can buffer multiple reports in a single SOAP header and upload them all to the server.

Figure 6 TransactionMetricsPayload (Close Figure 6)
/// <summary>
/// Defines Header class that will hold transaction metrics
/// data received from SOAP clients
/// </summary>
using System;
using System.Web.Services.Protocols;

namespace TransactionMetricsPayload
{
    public class AVPair
    {
        public string attribute;
        public string value;

        //Default public constructor for WSDL
        public AVPair()
        {
        }
        public AVPair(string a, string v)
        {
            attribute = attribute;
            value = value;
        }
    }

    public class ClientReport
    {
        public DateTime clientStartDateTime;
        public DateTime serverStartDateTime;
        public Int32 clientElapsedMS;
        public string type;
        public string status;
        public string transactionID;
        public string clientID;
        public AVPair[] transactionAttributes;
        public AVPair[] clientAttributes;

        //Default public constructor for WSDL
        public ClientReport()
        {
        }
    }

    public class Header : SoapHeader
    {
        public ClientReport[] ClientReports;

        //Default public constructor for WSDL
        public Header()
        {
        }
    }
}

Web Services

The ASP.NET Factor service defines two WebMethods: Factor and CheckJob. A class, FactorAck, is defined as the return type of the Factor service. This is used to return the transaction GUID and the server start time to the client. The server start time is needed so that the metrics processing components on the server don't require access to the Factor application database.

To create the WebMethods in Visual Studio .NET, you first create an ASP.NET Web services project called Factor, which will generate a default Service1 service. Use the Server Explorer to drag the factorjob and transactionmetrics queues onto the component workspace, causing Visual Studio .NET to declare and configure two member variables in the service: messageQueue1 and messageQueue2 (see Figure 7).

Figure 7 Factor and CheckJob (Close Figure 7)
/// <summary>
/// Provides two Web services:
/// Factor:  adds factoring problem  to message queue
/// CheckJob:  checks Factor database for completion
/// </summary>
using System;
using System.Collections;
using System.ComponentModel;
using System.Diagnostics;
using System.Web;
using System.Web.Services;
using System.Web.Services.Protocols;
using DataAccessLayer;
using TransactionMetricsPayload;

namespace Factor
{
    public class Service1 : System.Web.Services.WebService
    {
        public TransactionMetricsPayload.Header mHeader;

        public Service1()
        {
        InitializeComponent();
        }

        private System.Messaging.MessageQueue messageQueue1;
        private FactorServer factorServer = new FactorServer();
        private System.Messaging.MessageQueue messageQueue2;

        #region Component Designer generated code

        private IContainer components = null;

        private void InitializeComponent() {
            this.messageQueue1 = new System.Messaging.MessageQueue();
            this.messageQueue2 = new System.Messaging.MessageQueue();

            this.messageQueue1.Path = "myserver\\Private$\\FactorJob";
            this.messageQueue2.Path =
               "FormatName:DIRECT=OS:myserver\\private$\\transactionmetrics";
        }

        protected override void Dispose( bool disposing )
        {
            if(disposing && components != null) {
                components.Dispose();
            }
            base.Dispose(disposing);
        }

        #endregion

        public class FactorAck
        {
            public Guid transactionId;
            public DateTime serverStartDateTime;
            public FactorAck(){}
        }

        [WebMethod]
        [SoapHeader("mHeader", Required=false)]
        public FactorAck Factor(decimal bignumber)
        {
            if(mHeader != null) {
                try {messageQueue2.Send(mHeader);}
                catch {}
            }
            FactorAck fAck = new FactorAck();
            fAck.transactionId = factorServer.NewTransaction();
            fAck.serverStartDateTime = DateTime.Now;

            messageQueue1.Send(new WorkItem(fAck.transactionId,bignumber));
            return fAck;
        }

        [WebMethod]
        [SoapHeader("mHeader", Required=false)]
        public decimal[] CheckJob(System.Guid Handle)
        {
            if (mHeader != null) {
                try {messageQueue2.Send(mHeader);}
                catch {}
            }

            decimal[] factors = new decimal[0];
            factorServer.CheckStatus(Handle, out factors);
            return factors;

        }
    }
}

Note the mHeader member variable. This will hold any SOAP header sent by the client. The SoapHeader attribute for each WebMethod maps the header to this variable and also specifies that the header is optional (it need not be present in a message). The first thing each WebMethod does is check for an mHeader and if found, it's placed on the transactionmetrics queue.

If an exception is raised when queuing the header, it is caught and handled so that you can continue to process the business transaction even if there is a problem with the metrics infrastructure. Recall that one of the design objectives was to gather metrics in a manner that doesn't make the overall system less reliable than it would be without metrics collection. In a production system, the application should still continue after header processing errors, but any errors should be reported to the event log.

ConsoleReader—the Application "Server"

Normally, an application server is implemented as a Windows service, and a large transaction-based system will have a pool of application server machines. In this article, I chose to model the application server as a console application because it will allow me to run any number of application servers easily, together with multiple clients and Web services, on a single machine. Even though it is a console application, it is functionally equivalent to an application server since it performs the same basic operations—it reads work from a queue and deposits results into a database.

The code for the ConsoleReader application is available in the download. Each ConsoleReader application reads a WorkItem from the factorjob queue, computes the prime factors of its System.Decimal mItem object, and uses the DataAccessLayer to place the results into the Factor database. Multiple application servers can be simulated by creating multiple instances of ConsoleReader. Note that the factoring algorithm is very rudimentary and will take considerable time to factor extremely large numbers.

MetricsReader—the Metrics Logger

MetricsReader, shown in Figure 8, was also developed as a console application for simplicity. In a manner similar to the ConsoleReader, it reads TransactionMetricsPayload.Header items from the transactionmetrics queue and adds the array of ClientReport objects to the Metrics database.

Figure 8 MetricsReader (Close Figure 8)
/// <summary>
/// Reads from Metrics queue and posts metric data to Metrics database.
/// </summary>
using System;
using TransactionMetricsPayload;
using System.Data.SqlClient;
using System.Messaging;
namespace MetricsReader
{
    class MetricsReader
    {
        [STAThread]
        static void Main(string[] args)
        {
            string connectionstring =
                @"Integrated Security=SSPI;server=(local);database=Metrics";
            SqlConnection connection = new SqlConnection(connectionstring);
            connection.Open();

            MessageQueue messageQueue2 = new MessageQueue();
            messageQueue2.Path =
                "FormatName:DIRECT=OS:myserver\\private$\\transactionmetrics";
            messageQueue2.Formatter = new XmlMessageFormatter(new Type[]
                {typeof(TransactionMetricsPayload.Header)});

            while(true)
            {
                Message myMessage = messageQueue2.Receive();
                Header header = (Header)myMessage.Body;

                SqlCommand command = new SqlCommand();
                SqlTransaction transaction = connection.BeginTransaction();
                command.Connection = connection;
                command.Transaction = transaction;
                foreach(ClientReport c in header.ClientReports)
                {
                    command.Parameters.Clear();
                    command.CommandText =
                        "if not exists(select * from tblClient " +
                        "where ClientID=@clientId) insert into tblClient" +
                        "(ClientID) values(@clientId)";
                    command.Parameters.Add(
                        new SqlParameter("@clientId", c.clientID));
                    command.ExecuteNonQuery();

                    command.Parameters.Clear();
                    command.CommandText =
                        "insert into tblTransaction(TransactionID, " +
                        "ClientID, ClientStartTime, ClientElapsedMS, " +
                        "ServerStartTime, Type, Status) " +
                        "values(@transactionId, @clientId, " +
                        "@clientStartDateTime, @clientElapsedMS,
                        "@serverStartDateTime, @type,@status)";
                    command.Parameters.Add(new SqlParameter(
                        "@transactionID", c.transactionID));
                    command.Parameters.Add(new SqlParameter(
                        "@clientId", c.clientID));
                    command.Parameters.Add(new SqlParameter(
                        "@clientStartDateTime", c.clientStartDateTime));
                    command.Parameters.Add(new SqlParameter(
                        "@clientElapsedMS", c.clientElapsedMS));
                    command.Parameters.Add(new SqlParameter(
                        "@serverStartDateTime", c.serverStartDateTime));
                    command.Parameters.Add(new SqlParameter(
                        "@type", c.type));
                    command.Parameters.Add(new SqlParameter(
                        "@status", c.status));
                   command.ExecuteNonQuery();
                   •••
                }
                transaction.Commit();
            }
        }
    }
}

Multiple MetricsReaders can be started. The app doesn't employ a data access layer because the design is simple and the performance of the metrics infrastructure is relatively unimportant. Metrics header logging would in practice be performed on distinct servers anyway to reserve other resources for application servers.

Client Design and Coding

The client application in this article is designed as a console application, one that creates multiple threads, each one simulating a client that is submitting factor transactions. It is easy to build because I use the proxy class that was automatically generated from the service's WSDL to handle communications with the server. A PayloadManager class is developed to handle metrics recording. This class is general enough to be used in any Web services client application. For example, to build a trading application client, you would use the server class generated from the trading service's WSDL and the identical PayloadManager class used in this sample.

WSDL-Generated Service1 Class

The WSDL for the factoring service can be viewed in Microsoft Internet Explorer by entering the URL http://yourserver/Factor/Service1.asmx?WSDL. When you open it, notice the definitions for Factor and the CheckJob services and the FactorAck structure returned by Factor. It also has the complete definitions for Header and the ClientReport and the AVPair arrays it contains. The .NET Framework wsdl.exe utility is used to generate this class by entering the following command line:

WSDL -l:cs http://yourserver/Factor/Service1.asmx?WSDL

I want to be able to move the client application to a different machine. Since the server name is embedded in the client class that wsdl.exe generates, this should be a name that can be resolved to the server host I want to use regardless of the machine on which the proxy will be used. The \ConsoleClient\Service1.cs file has the WSDL-generated code for the sample application.

PayloadManager—The Client Metrics Recording Library

PayloadManager is the central part of the metrics recording design. It allows any client application to construct useful metrics payloads for response time analysis and system capacity planning (see Figure 9). It was designed with a number of objectives in mind. It must not require any new message exchanges between the client and server. It should be general enough to be used within any client application that implements business transactions with SOAP requests. In addition, it should provide a facility to buffer a configurable number of metrics before uploading them, and it should allow the application to choose which SOAP requests might have metrics headers. This allows the implementation to be tuned according to server and network resource limits.

Figure 9 PayloadManager (Close Figure 9)
using System;
using System.Collections;
/// <summary>
/// Utility class for managing transaction metrics payloads
/// </summary>
public class PayloadManager
{
    private string mClientID; // unique ID for this client.
    private int mBufferSize;  // num of reports to hold
                              // before sending

    private ClientReport currentReport;

    private ArrayList currentTransactionAttributes = new ArrayList();
    private ArrayList currentClientAttributes = new ArrayList();
    private ArrayList clientReports = new ArrayList();

    public PayloadManager(Guid clientID, int numToBuffer)
    {
        mClientID = clientID.ToString();
        mBufferSize = numToBuffer;
        currentReport = new ClientReport();
    }

    public void AddAttributeToCurrentClient (string a, string v)
    {
        AVPair avPair = new AVPair();
        avPair.attribute = a;
        avPair.value = v;
        currentClientAttributes.Add(avPair);
    }

    public void BeginTransactionRecord(string type)
    {
        currentReport.clientStartDateTime = DateTime.Now;
        currentReport.type = type;
        currentReport.clientID = mClientID;
        currentTransactionAttributes = new ArrayList();
    }

    public void SetTransaction(Guid id, DateTime serverstart)
    {
        currentReport.transactionID = id.ToString();
        currentReport.serverStartDateTime = serverstart;
    }

    public void AddAttributeToCurrentTransaction (string a, string v)
    {
        AVPair avPair = new AVPair();
        avPair.attribute = a;
        avPair.value = v;
        currentTransactionAttributes.Add(avPair);
    }

    public void AddAttributeToClient (string a, string v)
    {
        AVPair avPair = new AVPair();
        avPair.attribute = a;
        avPair.value = v;
        currentClientAttributes.Add(avPair);
    }

    public void EndTransactionRecord(string status)
    {
        //transaction attributes
        currentReport.transactionAttributes = new
             AVPair[currentTransactionAttributes.Count];
        currentTransactionAttributes.CopyTo
            (currentReport.transactionAttributes);
        currentTransactionAttributes = new ArrayList();
        //client attributes
        currentReport.clientAttributes = new
            AVPair[currentClientAttributes.Count];
        currentClientAttributes.CopyTo
            (currentReport.clientAttributes);
        currentClientAttributes = new ArrayList();

        currentReport.status = status;
        currentReport.clientElapsedMS = (Int32)DateTime.Now.Subtract
                (currentReport.clientStartDateTime)
                .TotalMilliseconds;
        clientReports.Add(currentReport);
        currentReport = new ClientReport();
    }

    public Header GetHeader()
    {
        if (clientReports.Count >= mBufferSize)
        {
            Header header = new Header();
            header.ClientReports =
                new ClientReport[clientReports.Count];
            clientReports.CopyTo(header.ClientReports);
            clientReports = new ArrayList();
            return header;
        }
        return null;
    }
}

The PayloadManager class constructor is used to specify the unique identifier of this client instance as well as to specify the number of ClientReports to buffer before sending. Client applications can use AddAttributeToClient to set client attributes.

BeginTransactionRecord is used to identify the start of a new business transaction and its type. A transaction will consist of several Service1 calls, the first of which must return a unique transaction identifier and the server time when the transaction was initiated. The client invokes SetTransaction to save these in the current ClientReport.

The client should precede any Service1 calls with a call to GetHeader to assign the HeaderValue member of the Service1 object. If the number of ClientReports is less than the numToBuffer threshold set in the PayloadManager constructor, the call will return null and no header will be sent in the next message.

AddAttributeToCurrentTransaction is used to add attributes to the transaction that is currently in progress. When a transaction is completed, the client application records the transaction status by invoking EndTransaction. Internally, PayloadManager records the elapsed time from the BeginTransaction call. It is important that both BeginTransaction and EndTransaction are invoked as close as possible to any user activity, so that elapsed times reflect the subjective experience of users.

In my implementation, PayloadManager is not thread safe. Even when a client application is multithreaded, server communications are almost always performed within a single thread.

ConsoleClient—the Client Simulator

A console application is used to simulate a population of clients submitting transactions of varying complexity at fixed intervals. ConsoleClient uses a number of command-line arguments, which are listed in Figure 10.

Figure 10 Command-Line Arguments (Close Figure 10)

Argument | Definition
scale | Size of maximum number to factor
count | Number of factoring problems/client (of size 0 .. scale) to submit
gdelay | Milliseconds between problem submissions
cdelay | Milliseconds between problem completion checks
buffer | Number of reports buffered before sending
threads | Number of clients to start
tdelay | Seconds between client starts

The tdelay parameter allows a client population to be introduced slowly to the system so that the rate at which server capacity is exceeded can be determined. Note that buffer is an upper limit—when a client thread is created, it is passed a random buffering factor to use whose value is between 1 and buffer. Since each client reports the actual buffer threshold it was given as a client attribute, the metrics database can be used to investigate any effect this value has on client response time.

Each client thread creates its own instances of Service1 and PayloadManager. Each thread submits count number of factoring problems between 0 and scale. It uses Thread.Sleep to wait gdelay milliseconds after a factoring problem is completed before submitting another. When a transaction is begun, a call to checkjob is made every cdelay milliseconds until the results are available.

The Sample Application

The download consists of C# projects and associated database SQL files. It is configured so that the entire sample transaction system can be run on a single Windows XP Professional installation, although in this example I will move the client simulator to a different machine. It contains several items, including a solution developed in Visual Studio .NET named ClientTransactionMetrics which consists of seven projects: ConsoleClient, ConsoleReader, Factor, DataAccessLayer, MetricsReader, TransactionMetricsPayload, and TransactionMetricsServer. They are all located in subdirectories of the main solution directory, except for Factor and its directory which should be placed in the default IIS root directory. Also included are FactorDatabase.SQL and MetricsDatabase.SQL, found in ClientTransactionMetrics\DB.

In order to run the system, you will need to create and populate the databases, create the message queues, and modify the source code server machine names as necessary. See the README.txt file included with the download for complete setup information.

Running the Transaction System

Create a command window and start the MetricsReader application from the command line. Create another command window and start an instance of ConsoleReader. If possible, move the ConsoleClient executable to a different machine (one that has the .NET Framework installed) and start it there with the following command line:

>ConsoleClient 1000000 100 333 2 6 1 10
(Note that if the client reports a "401 access denied error" it can be corrected by using CredentialCache.DefaultCredentials with the client proxy to authenticate to IIS using your domain credentials.)

This will start a single client thread, submitting 100 random factoring problems. The numbers to be factored will be between 0 and 1,000,000. It will wait a third of a second between problem submissions and it will wait two milliseconds after submitting a problem before checking results. The client will choose a random buffering factor between 1 and 6.

Each ConsoleClient thread reports the completion of 10 transactions at a time, so the output from the command just shown will be something like the following:

Thread:1 is at job:10
Thread:1 is at job:20
Thread:1 is at job:30
•••
If the system is running properly, ConsoleReader will report the count of completed factoring problems in increments of 50:
Jobs completed:2
Jobs completed:52
•••

Since ConsoleReader waits on the factorjob queue, it won't exit until it is shut down manually. MetricsReader will display every metrics report it receives after it logs it in the Metrics database. A sample metrics report follows. It shows the transaction and client IDs, the transaction start time, its elapsed time in milliseconds, and the attributes the client assigned to the transaction:

Metrics Report for Transaction: 9d873634-5f37-42c5-8348-730afe23b567
    ClientID: 9d873634-5f37-42c5-8348-730afe23b567
    Started at: 12/11/2002 1:49:29 PM
    Elapsed Time: 380
   Tx  Attribute: Number to factor Value:200472
   Tx  Attribute: Number of Factors Value:6
•••

Using the Metrics Database

In this section, I will demonstrate the usefulness of the collected metrics data by using it to determine the throughput that the system as a whole is capable of achieving. This measurement will account for client processing, network I/O, IIS, ASP.NET, queuing effects, database logging, and the CPU resources that are needed to factor a given problem.

This systemwide view must be distinguished from the analysis you would make to evaluate and optimize a particular component. Profiling and design analysis are necessary to ensure components make the best possible use of available resources. While this design will reveal capacity problems, you'll still have to use analysis to resolve them. The key benefit of client-oriented metrics gathering is that it allows overall transaction system capacity to be measured against its key performance target: client response time.

Performing the Test

In this section, I will use ConsoleClient to test the capacity of this system. I run two tests. In the first test, I run a single client with a low input rate. The purpose of this test is to examine response time in an unloaded system so I can study the effect that factoring problem complexity has on application response time.

In the second test, I run many clients in an attempt to overload the system. This is designed to measure server throughput, which is the maximum number of transactions a server can complete per unit of time. It is important to distinguish throughput from expected response time. Consider a system that has a maximum throughput of 100 transactions/second. In an unloaded system, a single client may expect a response time of approximately 1/100th of a second (ignoring network delays). That said, if I load this system with a real-world population that submits transactions at an average rate of 100 transactions/second, those clients will experience response times far slower that 1/100th of a second. While the average rate is 100/second, the random behavior of real-world client populations means that some intervals may see request rates of 80/second while others see 120/second. Backlog queues accumulate in the server during the peak arrival intervals, leading to increased response times.

To begin this example, move the ConsoleClient application to another machine. While ConsoleClient will use only a small amount of CPU (less than five percent) to load the system, it has to be running on a separate machine in order to accurately measure server capacity. Otherwise, the simulated clients will have difficulty generating new transactions when the system is heavily loaded with request traffic.

To make the analysis easier, first remove any existing client metrics from the Metrics database. Start the SQL Query Analyzer, connect to the Metrics database, and execute the following SQL:

delete from tblTransaction
delete from tblClient
delete from tblClientAVPair
delete from tblTransactionAVPair

On the server machine, start two instances of the ConsoleReader and shut down any instances of MetricsReader, which is tangential to the test. Since the metrics reports are queued, you can run it after the factoring load test is complete. I use two instances of ConsoleReader so that one can be computing factors if the other is waiting for a database update to complete. Start the ConsoleClient with the following command:

>ConsoleClient 1000000 100 333 2 6 1 10

This creates a single client load on the system. The client submits a factoring problem, checks back frequently for the results, and then waits a third of a second before submitting the next problem. The use of a single client prevents the accumulation of work queue jobs in the system so it can be used as a baseline estimate of the total server time needed to receive factoring problems for random numbers between 1 and 1,000,000 and deliver them back to the client. The client will finish after it submits 100 factoring jobs.

After the first ConsoleClient simulation ends, start another with the following command:

>ConsoleClient 1000000 700 333 2 6 10 30
This will create a total of 10 threads, each behaving like the single client in the last simulation. The threads are started at 30 second intervals in order to slowly increase the load on the system so I can test the point at which response times degrade. Since there are multiple clients submitting factoring jobs, a backlog will accumulate in the factorjob queue when the two ConsoleReader application servers are busy.

This simulation will take approximately 15 minutes to complete. When the client finishes, start a MetricsReader on the server and let it run until it has logged all the queued metrics reports into the Metrics database.

Analyzing the Results

Now I use the Metrics database to estimate the capacity of my simple factoring system. Recall that I designed the Metrics database for extensibility. I left the client and transaction attribute tables denormalized so that I could easily record new attributes in the client application without making schema or server design changes. This makes the analysis more difficult, but since analysis is performed offline on systems that don't affect production capacity, it's a manageable price to pay for the extensibility that is gained.

You can make the Metrics database more intuitive by creating a view. This view joins tblTransaction, tblClientAVPair, and tblTransactionAVPair into a single entity that relates transactions, response times, and both client and transaction attributes (see Figure 11 for an example).

Figure 11 View Several Tables (Close Figure 11)
create view vTransaction
as
select
   t.*,
   Buffering=a1.Value,
   Problem=a2.Value,
   ProblemFactors=a3.Value
from
   tblTransaction t,
   tblClientAvPair a1,
   tblTransactionAvPair a2,
   tblTransactionAvPair a3
where
   t.ClientID = a1.ClientID and a1.Attribute='Buffer Size' and
   t.TransactionID = a2.TransactionID and a2.Attribute='Number
   to factor' and
   t.TransactionID = a3.TransactionID and a3.Attribute='Number
   of Factors'

For convenience, I define a summary view for clients:

create view vClient
as
    select
       ClientID,
           submitted=count(*),
           first=min(ClientStartTime),
           last=max(ClientStartTime)
    from vTransaction
    group by ClientID
This view summarizes all the distinct clients that have reported metrics, showing the time of their first and last transactions, and the total number each client has submitted. I ran all simulated clients on the same client machine so their times are synchronized. However, a real population of clients would not have synchronized times, so for this reason I also gathered ServerStartTime along with the metrics for each transaction. I could substitute it in the query just shown to get the best measure of relative time that is available for those situations.

The command "select * from vClient order by first asc" should display the client from the first test (the one where I only simulated one client) at the top of the list. I'll work with that client and study the behavior of its transactions. I embed the query (adding the "top 1" qualifier) in a dynamic table within an outer query to look at all the transactions that client reported on:

select *
from vTransaction t,
(
    select top 1 *
    from vClient
    order by first asc
) loneclient
where
   t.ClientID = loneclient.ClientID
Note that all the buffering values are the same. A simulated client chooses a random buffering value and uses it to control all of its uploads. Also note that if the buffering value is greater than 1, there will be missing requests. This is because the final request issued by a client will not be reported. Once again, this is a sampling design, intended to give data on almost all operations. In my case, the client used a buffering factor of 3, so it only reported 99 of the 100 transactions it submitted.

Since that client ran alone, there was no system overload and no queuing, so I can study the relationship between the number of prime factors computed and the elapsed time experienced by that client. I ran the following query to look at the time that client experienced for each factoring problem:

select NumberOfFactors=ProblemFactors, ClientElapsedMS
from vTransaction t,
(
    select top 1 *
    from vClient
    order by first asc
) loneclient
where
   t.ClientID = loneclient.ClientID
order by ClientStartTime

The first transaction takes almost eight seconds, an anomaly that can be attributed to the overhead of new object and connection creation when the server receives the first call to the Factor service. Almost all transactions are completed in well under a second; a handful, associated with factor counts of two, take approximately two seconds. This relationship is confirmed with the query shown in Figure 12. The output shows the average response time by the number of prime factors. Note the large difference between 2 ProblemFactors and the others. The ConsoleReader prime number factoring algorithm returns an array of two items when it is given a prime number to factor. The number to be factored appears in both slots. The algorithm must work considerably harder attempting to factor a prime number because its main For loop is executed n times where n is the number to be factored.

Figure 12 Response Times in Unloaded System (Close Figure 12)

Query
select ProblemFactors, occurences=count(*),averagems=avg(ClientElapsedMS)
from vTransaction t,
(
    select top 1 *
    from vClient
    order by first asc
) loneclient
where
   t.ClientID = loneclient.ClientID
group by ProblemFactors
Output
ProblemFactors   occurences  averagems
---------------- ---------- ----------
7                3           390
5                25          405
3                17          498
8                4           397
6                11          432
4                26          762
2                12          1076
10               1           400

The average client response time in my test for the last 98 transactions (excluding the anomalous one at the start) is reported as 524 milliseconds. However, this is not a true measure of the capacity that the server machine is capable of. Recall that two instances of ConsoleReader were running in both the first test and the second, multiclient one. However, in the first test, the behavior of the client—waiting until a transaction was completed until submitting the next—meant that there was only one transaction in the system at a time. As a result, there was always an idle ConsoleReader and no opportunity within the system to use available CPU time while one process in the chain was I/O bound.

The second test, which was designed to overload the system, represents a better test of system capacity. Each client in that test behaved in the same manner as the lone client in the first test. But since there were many clients, there was always sufficient work queued to take advantage of any available resource. Overloading a system like this gives a measure of throughput—the maximum rate of work a server can complete per unit of time.

The second test introduced 10 clients at the rate of 1 new client every 30 seconds. Each client submits factoring problems at the rate of 3 per second. Each client submits 700 transactions so that for several seconds in the middle of the run, all 10 clients are actively loading the server. A test like this is designed to slowly ramp up the number of clients and slowly increase the transaction rate in order to reveal the point at which response times degrade.

Now I'll create a new view of tblClients, one that selects only the clients that ran during the second test:

create view vClientM
as
    select
       ClientID,
           submitted=count(*),
           first=min(ClientStartTime),
           last=max(ClientStartTime)
    from vTransaction
    group by ClientID
    having count(*) > 650
Notice that the number of reported transactions returned by "select * from vClientM" differs among the clients—each used a different buffering factor, leading to a different number of concluding transactions that were unreported.

The query in Figure 13 uses vClientM. The output lists average transaction response times by the number of prime factors. While prime number problems (those with two factors) still take longer than the others, the difference is smaller than in the first test. There are two reasons for this. In the first test, since there was only one problem in the system at a time, the resulting response times are serially independent. This means that the time required for one problem is independent of the preceding one. This is no longer true in the second, multiclient test. Since there are many clients submitting transactions, the time required for each is dependent on two conditions. The first condition is the difficulty of the problem (whether the number to be factored is prime and also, to some extent, its magnitude), and the second is the state of the system (whether there is a backlog of unfinished work queued in the server that has to be completed first).

Figure 13 Response Times in Overloaded System (Close Figure 13)

Query
select ProblemFactors, occurences=count(*),averagems=avg(ClientElapsedMS)
from vTransaction t,
(    select *
    from vClientM
) multiclient
where
   t.ClientID = multiclient.ClientID
group by ProblemFactors
Output
ProblemFactors   occurences  averagems
---------------- ---------- ----------
7                462         961
5                1379        979
3                1453        1113
15               4           888
13               6           1121
11               28          863
8                256         899
6                854         984
4                1746        1013
2                590         1893
14               5           1601
12               7           1012
10               57          849
9                122         884

Let's examine the average response times against the number of input transactions over time. First, I define a view (vTransactionM) of only those transactions that occurred within the multiclient test. Then I do an inner join, getting the time offset from the beginning of the test. For each one-second interval I get the number of transactions that have been submitted and the average client response time (see Figure 14).

Figure 14 Get Transaction Rates and Response Times (Close Figure 14)
create view vTransactionM
as
   select t.*
   from tblTransaction t, vClientM c
   where t.ClientID = c.ClientID
go

select
   second = datediff(ss,earliest.FirstTX, t.ClientStartTime),
   elapsedMS=avg(t.clientElapsedMS),
   numberSubmitted=count(*)
from vTransactionM t,
   (
    select firstTX=min(ClientStartTime)
from vTransactionM
    ) earliest
group by datediff(ss,earliest.FirstTX, t.ClientStartTime)
order by datediff(ss,earliest.FirstTX, t.ClientStartTime)
Figure 15 Graphing Response Times
Figure 15 Graphing Response Times

In Microsoft Excel, I take the query output and add 10 period trend lines, represented in Figure 15. Notice the correspondence between the input load trend and the response time trend. Normally, an overload test like this would saturate a transaction system, leading to exponentially increasing delays as the input load continued to exceed system capacity. But in this case, all server components run on a single machine. When the system reaches its capacity, it is unable to accept new work because the Web services have no available CPU time to accept an input transaction and queue it for processing. Recall also that in my simple example, the Web services check the same database that the application server uses to post results, leading to additional coupling between the two layers. This is why the input load rate in the example has the long plateau in the middle of the run.

In a real production system, the Web server, the application servers, and the database might all be running on different machines. In that architecture, the Web components can accept new work and queue it for delivery to internal servers. Queues can accumulate, leading to exponentially increasing delays when the capacity of server machines is exceeded.

Limitations

The design I've described makes it difficult to add application data to SOAP headers. For example, a system that uses SOAP headers to contain authentication credentials would have to modify the Header class that is implemented here in order to add application-specific fields. PayloadManager processing would have to be modified as well, since it would no longer be the sole owner of the Header. A better approach in this case might be to use a SOAP attachment, rather than a header, for metrics data. Alternatively, applications services could implement an optional general-purpose metrics block, one similar to the Header class, within each Web services call. The Header class is also not optimized for size. A redesign might be necessary for very low-bandwidth connections.

The ClientID in this design really represents a session; it doesn't represent a client application. Recall that a new GUID is generated whenever a new PayloadManager instance is created. In practice, it would really be better to use a machine identifier for ClientID. The Header should be extended to contain a SessionID in addition to the ClientID, and a new session entity should be added to the Metrics database as well.

As I mentioned previously, the DataAccessLayer should not be used as a model for a production data access layer. It doesn't employ connection pooling, and it doesn't use transactions to guarantee referential integrity in the Factor database.

Further Work and Possible Extensions

This design should be evaluated against other Web service implementations to ensure that the Header class works with other SOAP client implementations such as the Microsoft SOAP Toolkit.

This has been presented as a sampling design—a program that gathers metrics on most, but not all of the transactions. If deemed necessary, coverage could be increased by having PayloadManager save any buffered metrics reports to the client file system when an application session ends.

This design is not limited to response-time metrics; it can also be used for general-purpose quality reporting. Examples of other uses might be to add software version and operating system environment items as client attributes. Application faults could also be added as client or transaction attributes.

The Metrics database can be extended to include summaries of system resource utilization over time by loading operating system performance counters into the database. Client response time delays that indicate system bottlenecks could be matched against machine resource utilization to identify the limiting machine or application. Client application resources can contribute to delays as well, and so the items, such as client CPU utilization or available bandwidth for example, can be included as transaction attributes.

The MetricsReader implementation can be extended to allow near real-time intervention when clients experience problems. For instance, a metrics report that indicates an application software problem or an extremely long wait for a transaction could utilize SNMP to raise an alarm.

Brian Connolly is an independent consultant specializing in enterprise transaction system design and performance analysis. He has led development teams that have produced international trading systems and data warehouses. Contact Brian at brian@ideajungle.com.


Editor's Note: Topic Coverage Update

Editor's Note
Topic Coverage Update

Over the past couple of months, we've received a lot of mail about our recent issues. Although opinions are mixed, several people have let us know that they feel we've covered too much forward-looking content recently. This is a valid concern, and we'd like to address it.

Dev magazine coverage around products always goes in cycles. There's the beta build-up, the early adopter coverage, and the maintenance mode coverage. We've always felt that each of these areas is important to you especially because many of you have told us you keep issues for a year or more, referring back to them when an article becomes relevant to your current projects.

Last autumn, we had a unique opportunity to provide early coverage of two long-term betas that were announced at the Los Angeles PDC: "Longhorn" (the next version of Windows) and "Yukon" (now known as SQL Server 2005). These products will both be extremely important in the years to come, and we wanted to give you the first in-depth look at them, which we did in our January and February 2004 issues.

In the May and June issues, we devoted considerable space to Visual Studio 2005 and ASP.NET 2.0, two more major products that will soon be affecting the way you program. Three factors went into our thinking when planning these issues:

  1. More of you than ever are MSDN Universal subscribers, and therefore have access to these products today.
  2. We are still providing plenty of "right now" content, especially in our columns. For example, our June issue covered Visual Studio 2005, but it also featured lots of pages on current real-world solutions.
  3. You have a lot of say in your company's future decisions, and it's important that we bring you the best coverage of what's coming up so you can make those decisions wisely.

There are other factors in our content decisions as well. As one reader said, it's good to see products early on so that he can lend his voice to modifications and features that could make it into the final release. The past year has been unusual in that there are many more product cycles coming to the forefront—an updated OS, DBMS, Internet extensions, and development IDE. That's a lot of stuff to take in, and it's part of our role as the official Microsoft magazine for developers to bring you not only what you need to know now, but what you'll need to know in the next year or so. Since many of you may not have access to these products yet, we're wondering if the coverage was too detailed, too early. We have heard quite a few opinions on both sides of the fence, and we'd like to hear from you too. Drop us a line at mmeditor@microsoft.com and let us know what you think.

On another note, we are making some exciting improvements in our content. You'll notice a new column in the magazine this month: Test Run. Testing is as important to software development as the initial coding of an application, and we're putting our money where our pages are. Dr. James McCaffrey, an articulate expert on testing methodology, will be writing a regular column on, you guessed it, testing. This isn't the standard "run a unit test" kind of stuff, either. If you're the type of dev who disdains test, give Test Run a look anyway. We bet you'll change your mind.

We are also pleased to welcome some new regular authors to our columnist roster. Duncan Mackenzie, MSDN online's Visual Basic expert extraordinaire, will be sharing the writing credits of the revamped Advanced Basics column with two other industry leaders, Ken Getz and Carl Franklin. You may know Ken and Carl from industry conferences. (Carl's the one who cracks the bad jokes, and Ken's the one who politely pretends to laugh.)

Finally, we'd like you to do a little exercise with the magazine. Place this issue on a flat surface and turn to the last page. What do you see? Hmmmm? Our back page has been overhauled. The new column, { End Bracket }, will feature a lively mix of commentary from key industry figures, resource information for developers, and maybe an occasional interview. Don't like the way we put spaces around the curly braces? Don't take it out on the guest columnists; they didn't know it was happening.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Uwa Agbonile, Mike Ammerlaan, Eugene Bobukh, Brandon Bray, Mike Downen, Mike Fitzmaurice, Bill Gibson, Chris Hollander, Kannan C. Iyer, John Koropchak, Min Kwan Park, Ronald Laeremans, Ivan Medvedev, Patrick Ng, Gus Perez, Joel Pobar, Matt Powell, Ramesh Rajagopal, Yasser Shohoud, Keith Short, John Stallo, Alex Torone, Dmitriy Vasyura, and Daniel Witriol.

Active Directory, FrontPage, IntelliSense, JScript, Microsoft, MSDN, Outlook, SharePoint, Visual Basic, Visual Studio, Windows, Windows NT, and Win32 are registered trademarks of Microsoft Corporation. Windows Server and Xbox Live are trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Nancy Michell

Contents
Enforce Cross-Team Coding Standards
Read RSS Feeds on TV
Ready-Built ASP.NET Components
Powerful Listboxes at Your Disposal
Keep an Eye on Your Network
User-Managed Logon Credentials
Manage Your Internet Passwords Safely
Get More Out of Your Windows Clipboard
Get Clean Screen Shots for Technical Documentation

Enforce Cross-Team Coding Standards

FlexiSoft Solutions Inc. has just released the latest version of their coding standards enforcement tool, Standards Master 2004, an add-in for Visual Studio® .NET. Standards Master 2004 brings many of the best features of Standards Master 2.0 to Visual Studio .NET and adds several new features to easily enforce standards compliance.

Standards Master 2004 checks your .NET-based code against flexible, configurable coding standards that you define, reporting any violations it finds. It supports just about any naming convention and can even correct the names of your variables automatically as you type. Improved suggestion logic provides numerous choices for violations. When you have misnamed an identifier, simply choose one of the corrected names from the list of suggestions and Standards Master 2004 does the rest.

With Standards Master 2004 you may also define and enforce dozens of code construction standards, including checks for use of gotos, required exception handling logic, lines per file/class/function limits, and many more. Standards Master 2004 can fix many coding standards violations in your code automatically.

In addition to its standards checking options, Standards Master 2004 also includes a full-featured commenting tool, allowing developers to insert header comments (flowerboxes), revision histories, and inline comments at the touch of a button. The new Comment All functionality searches your code for missing comments and prompts you to enter them as needed.

Standards Master 2004 also includes a robust code formatting tool that allows you to indent and unindent your code in a consistent fashion, based upon the options you define. Automatic whitespace control, region creation, and outlining features keep your code neat and tidy.

http://www.flexisoftsolutions.com

Read RSS Feeds on TV

NewsGator Technologies has just launched NewsGator Media Center Edition, which allows users to read syndicated content feeds on their television with Windows® XP Media Center Edition. Both text and multimedia content is supported, with an interface designed to be used with a remote control.

Audio and video content is only one click away. Users see visual cues when a feed contains multimedia content; they can then instantly view this content using the remote control. There are a number of featured feeds for the launch, including a video feed from Microsoft.

Users typically use more than one device throughout the day, such as Outlook® at work, a mobile phone on the road, a Web browser in their home office, and a TV in their living room. NewsGator Online Services allows users to read one set of syndicated content from any of these devices, without any duplication.

The deep integration of technologies like XML and RSS, combined with rich multimedia, and presented on a device as pervasive as the television, gives consumers access to the information they care about, where and when they want to view it, truly delivering on the vision of a connected home.

NewsGator Online Services also includes three other content reader editions: Web Edition, POP Edition, and Mobile Edition. The service also provides subscriber-only content, as well as the ability to search for content that matches a specific keyword or URL, and return that content in a feed.

http://services.newsgator.com

Ready-Built ASP.NET Components

As an ASP programmer you've no doubt noticed that many of the functions and features of the desktop development environment are not available in ASP.NET. In order to implement many of the user interface features provided in desktop languages, you would normally use DHTML and JavaScript.

But using DHTML means that if you're new to ASP.NET you not only have to get up to speed with the many changes in the latest version of Visual Studio such as the big leap from ADO to ADO.NET, the syntax changes, and so forth, but you also need to learn JavaScript and DHTML. That is where the idea for ASPXpand came from.

ASPXpand is a server-side .NET component from Can't Be Done Software that does the DHTML dirty work for you. ASPXpand is a library of functions that literally expands the capability of the native ASP.NET components and includes some vital, missing features for the codebehind environment. The goal of this component is to provide a wide variety of functions that are very useful and flexible.

Flexibility is apparent in the data entry functions, which can perform validation of content, among other tasks; in visual functions that can resize windows, provide tooltips, and set the focus; and in non-visual functions that can export data to XML, add events, and lots more.

ASPXpand allows developers to worry less about interfacing questions and focus more on learning the many new changes to their development environment.

http://www.cantbedone.com

Powerful Listboxes at Your Disposal

EasyListBox, recently released by EasyListBox.com, is a custom control that helps ASP.NET developers build cross-browser listbox applications in very little time, with very little code. The Microsoft® Internet Explorer rendering engine provides features that have been sought after for years, like multiple columns, HTML and image support, combobox functionality, and cooperation with overlapping DHTML.

Version 2 provides dependent lists and linked lists with absolutely no code to write. Extra add-on controls like the ActionButton and DisplayField give even more power to the user with no additional time investment.

http://www.easylistbox.com

Keep an Eye on Your Network

PAC Software has released Network Console to help network managers monitor the status of their networks. When a failure occurs, information about it is displayed graphically in the Network Console window, and an alarm message can be sent to e-mail addresses, pagers, cell phones, and other devices so network managers can respond immediately rather than waiting for users to report an outage.

Network Console allows managers to draw a chart showing the nodes in their network and how they are connected. The chart shows a real-time view of the status of each node, with failing nodes displayed in yellow or red. Network managers can view the chart from any Windows-based computer in the network or from a Web browser.

In addition to displaying the status of the network, Network Console sends alarms to system managers whenever nodes change status. Network Console can test the status of network nodes through simple pings, or by opening sockets to test such services as HTTP, FTP, and SMTP.

Administrators can customize the frequency of testing for each node, as well as how long a failure must persist before an alarm is sent. Although Network Console can only be run on Windows-based computers, it is not limited to testing Windows nodes—any node can be tested if it can ping or open a socket.

http://www.NetworkConsole.com

User-Managed Logon Credentials

Wouldn't it be great if your users were able to recover their own forgotten passwords without having to call helpdesk? The folks at ANIXIS thought so, so they have released ANIXIS Password Reset v. 1.0 for Windows. The software package lets users manage their own access credentials by initially asking them a number of personal questions and hashing the answers to those questions. The number of questions is configurable and can range from one to ten.

Later, when a user forgets his password, he is asked to answer the questions. The answers are then hashed and those hashes are compared with the ones stored. For safety, Password Reset does not store the users' passwords or the answers to their questions, just the hashes of each user's answers.

Password Reset saves money with minimal implementation and maintenance costs. There are no database engines to patch or upgrade, and no third-party runtime programs to install. In addition, system overhead is reduced because users can simply use a standard Web browser to access Password Reset.

ANIXIS Password Reset v. 1.0 runs on Windows NT® 4.0, Windows 2000, and Windows Server™ 2003.

http://www.anixis.com

Manage Your Internet Passwords Safely

TK8 Productions has released TK8 Safe, a password manager for Windows that makes it easy to safely store and retrieve user IDs, passwords, serial numbers, and other confidential information required by Internet sites and software applications. All of your private information is stored in a single, encrypted database, and is accessible only to its owner. Unlike password managers that are difficult and time-consuming to use, TK8 Safe's intuitive interface and extra features make it simple to sort, search, and use the confidential information you need.

TK8 Safe's autofill feature lets you log in to password-protected Web pages automatically by locating the required entry and answering the login prompt for you. This feature works with all browsers and programs that ask for passwords. Alternatively, you can click the Web site's URL and use traditional cut-and-paste operations for your passwords.

For ongoing protection, you can lock your password file or set a timer so that the file locks itself during periods of inactivity. With its multi-user support, TK8 Safe can be used in the office, classroom, family room, or any setting where several people need to use a single computer. TK8 Safe runs on all versions of Windows from Windows 95 and above.

http://www.tk8.com

Get More Out of Your Windows Clipboard

How often do you wish you could recover data left on your Windows clipboard before you logged off? With X2Net SmartBoard V4, from X2Net Limited, you and your users can recover that data and even share clipboard contents with other users. This major revision of the utility that enhances the capabilities of the Windows clipboard provides significant new functionality including full multi-user capability and a completely revamped user interface.

X2Net SmartBoard's basic function is to extend the data storage capabilities of the Windows clipboard, storing multiple items even across a Windows restart. All clipboard formats are supported. Captured clipboard items can be organized using the familiar folder and subfolder concept.

X2Net SmartBoard is the ideal repository for commonly used boilerplate text items that you'll paste into a document over and over again. In addition it has spell checking, syntax highlighting, speech, text reformatting, clip analysis, rule processing, and a host of other features. Its capabilities can be extended using scripts written in any popular scripting language.

http://www.x2net.com

Get Clean Screen Shots for Technical Documentation

TNT Screen Capture, recently released by EC Software, is a screen capture and image editing tool designed to create enhanced screenshots for technical documentation and illustration.

The tool captures virtually everything from application interfaces to parts of the desktop to popup menus. Available options allow you to add soft shadows, cursor symbols, and image effects like a roughly cut paper edge. An easy screen capture wizard guides you through these options for a positive experience even at the very first try.

Once you take the screen shot, you can add text, bubbles, frames, icons, symbols, and other images quickly from a tool window. Every object has a variety of shadow and transparency options, including a fading semi-transparent overlay. The program also includes standard image manipulation functions for resize and crop, as well as color and effects filters.

TNT Screen Capture even includes a built-in raytracing unit that renders a screenshot in 3D. This is not the usual skew function that you find in many image editors. It is a simple but very efficient raytracer that lets you rotate the screenshot, move the camera, turn lights on and off, and adjust shadows and surface reflection. The result is a screen shot that looks photorealistic, especially on a white background. When you're ready to save, a variety of formats are supported, including bitmap, JPEG, PNG, and TIFF.

TNT Screen Capture runs under all versions of Windows from Windows 95 and above.

http://www.ec-software.com

Send your New Stuff to newstuff@microsoft.com.

Nancy Michell is a developmental editor at MSDN Magazine where she spends her time manipulating bits of string literals for the good of mankind.


Web Q&amp;A: Pop-Ups, Encrypting an ADO.NET Data Stream, and More

Web Q&A
Pop-Ups, Encrypting an ADO.NET Data Stream, and More
Edited by Nancy Michell

Q I am using a WebBrowser control in a C# application. When using the control to navigate to a Web page, some pop-up windows appear, which I'd like to disable. I tried the following code:

private void axWebBrowser1_NewWindow2(object sender,
    AxSHDocVw.DWebBrowserEvents2_NewWindow2Event e)
{
    e.cancel = true;
}

This causes another problem: when I click a link in the page, if the link is opened in a new window, this window is also killed.

A It's really difficult to determine whether a new window is one the user wants to see or not; she could be browsing to a page or Web app that's supposed to open a new window or she could have right-clicked and selected "open in new window." Ideally you would probably want to parse the Web page that's being loaded in the new window and determine its relevance, but that's nearly impossible.

So, it's a tough problem. You have to consider things such as targeting unknown frame names, using NavigateAndFind, and so forth. You could set a flag when you get the click message, but when do you unset the flag? Some pop-up blockers use timers. When you get a click, the first new window opened in the next 2500 ms is allowed; everything else is blocked. But even this is not foolproof.

Windows® XP Service Pack 2 contains a new event, NewWindow3, which adds a parameter that includes flags indicating whether or not mshtml.dll thinks a pop-up window is the result of a user-initiated action. Using that flag would be the optimal solution, once you upgrade to Windows XP Service Pack 2.

For more information, take a look at Changes to Functionality in Microsoft Windows XP Service Pack 2 and also check out Jeff Davis' blog at XP SP2 RC1.

Q Is there any way to encrypt an ADO.NET data stream as part of SQL replication? My SQL server is hosted at an ISP and I want to use SQL replication to replicate the database to my internal site.

A Take a look at the Encrypt connection string keyword (SqlConnection.ConnectionString Property). When this value is true, SQL Server™ uses Secure Sockets Layer (SSL) encryption for all data sent between the client and server if the server has a certificate installed. Recognized values are true, false, yes, and no.

Q My group is porting a legacy ASP application to ASP.NET. The old version uses lots of calls to the JScript® escape and unescape methods. It doesn't look like there are direct C# equivalents in the ASP.NET world for server-side calls to escape.

What is the best practice for porting the JScript server-side escape method to C#? Ideally, we don't want to have to modify our client-side script calls to unescape.

A You probably shouldn't try to port it to C# in the first place. Why not just use JScript .NET? JScript .NET was specifically designed for your scenario—you've got a bunch of JScript code that has to be ported to ASP.NET. Why go through all the expense and difficulty of porting working code to a new language?

If you're set on porting it to C#, and you have determined that you meant to call escape (not UrlEncode, UriEncode, or any other encoding methods) then probably the easiest thing for you to do would be to call Microsoft.JScript.GlobalObject.escape.

Q I've got a client that creates a proxy and calls a Web service. It works fine until I deploy it on another server, but if the time difference between the client and server is more than five minutes, the Message Expired exception is thrown from SoapHttpClientProtocol.ReadResponse. I don't want to sync the time between the client and the server. Where can I set a value for Microsoft.Web.Services.HttpSoapContext.ResponseContext.Timestamp.Ttl?

A You can set the Ttl (time-to-live) value on the response inside the Web service before sending the response back if the message is expiring on the response. To be clear, set RequestSoapContext.Timestamp.Ttl from the client if the message is expiring on the request; set ResponseSoapContext.Timestamp.Ttl inside the Web service if the message is expiring on the response.

Q What's the best way to load site configuration settings stored in a database into an ASP.NET Web site before the Web site is accessed by users? I want to run an initialization process on a subsite before it is accessed by users.

A Within Global.asax you will find the empty shell method:

protected void Application_Start(Object sender, EventArgs e)
{

}
This method is fired when the ASP.NET worker process or Microsoft® Internet Information Services (IIS) starts up an instance of this application, and this method will also fire every time the application is recycled (in IIS 5.0 and IIS 6.0) or when starting another process to handle requests (IIS 6.0).

One caveat to address is to make sure any resource (config file, database, share, for example) that you are accessing from the Application_Start is safe for concurrent access. By that I mean that the resource is read-only and specifically allows concurrent access or has some mitigation to resource contention.

You should also be aware that when the application has not yet executed on a server (after a shutdown or recycle), the first client will trigger the application to be started and loaded into memory, and this will trigger the App_Start. Depending on how long your configuration resources take to load and process, the first client will take significantly longer to load the page.

Q Can I use Active Directory® to store relatively static application data? My business application has a set of reference tables (about 10 tables with a total of about 2,000 to 5,000 rows). Can I place those tables into Active Directory? I plan to store pure business application data such as a set of permitted operations and the code to perform these operations.

A You should have no problem at all. Keep in mind that you have to define a proper schema definition for your attributes. You also have to decide whether you want them to replicate with the Global Catalog. If you do, you need to ensure you are prepared for the schema update and possible Global Catalog rebuild (if you are running domains with Windows 2000 domain controllers). You could also store your application-specific data through an application partition in Active Directory or in a separate Active Directory Application Mode (ADAM) instance. These both have different implications for your design that you will have to evaluate, but both will give you more control over replication.

The other thing you'll want to keep in mind is that whatever data you store in the directory should be fairly static. If you know that you're going to be writing more than reading the data, you're probably better off with a SQL store.

If you're trying to implement role-based access control for the application, an alternative approach would be to use AzMan which enables you to create the operations, tasks, and roles and store them in Active Directory or in an XML file store, depending on your application requirements.

Q I have a problem regarding the default HTML Submit button. I need the Cancel button to appear in the left side of the OK button, which is the opposite of the default behavior. I also need the default Submit button to say "Okay," not "OK." I found Microsoft Internet Explorer will set focus to the first declared button (Cancel in this case) when users start to input data. When users hit the Enter button, a Cancel.Click is fired.

A Here's one solution: don't use two submit inputs in the form; use one submit and, if needed, add buttons. If you need your Cancel button to submit the form, implement this in client-side code, as shown in Figure 1.

onclick="javascript: Form1.Submit();"

Figure 1 Submit Button (Close Figure 1)
<HTML>
    <HEAD>
    </HEAD>
    <body>
        <form name="Form1" id="Form1">
        <table cellspacing="0" cellpadding="0" border="0">
            <TR>
                <TD colspan="2"><input name="TextBox1" type="text"
                    id="TextBox1" style="width:150px;"></TD>
            </TR>
            <TR>
                <TD><input type="button" name="Cancel"
                     onclick="javascript: Form1.Submit();" value="Cancel"
                     id="Cancel" tabindex="2" style="width:60px;"></TD>
                <TD><input type="submit" name="Okay" value="Okay"
                     id="Okay" tabindex="2" style="width:60px;"></TD>
            </TR>
        </form>
    </body>
</HTML>

Q I have a form that has a server-side textbox control and a Submit button. The textbox is a server control because some data validation needs to happen right away. If the user makes an edit in the textbox and doesn't tab off but rather clicks the Save button, only the change event for the textbox fires. Is there a way to get both events to fire on the server?

A If you have AutoPostback turned on for the textbox, it will submit as soon as the focus is moved out of that textbox, causing you to lose the next Submit button event. In order to get both events, you need to turn off the AutoPostBack for the textbox. Then when you click the Submit button, the TextChanged event will fire first, then the button click event will fire. As far as providing user feedback as they tab off the field, if you use a validator control for the textbox, you can get immediate feedback using script but still preserve the textbox's event on postback.

Q Is it possible to redirect the client browser to a new page by passing parameters via an HTTP POST request? If so, how? I don't want to use a meta tag refresh to redirect the page.

A You can process the parameters through the Request.Forms property, then use Response.Redirect or Server.Transfer to perform a redirect. An article comparing Server.Transfer and Response.Redirect can be found at Server.Transfer Vs. Response.Redirect. If you are looking to post to a page not on your server, then you won't be able to use either of them. In that case, you may have to go for a page with hidden forms and a JavaScript onload event. As soon as the onload event fires, submit your hidden form. You can build the form on the fly, parameterizing the required fields and the action. The only caveat is that there will be tracking information about this page stored in the browser's URL history.

Q Can I run an exe from a button click in ASP.NET?

A Yes, if the requester has adequate permissions, it is possible to run an exe on the server. You can use the System.Diagnostics namespace to launch it. The code in Figure 2 shows how to use this object to display the directory information using cmd.exe. First create a wrapper class for this functionality. Then you can execute the command from any event or function like so:

dim myp As New myproc("cmd.exe", "/C Dir e:\")
myp.Start()

Figure 2 Imports System.Diagnostics (Close Figure 2)
Imports System.Diagnostics

Public Class myproc
    Public WithEvents p As System.Diagnostics.Process

    Public Sub New(ByVal ExeName As String, ByVal params As String)
        p = New System.Diagnostics.Process()
        p.StartInfo.FileName = ExeName
        p.StartInfo.Arguments = params
        p.StartInfo.UseShellExecute = False
        p.StartInfo.RedirectStandardOutput = True
        p.EnableRaisingEvents = True

        ' Handle the Exited event that the Process class fires.
        AddHandler p.Exited, AddressOf p_Exited
    End Sub

    Public Sub Start()
        p.Start()
        p.WaitForExit()
    End Sub

    Public Sub p_Exited(ByVal sender As Object, ByVal e As System.EventArgs)
        Dim output As String
        output = p.StandardOutput.ReadToEnd()
        HttpContext.Current.Response.Write(output)
    End Sub
End Class

When the process finishes executing, the Exited event will be raised and the standard output of the process will be written to the response stream for the current request.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Dimitri Artemov, Greg Bybee, Todd Carter, Jim Cheshire, Alex Chiang, Jeff Davis, Tamer Demir, Peter Foote, Andrew Goodsell, Robert Gruen, William Li, Eric Lippert, David Lovell, Will Martin, Herbert Mauerer, Jose Tomas Montelongo, David Qiu, Brian Roder, Zack Runner, Angel Saenz-Badillos, Luis Salazar, Ken Stanfield, Uma Subramanian, Stephen Toub, Michael Whalen, and Parker Zhang.


Data Points: Migrating from ADO to ADO.NET

Data Points
Migrating from ADO to ADO.NET
John Papa

Contents
The Evolution of ADO
Connections
Recordset to Readers
Single-Value Commands
Until Next Time

In recent years ADO has taken the lead as the preferred method for implementing data access in Windows®-based applications. Huge numbers of ADO applications are in use today and many developers are well versed in ADO development. With the introduction of the Microsoft® .NET Framework came ADO.NET—the evolution of ADO. While there are similarities between ADO and ADO.NET, the way they operate and their foundations are quite different. To help you make a smooth transition to ADO.NET, I'm going to take a look at how some common tasks are performed in ADO.NET.

I will discuss several data access situations, show how to tackle them with ADO, and show how they are accomplished with ADO.NET in ASP.NET using C#. I'll begin with the similarities in connecting to a data source. Then, I'll look at how the ADO Recordset object has evolved into many distinct and focused objects and methods in ADO.NET. Finally, I will explore firehose cursors, returning a single value from a rowset, and working with XML.

The Evolution of ADO

Some features of traditional ADO, such as establishing a connection to a data source, have changed little between versions. Other features have evolved considerably such as the ability to represent a disconnected rowset, to persist it to XML, and to transform it into a hierarchical rowset. One reason for the significant changes is that XML and data-shaping features were late additions to ADO; in ADO.NET they were built into the design up front.

Traditional ADO is very lightweight when compared with the data access tools of old such as DAO and RDO. One feature that made ADO so popular in n-tiered development with Visual Basic® 6.0 is its simple, easily navigated object model. The ADO Connection and Command objects translate to ADO.NET Connection and Command objects in a relatively straightforward manner, but the ADO Recordset object's features translate to several different objects and methods in ADO.NET.

ADO has remained a powerful and popular data access tool in part because of its XML features and its ability to manage disconnected rowsets. An ADO Recordset can be disconnected from its data source and Connection object by setting its CursorLocation property to adUseClient, its CursorType to adOpenStatic, and its LockType to adLockBatchOptimistic. Then, once the recordset has been opened and loaded, it can be disconnected by setting the recordset's ActiveConnection property to Nothing:

'---Disconnecting an ADO Recordset
oRs.CursorLocation = adUseClient
oRs.CursorType = adOpenStatic
oRs.LockType = adLockBatchOptimistic '— Or use adLockReadOnly
oRS.Open
Set oRS.ActiveConnection = Nothing

XML features were not integrated into ADO from the start, but as XML became more prevalent, its features were stapled onto later versions of ADO. The Save method of the ADO Recordset object can take the recordset's rows and columns and translate them to a predefined XML schema, then persist them to a file or a stream. The XML schema is not very flexible, but it was the first real attempt at persisting XML from a ADO rowset and it gave developers a glimpse of where development was heading. The recordset can also be loaded from an XML file as long as it uses the same XML schema that it expects. The following code shows how ADO can save a recordset to an XML file:

.Save "c:\MyRowSet.xml", adPersistXML

The rowset can be saved to a stream such as the output of the Response object from an ASP page. For example, some ASP code could field a request, retrieve a rowset, and stream the rowset to the Response object, which would return it to the client. The following code demonstrates how to save a recordset's contents to a Stream object. Alternatively, the Save method's first argument could be replaced with the Response object so it would stream the XML rowset to the calling browser:

Dim oStm As ADODB.Stream
Set oStm = New ADODB.Stream
oRs.Save oStm, adPersistXML

Traditional ADO and ASP have the ability to stream. Of course, with .NET this can be accomplished using Web services, which offer more features than any solution you can easily build using traditional ADO and ASP. In fact, Web services is a technology designed to process requests for data and transmit data back in XML across HTTP. While ADO can stream XML, this was an afterthought in its design. The ADO.NET DataSet object has WriteXml and WriteXmlSchema methods that allow its contents to be exported to an XML file or stream. So while the ADO Recordset's Save method has been retrofitted to be able to export its contents to XML, the ADO.NET DataSet had this functionality from the start.

Not only can ADO.NET persist a DataSet to XML and load one from XML, it also uses its XML-like structure to its advantage in other ways. For example, because the DataSet is fundamentally represented as XML, it can be passed between physical and logical layers with ease. This means that the XML can be passed through secured networks via HTTP as it takes the form of text-based XML.

Since it's built upon XML, ADO.NET can work disconnected. While a traditional ADO Recordset could either be connected or disconnected depending on several property settings (such as CursorType = adOpenStatic and CursorLocation = adUseClient), ADO.NET has dedicated RowSet objects that are either connected to a data source (the DataReader) or disconnected (the DataSet).

Connections

Creating connections with traditional ADO and with ADO.NET is a very similar process. Basically, you declare your connection object, instantiate it, set its connection string, and open it. Take a look at Figure 1. The first example demonstrates how to open a connection using ASP and ADO, and the second example shows how to open a connection using ASP.NET and ADO.NET.

Figure 1 Opening a Connection (Close Figure 1)

Traditional ASP and ADO in VBScript
dim oCn
dim sCn
set oCn = Server.CreateObject("ADODB.Connection")
sCn = "Provider=SQLOLEDB; Data Source=(local);" & _
      "Initial Catalog=Northwind;Integrated Security=SSPI"
oCn.Open sCn
ASP.NET and ADO.NET in Visual Basic .NET
Dim sCn As String = "Provider=SQLOLEDB; Data Source=(local); Initial" & _
             "Catalog=Northwind;Integrated Security=SSPI"
Dim oCn As New SqlConnection(sCn)
oCn.Open()
ASP.NET and ADO.NET in C#
String sCn = "Provider=SQLOLEDB; Data Source=(local); Initial" +
             "Catalog=Northwind;Integrated Security=SSPI";
SqlConnection oCn = new SqlConnection(sCn);
oCn.Open();

A major difference in creating connections with ADO and ADO.NET is that ADO fits all connections to all types of data sources into a single Connection object. ADO.NET can have separate objects that represent connections to different data sources. For example, ADO.NET has a namespace called System.Data.SqlClient which houses all the SQL Server™-specific ADO.NET objects including the SqlConnection object. The SqlConnection object is built specifically to communicate with SQL Server databases and thus is the fastest and most feature-rich way of interacting with SQL Server. There is also a more generic namespace called System.Data.OleDb which can communicate with any OLE DB-compliant data source. In ADO.NET you can create multiple data provider namespaces to connect specifically with a particular data source, making access faster and more efficient and allowing each namespace to exploit the features of its targeted data provider. If an application must either change the kind of data provider on the fly or focus on a provider that does not have a specific ADO.NET connection object, it can use the OleDbConnection.

Recordset to Readers

The ADO Recordset, which behaves differently depending on how its properties are set, has been broken up into many different objects and methods in ADO.NET. This allows ADO.NET's objects to work more efficiently since its objects can concentrate on what they do best instead of trying to be a jack of all trades. The ADO Recordset can be both a connected rowset and a disconnected rowset. It can act as a forward-only, read-only (firehose) cursor or it can allow positional moves backwards, forwards, and in the middle of the rowset. An ADO Recordset can allow data changes to be made directly to the database and it can allow data changes to be saved up and sent to the database in a batch. The point is that an ADO Recordset can wear many hats. ADO.NET splits this functionality into objects that are optimized to accomplish their specific tasks. Thus, the ADO Recordset's features have been split into the DataSet object, the DataReader object, and parts of the DataAdapter and Command objects.

Forward-only, read-only cursors require a continuous connection to a data source. ADO.NET has a DataReader object that, when open, is always connected. It was written specifically for a data provider, like SQL Server, Oracle, and the more generic OLE DB provider. Thus, the SqlDataReader object can connect to a SQL Server database and act as a firehose cursor looping through tons of records if need be. The SqlDataReader provides fast-forward-only access to query results. It retrieves a record from the query results from the database and leaves the connection open so it can get the next record in succession. The ADO.NET DataReader is extremely efficient because it does not support all of the features of the ADO Recordset. It's a lean firehose-cursor machine. The examples in Figure 2 show how to implement a forward-only cursor in classic ADO and in ADO.NET. Notice that in ADO.NET, the DataReader object's Read method automatically moves the position to the next record. This eliminates the common infinite loop problem that developers fall into with traditional ADO of omitting the ADO Recordset's MoveNext method.

Figure 2 Implementing a Forward-Only Cursor (Close Figure 2)

ASP and ADO
oRs.ActiveConnection = oCn
oRs.LockType = adLockReadOnly
oRs.CursorType = adOpenForwardOnly
oRs.CursorLocation = adUseServer
oRs.Open sSQL
Do While Not oRs.EOF
    Response.Write oRs("CompanyName") & "<br>"
    oRs.MoveNext
Loop
ASP.NET and ADO.NET in Visual Basic .NET
Dim oDr As SqlDataReader = oCmd.ExecuteReader()
While oDr.Read()
    Response.Write(oDr(0) & "<br>")
End While
ASP.NET and ADO.NET in C#
SqlDataReader oDr = oCmd.ExecuteReader();
while(oDr.Read())
{
    Response.Write(oDr[0] + "<br>");
}

Another difference in ADO and ADO.NET is how the forward-only cursor is filled. In ADO, all rowsets, forward-only or otherwise, are contained within the Recordset object which are opened via the Recordset.Open method or the Connection or Command's Execute methods. In ADO.NET there is a specific method built to retrieve forward-only data to a DataReader: the command object's ExecuteReader method. This method tells the command object to specifically set up the DataReader object for the data provider to process the results in an optimized forward-only manner, as shown in the previous example. There is also a method of the command object called ExecuteXmlReader which tells the command object to set up the query results to be processed by an XmlReader object. The XmlReader object can be used to translate and process XML query results as shown in the following ASP.NET example:

' Visual Basic .NET
oCn.Open()
Dim oCmd As New SqlCommand("SELECT * FROM Orders FOR XML AUTO",oCn)
oCmd.CommandType = CommandType.Text
Dim oXR As XmlReader = oCmd.ExecuteXmlReader()
While oXR.Read()
    Response.Write(oXR.ReadOuterXml())
End While

// C#
oCn.Open();
SqlCommand oCmd = new SqlCommand("SELECT * FROM Orders FOR XML AUTO",oCn);
oCmd.CommandType = CommandType.Text;
XmlReader oXR = oCmd.ExecuteXmlReader();
while(oXR.Read())
{
    Response.Write(oXR.ReadOuterXml());
}

This example shows how the XML can be sent to the calling browser. However, the XML could be gathered and streamed to the Response as a whole, as well.

Single-Value Commands

At some point, most apps need to retrieve a single value from a database query. The standard way to do this in classic ADO is to set up the SQL statement and open a recordset containing the query results. Since there is only one row and one column in the query's results, it seems like overkill to get that one value. The following example retrieves a single value from a query. This example grabs the number of rows in the Orders table:

'-- ASP and ADO
Set oRs = Server.CreateObject("ADODB.Recordset")
oRs.ActiveConnection = oCn
oRs.Open "SELECT COUNT(*) As iRowCount FROM Orders"
iCount = oRs.Fields("iRowCount").Value

ADO.NET introduces a new way of getting a single value from a query's results when you expect only one row and one column to return. The ADO.NET command object has an ExecuteScalar method which returns the first row and column's value from its associated query. This generates very little overhead as it does not have to create a rowset, find the value, and close the rowset. The ExecuteScalar method is optimized for the specific situation in which retrieving a single value is required. The following example accomplishes the same thing as the previous example, only using ASP.NET and ADO.NET with the ExecuteScalar method:

' Visual Basic .NET
Dim sSql As String = "SELECT COUNT(*) As iRowCount FROM Orders"
Dim oCmd As New SqlCommand(sSql, oCn)
oCmd.CommandType = CommandType.Text
Dim iCount As Integer = CType(oCmd.ExecuteScalar(), Integer)

// C#
string sSql = "SELECT COUNT(*) As iRowCount FROM Orders";
SqlCommand oCmd = new SqlCommand(sSql, oCn);
oCmd.CommandType = CommandType.Text;
int iCount = (int)oCmd.ExecuteScalar();

Another way to retrieve a single value is to use an output parameter of a stored procedure. This technique can also retrieve a number of values from a single row, for example. It's available in both ADO and ADO.NET, although ADO.NET has expanded output parameter abilities. To grab an output variable's value in ADO.NET from a command object, execute the query using the ExecuteNonQuery method. This method tells ADO.NET that the query will not return a rowset, so the overhead of a DataSet or DataReader will be avoided:

' Visual Basic .NET
oCmd.ExecuteNonQuery()
oCmd.UpdatedRowSource = UpdateRowSource.OutputParameters
Dim iOrderID As Integer = _
    CType(oCmd.Parameters("@OrderID").Value, Integer)

// C#
oCmd.ExecuteNonQuery();
oCmd.UpdatedRowSource = UpdateRowSource.OutputParameters;
int iOrderID = (int)oCmd.Parameters["@OrderID"].Value;

This code then sets the UpdatedRowSource property to point to the output parameters, assuming they have been set. Then, the output values can be retrieved. This was accomplished in traditional ADO using an obscure parameter on the Connection object's Execute method, but it's very straightforward in ADO.NET, as you see. Of course, ADO.NET also has a method optimized for returning a standard rowset—the command object's Execute method.

The Recordset object in traditional ADO can figure out the UPDATE, INSERT, and DELETE statements that are required for the Recordset to be able to apply to the underlying database any changes that have been made to the Recordset. While this feature is handy, it also comes with a lot of overhead as it has to go back to the database to figure out how to do this.

The ADO.NET objects can do this as well through the CommandBuilder object; however, it is also wrought with overhead. In most development situations the exact SELECT, INSERT, UPDATE, and DELETE statements are known at design time. In classic ADO there was no easy way to associate the action queries with a Recordset so that it would take advantage of them. But in ADO.NET a DataAdapter has four separate command objects associated with it to represent each action query and the SELECT statement. This allows the DataAdapter to help fill a DataSet with a query's results and tells the DataAdapter what action queries to issue against the database ahead of time. While this requires a bit more coding at design time, the extra code is a good trade-off for the performance gains (not to mention the ease of maintenance since the code is self explanatory). My May 2004 Data Points column demonstrates how to use the DataAdapter in detail.

Until Next Time

In this month's column I discussed the evolution of some key features from ADO to ADO.NET, including firehose cursors and persisting XML. In upcoming columns I will talk more about the evolution of cursors and demonstrate how to deal with concurrency issues and batch updates in traditional ADO and translate that to work with ADO.NET.

Send your questions and comments for John to mmdata@microsoft.com.

John Papa is a baseball fanatic who spends most of his summer nights rooting for the Yankees with his two little girls, wife, and faithful dog, Kadi. He has authored several books on ADO, XML, and SQL Server and can often be found speaking at industry conferences such as VSLive.


Cutting Edge: Implement Custom Cache Dependencies in ASP.NET 1.x

Cutting Edge
Implement Custom Cache Dependencies in ASP.NET 1.x
Dino Esposito

Code download available at: CuttingEdge0407.exe (128KB)
Contents
What's a Cache Dependency, Anyway?
Cache Dependencies in ASP.NET 2.0
Custom Cache Dependencies in ASP.NET 1.x
Setting Up the Timer
Creating a Web Service Dependency
Database Dependencies in ASP.NET 1.x
Conclusion

One of the most compelling improvements that ASP.NET brought to ASP programming was the Cache object. The Cache has some similarities to the Application object and is a container of global data (as opposed to session-specific data) that features a fair number of innovative characteristics. At its core, the ASP.NET Cache is a sealed data container class built around a hashtable and defined in the System.Web.Caching namespace. The Cache object is central to the whole ASP.NET infrastructure. Various runtime components use the ASP.NET Cache to store working data. For example, the output of cached pages and controls is stored there. Likewise, applications that keep session state in memory use the ASP.NET Cache to store it.

The Cache object is made up of two independent data stores—a public and a private cache. The private cache is reserved for system components and is accessed through a private API. The public cache is the one that is available to developers. In order to enable you to loop through the public store, the Cache class implements the IEnumerable interface.

Unlike data stored in the Application object, items stored in the Cache object can have a number of additional attributes applied to them, including an expiration policy, a priority, and one or more dependencies. These attributes form the substrate of the advanced Cache features such as the ability to remove least-used items, have items expire at a given time, and set logical connections between a group of items and disk files.

Like many other ASP.NET features, the Cache object has whetted the appetites of developers and given them more ideas for features to request in future versions. The most requested new Cache features are most likely support for database-driven data invalidation and custom dependencies. And if they aren't, they should be. The good news is that these two features will be available in ASP.NET 2.0 through a dazzling new class, SqlCacheDependency, which links cached items to database tables, and through the newly unsealed CacheDependency class.

The SqlCacheDependency class detects changes to tables in SQL Server™ 7.0, SQL Server 2000, and SQL Server 2005 (formerly code-named "Yukon") and invalidates all dependent cache entries. In ASP.NET 2.0, CacheDependency has new virtual members and, more importantly, is unsealed and can be derived from. SqlCacheDependency derives from CacheDependency.

In this column, I won't go through all the new caching features of ASP.NET 2.0, but I'll demonstrate how to implement similar custom cache dependencies in ASP.NET 1.x. A good understanding of cache dependencies is crucial if you are to create custom dependencies. So without further ado, let's start with a quick refresher of what a cache dependency is and how it works.

What's a Cache Dependency, Anyway?

The whole ASP.NET cache dependency mechanism is encapsulated in the CacheDependency class. This class represents a relationship between a cached item and an array of objects like files, directories, and other cached objects. To establish a dependency between a cached item and an external component, you add the item to the ASP.NET Cache object using a specific overload of the Insert method, like so:

CacheDependency dep = new CacheDependency(fileName);
cache.Insert(key, value, dep);
In this code snippet, the item is fully identified by the key/value pair and its lifetime is bound to the timestamp of the specified file name. The item will automatically be removed from the Cache when the timestamp of the specified file changes. To add a new item to the Cache you can also use the Add method or the set accessor of the Item property:
Cache[key] = value;

The Add method is not overloaded and it differs from the Insert method in that it throws an exception if the specified item already exists. (Insert, on the other hand, will overwrite the existing item.) You should note that if you use the set accessor of the Item property to add a new item, the item is correctly inserted into the cache, but no dependency is created. As a result of this, the item will be removed only when the application shuts down or if the item is explicitly removed.

A cached item can also be bound to other cached items instead of or in addition to file dependencies. The following code snippet shows how to accomplish this:

CacheDependency dep;
dep = new CacheDependency(fileNames, otherKeys);
Cache.Insert(key, value, dep);
When either the files or the other cached items change, the recently added cache object is invalidated and then removed. To make an item dependent only upon a cached item, you set the file name parameter to null. A cache dependency can be subordinate to another cache dependency. This feature is particularly useful for implementing cascading changes to stored items. So much for the programming interface of the CacheDependency class; now it's time to take a closer look at its internal implementation and interaction with the Cache data store.

Overall, items are automatically removed from the cache based on time, file, and key dependencies. It is interesting to look at how each type of dependency is handled. Time and key dependencies are managed by the Cache object itself. Items with an expiration are associated with a timer and removed at the end of the specified countdown interval.

Resolving key dependencies is just one of the many tasks important to the update of the cache. Whenever a write operation occurs, an internal update process fires and frees all of the items that have a broken dependency.

The file change notification mechanism keeps track of file dependencies. It's an operating system function that various ASP.NET modules, including the HTTP runtime, use extensively. When a file dependency is created, the ASP.NET Cache starts to monitor that file or directory. Thanks to the capabilities of the OS, any change on a monitored resource results in an event raised to the Cache object. The handler of this internal event takes care of the removal of the linked item.

That's the only way in ASP.NET 1.x to force the Cache to evict stored elements. This means that any form of custom dependency must be implemented in terms of file, time, or, more likely, key dependencies. So what's a custom cache dependency? It is primarily a class that listens to a data source for changes. When a change is detected, the class bubbles that change up to a particular stored cache item so that the item is evicted from the cache. This loose description can be translated into three actions that any custom cache dependency class should take in ASP.NET 1.x.

  • Define the data source to listen to
  • Define a channel with the Cache to bubble changes up
  • Define a custom API

The first two points are common to all versions of ASP.NET. However, only ASP.NET 2.0 provides an API that will create and manage custom dependencies. Let's proceed with a quick look at ASP.NET 2.0 cache dependencies before taking the plunge into my ASP.NET 1.x implementation.

Cache Dependencies in ASP.NET 2.0

As I mentioned, in ASP.NET 2.0 the CacheDependency class is unsealed and therefore inheritable. Subsequently, a custom dependency is simply a class that inherits from CacheDependency and implements a custom algorithm to detect changes in a given data source. Each detected change then causes an appropriate action to invalidate items in the cache. The use of inheritance guarantees that no breaking changes will ever be introduced in code ported from ASP.NET 1.x applications. In addition, there is no risk that your class misbehaves with the Cache object. The base class will handle all the wiring of the dependency object to the ASP.NET Cache object and all the issues surrounding synchronization and disposal. On the other hand, inheritance means that the memory footprint of your dependency class might be bigger than needed because your cache dependency class picks up all base class functionality, whether it needs it or not. Such functionality includes constructors that accept arrays of files or create dependencies on other cached items.

A good example of a custom dependency class is the aforementioned SqlCacheDependency. It implements database dependencies by listening to table changes on SQL Server 7.0, SQL Server 2000, and SQL Server 2005. Figure 1 lists the new members added to the CacheDependency class to support custom dependencies.

Figure 1 New Members of CacheDependency Class (Close Figure 1)

Member | Description
DependencyDispose | Releases the resources used by the class
FinishInit | Informs the base class that it's safe to call DependencyDispose
GetUniqueId | Retrieves a unique string identifier for the object
NotifyDependencyChanged | Notifies the base class that the dependency represented by this object has changed
SetUtcLastModified | Marks the time when a dependency last changed
HasChanged | Indicates whether the dependency has changed; this property also exists in version 1.x, but is not virtual
UtcLastModified | Gets the time when the dependency was last changed; this property also exists in version 1.x, but is not publicly accessible

A custom dependency class relies on its parent for any needed interaction with the Cache object. The NotifyDependencyChanged method is called by classes that inherit from CacheDependency to tell the base class that the dependent item has changed. In response to this, the base class updates the values of the HasChanged and UtcLastModified properties. Any cleanup code needed when the custom cache dependency object is dismissed should go into the DependencyDispose method.

The structure of a custom dependency object follows the pattern outlined in Figure 2. The class uses a timer to poll the data source at regular intervals and maintains a reference to the current data. When the data downloaded from the source is newer than the current copy, the corresponding item in the cache is invalidated. A call to the new NotifyDependencyChanged method breaks the internal dependency so that the Cache update process can evict the item.

Figure 2 Custom Dependency Object (Close Figure 2)
public class YourCacheDependency: CacheDependency
{
    static System.Threading.Timer _timer;
    int _pollTime;
    object _currentValue;

    public YourCacheDependency(int pollTime)
    {
        // Set some internal members using input parameters
        _pollTime = pollTime;

        // Get the current value from the source
        _currentValue = GetCurrentValue();

        // Set up the timer for next reads
        if (_timer == null)
        {
            TimerCallback fun  = new
                TimerCallback(CheckDependencyCallback);
            _timer = new Timer(func, this, pollTime, pollTime);
        }
        FinishInit();
    }

    public void CheckDependencyCallback(object sender)
    {
        YourCacheDependency dep = (YourCacheDependency) sender;

        // Get the current value from the source
        object _value = GetCurrentValue();

        // Compare to the current value
        if (!_value.Equals(_currentValue))
            dep.NotifyDependencyChanged(dep, EventArgs.Empty);
    }

    private object GetCurrentValue()
    {
        // Check the data source for changes
    }

    protected override void DependencyDispose()
    {
        if (_timer != null)
        {
            _timer.Dispose();
            _timer = null;
        }
        base.DependencyDispose();
    }
}
Figure 3 SqlCacheDependency with SQL Server 2005
Figure 3 SqlCacheDependency with SQL Server 2005

Note that in ASP.NET (no matter what the version) there are only two possible models for detecting changes—polling and notification. Polling means that the dependency class will periodically check the data source for changes. The notification mechanism is based on an external component that pushes changes to the dependency class in an asynchronous manner. A good example is the aforementioned file change notification mechanism. In ASP.NET 2.0, an implementation of the notification model is represented by the SqlCacheDependency class when it is connected to a SQL Server 2005 database. The overall picture is shown in Figure 3. The following is the code to set up the dependency:

// data is a DataTable filled using this same command
SqlCommand cmd = new SqlCommand(
   "SELECT * FROM Customers WHERE country='USA'", conn);
SqlCacheDependency dep = new SqlCacheDependency(cmd);
Cache.Insert("SqlSource", data, dep);

In this case the SqlCacheDependency class listens to data coming from a SQL Server 2005 component—the Notification Delivery Service. The service monitors all the tables involved with the given query and invokes a callback function whenever something happens that modifies the resultset generated by the query. Internally, SqlCacheDependency exploits the new notification feature of SQL Server 2005 and runs code like the following:

// cmd is the SqlCommand object defined above
SqlDependency dep = new SqlDependency(cmd);
dep.OnChanged += new OnChangedEventHandler(OnDepChanged);
The callback function defined by SqlCacheDependency—the OnDepChanged method—calls NotifyDependencyChanged to invalidate the corresponding cache entry.

Custom Cache Dependencies in ASP.NET 1.x

At its core, a custom cache dependency is a class that incorporates a timer to periodically check the data source or that listens for event notifications pushed to it. You must link an instance of this class to a cached item. Because the CacheDependency class is sealed in ASP.NET 1.x, you can't rely on the Cache.Insert method to establish this link automatically. In addition, an instance of the custom dependency class must be stored in the ASP.NET Cache to survive for the application's lifetime. When the dependency class detects a change on the data source, it must do something to invalidate the cached item. To do so, you can only use one of the base ASP.NET 1.x expiration techniques—time, file, or key. Time just doesn't apply here. File or key dependencies are both acceptable and I don't see a reason to create or edit a temporary file. So I'll associate each item that has a custom dependency with an additional helper entry. By modifying the helper entry on changes to the monitored data source, you'll automatically invalidate the cached item. A new API is needed to transparently create the helper entry upon insertion of the item with a custom dependency:

AmazonBooksCacheDependency dep =
  new AmazonBooksCacheDependency(key, author);
CacheHelper.Insert(key, dataSet, dep);

The AmazonBooksCacheDependency class represents a dependency on the list of books written by a given author. Some of the information from this data source is relatively static (books, titles, publishers); some other data, however, may vary on a regular, even daily basis. Price, sales rank, reviews, and rating are parameters that may change more than once a day. The previous code sample inserts a DataSet with book information into the ASP.NET cache and makes it dependent on the raw information published on the Amazon Web site. Whenever new information is posted, the cached DataSet is invalidated and refreshed.

You can't just pass an instance of the AmazonBooksCacheDependency class to the Cache.Insert method. That's why I have written a helper method: CacheHelper.Insert. As you can see in Figure 4, the overall programming interface is nearly identical to that of ASP.NET 1.x for standard dependencies and to that of ASP.NET 2.0 for custom dependencies.

Figure 4 Creating Custom Cache Dependencies (Close Figure 4)
using System;
using System.Web;

namespace MsdnMag
{
    // API to create custom cache dependencies
    public class CacheHelper
    {
        internal static string GetHelperKeyName(string key)
        {
            return key + ":MsdnMag:CacheDependency";
        }

        public static void Insert(string key, object keyValue,
                                  MsdnMag.CacheDependency dep)
        {
            // Create the array of cache keys for the CacheDependency ctor
            string storageKey = GetHelperKeyName(key);

            // Create a helper cache key
            HttpContext.Current.Cache.Insert(storageKey, dep);

            // Create a standard CacheDependency object and
            // link it to the previously created
            string[] rgDepKeys = new string[1];
            rgDepKeys[0] = storageKey;
            System.Web.Caching.CacheDependency keyDep;
            keyDep = new System.Web.Caching.CacheDependency(null, rgDepKeys);

            // Create a new entry in the cache and make it dependent on a
            // helper key
            HttpContext.Current.Cache.Insert(key, keyValue, keyDep);
        }
    }
}

Figure 4 shows the source code of the helper API to cache items with a custom dependency. CacheHelper is a class with a couple of static methods, the most important of which is Insert:

static void Insert(string key, object value, MsdnMag.CacheDependency dep)
This method wraps the overload of the Cache's Insert method that takes a CacheDependency object. The Insert method accepts a custom dependency object and does some extra work. Each new cache entry is paired with a second key whose name is programmatically built by concatenating the unique name of the entry with a standard (but arbitrary) string:
internal static string GetHelperKeyName(string key)
{
   return key + ":MsdnMag:CacheDependency";
}

The GetHelperKeyName is marked as internal (called friend, in Visual Basic® .NET) because I want it to be accessible from within other classes defined in the same assembly, but I don't want it to be callable from just any client.

The Insert method in the CacheHelper class does two key things. First, it creates a helper key and stores the custom cache dependency object in it. The helper key is needed to convey notification changes to the principal item to which it is linked. Once placed in the cache, the custom dependency object is up and running all the time and can periodically query the data source looking for changes. When a change is detected, the custom dependency object updates its last modified date property in the cache. In doing so, it breaks the standard dependency between the helper item and the base item. As a result, the base item is evicted from the cache based on a change on an external data source.

Figure 5 contains the source code of the MsdnMag.CacheDependency class. The class is built around a timer and an abstract method—HasChanged. The timer is instantiated in the constructor and executes a given callback method at specified intervals whose length is controlled by the Polling protected member. Polling indicates the extent of the interval in seconds, whereas the .NET Framework timers count in terms of milliseconds.

Figure 5 ASP.NET 1.x CacheDependency Class (Close Figure 5)
using System;
using System.Web;
using System.Threading;

namespace MsdnMag
{
    // Base ASP.NET 1.1 class for custom cache dependencies
    public abstract class CacheDependency
    {
        // **************************************************************
        // The internal timer used to poll the data source
        protected Timer InternalTimer;

        // Seconds to wait between two successive polls
        protected int Polling;

        // Name of the dependent cache key
        protected string DependentStorageKey;

        // **************************************************************
        // Last update
        public DateTime UtcLastModified;

        // **************************************************************
        // Class constructor
        public CacheDependency(string cacheKey)
        {
            // Store the name of the cache key to evict in case of changes
            DependentStorageKey = cacheKey;

            // Poll every 30 seconds by default
            Polling = 30;

            // Set the current time
            UtcLastModified = DateTime.Now;

            // Set up the timer
            if (InternalTimer == null)
            {
                int ms = Polling*1000;
                TimerCallback func = new
                    TimerCallback(InternalTimerCallback);
                InternalTimer = new Timer(func, this, ms, ms);
            }
        }

        // **************************************************************
        // Built-in timer callback that fires an event to the caller
        private void InternalTimerCallback(object sender)
        {
            CacheDependency dep = (CacheDependency) sender;
            if (HasChanged())
                NotifyDependencyChanged(dep);
        }

        // **************************************************************
        // Must-override member that determines if the monitored source
        // has changed
        protected abstract bool HasChanged();

        // **************************************************************
        // Modify the helper key thus breaking the dependency in the Cache
        protected virtual void NotifyDependencyChanged(CacheDependency dep)
        {
            // Get the name of the helper key
            string key =
                CacheHelper.GetHelperKeyName(DependentStorageKey);

            // Modify the date
            dep.UtcLastModified = DateTime.Now;

            // Overwrite the helper key to trigger eviction on the linked
            // item
            HttpRuntime.Cache.Insert(key, dep);
        }

    }

The constructor of the CacheDependency custom class requires the name of the dependent key. This information is essential for establishing a link between the dependency object and the dependent item. With default ASP.NET dependencies (in both 1.1 and 2.0) this link is implicitly created by the Cache.Insert method when a cache entry is created or updated. The name of the linked item is stored in the DependentStorageKey protected property. Finally, the UtcLastModified date member contains the time of the last update to the dependency. When a change in the data source is detected, an update to this property triggers the eviction of the linked item from the cache.

Setting Up the Timer

As mentioned, there are two basic ways for a dependency object to know about changes in a monitored resource—polling and notification. Basically, either the dependency object sets up a timer and executes a method at specified intervals or registers to receive notifications about changes from an external module. The availability of a notification service is specific to the data source you're working with. SQL Server 2005 and Windows® have one for query and file changes, respectively. If you're going to create a dependency based on a message queue (for example, MSMQ), then a notification model might be appropriate to implement. For the sample dependency object covered in this column I'll use the polling model and implement it through a timer:

if (InternalTimer == null)
{
   TimerCallback func = new TimerCallback(InternalTimerCallback);
   int ms = Polling*1000;
   InternalTimer = new Timer(func, this, ms, ms);
}

The timer is an instance of the System.Threading.Timer class. The core of the timer is the delegate for the method that will be called periodically. In addition, the timer's constructor lets you specify a due time and a period. The due time is the time to wait before the first execution of the method; the period is the time to wait between subsequent executions of the callback method. It is important to note that you must keep a reference to the timer alive for as long as you use the object. Like any other managed object, the timer is subject to garbage collection if it goes out of scope. The timer callback is a TimerCallback delegate:

public delegate void TimerCallback(object state);

The parameter indicates an object containing specific information relevant to the method. The callback doesn't execute in the thread that created the timer; instead it executes in a separate thread that is provided by the system. The following code snippet illustrates the default implementation of the timer callback for a custom dependency object:

private void InternalTimerCallback(object sender)
{
   // CacheDependency is the custom dep object
   CacheDependency dep = (CacheDependency) sender;
   if (dep.HasChanged())
      NotifyDependencyChanged(dep);
}

The callback calls the HasChanged method on the dependency class and based on the return value invokes the NotifyDependencyChanged method to break the dependency with the cache item. The NotifyDependencyChanged method I wrote belongs to the ASP.NET 1.x custom dependency class, but I named it after the equivalent method in ASP.NET 2.0 just to help you get familiar with ASP.NET 2.0 features more quickly. The method retrieves the name of the helper key and modifies its content. The content is just the dependency object modified to reflect the date of the last update:

dep.UtcLastModified = DateTime.Now;
HttpRuntime.Cache.Insert(key, dep);

Since this is running on a background thread, in this case you must use the HttpRuntime object to get a valid reference to the ASP.NET Cache. If you try to reach the Cache through the current HttpContext object—what you normally would do from within a codebehind class—you'll run into a null exception:

HttpContext.Current.Cache.Insert(key, dep);
This line of code is equivalent to calling Cache through HttpRuntime but only within the context of a request, which is not necessarily the case in this instance.

The NotifyDependencyChanged method is invoked by the timer callback, and the timer works independently of page requests. Basically, the dependency class, once placed in the ASP.NET Cache, remains active and running as long as the application lives, as does the timer it incorporates. The timer lives in the context of an ASP.NET application, but not in the context of a particular HTTP request. For this reason, HttpContext.Current just returns null. However, this doesn't mean that the Cache object is unavailable or unreachable. You simply have to take the right route to it; you have to use the HttpRuntime.Cache property.

Creating a Web Service Dependency

As you can see, there's a difference between the ASP.NET 2.0 customized dependency objects and my sample class. In ASP.NET 2.0, the base class doesn't incorporate any timer; each derived class can create its own timer if needed. In this ASP.NET 1.x implementation, I tried to reduce the amount of code that characterizes a particular dependency. I've accomplished this by deriving a new class from MsdnMag.CacheDependency and overriding the abstract HasChanged method:

protected abstract bool HasChanged();

The method is expected to check the monitored data source and return true if changes have occurred. Let's put it all together and write a custom dependency that gets broken if the output of a Web service method changes. (Rob Howard presented a similar ASP.NET 2.0 example at the Microsoft PDC 2003. You can find slides and source code on the ASP.NET site at http://www.asp.net/whidbey/downloads/WSV330_rhoward_demos.zip.)

I'll build a dependency class to monitor books available on Amazon. In general, imagine you have an application that needs to work with relatively static data—data that changes, but not as frequently as to justify reading it back at every postback. So you download it the first time and place it in the ASP.NET Cache. How do you deal with changes? Well, if data is known to change often by your standards, you should use a time dependency that guarantees that your data is updated at regular intervals. If the frequency of changes is low, you can opt for a custom dependency—whenever the data changes on the server, your cached snapshot is automatically invalidated. When the cache entry is accessed next, it returns null and you know that it's time for you to get the records from the server. As you'll see in a moment, my sample code goes beyond this and automatically replaces the data in the cache. Figure 6 shows the full source code of the AmazonBooksCacheDependency class.

Figure 6 AmazonBooksCacheDependency Class for ASP.NET 1.x (Close Figure 6)
using System;
using System.Data;
using System.Net;
using System.IO;
using System.Web;
using System.Configuration;

namespace MsdnMag
{
    // Create a cache dependency on the results of a query to the Amazon
    // Web service for my books
    public class AmazonBooksCacheDependency : CacheDependency
    {
        protected string AuthorName;

        public AmazonBooksCacheDependency(string key, string author) :
                                          base(key)
        {
            AuthorName = author;
        }

        public AmazonBooksCacheDependency(string key, string author, int
                                          pollTime) : base(key)
        {
            AuthorName = author;
            Polling = pollTime;
        }

        protected override bool HasChanged()
        {
            // Make access to the Web service
            string response = GetBooksInfo();

            // GetBooksInfo reads all the data and compares that to the
            // cached snapshot.
            // If you have access to the Web service, you might want to
            // use a separate, ad-hoc method to check for changes. For
            // example, a method that simply returns a Boolean value.

            // Compare data
            bool hasChanged = (response != (string)
                HttpRuntime.Cache[DependentStorageKey]);
            return hasChanged;
        }

        private string GetBooksInfo()
        {
            // Get the data from the Amazon Web service
            string url = "http://xml.amazon.com/onca/xml2?t=webservices-20";
            url += "&f=xml";
            url += "&mode=books";
            url += "&type=heavy";
            url += "&dev-t=" + ConfigurationSettings.AppSettings["DevToken"];
            url += "&KeywordSearch=" + AuthorName;

            try
            {
                WebRequest req = WebRequest.Create(url);
                WebResponse result = req.GetResponse();
                Stream receiveStream = result.GetResponseStream();
                StreamReader rs = new StreamReader(receiveStream);
                string response = rs.ReadToEnd();
                rs.Close();
                result.Close():
                return response;
            }
            catch
            {
                return null;
            }
        }
    }
}

The class inherits CacheDependency and defines two constructors—one takes the author's name, and one takes both a name and a poll time. The author's name is a parameter specific to the task of this dependency. The HasChanged method downloads book information for a given author and compares that to the snapshot currently stored in the cache. If the two don't match, the method returns false and the current cached item is freed:

private void Refresh()
{
   // Is the author cached already?
   string author = AuthorName.Text;
   if (!IsAuthorCached(author))
      PrepareCacheForAuthor(author);

   BindData(author);
}

When the user clicks the Refresh button, the code verifies that the books of the specified author have been cached. If not, a new entry in the cache is prepared:

private void PrepareCacheForAuthor(string author)
{
   string key = GetAuthorKey(author);
   AmazonBooksCacheDependency dep;
   dep = new AmazonBooksCacheDependency(key, author);

   // Create the cache entry
   CacheHelper.Insert(key, GetBooksInfo(author), dep);
}

GetBooksInfo uses the Amazon Web service to grab up-to-date book information. The data is an XML string that can be easily loaded into a DataSet. It is stored as a string in an ASP.NET Cache item named after the author. As mentioned, CacheHelper.Insert adds two keys to the cache—one with the actual data and one with the dependency object. The dependency object checks the Amazon Web service periodically and invalidates the cached item when changes have been detected. BindData uses a special method to read from the cache, as shown here:

string ReadFromCacheForAuthor(string author)
{
   string key = GetAuthorKey(author);
   string data = (string) Cache[key];
   if (data == null || data == string.Empty) {
      data = GetBooksInfo(author);
      Cache[key] = data;
   }
   return data;
}

If the contents of the cached item is null or empty, the method refills it with a new call to the Web service. Note that in order to develop with the Amazon Web service you must get a developer token, which you can apply for at http://xml.amazon.com. You'll also find licensing and reuse information there.

It is worth noting that the solution outlined so far works, but could be improved upon. The key problem is with the implementation of the HasChanged method. In the Amazon example, book information is retrieved at every timer interval and compared to the contents currently stored in the cache. If a change is detected, the contents of the cache are freed and a second download occurs with the next read from the cache.

As I've learned from sample implementations in ASP.NET 2.0, you should design the Web service to expose information about data changes through a simple Boolean method. A round-trip is still needed, but this trick will significantly reduce the amount of data being moved over the wire. ASP.NET 2.0 applies this pattern to database dependencies. Triggers set on monitored tables write a flag on a small helper table whenever a change is recorded. The helper table has just a few records—one for each monitored table in a database. The timer callback queries the small table looking for all records with the flag set. The result is guaranteed and the performance is optimal.

If you don't have control over the Web service (which I don't have with Amazon), then I suggest a slightly different approach. Modify the HasChanged method so that it replaces data in the cache when changes are detected. In addition, have it always return false to prevent the custom dependency object from breaking the link and invalidating the data. This guarantees that no double read is needed when changes occur.

Database Dependencies in ASP.NET 1.x

Database dependencies are not supported in ASP.NET 1.1, but Jeff Prosise demonstrated how to achieve them in the April 2003 issue of MSDN® Magazine. The idea is that you define a trigger on a SQL Server table and configure it to monitor insertions, deletions, and updates. When fired, the trigger calls an extended stored procedure to modify a disk file. If you set a dependency between a cached item and this file, the ASP.NET Cache will dispose of it whenever the SQL Server table undergoes some changes. This solution can be reworked and optimized a bit in light of the ASP.NET 2.0 code. For example, you can create a helper table to record changes and a second cache entry to invalidate the cached data whenever changes to your data occur.

Conclusion

Once again, I have presented some code you can use now which is a much simplified version of code that will be completely built when ASP.NET 2.0 is released. You might as well get the functionality you want today, and you'll be ahead of the curve in understanding ASP.NET 2.0 tomorrow.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Espositois a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and the newest Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Reach Dino at cutting@microsoft.com or join the blog at http://weblogs.asp.net/despos.


The XML Files: Messages vs. Methods

The XML Files
Messages vs. Methods
Aaron Skonnard

Q What happens behind the scenes when you call a method on a Web service proxy object?

A Given that most Web services toolkits make heavy use of object-oriented abstractions, it's easy to ignore what's going on behind the scenes when interacting with a service through a generated proxy class. The Microsoft® .NET Framework makes it possible to generate proxy classes from Web Services Description Language (WSDL) definitions by running wsdl.exe on the command line or by selecting "Add Web Reference" from within Visual Studio® .NET. Then you can take the generated proxy class and simply make method calls in order to invoke operations on the Web service.

For example, the following code illustrates how to interact with a Web service that exposes the canonical Add operation:

MathService s = new MathService();
double sum = s.Add(3, 4);
This type of framework really simplifies the act of working with Web services for those already familiar with object-oriented programming. The generated MathService class represents the Web service endpoint of interest, but it's different from a traditional .NET class because the implementation of the Add method (or any method for that matter) isn't found on the object itself, but rather across the Internet on a different machine. At first this may feel a lot like using distributed objects in COM+, but it's actually quite different because you don't have to use objects on either side.

Web services are built on the foundations of XML messaging. You send a SOAP message to a Web service to invoke one of its operations. And if it's a request/response operation, the Web service will then send another SOAP message back to the original sender. The ability to work with a Web service through an object is simply a facade that hides the underlying XML and protocol details.

When you instantiate the MathService class I just discussed, you get a local object that serves as a gateway (or proxy) for future Web service invocations. The MathService object does not get instantiated on another machine like it would in COM+. In fact, nothing happens on the wire at this point.

The magic begins when you call one of the methods on the local object, like Add. Since the Web service resides on another machine, the implementation of Add must generate the appropriate SOAP message to send across the wire. It does this by using .NET reflection to inspect the method signature and any applied attributes that govern the underlying XML details.

Once it has generated the SOAP message, the proxy opens an HTTP connection and transmits the SOAP message. Upon receiving the message, the Web service endpoint will process the "Add" functionality using the supplied data. And in this case, the service will return a response SOAP message containing the result. The proxy waits for the HTTP response and then processes the returned SOAP message, mapping it back to the appropriate .NET type for the return value.

For the most part, the XML and HTTP details are hidden behind the proxy's object facade. It can't hide everything, however, because there are some things that you may need to control, such as location, authentication, or proxy server details. The generated proxy class, MathService, derives from a base class called SoapHttpClientProtocol, which exposes several properties for controlling some of these underlying details. For example, you can modify the URL used to establish the HTTP connection through the proxy's Url property, as shown here:

MathService s = new MathService();
s.Url = "http://skonnard.com/samples/sample.asmx";
double sum = s.Add(3, 4);
Or you might need to set the HTTP basic authentication credentials through the proxy's Credentials property:
MathService s = new MathService();
s.Credentials = new NetworkCredential("bob", "iL0veCheEse!");
double sum = s.Add(3, 4);

As you can see, the proxy object is responsible for performing the translation between the different worlds of objects and XML, and for transmitting the XML messages to the actual Web service endpoint. It does a good job of hiding most of these details but sometimes you have to peer through the veil and tweak things to achieve the exact functionality you need.

Q Is it possible to pass parameters by reference to a WebMethod?

A Passing parameters by reference is a programming concept found in most of today's object-oriented languages. For example, in C# you annotate a parameter with the ref modifier to indicate that it's a reference parameter. This means the parameter will not contain a copy of the supplied value (with a new storage location), but rather a reference to the same storage location used by the variable in the calling code. So if you change the value of the reference parameter within the method, the change will be reflected in the original variable after returning from the method.

Mapping this concept to Web services presents a challenge because you're not dealing with memory storage locations anymore. Web service invocations typically span process, machine, or organizational boundaries. When you invoke a Web service operation, you're ultimately generating a SOAP message that's transmitted across the wire to another application. The service then returns a response through another SOAP message.

Obviously this means that the Web service cannot have direct access to the caller's memory locations. However, it is possible to simulate the passing of parameters by reference in a distributed environment (for example, Web services, remoting, COM+, and so on) by returning "reference parameter" values back in the response message. The calling code can then update the original variables with any changes.

The WebMethod framework supports this approach by allowing you to use the ref modifier in your WebMethod signatures. Doing so only has an effect on the response message, which will then include the reference parameters. For example, consider a WebMethod that does not contain any reference parameters:

[WebMethod]
public double Add(double x, double y)
{
   return x + y;
}

If you invoke this operation, supplying the value of 3 for x and y, you'll get the following SOAP response message:

<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <AddResponse xmlns="http://example.org/sample">
      <AddResult>6</AddResult>
    </AddResponse>
  </soap:Body>
</soap:Envelope>

Let's modify the signature to make x a reference parameter and change the implementation to increment x's value before returning:

[WebMethod]
public double Add(ref double x, double y)
{
    return x++ + y;
}

Now when you invoke this operation using the same values as before, you'll get the following SOAP response message:

<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <AddResponse xmlns="http://example.org/sample">
      <AddResult>6</AddResult>
      <x>4</x>
    </AddResponse>
  </soap:Body>
</soap:Envelope>

Notice that x is returned in the response message along with the result. The WSDL file generated by the WebMethod framework contains the XML Schema definition (shown in Figure 1) to describe the elements found in the request and response messages.

Figure 1 XML Schema Definition (Close Figure 1)
<s:schema elementFormDefault="qualified"
    targetNamespace="http://ex_mple.org/sample">
    <s:element name="Add">
    <s:complexType>
        <s:sequence>
           <s:element name="x" type="s:double" />
           <s:element name="y" type="s:double" />
        </s:sequence>
    </s:complexType>
    </s:element>
    <s:element name="AddResponse">
    <s:complexType>
        <s:sequence>
           <s:element name="AddResult" type="s:double" />
           <s:element name="x" type="s:double" />
        </s:sequence>
    </s:complexType>
    </s:element>
</s:schema>

When you generate a proxy class from this WSDL definition, the .NET Framework is smart enough to figure out that since x is found in both the request and response messages, it can be represented in C# as a reference parameter in the generated proxy class. So the generated proxy class in this case will contain a method that looks like the following (attributes omitted):

public double Add(ref double x, double y) {
   object[] results = this.Invoke("Add", new object[] {
               x,
               y});
   x = ((double)(results[1]));
   return ((double)(results[0]));
}

To invoke this operation through the proxy class, you must now specify that it's a reference parameter when making the call. This ensures that you provide a variable that can contain the updated value after the call returns, as illustrated in the following example:

SampleService s = new SampleService();
double x = 3;
Console.WriteLine("sum={0}", s.Add(ref x, 4));
Console.WriteLine("x={0}", x);
Running this code outputs "sum=7" and "x=4", showing that the updated x value made its way back to the original variable.

Q I'm having a problem passing a linked list to a WebMethod by reference. I have the following class:

public class Widget
{
    public string Name;
    public Widget NextWidget;
}

I've defined a WebMethod that receives a Widget by reference. The implementation traverses the list and changes the name of each Widget to "MyWidget", as shown here:

[WebMethod]
public void Test (ref Widget widget)
{
    Widget next = widget;
    while (next != null) {
       next.Name = "MyWidget";
       next = next.NextWidget;
    }
}

On the client side, I run the code shown in Figure 2. When the call returns, only the first Widget's Name field has been set. What am I missing?

Figure 2 Widget WebMethod (Close Figure 2)
Widget w1 = new Widget();
Widget w2 = new Widget();
Widget w3 = new Widget();

w1.NextWidget = w2;
w2.NextWidget = w3;
w3.NextWidget = null;

WidgetService ws = new WidgetService();
ws.Test (ref w1);

Console.WriteLine(w1.Name);
Console.WriteLine(w2.Name);
Console.WriteLine(w3.Name);

A This problem, like many others, stems from the fact that you're thinking that Web services can be treated like traditional object systems in which code like this might work. In this example, you're passing w1 by reference to the Test operation. Widget w1 references w2, which references w3. However, when you invoke the Test operation, the object graph is serialized into the SOAP request message shown in Figure 3.

Figure 3 SOAP Request Message (Close Figure 3)
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <Test xmlns="http://example.org/sample">
      <widget>
        <Name></Name>
        <NextWidget>
          <Name></Name>
          <NextWidget>
            <Name></Name>
            <NextWidget xsi:nil="true" />
          </NextWidget>
        </NextWidget>
      </widget>
    </Test>
  </soap:Body>

The WebMethod will create a similar object graph when it deserializes this message, although it will be referenced by the parameter named widget instead of w1. The code within the WebMethod will then traverse the list of widgets and update the name of each one. The SOAP message returned from the WebMethod will now look like Figure 4.

Figure 4 SOAP Response Message (Close Figure 4)
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLSchema"
xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <TestResponse xmlns="http://example.org/sample">
      <widget>
        <Name>MyWidget</Name>
        <NextWidget>
          <Name>MyWidget</Name>
          <NextWidget>
            <Name>MyWidget</Name>
            <NextWidget xsi:nil="true" />
          </NextWidget>
        </NextWidget>
      </widget>
    </TestResponse>
  </soap:Body>

When the client-side proxy class deserializes this message, it instantiates three new Widget objects and sets up the list relationship. It then assigns the first Widget object to the w1 variable (originally passed by reference), but nothing changes with w2 or w3 since they weren't passed by reference. The thing to realize here is that w1 references a new Widget object now—this is not the same one that was passed in originally. However, after the call returns, the w2 and w3 variables still reference the original (empty) Widget objects. There are now two new Widgets hanging off of w1.

To get the results you want, you need to modify the client code to access the new objects attached to w1, as shown here:

ws.Test(ref w1);
Console.WriteLine (w1.Name);
Console.WriteLine (w1.NextWidget.Name);
Console.WriteLine (w1.NextWidget.NextWidget.Name);
This will give you access to the new objects deserialized from the response message with the modified Name values.

Q When I call the following WebMethod supplying the same Widget object for w1 and w2, it always returns false. Why?

[WebMethod]
public bool TestWidgets (ref Widget w1, ref Widget w2)
{
    return (w1 == w2); // always returns false!
}

A WebMethods use the document/literal style for SOAP messages by default. Literal means that a literal schema definition specifies exactly what goes in the SOAP body. The problem with using document/literal in this particular example is that XML Schema does not provide a built-in mechanism for maintaining object identity. Consider the following client code:

WidgetService ws = new WidgetService();
Widget w = new Widget("MyWidget");
bool same = Ws.TestWidgets(ref w, ref w);

When I supply the same Widget object to the proxy class, it gets serialized into the document twice (see Figure 5). Then, during deserialization (at the WebMethod) w1 and w2 become two separate Widget objects because there's no way to tell that they were originally the same object. You could define a mechanism on top of XML Schema for managing object identity, but there's nothing currently built into the WebMethod framework.

Figure 5 MyWidget Serialized Twice (Close Figure 5)
<soap:Envelope
   xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <TestWidgets xmlns="http://example.org/sample">
      <w1>
        <Name>MyWidget</Name>
      </w1>
      <w2>
        <Name>MyWidget</Name>
      </w2>
   </TestWidgets2>
 </soap:Body>
</soap:Envelope>

The SOAP 1.1 specification defined a set of encoding rules for dealing with this type of situation (section 5 of the specification at http://www.w3c.org). These rules make it possible to identify an object uniquely through a combination of id and href attributes. Using this technique will enable the WebMethod to behave appropriately (although this approach has a problem that I will discuss shortly).

You can use rpc/encoded for a particular WebMethod by annotating it with the [SoapRpcMethod] attribute:

[WebMethod]
[SoapRpcMethod]
public bool TestWidgets (ref Widget w1, ref Widget w2)
{
    return (w1 == w2); // behaves correctly!
}

Now when you pass the same Widget object to the proxy class, it will be serialized using the SOAP encoding rules, which maintains object identity (see Figure 6). Notice that the w1 and w2 elements both refer to the same Widget element with an id of "#id1". This makes it possible for the WebMethod framework to maintain object identity after deserialization.

Figure 6 Maintaining Object Identity (Close Figure 6)
<soap:Envelope
   xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"
   xmlns:tns="http://example.org/sample"
   xmlns:types="http://example.org/sample/encodedTypes"
>
  <soap:Body soap:encodingStyle=
    "http://schemas.xmlsoap.org/soap/encoding/">
    <tns:TestWidgets>
      <w1 href="#id1" />
      <w2 href="#id1" />
    </tns:TestWidgets>
    <types:Widget id="id1" xsi:type="types:Widget">
      <Name>MyWidget</Name>
    </types:Widget>
  </soap:Body>
</soap:Envelope>

Although the SOAP encoding rules solve this particular problem, they have also been the cause of many interoperability headaches over the years. As a result, the WS-I officially banned the use of SOAP-encoded messages in the basic profile. In general, you should avoid using the SOAP encoding rules and opt for using document/literal in order to maximize interoperability.

Q Is it possible to pass an endpoint reference to another Web service in a SOAP message?

A Yes, this is different from passing objects by reference as discussed in some of the previous questions. Passing a reference to a Web service is about passing enough information in the message to allow the receiver to use a particular endpoint. The most basic piece of information required to invoke an endpoint is its location, which can be identified by a simple URI.

You can pass a URI like you would pass any other information in a SOAP message. The receiver can pick up the URI and, assuming it already knows how to interact with the service ahead of time, begin using the identified endpoint. A true reference, however, would provide additional metadata which is needed to figure out how to interact with the given endpoint on the receiving end.

The WS-Addressing specification proposes a standard technique for representing Web service endpoint references. WS-Addressing defines a new XML Schema type called wsa:EndpointReferenceType and it also provides an element named wsa:EndpointReference of type wsa:EndpointReferenceType. The following document illustrates the basic structure of wsa:EndpointReference:

<wsa:EndpointReference>
   <wsa:Address>xs:anyURI</wsa:Address>
   <wsa:ReferenceProperties> ... </wsa:ReferenceProperties> ?
   <wsa:PortType>xs:QName</wsa:PortType> ?
   <wsa:ServiceName
    PortName="xs:NCName"?>xs:QName</wsa:ServiceName> ?
   <wsp:Policy> ... </wsp:Policy>*
</wsa:EndpointReference>
As you can see, this element allows you to specify the endpoint's address along with other information such as the port type, the service name, endpoint policies, and other custom properties.

WS-Addressing also defines several standard message information headers that are of type wsa:EndpointReferenceType (such as wsa:ReplyTo, wsa:FaultTo, wsa:From, and so forth). These headers will make it possible to build interesting message exchange patterns and systems that dynamically figure out which services to use at run time.

The .NET 1.1 Web services framework does not currently support WS-Addressing, but the new Web Services Enhancements (WSE) 2.0 technology preview does. With WSE 2.0 you can easily pass around endpoint references through an easy-to-use API.

Check out "Moving from WS-Routing to WS-Addressing" for more information on the latest WS-Addressing specification.

Q The SOAP response message generated by my WebMethod always includes an element with the same name as the method. This element then contains the result element. Is it possible to remove this wrapper element?

A Yes, you can control most of the details of the underlying SOAP message by applying additional attributes to your WebMethod signature. The attribute you want to use in this case is [SoapDocumentMethod]. You need to set the ParameterStyle to Bare instead of Wrapped (the default), as illustrated here:

[WebMethod]
[SoapDocumentMethod(ParameterStyle=SoapParameterStyle.Bare)]
public double Add(double x, double y)
{
   return x + y;
}
With this in place, a SOAP request message for this method will now look like this:
<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <x xmlns="http://example.org/sample">3</x>
    <y xmlns="http://example.org/sample">4</y>
  </soap:Body>
</soap:Envelope>
The response SOAP message will look like this:
<soap:Envelope
  xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
    <AddResult xmlns="http://example.org/sample">7</AddResult>
  </soap:Body>
</soap:Envelope>

Q I've developed a class hierarchy having a base class named Employee. If I define a WebMethod that takes an Employee as input, is it possible to supply Employee-derived objects at run time?

A You can substitute derived types to a WebMethod as long as you've annotated the base class with the [XmlInclude] attribute. You use the attribute to specify a derived type that's allowed for substitution at run time. The XmlInclude class hierarchy in Figure 7 illustrates how this works.

Figure 7 XmlInclude (Close Figure 7)
[XmlInclude(typeof(Manager))]
[XmlInclude(typeof(Programmer))]
public class Employee
{
   ...
}

public class Manager : Employee
{
   ...
}

public class Programmer : Employee
{
   ...
}

This example indicates that it's possible to substitute Manager and Programmer instances wherever an Employee is expected. When this class hierarchy is mapped to XML Schema, the derivation remains intact through xsd:complexType derivation.

Now it's possible to supply Manager or Programmer instances to the WebMethod in Figure 8. Within the WebMethod you can check to see what type was actually supplied and process the object accordingly. However, the element found in the SOAP message must contain enough information for the WebMethod framework to determine the actual type. This is typically accomplished through an xsi:type annotation.

Figure 8 ProcessEmployee (Close Figure 8)
[WebMethod]
public void ProcessEmployee(Employee e)
{
   if (e is Manager)
   {
      Manager m = e as Manager;
      // process as Manager here
   }
   else if (e is Programmer)
   {
      Programmer p = e as Programmer;
      // process as Programmer here
   }
   // process as Employee here
}

In this case, a .NET-generated proxy class will still show the method as taking an Employee, but you can supply Employee-derived objects in the traditional object-oriented way:

EmployeeService s = new EmployeeService();
Programmer p = new Programmer();
s.ProcessEmployee(p);

When the Programmer object is serialized into the SOAP message, it's annotated with xsi:type="Programmer". This is how the WebMethod framework knows to deserialize a Programmer object on the receiving end.

Q How do I know if a particular field was specified in the XML document or given a default value during deserialization?

A One way to tell is to enable XML Schema validation and specify in the schema that the elements are required. Check out the MSDN® Magazine article, "Extend the ASP.NET WebMethod Framework by Adding XML Schema Validation" for details on this technique.

A simpler approach, however, is to use an undocumented feature of the serialization engine that allows you to figure out if a given field was specified in the XML message. Here's how it works:

[WebMethod]
public double Add(double x, double y,
  [XmlIgnore]bool xSpecified, [XmlIgnore]bool ySpecified)
{
   if (!xSpecified || !ySpecified)
      throw new Exception(
        "you must provide both x and y values");
   return x+y;
}

You add a new Boolean parameter for every existing parameter with "Specified" appended to the end of the name. Then, if the particular field is found during deserialization, the "Specified" field is set to true; otherwise, it's set to false. This technique can also be used to determine if fields of a class were specified or defaulted during deserialization.

Send your questions and comments for Aaron to xmlfiles@microsoft.com.

Aaron Skonnard teaches at Northface University in Salt Lake City. Aaron coauthored Essential XML Quick Reference (Addison-Wesley, 2001) and Essential XML (Addison-Wesley, 2000), and frequently speaks at conferences. Reach him at http://www.skonnard.com.


Test Run: Using Combinations to Improve Your Software Test Case Generation

Test Run
Using Combinations to Improve Your Software Test Case Generation
Dr. James McCaffrey

Code download available at: TestRun0407.exe (132KB)
Contents
Combinations
A Combination Class
Calculating the Number of Combination Elements
Generating All Combination Elements
Conclusion

Testing has always been a vital part of the software development process, but three recent factors have caused it to take an even more central role. First, the introduction of the Microsoft® .NET environment has dramatically improved developers' ability to write custom test automation. Test programs that took weeks to create before the .NET Framework can now be written in just hours. Second, increasingly complex systems are being built that require more sophisticated testing. Finally, software security is no longer an afterthought of the development process; it is an absolute essential. It once was possible to push a product out the door without complete testing, but this is no longer a viable option. To help you meet today's testing challenges I will show you best practices, principles, and techniques of software testing right here in this column every few months.

This month I'll begin with the role of combinations in software testing. The ability to programmatically generate combinations gives you a powerful way to generate test case input. To see what I mean about combinations, suppose you are writing a poker program. Manually generating all possible five-card test inputs would not be pleasant. But with the code in this column you can do it in minutes:

string[] deck = new string[] { "Ac", "Ad", "Ah", "As", "Kc", (...) };
Combination c = new Combination(52,5); // 52 cards, 5 at a time
string[] pokerHand = new string[5];

while (c != null)
{
  pokerHand = c.ApplyTo(deck);
  PrintHand(pokerHand);
  c = c.Successor();
}

The number of test automation scenarios in which combinations are useful is astonishing once you learn to recognize them. As another example, suppose you are testing some system that accepts user input into a textbox that holds 10 characters. One input might be "ABCDEFGHIJ" and another might be "!@#$%^&*()". You want to know how many different test cases there are. Let's assume you've determined that the character input falls into 20 equivalence classes—categories that are equivalent as far as your system is concerned. One equivalence class might be uppercase A through Z and another equivalence class might be the digits 0 through 9.

Notice that you must select 10 characters and each character must come from one of 20 categories. So you have 20 items taken 10 at a time, or Choose(20,10)—a function I'll discuss a little later in this column. Note that I have simplified this scenario. In practice, you would also need to consider permutations of each combination along with boundary conditions and many other test concepts.

Here I will build a Combination class in C# and show you how to use combinations to improve your testing effort. I think you'll find that understanding and using combinations and their associated algorithms are valuable assets.

Figure 1 Combinations Demo
Figure 1 Combinations Demo

The best way to show you where I'm headed is with a screen shot. Figure 1 is a screen of a Windows®-based application that demonstrates the use of combinations. As you can see, a combination of items is a subset of those items in which order does not matter. In this example I have five items—the names Adam, Barb, Carl, Dave, and Eric—and I am interested in combinations of size 3. There are 10 possible subsets of the 5 items taken 3 at a time:

{ Adam, Barb, Carl }, { Adam, Barb, Dave }, . . . { Carl, Dave, Eric }

Notice that since order doesn't matter I don't count a subset like { Carl, Barb, Adam } because I consider it the same as { Adam, Barb, Carl }. The example in Figure 1 also illustrates that in addition to generating combinations, I need the ability to compute how many combinations there are for a particular item set size and subset size.

A mathematical combination is a generalization of this idea of subsets. Instead of being a subset of arbitrary items, a mathematical combination of order n is a subset of the integers from 0 up to n-1. So the six elements of a mathematical combination of four items taken two at a time are:

{ 0, 1 }{ 1, 2 }
{ 0, 2 } { 1, 3 }
{ 0, 3 } { 2, 3 }

As I said, combinations are enormously useful in a surprising number of areas of software test automation, development, and management. While the mathematical concepts behind combinations are old and deep, I recently discovered that, in general, combinations are not well understood by software engineers, and that the majority of combination-related code examples available on the Internet are terribly inefficient, or in many cases, just plain wrong.

Combinations

The three essential operations on mathematical combinations are illustrated in Figure 2. The output tells you that with n = 4 and k = 2, there are six combinations:

{ 0, 1 } { 1, 2 }
{ 0, 2 } { 1, 3 }
{ 0, 3 } { 2, 3 }
From this example you can see that I need to be able to create a combination, calculate how many total combination elements there are for a given set of items and subset size, and determine the successor to a particular combination element so that I can list all combination elements.

Figure 2 Combinations (Close Figure 2)
static void Main()
{
  long n = 4;
  long k = 2;
  Console.WriteLine("With n = " + n + ", and k = " + k);
  Console.WriteLine("There are " + Combination.Choose(n,k) +
      "combinations\n");

  Combination c = new Combination(n,k);
  Console.WriteLine("The mathematical combinations are:");
  while (c != null)
  {
    Console.WriteLine(c.ToString());
    c = c.Successor();
  }
  Console.ReadLine();
}

Looking a little closer at the examples, you can see that combinations have two important characteristics: the total number of items in the set (usually denoted by n in mathematical literature) and the size of the subset (usually denoted by k). Mathematical combinations can be 0-based or 1-based. I will use 0-based notation throughout this column, and use n and k for the total number of items and the number of items in the subset, respectively.

In my examples thus far I have listed the elements of the combinations in what is called lexicographic order (sometimes called dictionary order). For mathematical combinations this means that the elements, if interpreted as integers, are listed in increasing order. For example, if n = 5 and k = 3, the first element is { 0, 1, 2 } and the next element is { 0, 1, 3 } because 12 comes before 13. Notice too that the atoms (individual integers) of a combination element also appear in increasing order so there is a kind of dual ordered-ness.

An important function for combinations is the total number of elements for particular n and k values. This function is most often called Choose. So for the first example with 5 names of people I can write Choose(5,3) = 10, meaning that for 5 items taken 3 at a time there are 10 total combination elements. There are other notations and names for the Choose function that you might run into, particularly in math articles but I'll use Choose.

It is very easy to confuse a combination of n and k with a Choose function of n and k. A mathematical combination of n = 7 and k = 4 (7 items taken 4 at a time) has elements like { 0, 3, 4, 6 }, whereas the associated Choose(7,4) function returns 35 and is the total number of elements of 7 items taken 4 at a time.

Combinations are frequently confused with permutations which are all the possible arrangements of a set of items where order does matter. Say you have the names Alex, Bill, Cris, and Doug. The first three permutations in lexicographic order are { Alex, Bill, Cris, Doug }, { Alex, Bill, Doug, Cris }, and { Alex, Cris, Bill, Doug }.

A Combination Class

A mathematical combination lends itself very nicely to implementation as a class. For data members you need to store the values of n (total number of items), k (number of items in each subset element), and an array to hold the "atoms" of each combination element. The basic code that represents a Combination object and a constructor to create the first lexicographic element of the Combination object along with code to represent it as a string is shown in Figure 3. I decided to use C#, but you can easily adapt it to the .NET-based language of your choice.

Figure 3 Combination Class Definition (Close Figure 3)
public class Combination
{
  private long n = 0;
  private long k = 0;
  private long[] data = null;

  public Combination(long n, long k)
  {
    if (n < 0 || k < 0) // normally n >= k
      throw new Exception("Negative parameter in constructor");
    this.n = n;
    this.k = k;
    this.data = new long[k];
    for (long i = 0; i < k; ++i)
      this.data[i] = i;
  }

  public override string ToString()
  {
    StringBuilder sb = new StringBuilder();
    sb.Append("{ ");
    for (long i = 0; i < this.k; ++i)
      sb.AppendFormat("{0} ", this.data[i]);
    sb.Append("}");
    return sb.ToString;
  }

After I put this code into a Class Library and compile, I can add a Project Reference to it and call it from a .NET Console Application, like I've done here:

Combination c = new Combination(5,3);
Console.WriteLine("\nCombination c(5,3) is initially " +  c.ToString());
The following output would display on the screen:
Combination c(5,3) is initially { 0 1 2 }
When the Combination constructor is invoked like so:
Combination c = new Combination(5,3);
I get an object in memory that represents the first mathematical combination element in lexicographic order of five items taken three at a time. The object in memory can be represented as shown in Figure 4.

Figure 4 Object in Memory
Figure 4 Object in Memory

The constructor code that creates the first combination element is quite simple. The two parameters that represent the total number of items and the size of the subset are stored in data members n and k, respectively. Because the values I'm dealing with can be very large, I decided to use the C# type long instead of type int. If I wanted to I could have doubled the range of values by using type ulong (unsigned long). I use the size of the subset, k, to allocate space for an array of long-named data, and then fill each cell of data with integers ranging from 0 through k-1.

Calculating the Number of Combination Elements

Now that I've determined how to create a Combination object, let's look at the second of three fundamental operations on combinations—calculating the total number of combination elements for a given total number of items n, and the subset size k. For example, if you are dealing with n = 5 items taken k = 3 at a time, there are 10 possible combination elements:

{ 0, 1, 2 } { 0, 3, 4 }
{ 0, 1, 3 } { 1, 2, 3 }
{ 0, 1, 4 } { 1, 2, 4 }
{ 0, 2, 3 } { 1, 3, 4 }
{ 0, 2, 4 } { 2, 3, 4 }

I want to implement a Choose(n,k) function that returns the number of elements; Choose(5,3) returns 10. Looking for existing Choose implementations, I was surprised to find that the most common algorithms on the Internet are very weak. Before I show you my implementation of Choose, let's briefly examine the standard implementation of a Choose function.

The standard way of coding the Choose(n,k) function uses its definition formula directly. This is an obvious but poor solution. Here's the typical Choose(n,k) function coded using C#:

// poor implementation of Choose(n,k)
static int Choose(int n, int k)
{
  int numerator = Factorial(n);
  int denominator = Factorial(k) * Factorial(n-k);
  return ( numerator / denominator );
}
The helper function Factorial is implemented as follows:
static int Factorial(int m)
{
  int ans = 1;
  for (int i = m; i >= 1; —i)
  {
    ans = ans * i;
  }
  return ans;
}

But there are several problems with this implementation of Choose(n,k). The most serious is that it will overflow for relatively small values of n and k. Notice that this Choose(n,k) first calculates Factorial(n), which can quickly grow into a huge number even for relatively small values of n (for example, 21! will overflow an unsigned 64-bit number). Then Choose(n,k) divides by the product of two factorials, which could also be a huge number, bringing the result back down to a relatively small number. The point is that even if Choose(n,k) returns a reasonably small value, the intermediate calculations can easily overflow.

A better implementation of Choose(n,k) calculates its answer in a different way. It turns out that Choose(n,k) can be calculated using the following alternative formula

Choose(n,k) = (n * (n-1) * (n-2) * ... * (n-k+1)) / ( 1 * 2 * ... * k)
which looks ugly, but is more understandable with an example:
Choose(7,3) = (7 * 6 * 5) / (1 * 2 * 3)

Instead of computing the numerator (a big number), then the denominator (a big number), and then dividing, you can calculate partial products and divide as you go. For the Choose(7,3) example, you first calculate 7 * 6 and divide by 2, getting 24 (skipping the first 1 term on the bottom of the fraction because dividing by 1 has no effect). Then multiplying that partial product by 5 and dividing by 3, you get an answer of 35, as before.

There is a second optimization for the Choose(n,k) method that is a consequence of the following property:

Choose(n,k) = Choose(n, n-k).
For example, Choose(10,8) = Choose(10,2). This is not an obvious relationship, but if you experiment with a few examples you'll see why this is true. Calculating Choose(10,8) directly involves computing seven partial products and seven divisions, but calculating the equivalent Choose(10,2) requires only one multiplication and one division operation.

Putting these ideas together, I implemented Choose(n,k) as shown in Figure 5. In the Choose function, I do a quick check for n equal to k and return 1 when this is true. My Choose algorithm works without this check, but it improves the performance of a method that generates a specified Combination object element. The remainder of the Choose implementation calculates the total number of elements using the algorithm I just explained.

Figure 5 Efficient Choose Method Implementation (Close Figure 5)
public static long Choose(long n, long k)
{
  if (n < 0 || k < 0)
    throw new Exception("Invalid negative parameter in Choose()");
  if (n < k) return 0;
  if (n == k) return 1;

  long delta, iMax;

  if (k < n-k) // ex: Choose(100,3)
  {
    delta = n-k;
    iMax = k;
  }
  else         // ex: Choose(100,97)
  {
    delta = k;
    iMax = n-k;
  }

  long ans = delta + 1;

  for (long i = 2; i <= iMax; ++i)
  {
    checked { ans = (ans * (delta + i)) / i; }
  }

  return ans;
}

Generating All Combination Elements

The third fundamental operation on combinations is generating a list of all combination elements for a given number of items, n, and subset size, k. Just like the problem of implementing the Choose function in the previous section, a search on the Internet turned up less than optimal solutions. Let's briefly look at a typical solution for generating all combination elements for given n and k values, and then I'll improve on it.

Suppose you have four items—the names Adam, Barb, Carl, Dave—and you want all combinations of these four items taken two at a time. A fragment of C# code that will generate the six elements of this combination is shown here:

  // naive technique to generate all combinations
  Console.WriteLine("\nAll elements of 4 names, 2 at a time: ");
  string[] names = {"Adam", "Barb", "Carl", "Dave"};
  for (int i = 0; i < names.Length; ++i)
  {
    for (int j = i+1; j < names.Length; ++j)
    {
      Console.WriteLine( "{ " + names[i] + ", " + names[j] + " }" );
    }
  }

If you execute this code, the (correct) result will be:

{ Adam, Barb }, { Adam, Carl }, { Adam, Dave }, { Barb, Carl },
    { Barb, Dave }, { Carl, Dave }.

But there are three problems. First, the technique works well if you want to generate all elements of a combination, but what if you only want some of the elements or a particular element? Second, this technique is very specific to a particular problem and doesn't generalize well. And third, it works nicely when the number of items in each subset element, k, is small, but what if k is very large? If you were interested in n = 100 items taken k = 50 at a time, you'd have to code 50 for loops or use recursion.

A better solution to the problem of generating all combination elements is to implement a Successor method that returns the next lexicographic element of a given element. If you combine Successor with an ApplyTo method that maps a mathematical combination onto a string array, you'll have an efficient, general way to produce all combination elements.

The code in Figure 6 shows the Successor method. Successor begins by checking to see if there is a next combination element. Suppose, for example, you are dealing with n = 5 items taken k = 3 at a time. There are 10 combination elements:

{ 0, 1, 2 } { 0, 3, 4 }
{ 0, 1, 3 } { 1, 2, 3 }
{ 0, 1, 4 } { 1, 2, 4 }
{ 0, 2, 3 } { 1, 3, 4 }
{ 0, 2, 4 } { 2, 3, 4 }

Figure 6 Lexicographic Successor of an Element (Close Figure 6)
public Combination Successor()
{
  if (this.data.Length == 0 ||
      this.data[0] == this.n - this.k)
    return null;

  Combination ans = new Combination(this.n, this.k);

  long i;
  for (i = 0; i < this.k; ++i)
    ans.data[i] = this.data[i];

  for (i = this.k - 1; i > 0 && ans.data[i] == this.n - this.k + i; --i) ;

  ++ans.data[i];

  for (long j = i; j < this.k - 1; ++j)
    ans.data[j+1] = ans.data[j] + 1;

  return ans;
}

Notice that you can identify the last lexicographic element, { 2, 3, 4 }, because it is the only element which has a first atom of 2—which equals the value n-k, or in other words, it has a value of n-k at position 0. This relationship is true in general for all combinations. Likewise, you can identify the first lexicographic element, { 0, 1, 2 }, because it is the only one with a last atom of 2 or, in other words, it has a value of k-1 at position k-1. Again, this is true in general. The Successor method returns null if there is no valid next element. Alternatives to returning null would be to throw an exception or return an error code.

The algorithm to generate the Successor element does not use any special tricks. Essentially you start at the right-most atom and move left until you locate the left-most atom which should be incremented. Then you increment the atom at index i and reset all atoms to the right of i to be 1 larger than the value to its left. For example, suppose n = 5 and k = 3 and you want the successor to the combination { 0, 3, 4 }. Index i starts at cell 2 (pointing to atom value 4), and moves left until it hits cell 0 (pointing at atom value 0). That atom value is incremented to 1, and all atoms to the right (the 3 and 4) are incremented from the value to the left in the array, yielding the result { 1, 2, 3 }.

Once you have your Successor method, you need an ApplyTo method that will apply a combination element to an array of strings. The ApplyTo method is simple:

public string[] ApplyTo(string[] strarr)
{
  if (strarr.Length != this.n)
    throw new Exception("Bad array size");

  string[] result = new string[this.k];

  for (long i = 0; i < result.Length; ++i)
    result[i] = strarr[this.data[i]];

  return result;
}

After checking to make sure that the array of strings input parameter has the correct number of strings, you create a result array with the size of the subset k. You then iterate through the input strings and store a reference to each at the appropriate cell of the result array. Like many operations that are performed with combinations, it's not entirely obvious what is happening until you trace through an example or two.

After instantiating a Combination object of the appropriate size and subset size, create an array of strings to hold the result combinations. Use a while loop to iterate through all combination elements—recall that the Successor method returns null when there aren't any more next elements—and the ApplyTo method maps the current element onto the original array of strings.

Conclusion

Combinations are an indispensable tool in planning and executing configuration testing, especially in a sub-area called interaction analysis. Suppose, for example, you need to test your product on a machine with multiple browsers and multiple media players installed. You want to test your system in conjunction with three browsers installed from a pool of eight browsers, and with two media players installed from a pool of six players. How many configuration combinations are there? How can you programmatically list these configurations? The techniques presented in this column make it easy for you to calculate that there are Choose(8,3) * Choose(6,2) = 840 possible test configurations. It also allows you to easily list all of them programmatically.

Combinations are useful when examining and testing paths of execution. I'll illustrate with a classic problem that is a surrogate for analyzing paths of execution (problems like this example are often used in interviewing test engineer candidates at Microsoft). Suppose you are developing a game. Players enter the southwest corner of a room with a tiled floor. Players must move to the northeast corner of the room by moving one tile to the east or one tile to the north (in other words players are always moving in the direction of the exit and do not backtrack). If the room is small—only 10 tiles by 6 tiles—how many different paths can the player take? Can you test all of them? If a move to the east is represented using the letter E and a move to the north is represented by the letter N, one possible path to the exit where the player moves all the way to the east wall and then straight north is:

E E E E E E E E E E N N N N N N
A different path is:
E N E N E N E N E N E N E E E E

Observe that no matter how the player moves, there will always be exactly 16 moves. Also notice that you can think of a move as "E" or "not E." If you imagine a sequence of 16 blanks, you must fill in 10 of the 16 blanks with "E" (because the remaining blanks must be "N"). So, the answer to this question is that there are Choose(16,10) = 8,008 possible paths and you can easily generate them using the code in this column.

As I said earlier, testing is a vitally important aspect of software development. Join me here next time for more tips you can put to use in your testing process.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences Inc. where he manages technical training for software engineers at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Security Briefs: Mind Those Passwords!

Security Briefs
Mind Those Passwords!
Keith Brown

Code download available at: SecurityBriefs0407.exe (349KB)
Contents
Internet Explorer and Passwords
Writing Down Passwords
A Better Solution: The Password Multiplexer
Password Minder
From Master Password to Master Key
Picking a Good Master Password
Defense in Depth
More on Stretching
Conclusion

Passwords are necessary evils. The way most password-based authentication works these days, the length and complexity of your password is directly related to how difficult it would be for an attacker to impersonate you. For example, if you choose "password" as your password, an attacker will usually guess this on his second try (his first try would probably be an empty string). Even if you were to use a six-character randomly generated password like 2=w\Z9, the entropy would be less than 40 bits, meaning it would take less than 240 guesses to discover the password in a brute-force attack. Depending on how the system is implemented, a brute-force attack may not even be necessary.

Many Web sites store passwords in cleartext (data in its original and unencrypted form), or using simple non-keyed "ciphers" such as a Caesar-shift, which is reversible without a key. Even sites that encrypt user passwords have to store the key somewhere, and Web server compromise can lead to compromise of this key, which can lead to the unveiling of all the cleartext passwords stored there. The recommended approach for storing passwords is to hash them and only store the hash values (preferably salted and stretched, techniques which I'll cover later in this column). This way, passwords cannot be stolen (the site doesn't store them), but the site can still verify a user's credentials by hashing a password and comparing it to the stored hash value.

Even if you choose a really long, randomly generated password and use it everywhere, you're still at significant risk. It only takes a single Web site with lousy security to disclose your password to an attacker, and chances are good that he can then use your login name (typically your e-mail address) and password at other sites that you frequent. Plus, why trust the administrator of one site with the password you use at another site? This is really scary stuff considering that many people use the Web to do banking, stock trading, and so on.

What you really need to do is choose a unique password for each and every Internet site you visit. Ideally, this password should be as long as allowed by the site, complex, and preferably randomly generated. Of course, there are very few humans in the world who can randomly generate a complex password without the help of a tool. But don't worry, there are a lot of tools out there to help you. Of course, you should only use tools from sources you really trust.

This column describes a tool I wrote, Password Minder, to solve this problem. Even if you don't use it yourself, you'll find its implementation to be instructive.

Internet Explorer and Passwords

Microsoft® Internet Explorer has built-in support for helping you remember passwords. So why is it that most companies (ironically, including Microsoft) encourage their employees to avoid this feature? Well, let's think about how the feature might be implemented. Given that the browser never asks you for any other secret before auto-filling a password field, it's very likely that it's using your login credentials (your Windows® password) to derive an encryption key for the passwords that it remembers for you. There are a couple of Win32® APIs that make it really easy to encrypt data using your login credentials as the source of an encryption key: CryptProtectData and CryptUnprotectData (this functionality is provided to managed applications via a new class in the .NET Framework Library called ProtectData in the .NET Framework 2.0).

But think about this for a moment. If the browser can decrypt these passwords at any time without requiring any further secret from you, then it's possible for any other code on your machine running in your logon session to do the same thing! Imagine a virus that used the same algorithm as Internet Explorer to decrypt your passwords, uploading them to an FTP site of the attacker's choosing. If you were to inadvertently run the virus code, perhaps by opening a malicious attachment in an e-mail, any passwords that Internet Explorer manages for you would be up for grabs. Remember that the virus can call CryptUnprotectData just as easily as your browser can. Frankly I'm amazed that some of the more successful viruses haven't already done this (Nimda, for example). Certainly the ubiquity of this feature makes it a very juicy target. The thought that accidentally opening a malicious e-mail attachment could unleash an attack like this should make you reconsider whether you'd want to have your browser managing passwords for you.

Another problem with browser-managed passwords is that you are the one who chooses each password in the first place, and humans typically pick really bad passwords. "asdf7890" is not a good password unless you qualify goodness as how easy it is to type. You should have a good random source of password material. One technique I've seen is to throw dice to generate each character in a password. That's a great method, but I don't know of many people who would be patient enough to use this technique on a daily basis. Besides, I can just see you trying to explain to your boss that you're not playing games when he walks in on one of your password-generation sessions.

Writing Down Passwords

A lot of people joke about writing down passwords, but sometimes it's a reasonable approach. As long as you don't do something silly like use a sticky note on your monitor to store your passwords, writing them down is arguably a better idea than letting the browser remember them. A virus that can decrypt your passwords and ship them wholesale to an attacker cannot pull that little piece of paper out of your wallet and read it.

But once again, you've probably got way too many passwords to write them all down and store them in your wallet. Besides, if you use really good passwords, it's very difficult to read them and type them without making any mistakes. And what happens if that little piece of paper goes through the wash? What happens if your wallet is stolen or misplaced?

I've seen some people keep passwords in a file on their computer. When done carefully, this isn't a half bad approach either, and it's easier because you can transfer passwords via the clipboard. You can use the encrypting file system to achieve about the same level of security that your browser would give you, effectively encrypting your password file with your login credentials. One of my friends used this approach until the day I told him that I'd just accidentally glanced at his list of passwords (he'd left it sitting open on his laptop and walked away).

A Better Solution: The Password Multiplexer

A couple of years ago, I decided to solve my own password management problem by writing a password multiplexer. My requirements were as follows: I only wanted to have to remember one really good master password (this is different from my Windows login password, so technically I have to remember two strong passwords). Once logged into Windows, whenever I would find myself prompted for a password, I wanted to simply type in my master password and somehow have the system automatically convert this into a unique password for the site I'm visiting. My goal was to have a long, unique, randomly generated password for each site I'd visit, but I didn't ever want to have to actually see those passwords or type them manually. I also wanted this to work in other apps in addition to the browser. For example, when I set up a VPN connection to a client's network, I wanted to use my password multiplexer.

So, what would be the difference between my password multiplexer and letting the browser remember my passwords? Well, assuming my collection of real passwords is encrypted using a key derived from my single master password, my multiplexer would require me to type my master password each time I wanted to retrieve one of my real passwords. Because I would not be using my Windows login credentials to encrypt my passwords, they wouldn't automatically be available to any malicious software that might slip into my system. It's not perfect, of course. A root compromise of my system would allow an attacker to install a keyboard driver that reads all of my keystrokes, for example, but it's a heck of a lot safer than having my browser remember my passwords.

Frankly I didn't know how to build a password multiplexer that satisfies all of these requirements, but the final design turned out to require only a few extra keystrokes beyond my original stated goal, and I have been using this tool for a couple of years now and have been really happy with it.

Before settling on my final design, I considered simply deriving passwords for each site directly from my master password. For example, for each site I'd simply store a salt value, which is just a bunch of randomly generated bits. I could then hash this salt plus my master password to obtain password material for the site. This would make the multiplexer implementation really easy. But it also would make it rather difficult to change my master password: I'd need to change my passwords at all the sites at the same time. And I have about 50 passwords that I manage at the moment.

Another problem with this solution is that many sites have restrictions on the passwords they allow. A lot of sites won't allow you to have punctuation characters in a password. Even more sites place arbitrary restrictions on the length of the password; I've seen some sites that won't allow longer than six characters. What's worse is that often these limitations aren't even documented. Apparently, nobody expects you to use a really long password. So, if you happen to be developing a Web site that requires a password, please make it very clear what types of passwords you'll accept. And if at all possible, allow very long passwords. What's the harm in allowing someone to give you a 100-character password? Few people will do it, but you should allow those of us who want really good security to be able to get it.

Finally, I have a few smartcards that I use from time to time, and they require PIN codes that are four characters long. With all of these problems, it quickly became clear that I wasn't going to be able to simply generate my passwords directly from a single master key. But it was a fun idea. From a cryptographic perspective, it was also not the best idea. If someone got access to a large sample of the derived passwords, they could try to guess the master password.

Password Minder

What I ultimately built was a tool called Password Minder. I probably should have named it Password Multiplexer, but I thought that it might be used by nontechnical folks who might shy away from such an odd name. Because there was no safe way for me to integrate my software with the operating system so that I could hook every possible password edit field automatically, I opted for a very simple approach: I just wired up Password Minder to a hotkey combination by creating a shortcut on my desktop and configuring the "Shortcut Key" setting via the shortcut's property sheet. Whenever I find myself on a password field in a Web form or dialog box, I just hit my hotkey combination (Control-Shift-P), and up pops a little prompt asking me for my master password (see the dialog box in Figure 1).

Figure 1 Master Password Prompt
Figure 1 Master Password Prompt

Once I enter my password, the system performs some calculations to obtain a master key. Later I'll go into detail about the motivation and mechanics for these calculations, but for now, suffice it to say that after a couple of seconds another form pops up showing a list of Web sites and applications for which Password Minder is managing my passwords. Most sites simply use my e-mail address as my user name, but other sites like my bank require something different, so my list reminds me at a glance what user name I use at these more finicky sites. Figure 2 shows what my list looks like (blanking out the user names I've chosen). Note the scrollbars. I wasn't kidding when I said that I've got tons of passwords that I have to manage. But each one is unique, random, and as strong as the site or application will allow.

Figure 2 Password List
Figure 2 Password List

I hate taking my hands off the keyboard, so at this point I'll just hit the first letter of the site name to scroll through the list and quickly get to the password I need, pressing Enter to select it. Password Minder then uses my master key to decrypt just that one password, and types it for me by inserting keystroke messages into my input queue. Password Minder then immediately exits to reduce the exposure of my master key. Because any given use of Password Minder typically lasts just a few seconds, my master key is only exposed in memory for very a brief period during the day.

In reality, because Password Minder is a managed application, it's very difficult to ensure that the master key is wiped from memory once I'm done with it. This is a big problem in any language (such as .NET and Java) that uses garbage collection (though in the .NET Framework 2.0, the new SecureString class would make it easier). This is why I designed Password Minder to immediately shut down after typing a password, at least by default. I can then rely on the operating system's guarantee that any other process that ends up using the same memory that Password Minder just vacated will have the memory zeroed before being mapped into the new process. This isn't perfect (the master key could end up sitting in the swap file some time), but it's good enough for me. If my computer is compromised so badly that the swap file is up for grabs, my keystrokes are probably also being monitored as well, which means the attacker already has my master password anyway.

Oddly enough, I benefit by using a password management tool that is not built into the operating system. Not everyone who owns Windows has this tool, which makes it a less likely target for attackers. Not that I rely on this, though. As you'll see shortly, Password Minder takes security very seriously, and even if an attacker were to obtain my password file, since I use a strong master password and a password-stretching technique, even an offline brute-force or dictionary attack against my password file should ultimately fail.

From Master Password to Master Key

Once Password Minder gets my master password from the dialog in Figure 1, it must solve a little puzzle to figure out the master key for my password file. This technique is called password stretching, and it can take a good password and strengthen it so even the most well-funded attacker cannot discover it via a brute-force attack. The technique Password Minder uses is documented in the Public Key Cryptography Standards (PKCS) for password-based cryptography. It's technically called the Password Based Key Derivation Function 2 (PBKDF2), and it works by running a cryptographic pseudo-random number generator over and over again, seeded with your master password and a salt value. At the heart of this pseudo-random number generator is a hash algorithm, and you can use any hash algorithm you like, so I've chosen the 256-bit version of the Secure Hash Algorithm (SHA-256) based on recommendations from well-respected cryptographers Niels Ferguson and Bruce Schneier in their fine book, Practical Cryptography (John Wiley & Sons, 2003).

I'm currently using 218 iterations of this algorithm, which effectively adds 18 bits of security to my password. Why? If an attacker somehow gets a copy of my password file, he can mount a brute-force attack against my master key by simply trying every possible password, calculating the master key, and decrypting one of the passwords in my list. If he eventually guesses my master password, he'll know he's found it because the decryption will result in a plaintext byte stream that represents a password in UTF-8 encoding, and only certain characters will be present in any of my passwords. He can then further confirm that he's found the correct password by using the master key to decrypt the other passwords in the same fashion, checking to see if they are also valid UTF-8 encodings of passwords.

But how many attempts will the attacker have to make before he happens upon the correct password? The master password I'm using this month is 12 characters long, does not contain any words found in a dictionary, has both uppercase and lowercase letters, numbers, and punctuation. This means each character comes from a possible set of 26+26+10+32=94 alternatives, if you count the punctuation characters you'll find on my English keyboard. So my password comes from a possible set of 9412 passwords, which is pretty close to 279, and a 79-bit key is pretty strong. But since I don't really use all of the possible letters, numbers, and punctuation characters in a uniform distribution, I really can't rely on having the true strength of 79 bits. That's where the puzzle comes in.

By forcing the attacker to perform 218 iterations of a pseudo-random number generator based on SHA-256, I'm significantly increasing the work he must perform in a brute-force attack. For each password guess, he has to perform 218 extra steps. So you can think of this as "stretching" the password strength by 18 bits. Instead of 279 steps to find my password, a brute-force attack takes up to 297 steps. Now, as I said before, I don't really have 279 bits of entropy in my master password. But you can see that this stretching helps strengthen any password that I choose. By combining a really strong password with a stretching technique like PBKDF2, you can get some serious protection against anyone who might obtain a copy of your password file. This is important if you want your password file to be mobile because you'll often carry it with you or post it on a network share to make it easy to access.

Picking a Good Master Password

Password Minder also offers a measure of entropy whenever you provide a password. For example, when I type a new master password, Password Minder displays the maximum possible entropy the password can have by checking the types of characters I am using and the length of the password. Password Minder won't accept anything less than 60 bits of entropy for a master password. Of course, nothing stops me from using a lousy master password like P@ssw0rd!!, which will register with a maximum entropy value of 66 bits since it uses all the categories of characters that the program checks for.

To help me pick a long master password that I can remember, I imagine a picture and tell myself a little story about it: "Dark leaves fall to the ground as the Autumn wind howls around my house," collapses to DlfttgatAwhamh. Then I throw in a little dance around the keyboard somewhere in there, say @#32 and I'll end up with a nice long password: Dlfttg@#32atAwhamh. This is surprisingly easy to do. I've even pasted a real picture in my office for the first few days while I was memorizing a new passphrase. They say a picture paints a thousand passwords, right? Well, something like that, anyway.

Defense in Depth

Given the high strength of my long master password coupled with password stretching, I must say that I still won't post my password file on my Web site for all to see. I think it's important to have a layered defense, so I guard my password file carefully. Password Minder stores password files in my user profile, where the default access controls on the directory only allow me or an administrator to view the files stored there (note, though, that if your enterprise has set up "roaming profiles," this data will roam between machines you log into and could conceivably be captured en-route). And if I choose to store my password file somewhere else, Password Minder will manually set the access control list the same way (granting only myself and administrators permission to read and write the file) each time it's used. Of course, for password files that I keep on a FAT partition such as a USB memory stick that I carry around with me, Password Minder skips this step. By limiting access to the file, encouraging strong master passwords, and stretching those passwords before using them, Password Minder provides you with defense in depth.

More on Stretching

Since stretching the password takes some time, and computer speeds differ wildly these days, Password Minder allows me to configure the number of iterations it performs during the stretch. I can change this value each time I reset my master password, which Password Minder reminds me to do every 90 days. If you try it out, you'll notice that on its first run, Password Minder will present you with a wizard to help you set up your password file. That wizard will allow you to test your computer speed to figure out what level of stretching you're comfortable with, as shown in Figure 3.

Figure 3 Testing Computer Speed
Figure 3 Testing Computer Speed

Each tick on the slider bar represents another bit of stretching. The current implementation allows a minimum of 216 iterations, the lowest setting on the slider bar. On my laptop, which is about a year old, this takes a second to calculate. The highest slider setting would be 225, and I've honestly never waited long enough for this to complete. But since Moore's law results in faster computers, someday that setting might make sense.

Conclusion

Next time I'll show you the guts of Password Minder, including the class that implements the PBKDF2 algorithm, and describe how the program encrypts user names and passwords using unique keys derived from the master key. If you can't wait to see the code, which is primarily in C#, you can download it, along with the binaries for Password Minder and instructions for using it from the link at the top of this article. The current version is built with .NET Framework 1.1, which you'll need if you want to run Password Minder.

If you don't have the .NET Framework, consider using Bruce Schneier's Password Safe program, which you can find via http://www.schneier.com/passsafe.html. What really matters is that you pick a password management tool that you trust and that you use it consistently. Don't settle for using the same password everywhere you go!

Send your questions and comments for Keith to briefs@microsoft.com.

Keith Brown is an independent consultant specializing in application security. He authored Programming Windows Security (Addison-Wesley, 2000) and has just finished his new book, A .NET Developer's Guide to Windows Security (both published by Addison-Wesley). Read the new book online at http://www.pluralsight.com/keith.


Advanced Basics: Data Binding Radio Buttons to a List

Advanced Basics
Data Binding Radio Buttons to a List
Duncan Mackenzie

Code download available at: AdvancedBasics0407.exe (139KB)
Contents
Why Not Radio Buttons in a Panel?
Don't Reinvent the Wheel
Drawing the Radio Buttons
Let the User Interact with the Control
Let's Handle Resizing and Scrolling
Where to Go from Here

I recently heard someone making skeptical statements about a Windows® Forms application and proclaiming surprise that "it even used data binding!" This made me reflect on the history of data binding over the past few generations of development tools. It brought to mind issues that existed across multiple versions of Visual Basic® and Microsoft® Access that left data binding with a bad reputation. Developers, especially the experienced ones, often see data binding as an inflexible and unreliable means of connecting a UI to data, as a technology best suited for demos but not for real-world use. In addition to these concerns, one of the biggest issues many people have with data binding is that it implies a tight coupling between the data and presentation layers of an application—a bad thing in many situations.

While some of these issues were valid in the past, data binding in Windows Forms has been designed to avoid or mitigate most of these concerns. The data-binding technology in the Microsoft .NET Framework is customizable, flexible, and (since you can bind to objects, not just to database systems) it does not require you to tie your interface directly to your back-end database. Thanks to these changes, I personally use data binding whenever possible, binding to a DataView or to a strongly typed collection in almost every single one of my projects.

With so many issues resolved I am able to concentrate on my wish list of features. The first I'd like to tackle is the absence of a data-bound control that would allow users to select from a list of radio buttons. In this column, I'm going to look at how you can create your own control to get around this omission from the Windows Forms classes.

Why Not Radio Buttons in a Panel?

Binding works by hooking to a single property value on a control, and there is no sort of "selected value" property on a group of radio buttons, so if you want to use this type of interface in your application, you have to handle it yourself. Having one set of controls that isn't data bound isn't a horrible situation, but it leads to special cases in your presentation layer code where you loop once for all of your bound controls, then add a little bit of special code for the one or more groups of radio buttons in your interface. In my experience, code in which you try to handle each case differently is difficult to maintain and the source of a majority of bugs.

Figure 1 ASP.NET RadioButtonList
Figure 1 ASP.NET RadioButtonList

A better solution would be to create a completely new type of control, a RadioButtonList, which would be the Windows Forms equivalent of the ASP.NET RadioButtonList control (see Figure 1), containing a SelectedValue property that you can bind to. This control would be bound in two ways, through a data source property for its list of options (the text and values for all of the radio buttons) and through the SelectedValue property for the value that has been selected. This type of data binding is used for all of the data-bound list controls in Windows Forms, including the ListBox and the ComboBox control, so this new control will be consistent with what is currently available.

Don't Reinvent the Wheel

Using Visual Basic 6.0, you had only one choice when developing your own control; you had to start with a blank User Control and add all of the functionality you needed. That is still an option in .NET, but it is generally not the best choice. You should try to base your new control on existing ones by inheriting from another control or class instead of from Control or UserControl. Of course, it isn't always possible, but in this case there is a class, ListControl, that provides the base implementation for both ComboBox and ListBox, and that will provide my new RadioButtonList control with all of the basic data-binding functionality I need:

   Public Class RadioButtonList
       Inherits ListControl
       •••
   End Class

Although ListControl provides great functionality (including managing the bound data source and providing a CurrencyManager instance to interact with), I still need to write the code to handle a few key issues. First I have to draw the control, which in this case means drawing a series of radio buttons. Also I must allow keyboard and mouse interaction with the control, including tracking which item in the list has focus (different from which item is selected). Finally, I have to handle resizing and need to provide a scrollbar if the control is too small for the number of items in the list.

Drawing the Radio Buttons

The first task I'll tackle is the graphics work because until that is done I won't get anything drawn on the screen at all, and where's the fun in that? In my new control class, let's override the OnPaint routine and add a loop to draw the list of items as I did in Figure 2.

Figure 2 Draw Items (Close Figure 2)
Protected Overrides Sub OnPaint(ByVal e As PaintEventArgs)
    Dim myList As IList
    Dim gr As Graphics = e.Graphics
    gr.Clear(Me.BackColor)
    ControlPaint.DrawBorder3D(gr, Me.DisplayRectangle, m_borderStyle)
    gr.TranslateTransform(0, -1 * Me.m_currentTopPosition)

    If Me.DataManager Is Nothing Then
        myList = Nothing
    Else
        myList = Me.DataManager.List
        Dim itemCount As Integer
        itemCount = myList.Count

        Dim i As Integer 'loop indexes

        For i = 0 To itemCount - 1
            DrawOneItem(i, gr)
        Next
    End If
End Sub

Walking through this code from top to bottom, you'll see that the first step is to clear the entire control to remove any pixels from having drawn a previous set of buttons or a different size control. After that, the border is drawn, using an internal variable m_borderStyle (I also added a property to the control to enable the user to set the border style, but I won't cover that in this column). Next, a transform is applied to the Graphics object (using the TranslateTransform method) to shift it according to the current position of the scrollbar (more on that in a bit). Finally, with everything all set up, the actual items (radio buttons) are drawn onto the control by looping through the data source (represented by the myList variable) and drawing each item out one at a time.

The DrawOneItem routine, as its name suggests, is called to draw each individual RadioButton onto the control:

Private Sub DrawOneItem(ByVal index As Integer, ByVal gr As Graphics)

To start, the code grabs the Font and ForeColor values from built-in properties of the control and uses them for all of its drawing, which ensures that the user has full control over the appearance of each item. Using the built-in properties has several advantages: you don't have to write any of your own code, the control doesn't get cluttered with additional properties, and you get any built-in behavior of the base properties. In the case of the Font and ForeColor properties, your control will automatically stay in sync with its container; changes to the corresponding properties on the form will be reflected by your control (assuming the user hasn't overridden that property on the control already):

Dim textFont As Font = Me.Font
Dim textBrush As New SolidBrush(Me.ForeColor)

After grabbing the Font and ForeColor values, a StringFormat variable is created and configured to control how the Radio Button's caption will be drawn. In this case, the StringFormat is configured to align the text Near (which corresponds to left alignment if you are using a US English machine), but that alignment setting won't really have the desired effect because when DrawString is called later only a starting point is given, not a complete text area. Try changing it to Far if you want to see what happens. The text will be drawn so that it ends at that given starting point, which means that most of it won't even be visible on the control. FormatFlags controls how the text is clipped if it consists of more than one line; LineLimit indicates that only completely visible lines should be drawn:

Dim myStringFormat As New StringFormat
myStringFormat.Alignment = StringAlignment.Near
myStringFormat.FormatFlags = StringFormatFlags.LineLimit

This routine uses a variety of private variables to configure the layout of the button and the text (m_HorizontalSpacing and others), which is done to provide a bit more control to the user (a developer using this control), allowing him to use the corresponding control properties to tweak the control's appearance as he likes. Another internal variable, the rowStarts array, contains the vertical position of each item in the list and, since it is only calculated once when the control is resized, it is used to save time instead of having to recalculate a top position when each item is drawn. If you want to see all of the code that exposes these values as properties or that does the starting position calculation, then check out the complete source code for this class in the download available at the link at the top of this article.

Dim leftPos As Integer = m_HorizontalSpacing
Dim topPos As Integer = Me.rowStarts(index)

The display text for the current item is the value of the DisplayMember property for the current object and accessing that value (which would involve property descriptors and a few lines of code) is simple thanks to the existence of a GetItemText function on the control's base class (ListControl).

'grab the display member text from the data source
Dim itemText As String = Me.GetItemText(Me.DataManager.List.Item(index))

Now that I have the item text and can determine how much space it is going to take up on the control, it is possible to draw a focus rectangle to indicate if the current item is selected. Using a method of the ControlPaint class, which is a powerful set of utility functions designed for use when drawing your own controls, a dotted-line rectangle is drawn around the text if the item currently has the focus, as shown here:

'draw focus rectangle (dotted line)
If Me.focusedItem = index Then
    Dim rowSize As SizeF
    rowSize = gr.MeasureString(itemText, textFont, _
        New PointF(leftPos, topPos), myStringFormat)
    Dim focusRect As New Rectangle(leftPos, topPos, _
        rowSize.Width + m_buttonSize, rowSize.Height)
    focusRect.Inflate(m_focusRectInflation, m_focusRectInflation)
    ControlPaint.DrawFocusRectangle(gr, focusRect)
End If

Then the RadioButton itself is drawn after setting up a ButtonState variable. ButtonState is an enumeration used with several methods of the ControlPaint class to indicate how a button (radio button, checkbox, and so on) should be drawn. In this case, the state of this RadioButton is affected by two things—whether or not the control is enabled and whether or not this particular radio button is selected:

'the ButtonState indicates how the Radio Button should be drawn
Dim bs As ButtonState

'should it be grayed out?
If Not Me.Enabled Then
    bs = bs Or ButtonState.Inactive
End If

'should it be selected?
If Me.SelectedIndex = index Then
    bs = bs Or ButtonState.Checked
End If

The combination of settings is then used to draw the actual control through the ControlPaint.DrawRadioButton method and then the Text is drawn to the left of the radio button. A nice addition to this control, which you might want to make if you were going to ship it out as part of a control library, would be to allow the user to configure the relative position of the text and of the radio button (enabling the button to be on the right-hand side of the text, for example):

ControlPaint.DrawRadioButton(gr, New Rectangle(leftPos, topPos, _
    m_buttonSize, m_buttonSize), bs)

gr.DrawString(itemText, textFont, textBrush, leftPos + m_buttonSize, _
    topPos, myStringFormat)

Let the User Interact with the Control

Your control isn't much use if the user can't click on it to select an item or navigate to it with the arrow and Tab keys on his keyboard. To handle both of these cases, you have to override the OnClick, OnKeyDown, and IsInputKey routines of your control. Focusing on the mouse interaction first, by capturing a mouse click via OnClick, you can determine which item in your list was clicked and make that item the currently selected one (see Figure 3).

Figure 3 Set Currently Selected Item (Close Figure 3)
Protected Overrides Sub OnClick(ByVal e As System.EventArgs)
    Dim mouseLoc As Point = Me.PointToClient(Me.MousePosition())
    mouseLoc.Y += Me.m_currentTopPosition
    Dim itemHit As Integer = HitTest(mouseLoc)
    If itemHit <> -1 Then
        Me.SelectedIndex = itemHit
        Me.Focus()
    End If
End Sub

Private Function HitTest(ByVal loc As Point) As Integer
    Dim i As Integer
    Dim found As Boolean = False
    i = 0
    Do While i < Me.DataManager.Count And Not found
        If GetItemRect(i).Contains(loc) Then
            found = True
        Else
            i += 1
        End If
    Loop
    If found Then
        Return i
    Else
        Return -1
    End If
End Function

Keyboard handling is a bit more complex. First, you have to override the IsInputKey routine to let the control know which key presses you are capable of handling, like so:

Protected Overrides Function IsInputKey( _
        ByVal keyData As System.Windows.Forms.Keys) As Boolean
    Select Case keyData
        Case Keys.Down, Keys.Left, Keys.Up, _
                Keys.Right, Keys.Enter, Keys.Return
            Return True
        Case Else
            Return MyBase.IsInputKey(keyData)
    End Select
End Function

Once you override IsInputKey, all of the indicated key presses will arrive through the OnKeyDown routine. Inside that routine, I don't do very much at all; I just pass the pressed key onto another routine, KeyPressed. That routine decides how the information about the pressed key should affect which item in the control gets the focus and what to do with the selected item if the user presses the return or space keys (see Figure 4).

Figure 4 Who Gets the Focus (Close Figure 4)
Private Sub KeyPressed(ByVal Key As System.Windows.Forms.Keys)
    Try
        Dim currentSelectedItem As Integer = Me.focusedItem
        Dim newPosition As Integer = currentSelectedItem
        Dim selectedRow As Integer
        Dim selectedColumn As Integer

        If Not Me.DataManager.List Is Nothing _
            AndAlso Not Me.Parent Is Nothing Then

            Select Case Key
                Case Keys.Up
                    If currentSelectedItem = 0 Then
                        'go to the previous control
                        Me.Parent.SelectNextControl( _
                            Me, False, True, True, True)
                    Else
                        'go up by one
                        newPosition -= 1
                    End If

                Case Keys.Down
                    Dim selected As Boolean
                    If currentSelectedItem >= _
                        Me.DataManager.Count - 1 Then
                        'go to the next control
                        Me.Parent.SelectNextControl( _
                            Me, True, True, True, True)
                    Else
                        'go down by one
                        newPosition += 1
                    End If

                Case Keys.Left
                    'go to the previous control
                    Me.Parent.SelectNextControl(Me, _
                        False, True, True, True)

                Case Keys.Right
                    'go to the next control
                    Me.Parent.SelectNextControl(Me, _
                        True, True, True, True)

                Case Keys.Enter, Keys.Return, Keys.Space
                    Me.SelectedIndex = newPosition
            End Select

            If newPosition < 0 Then
                newPosition = 0
            ElseIf newPosition >= Me.DataManager.Count Then
                newPosition = Me.DataManager.Count - 1
            End If

            If Me.focusedItem <> newPosition Then
                Me.focusedItem = newPosition
            End If
        End If
    Catch except As Exception
        Debug.WriteLine(except)
    End Try
End Sub

I actually changed this keyboard-handling routine completely just before writing this column. Originally the selected item changed if the user moved up and down the list of radio buttons using the arrow keys, but I modified it so that the user had to explicitly select an item (using space or enter) and so that the arrow keys just moved the focus. This seems like a better user experience to me, but you can easily modify the code to match my original design if you prefer the other behavior.

Let's Handle Resizing and Scrolling

Whenever you are creating a control that can be bound to a data list, you have to handle the situation in which you have more items than can fit in the control (see Figure 5).There, a vertical scrollbar is added if AutoSize = False and if the control is smaller than is needed to display the list of items.

Figure 5 Too Many Items
Figure 5 Too Many Items

For this control, I decided to handle the situation in two different ways: by automatically resizing the control (configurable with an AutoSize property) or by adding a vertical scrollbar as necessary (see Figure 5). Whenever the data source changes (because the number of items could have changed) or whenever the control is resized, the RecalcSizing routine is called to automatically resize the control or to add a scrollbar (see Figure 6).

Figure 6 Handling Control Resizing (Close Figure 6)
Private WithEvents vSB As VScrollBar

Private Sub RecalcSizing()

    If Me.AutoSize Then
        Me.Height = m_textHeight
    Else
        If Me.Height < m_textHeight Then
            If vSB Is Nothing Then
                vSB = New VScrollBar
                Me.Controls.Add(vSB)
            End If
            With vSB
                .Top = 0
                .Left = Me.Width - .Width
                .Height = Me.Height
                .Visible = True
                .Maximum = m_textHeight - Me.Height
                .Minimum = 0
                .Value = 0
                .LargeChange = .Maximum \ 10
                .SmallChange = 1
            End With
        Else
            If Not vSB Is Nothing Then
                Me.Controls.Remove(vSB)
                vSB = Nothing
            End If
        End If
    End If

End Sub

Private Sub vScrollChanged(ByVal sender As Object, _
    ByVal e As EventArgs) Handles vSB.ValueChanged
    Dim sb As VScrollBar
    sb = DirectCast(sender, VScrollBar)
    Me.m_currentTopPosition = sb.Value
    Me.Invalidate()
End Sub

Alternatively you could use AddHandler/RemoveHandler for tracking the ValueChanged event of the scrollbar, but by using WithEvents and Handles, Visual Basic .NET takes care of the event wiring for you.

Where to Go from Here

The end result of all this code (and a few small snippets that are included in the download but don't merit listing within this column) is a data-bindable RadioButtonList control for Windows Forms. Almost any application can benefit from more features, and this control is no exception. You could expand it to handle checkboxes as well as radio buttons, you could add horizontal scrolling or word wrap on the captions, and lots more. As it's built now, it shows how you can achieve consistent use of data binding across your entire interface. For more information on Windows Forms development, check out Developing Custom Windows Controls Using Visual Basic .NET and http://www.windowsforms.net.

Send your questions and comments to basics@microsoft.com.

Duncan Mackenzie is the MSDN Content Strategist for Visual Basic and C# and the author of the Coding 4 Fun column on MSDN online. He can be reached through his blog at http://weblogs.asp.net/duncanma or personal site at http://www.duncanmackenzie.net.


.NET Matters: How Microsoft Uses Reflection

New information has been added to this article since publication.
Refer to the Editor's Update below.

.NET Matters
How Microsoft Uses Reflection
Edited by Stephen Toub

Code download available at: NETMatters0407.exe (153KB)
Contents
Steve Lamb, Xbox Live Team
Adi Oltean, Core File Services Team

In this fourth installment of .NET Matters, I'm taking a breather from the Q&A format. Instead, this month I bring you stories from the trenches; developers in product groups within Microsoft describe how they use .NET technologies to get their jobs done and develop their products. This month, Steve Lamb, a lead software design engineer on the Xbox team, and Adi Oltean, a software design engineer from the core file services team, discuss ways they've used reflection (System.Reflection and System.Reflection.Emit) in their particular products.

Steve Lamb, Xbox Live Team

The Xbox Live™ architecture consists primarily of services written using ASP.NET. The many benefits of ASP.NET and the .NET Framework, including rapid development, easy deployment, stability and maintainability, made them ideal technologies to meet the stability, security, and flexibility requirements of Xbox Live.

One of the challenges of using managed code on our servers was in developing the network protocol used to talk with Xboxes. These Xbox clients are written primarily in C++ and are optimized for low memory overhead and network bandwidth usage. We needed a simple way of converting data structures written in C# to and from the binary equivalents used by the C++ clients.

In the simplest cases, this can be done by using the functionality provided in the System.Runtime.InteropServices namespace, such as the Marshal.StructureToPtr and Marshal.Copy routines. However, these methods are very limited in the data types that can be converted, there's little control over how a data type is serialized, and there's added risk of causing crashes or leaking memory when improperly used. It's also possible in some scenarios to use unsafe code to read and write byte arrays, casting them to and from the appropriate C# data structures. On our team, however, we try very hard not to use unsafe code as it increases our risks, just as with the Marshal options.

Most of the 200+ protocol structures defined for Xbox Live are anything but simple. Many contain reference types, embedded structures, and variable length arrays and strings. Dates are transmitted as FILETIME values. Strings are UTF8-encoded before transmitting over the network. Most strings and arrays also include a length field specifying the number of bytes in a string or the number of elements in an array.

Writing custom marshaling code for each protocol structure would quickly become prohibitively expensive and error prone. To address this, we developed a fast, flexible, and type-safe way of serializing C# structures and classes, which maintains most of the advantages of custom serialization code without having to actually write custom serialization code. This was done using the powerful Reflection and Reflection.Emit features of the Framework SDK.

By way of example, take a look at the protocol structure written in C++ and C# in Figure 1. These structures may be sent when an Xbox client requests the latest version of a user's Friends list. The "header" structure, XBOX_FRIENDS_LIST, contains a count of the number of friends (cFriends) and is followed by a list of XBOX_FRIEND structures, each describing the user ID and gamertag of that particular friend, and the date they were added. The structure definitions are surrounded by #pragma pack() instructions, which are used to tell the C++ compiler not to add the usual padding to byte-aligned fields in the structure.

Figure 1 XBOX_FRIEND (Close Figure 1)

C++
#pragma pack(push, 1)
struct XBOX_FRIENDS_LIST
{
    ULONGLONG qwUserId;
    DWORD     dwListVersion;
    USHORT    cFriends
    // XBOX_FRIEND[] friends;
};
struct XBOX_FRIEND
{
    ULONGLONG dwUserId;
    BYTE      bGamertagLen;
    CHAR*     szGamertag;
    FILETIME  ftAdded;
}
#pragma pack(pop)
C#
class XBOX_FRIENDS_LIST
{
    ulong         userId;
    uint          listVersion;
    ushort        numFriends;
    XBOX_FRIEND[] friends;
}
class XBOX_FRIEND
{
    ulong         userId;
    byte          gamertagLen;
    string        gamertag;
    DateTime      added;
}

This XBOX_FRIENDS_LIST, as defined, would not be serializable by the Marshal.StructureToPtr method, due to the inability to serialize the array of structures in the friends field. Furthermore, the XBOX_FRIEND class could not be correctly serialized by the same Marshal method, as the DateTime member added would not be converted to a FILETIME before conversion.

In order to marshal these structures, the developer would have to write custom serialization methods. Reading these structures from a byte stream would typically utilize the System.IO.BinaryReader object, reading each individual field sequentially from the stream. Appropriate conversions would be performed along the way, such as converting strings from their UTF8 form to Unicode and converting FILETIME values to the corresponding DateTime value. This method would consist of code similar to that in Figure 2.

Figure 2 Custom Deserialization (Close Figure 2)
BinaryReader reader = new BinaryReader(input);
XBOX_FRIENDS_LIST friendsList = new XBOX_FRIENDS_LIST();

friendsList.userId = reader.ReadUInt64();
friendsList.listVersion = reader.ReadUInt32();
friendsList.numFriends = reader.readUInt16();

friendsList.friends = new XBOX_FRIEND[friendsList.numFriends];
for (int i = 0; i < friendsList.numFriends; i++)
{
    friendsList.friends[i].userId = reader.ReadUInt64();
    friendsList.friends[i].gamertagLen = reader.ReadByte();

    byte [] bytes =
        reader.ReadBytes(friendsList.friends[i].gamertagLen);
    friendsList.friends[i].userId = Encoding.UTF8.GetString(bytes));

    long filetime = reader.ReadInt64();
    friendsList.friends[i].added = DateTime.FromFileTime(filetime);
}

Custom marshaling code to serialize the structures to a byte stream would consist of essentially the same process, instead using the System.IO.BinaryWriter class. Furthermore, fields such as numFriends and gamertagLen could be set to the correct values just before serialization to ensure accuracy.

Obviously, repeating this pattern for the large number of structures we currently use would be tedious, error prone, and difficult to maintain—the antithesis of what we've come to expect when developing in C#. This becomes a great application for Reflection and Reflection.Emit, since each of the fields are serialized based on their individual types. The Xbox team's approach was to define a base class, Serializer, from which any object to be serialized inherits and which provides two methods, ReadFromStream and WriteToStream, to support reading to and writing from a byte stream. These methods, in turn, invoke dynamically generated code which performs the actual calls to the field serialization methods. The type-specific work performed by these methods is generated at run time as a part of the first invocation of the method. This is implemented by declaring delegates for the dynamically generated read and write methods. The delegate methods for each type is maintained in a static hashtable.

On each invocation of ReadFromStream or WriteToStream, the method checks this hashtable for an existing delegate for the type. If it doesn't exist, one is created and is added to the hashtable. The delegate is then invoked, taking as arguments the specific object instance and the stream to be read from or written to:

public void ReadFromStream(BinaryReader reader) {
    ReadStreamDelegate readStream;
  string key = this.GetType() + ".ReadStream";
  readStream = (ReadStreamDelegate)SerializerDelegates[key];
  if (readStream == null) {
      readStream = GenerateReadStream();
      SerializerDelegates[key] = readStream;
  }
  readStream(this, reader);
}

The process of dynamically generating the method consists of creating a container assembly and class, creating the method, and emiting the Microsoft® intermediate language (MSIL) opcodes to read or write the individual fields of the provided type. The dynamically generated MSIL looks very similar to the MSIL that would be generated by the C# compiler if a custom serialization method was written to resemble the code shown in Figure 2. Reflection is used to discover the fields of the object to be serialized. [ Editor's Update - 8/17/2004: This solution relies on assumptions concerning the functionality of Type.GetFields, assumptions which are not guaranteed to hold in future versions of the .NET Framework, including v2.0. Please note that GetFields is not guaranteed to return the fields of a struct or a class in the same order in which they're defined in that struct or class. However, in v1.x of the .NET Framework, the C# compiler usually maintains source-order when emitting metadata, and in v1.x the order of FieldInfo instances returned from GetFields will always be the same for a given Type. The solution provided here can be remedied by sorting the FieldInfo objects returned by GetFields into the proper order for serialization and deserialization; one approach for doing so would be to tag each field with a custom attribute specifying the ordering and then use that attribute to sort the FieldInfo instances returned from GetFields.] For simplicity, we just serialize all of the public instance fields of the type, as you can see here:

ILGenerator gen = methodBuilder.GetILGenerator();
FieldInfo fields = this.GetType().GetFields(
    BindingFlags.Public | BindingFlags.Instance);
EmitHeader(gen);
foreach(FieldInfo field in fields) EmitReadField(gen, field);
EmitFooter(gen);

To generate the MSIL for reading a field, we first determine the BinaryReader method that will be invoked. To simplify this code, the team built a hashtable to map field types to the specific method to be called, as shown in Figure 3.

Figure 3 Hashtables of Helper Methods (Close Figure 3)
public class SerializationMethods
{
    public static Hashtable ReadMethods = new Hashtable();
    public static Hashtable WriteMethods = new Hashtable();
    public static Hashtable EncodingMethods = new Hashtable();

    static SerializationMethods()
    {
        // Read methods for primitives
        ReadMethods.Add(typeof(bool),
            typeof(BinaryReader).GetMethod("ReadBoolean"));
        ReadMethods.Add(typeof(byte),
            typeof(BinaryReader).GetMethod("ReadByte"));
        ReadMethods.Add(typeof(byte[]),
            typeof(BinaryReader).GetMethod("ReadBytes"));
        // ... etc
    }
}

Generating the MSIL code to read each field is then simply a matter of pushing the BinaryReader onto the stack and then calling the helper method, setting the field to the result:

MethodInfo method = (MethodInfo)
    SerializationMethods.ReadMethods[fieldInfo.FieldType];
gen.Emit(OpCodes.Ldarg_0);
gen.Emit(OpCodes.Ldarg_1);
gen.EmitCall(OpCodes.Callvirt, method, null);
gen.Emit(OpCodes.Stfld, fieldInfo);

Writing primitives to a BinaryWriter takes a similar approach, first pushing the BinaryWriter on the stack, followed by the locally cast object and field as parameters. Reading and writing more complex types such as strings, DateTime values (which must be converted to or from FILETIME for our purposes), and arrays takes a little more code than the basic primitives. However, because dynamically generated MSIL is magnitudes more difficult to debug and maintain, the team tries to keep as much of this added complexity as possible in C#. This is done by adding a series of static helper methods to the SerializationMethods class to perform the necessary functions. For example:

public static DateTime ReadDateTime(BinaryReader reader)
{
    return DateTime.FromFileTime(reader.ReadInt64());
}

Now, the dynamically generated MSIL for reading more complex types is almost identical to that for the primitive types.

The MSIL code for arrays and strings is slightly more complex. Helper methods to read arrays and strings are defined as taking both the BinaryReader and the length as parameters. The length to use is determined while generating the dynamic code, based on a custom attribute specified on the field declaration. Lengths can either be declared as a constant length or as a dynamic size specified in another field on the object. If it's a constant length, the value is simply pushed on the stack before invoking the helper method. Otherwise, the field's value is loaded onto the stack. The helper method is then invoked:

FieldInfo sizeField = GetLengthField(fieldInfo, out staticSize);
if (sizeField == null) {
    gen.Emit(OpCodes.Ldc_I4, staticSize);
} else {
    gen.Emit(OpCodes.Ldloc_0);
    gen.Emit(OpCodes.Ldfld, sizeField);
}
gen.EmitCall(OpCodes.Callvirt, method, null);

Of course, if you're using a field to define the length of another field, you need to ensure that the field storing the length is correct before the object is serialized. When writing arrays and strings which may have variable-length fields, we first generate the code to set the corresponding length field to the correct value. This is done by checking each field on a type for a custom "Serialize" attribute with the SizeField parameter set. As shown in Figure 4, when such a field is found, code is emitted to set the value of the SizeField to the length of the UTF8-encoded string or to the number of elements in the array, depending on the field's type. The code to write arrays and strings is similar to the code that reads them, except it takes as parameters a BinaryWriter instead of a BinaryReader along with the actual array or string.

Figure 4 Handling Fields with Sizes (Close Figure 4)
int staticLen;
•••
FieldInfo sizeField = GetLengthField(field, out staticLen);
gen.Emit(OpCodes.Ldloc_0);

// if the field is a string, the size is the byte count
if (field.FieldType == typeof(string))
{
    gen.Emit(OpCodes.Call, (MethodInfo)
        SerializationMethods.EncodingMethods["get_UTF8"]);
    gen.Emit(OpCodes.Ldloc_0);
    gen.Emit(OpCodes.Ldfld, field);
    gen.Emit(OpCodes.Callvirt, (MethodInfo)
        SerializationMethods.EncodingMethods["GetByteCount"]);
}
// if the field is an array, the size is the length
else if (field.FieldType.IsArray)
{
    gen.Emit(OpCodes.Ldloc_0);
    gen.Emit(OpCodes.Ldfld, field);
    gen.Emit(OpCodes.Ldlen);
}
else throw new Exception("Expected string or array.");

// convert the result of either action to the destination type
if (sizeField.FieldType == typeof(byte))
    gen.Emit(OpCodes.Conv_U1);
else if (sizeField.FieldType == typeof(ushort))
    gen.Emit(OpCodes.Conv_U2);
else if (sizeField.FieldType == typeof(short))
    gen.Emit(OpCodes.Conv_I2);
else if (sizeField.FieldType == typeof(int))
    gen.Emit(OpCodes.Conv_I4);
else if (sizeField.FieldType != typeof(uint))
    throw new Exception("Unsupported type for size field!");

// store the size
gen.Emit(OpCodes.Stfld, sizeField);

When reading fields that inherit from Serializer (since Serializer-derived classes can have fields whose types are also derived from Serializer), the MSIL code first creates an instance of the type and then calls ReadFromStream on the new object:

ConstructorInfo ctor =
    fieldInfo.FieldType.GetConstructor(Type.EmptyTypes);
readMethod = fieldInfo.FieldType.GetMethod("ReadFromStream");
gen.Emit(OpCodes.Ldloc_0);
gen.Emit(OpCodes.Newobj, ctor);
gen.Emit(OpCodes.Stfld, fieldInfo);
gen.Emit(OpCodes.Ldloc_0);
gen.Emit(OpCodes.Ldfld, fieldInfo);
gen.Emit(OpCodes.Ldarg_1);
gen.Emit(OpCodes.Callvirt, readMethod);
Writing is even simpler (since no new object needs to be created) and we simply invoke the subobject's WriteToStream method. Reading and writing arrays of Serializer-derived objects involve the same process as for a single object, repeated for each element that is found in the array.

This approach for marshaling objects was chosen for the Xbox Live service primarily for its flexibility in handling a large variety of data types and structures. It's fairly easy to add support for new types, and it helps to avoid common bugs such as specifying the incorrect length for an international string. Furthermore, the custom attributes allow us to provide other code-enforcement checks, such as ensuring that variable-length strings can be no longer than a specific length. This provides another means of defense against common attacks such as buffer overruns.

Interestingly, dynamically generated assemblies are also relatively fast. Unscientific tests show that reading and writing structures that have six to eight fields with the dynamically created objects perform almost twice as fast as marshaling using the System.Runtime.InteropServices.Marshal methods. Custom code, however, could be an even faster route than the dynamically generated methods. Thus, the ReadFromStream and WriteToStream methods are declared as virtual; if speed is of the utmost importance, these can always be overridden with custom code. The dynamically generated code, however, has been more than fast enough for the vast majority of the needs of the Live service.

Note that the approach I've outlined here is similar to how XmlSerializer works, although rather than requiring all serializable types to derive from a base class, XmlSerializer takes the Type of the object to be serialized as a constructor parameter. The XmlSerializer's instance methods for serializing and deserializing then delegate to the dynamically created read and write methods for that Type. For more information on the code generated by XmlSerializer, see the last question in the April 2004 .NET Matters column.

Adi Oltean, Core File Services Team

One of the questions most frequently asked by developers is how to write reusable code. There are many answers to this question. Still, writing a reusable set of classes or templates can be quite difficult. If you write a generic "data grid" class, for example, you can't possibly know in advance all of the scenarios in which your class will be used. If your class will be used by thousands of developers, your implementation will invariably not satisfy all of them. Some people will need a faster version of your class, or a thread-safe version, or will require any of the plethora of other features that your implementation does not provide.

Generally, the problem with flexibility resides in the contract between the class/template being used and the code that is using this module. Sometimes this contract is too restrictive because this is how it was designed in the first place. Sometimes, this contract does not address certain usage scenarios, and developers will have a hard time using your class.

One solution to this new challenge is to design reusable code in a more flexible and configurable way. Instead of designing a C++ class, you could instead create a template. This template would accept a number of parameters that can be used to configure the piece of code that is being reused. A standard example of this kind of flexibility is the memory allocation scheme in STL—there, many classes can be configured to use a custom memory allocator (though a default allocator is provided).

C++ templates represent a simple way to design configurable, reusable code. Still, in some scenarios, templates might offer limited configurability. For example, a template has a fixed number of parameters, which can be very restrictive in certain situations.

The .NET Framework adds a new dimension to the usual array of techniques for creating reusable code. The fact that you can attach attributes to methods and classes gives you immense power. While certainly not a parallel, in some sense .NET attributes can be viewed as a more powerful equivalent of C++ template arguments. For example, you could "configure" a .NET class with a custom attribute in the same way you configure a template by specifying a certain argument. You could write a C# class that would look like the following:

[Key(typeof(String)), Value(typeof(String))]
class MyHashTable {
    •••
}
This resembles the following C++ template:
template<class Key, class Value>
class MyHashTableTemplate {
    •••
};
typedef MyHashTable<string, string> MyHashTable;
Of course, there are many significant differences between these C# and C++ examples, but in both cases the hashtable implementation is able to vary its behavior based on the attributes/template parameters that are supplied.

Let's continue this comparison between .NET attributes and C++ templates. The first thing to note is that you can specify a variable number of custom .NET attributes, as opposed to a fixed number of arguments in a template. On the other hand, there might be a performance penalty when you choose the reflection route: C++ templates are expanded at compile time, whereas .NET attributes need to be reflected over at run time. That said, custom .NET attributes are a great extensibility mechanism.

As an example, my team has designed and created a reusable framework for command-line parsing. If you have ever written a command-line application that takes a lot of parameters, you undoubtedly had to write a lot of embedded switch statements with complex parsing rules (depending on whether the parameters are optional, repeating,and so on). Usually you end up with spaghetti code. We developed a simple, straightforward, reusable codebase that parses the command line according to simple grammar-based rules, then invokes the appropriate methods in our classes.

When designing the system, we decided the command-line tool should be able to deal with multiple tasks, depending on the specified command-line options. For every separate task, we would like to invoke a certain callback, and we can establish the link between callbacks and tasks using custom attributes. Here is an example:

// Binding the ListFiles routine to the "/l" option
[Pattern("/l <directoryName>")]
[Description("Lists the directory contents.")]
public int ListFiles([DirectoryPath]string directoryName) {
    Console.WriteLine("Listing files under {0}...", directoryName);
    foreach(FileInfo fi in new DirectoryInfo(directoryName).GetFiles())
    {
        Console.WriteLine(fi.Fullname);
    }
    return 0;
}

In this code, I've bound the method ListFiles with the command-line option "/l <directoryName>". When the arguments match the pattern, the infrastructure will extract the parameter "directoryName" and then invoke the method, supplying the parameter parsed from the command line to the method.

A simplified version of our parsing and dispatch routine is shown in Figure 5. We use the Regex class to match the command-line parameters against the bound pattern. If we find a match, we enumerate the method parameters and, for each parameter name, we extract the value of the corresponding Regex named group. With these values we build a parameter array which will be used to invoke the matching method. The GetAttribute routine supplies the magic ingredient to get the value of our custom parameters in the first class. GetAttribute extracts the attribute value in string format using the standard .NET GetCustomAttributes API. The attributes used by our infrastructure (namely PatternAttribute and DescriptionAttribute) are derived from an attribute class GenericTextAttribute. This attribute defines a Text property that stores the value of our attribute:

public class GenericTextAttribute : Attribute {
    public GenericTextAttribute(string textParam) {
        _text = textParam;
    }
    public string Text { get { return _text; } }
    private string _text;
}

Figure 5 Parsing the Command Line and Invoking a Method (Close Figure 5)
public int MainFunction(string[] args, object target) {
    // Display help, if needed
    if (args.Length == 1 && (args[0] == "/?"))
        return DisplayUsage(target);

    // Get the command line in one string.
    string arguments = "";
    foreach(string arg in args) {
        if (arguments.Length != 0) arguments += " ";
        if (arg.IndexOfAny("\t ".ToCharArray()) == -1)
            arguments += arg;
        else arguments += '"' + arg + '"';
    }

    // Enumerate methods in the class
    foreach(MethodInfo method in target.GetType().GetMethods()) {
        string pattern = GetAttribute(typeof(PatternAttribute), method);
        string description = GetAttribute(typeof(DescriptionAttribute),
                                          method);
        if (pattern == "") continue;

        // Replace the parameter name with its regular expression
        foreach(ParameterInfo param in method.GetParameters()) {
            string paramPattern = GetAttribute(typeof(PatternAttribute),
                                               param);
            if (paramPattern != "")
                pattern = Regex.Replace(pattern,
                    string.Format("<{0}>", param.Name),
                    string.Format("(?<{0}>{1})", param.Name,
                              paramPattern));
        }

        Regex r = new Regex(pattern, RegexOptions.IgnoreCase);
        Match m = r.Match(arguments);
        if (m.Success) {
            // Try to match each method parameter
            ArrayList paramList = new ArrayList();
            foreach(ParameterInfo param in method.GetParameters()) {
                string paramValue = m.Groups[param.Name].Value;
                if (paramValue.StartsWith("\"") &&
                    paramValue.EndsWith("\""))
                    paramValue = paramValue.Substring(1,paramValue.Length
                                                      - 2);
                paramList.Add(paramValue);
            }

            // Invoke the method
            return (int)method.Invoke(target,
                (object[])paramList.ToArray(typeof(object)));
        }
    }

    Console.WriteLine("\nError: Invalid option\n");
    return DisplayUsage(target);
}

As a bonus, the command-line framework implements a usage method that takes advantage of reflection to dynamically create a usage message for the user. This method will be called automatically whenever we invoke our application with the command-line option "/?" and will simply iterate through all methods that are bound to a PatternAttribute attribute, writing the description and the pattern to the console.

As you have seen, .NET offers many powerful ways to write reusable code. Not only do you have all the traditional techniques like inheritance and components (and generics in the .NET Framework 2.0), but reflection can offer simple and elegant solutions where classic techniques fail to meet these requirements.

Send your questions and comments to netqa@microsoft.com.


C++ Q&amp;A: Message Values, Managed String Literals, Obfuscating Code, and More

C++ Q&A
Message Values, Managed String Literals, Obfuscating Code, and More
Paul DiLascia

Code download available at: CQA0407.exe (360KB)

Anumber of C++ fans have expressed concern that too much C# has crept into my columns of late. I agree! My only defense is that as the Microsoft® .NET Framework has gained wider acceptance, more readers have been sending me questions about C#, and since C# is so similar to C++, well, I've sort of been answering them. It was never my intent to alienate C++ fans—lord knows, I'm one of them! However, by popular demand, starting this month the C++ column will focus more exclusively on C++, including managed extensions and old favorites like MFC. So bring on your C++ questions! I especially encourage you to submit questions about managed C++. What experiences have you had with it?

Q In your March 2004 column, you implemented your CMyOpenDlg initialization using a redefined WM_USER+1. I think this is a misuse of the WM_USER range in general (it is reserved for use by whomever does the RegisterClass), but also wrong inasmuch as WM_USER+1 is already the predefined dialog message DM_SETDEFID. Shouldn't you have used a different value for the message?

Jeff Partch

A You're perfectly right! WM_USER is reserved for whomever implements the window class—whether it be you, the friendly Redmondtonians, or aliens from Planet Gleepoid. Figure 1 shows the official breakdown of Windows® message values, which everyone should review at least once a decade. WM_USER through 0x7FFF is reserved for private window classes. You can think of this range as dedicated to messages that have meaning within the particular window class. For example, the status bar control uses WM_USER+1 for SB_SETTEXTA. And as you point out, dialogs use WM_USER+0 and WM_USER+1 for DM_GETDEFID and DM_SETDEFID. My use of WM_USER+1 in the March 2004 column conflicts with DM_GETDEFID.

Figure 1 Message Ranges in Windows (Close Figure 1)

Range | Description
0 - (WM_USER-1) | Messages reserved for use by the system. Examples: WM_CREATE and WM_SCROLL.
WM_USER - 0x7FFF | Integer messages for use by private window classes. Examples: DM_GETDEFID (dialog), TB_ENABLEBUTTON (toolbar), and PBM_SETRANGE (progress bar).
WM_APP - 0xBFFF | Messages available for use by applications.
0xC000 - 0xFFFF | String messages for use by applications (RegisterWindowMessage).
> 0xFFFF | Reserved by Windows for future use.

Applications that want to define their own messages should use WM_APP. WM_APP is guaranteed not to conflict with the system (WM_CREATE, and so on) or class/control-specific messages like DM_GETDEFID. Windows defines WM_APP like so:

#if (WINVER >= 0x0400)
#define WM_APP 0x8000
#endif

As every Windows geek knows, WINVER 0x0400 is Windows 95, Windows 98, and Windows NT®. So WM_APP is just under a decade old, which explains why I overlooked it—I'm not due for my next decennary message-range review until 2005!

But should I really have used WM_APP for CMyOpenDlg? CMyOpenDlg lives somewhere between Windows and the application. I'm writing it as an extension for other programmers to use in their apps. If a programmer is already using WM_APP and uses CMyOpenDlg, is there a conflict? It's unlikely in the case of the file open dialog—but what if I were writing a custom control like a toolbar or progress bar and needed to define my own messages? I could base them on WM_APP to avoid conflicting with the base control (but then risk conflicting with the app) or I could pick a number like WM_USER+400 that's well out of the range of the Windows control. Unfortunately, there's no right answer. It's a problem that's plagued control and class library designers ever since Windows was released. There's simply no perfect way to divide the message space that's guaranteed never to cause a conflict. You have to make a practical decision in each case. For CMyOpenDialog, I should have chosen something like WM_USER+10, which doesn't conflict with the DM_XXX messages. Using a registered message is probably overkill.

Speaking of registered messages, what's the difference between the WM_APP range and the registered message range (0xC000 - 0xFFFF), and when should you choose to use one over the other? If you're only talking to yourself—that is, if you plan to send the message only to windows that live within your own application, you can use the WM_APP range. Registered messages are for global messages that you want to send to other apps written by other programmers who know the name of the message. For example, if you're writing a conspiracy-management app that identifies itself to like-minded programs by sending a special message, you should use a registered message:

UINT WM_SAYHELLO = RegisterWindowMessage(_T("WmSayHello"));

Now other conspiracy apps can register using the special name "WmSayHello" and get the same value, so they can all communicate. The actual value WM_SAYHELLO won't be the same for each Windows session, but within a session it will have the same value for all apps that register it. And, of course, the value will always be in the range 0xC000 - 0xFFFF. Gosh—MSDN® Magazine has the sharpest readers!

Q I'm writing an app in managed C++ and have a question about string literals. I know I can create a string with _T("sample string") or S"Sample String". I've read that the S string literal is more efficient, but no one seems to know why or how much more efficient. What is the difference between the two types of literals? What exactly is a managed string literal, and why is it better?

Randy Goodwin

A The best way to understand the difference between managed string literals and ordinary C/C++ literals is to write some code that uses both and look at the compiled results. I wrote a simple managed C++ program, StrLit.cpp, that creates some Strings using both C++ and .NET-style literals. Figure 2 shows the code; Figure 3 and Figure 4 show Microsoft intermediate language (MSIL) code disassembled with ILDASM. As many of you probably know, the primary difference is that when you use a C++ literal, the compiler generates a call to the appropriate String constructor:

// String* c2 = new String(_T("This is a TCHAR string"));
ldsflda valuetype $ArrayType$0x11197cc2
   modopt([Microsoft.VisualC]Microsoft.VisualC.IsConstModifier)
   '?A0x1f1e2151.unnamed-global-0'
newobj instance void [mscorlib]System.String::.ctor(int8*)

Figure 4 StrLit Disassembled (Close Figure 4)
//////////////////
// .asm file for strlit.cpp, edited to show highlights.
// Produced with cl /FAs (generate assembly with source).
//
.method public static int32
  modopt([mscorlib]System.Runtime.CompilerServices.CallConvCdecl)
  main() cil managed
{
  .vtentry 1 : 1
  // Code size 111 (0x6f)
  .maxstack  1
  .locals init ([0] string c1,
           [1] string s3,
           [2] string s2,
           [3] string s1,
           [4] string c2)
           •••
  // String* c1 = new String(_T("This is a TCHAR string"));
  IL_000b:  ldsflda    valuetype $ArrayType$0x11197cc2
      modopt([Microsoft.VisualC]Microsoft.VisualC.IsConstModifier)
      '?A0x1f1e2151.unnamed-global-0'
  IL_0010:  newobj instance void [mscorlib]System.String::.ctor(int8*)
  IL_0015:  stloc.0
  •••
  // String* s1 = new String(S"This is a String literal");
  IL_002e:  ldstr      "This is a String literal"
  IL_0033:  stloc.3
  •••
  // Console::WriteLine(_T("Goodbye,"));
  IL_0052:  ldsflda    valuetype $ArrayType$0x51890b12
      modopt([Microsoft.VisualC]Microsoft.VisualC.IsConstModifier)
      '?A0x1f1e2151.unnamed-global-2'
  IL_0057:  newobj  instance void [mscorlib]System.String::.ctor(int8*)
  IL_005c:  call       void [mscorlib]System.Console::WriteLine(string)

  // Console::WriteLine(S"World");
  IL_0061:  ldstr      "World"
  IL_0066:  call       void [mscorlib]System.Console::WriteLine(string)
  •••
// end of method 'Global Functions'::main
}
Figure 2 StrLit.cpp (Close Figure 2)
////////////////////////////////////////////////////////////////
// MSDN Magazine -- July 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.

// To see the difference between using C++ string literals (TCHAR) and
// .NET (managed) string literals, compile this program and then
// disassemble the .exe with ILDASM. You can also compile with /FAs to
// view the assembly code.
//
//     cl /clr /FAs strlit.cpp
//
#include "tchar.h"
#using <mscorlib.dll>

using namespace System;

int _tmain()
{
   // First two TCHAR strings are the same.
   String* c1 = new String(_T("This is a TCHAR string"));
   Console::WriteLine(c1);

   String* c2 = new String(_T("This is a TCHAR string"));
   Console::WriteLine(c1);

   // First two String literals are the same, third is different.
   String* s1 = new String(S"This is a String literal");
   Console::WriteLine(s1);

   String* s2 = new String(S"This is a String literal");
   Console::WriteLine(s2);

   String* s3 = new String(S"This is a different String literal");
   Console::WriteLine(s3);

   // What happens when you pass to managed functions?
   Console::WriteLine(_T("Goodbye,"));
   Console::WriteLine(S"World");

   return 0;
}

I'll explain the gobbledy-gook in a minute, but the basic idea is that the compiler loads the address of your C++ string literal, then generates a newobj instruction that passes it to the String constructor. In addition, the compiler creates a special Array type for your unmanaged literal—that's the strange-looking $ArrayType$0x11197cc2 which appears in the previous snippet and also in Figure 3, with a static instance named ?A0x1f1e2151.unnamed-global-0.

Figure 3 StrLit in the Disassembler
Figure 3 StrLit in the Disassembler

If you spend much time looking at MSIL, you'll discover modopts all over the place. A full description of modopt is beyond the scope of this column, but the basic idea is that compilers can use modopt to supply additional information about types not understood by the common language runtime (CLR), but which other consumers of the assembly may understand—in this case, the info is that the global is const. The CLR itself has no notion of const, but other consumers of the assembly might. If the assembly were imported into C++, the C++ compiler would know the variable is const; whereas if the assembly were used by C#, the C# compiler would ignore modopt const since C# has no notion of const. As you can see, there's a lot of voodoo that goes on when you use native types in managed code.

When you use a managed literal using the S syntax, however, everything is simple:

// String* s1 = new String(S"This is a String literal");
  ldstr "This is a String literal"
There's no function call, just an ldstr instruction. This instruction pushes a new object reference to a string literal stored in the program's metadata. This is true whether you use the literal to create a String object or pass it directly to some function that requires a string:
// Console::WriteLine(S"World");
ldstr "World"
call void [mscorlib]System.Console::WriteLine(string)

Another way to see what's happening is to compile your program with /FAs to generate an assembly listing. (Yes, it works for MSIL too!) Figure 5 shows the edited results from compiling StrLit with /FAs. What's interesting here is that you can see that the compiler generates two strings ($SG1883 and $SG2266) for the TCHAR strings "This is a TCHAR string", even though they have the same content. However, there's only one managed string literal ($StringLiteralTok$70000001$) for S"This is a String literal", which is used twice. Most programmers call this process "uniquification," but for some reason the Redmondtonians call it "string interning." As the documentation states: "The Common Language Infrastructure (CLI) guarantees that the result of two ldstr instructions referring to two metadata tokens that have the same sequence of characters return precisely the same string object."

Figure 5 StrLit.asm (Close Figure 5)
; Edited to show only highlights—Paul DiLascia
; Listing generated by Microsoft Compiler Version 13.10.3077

; Generated by Visual C++ for Common Language Runtime
.file "StrLit.cpp"
   .rdata
$SG1883:
   .ascii   "This is a TCHAR string\000"
$SG2266:
   .ascii   "This is a TCHAR string\000"
.global
   .bss
.local   $StringLiteralTok$70000001$,4
.local   $StringLiteralTok$70000001$,4
   •••
   .text
.global  ?main@@$$HYAHXZ ; main
   •••
; String* c1 = new String(_T("This is a TCHAR string"));
   ldsflda     $SG1883
   newobj      ??0String@System@@$$FQ$AAM@PAC@Z
   stloc.0           ; _c1$
   •••
; String* c2 = new String(_T("This is a TCHAR string"));
   ldsflda     $SG2266
   newobj      ??0String@System@@$$FQ$AAM@PAC@Z
   •••
; String* s1 = new String(S"This is a String literal");
   ldstr    $StringLiteralTok$70000001$
   stloc.3           ; _s1$
   •••
; String* s2 = new String(S"This is a String literal");
   ldstr    $StringLiteralTok$70000001$
   stloc.2           ; _s2$
   •••
   ret
 .end ?main@@$$HYAHXZ ; main
text  ENDS

The main point of all this is that managed literals are faster. In an attempt to find out how much faster, I wrote another managed C++ program, StrTime, that allocates a bunch of strings and reports how long it took. I used the ShowTime class from my June 2004 column:

_stprintf(msg, _T("Test 1: Allocate %d strings using LPCTSTR"), num);
ShowTime st(msg);
for (int i=0; i<num; i++) {
   String* s = new String(_T("This is a TCHAR string"));
}
StrTime then does the same thing using a managed S literal. Here are the results:
C:\>StrTime 50000000
Test 1: Allocate 50000000 strings using LPCTSTR: 32797 msec
Test 2: Allocate 50000000 strings using .NET S literal: 100 msec

Wow! As you can see, S literals are much faster. But hey—wait a minute, what's really going on here? Is ldstr really 327 times faster? This test is suspicious for a couple of reasons. First, a function call shouldn't be so expensive; second, did I really allocate 50 million strings when my machine has only 785MB? Of course not! Since the variable s lives inside the for loop, it goes out of scope with every iteration—which makes it eligible for garbage collection. In fact, you could write the following program and it would loop forever without running out of memory:

while (1) {
   String* s = new String(L"foo");
}

The main reason the S literal appears so much faster in the test is that the C++ literal calls newobj to create a new object whereas the S literal generates ldstr, which simply pushes the same object reference on the stack over and over again, 50 million times (remember "string interning"?) To prove this, I modified StrTime (StrTime2) to display the actual address of the managed String. To get it, you need to use GCHandle:

String* s = new String(S"foo");
GCHandle h = GCHandle::Alloc(s, GCHandleType::Pinned);
IntPtr ptr = h.AddrOfPinnedObject();
_tprintf(_T(" string at %p\n"), ptr.ToInt32());
h.Free();
This creates a pinned handle, whereupon you can call GCHandle::AddrOfPinnedObject to get the address of the pinned object. Here's the result of running this new test:
C:\>StrTime2 5
Test 1: Allocate 5 strings using LPCTSTR
 string at 04AC21E0
 string at 04AC22D8
 string at 04AC2318
 string at 04AC2358
 string at 04AC2398
Test 2: Allocate 5 strings using .NET S literal
 string at 04AC218C
 string at 04AC218C
 string at 04AC218C
 string at 04AC218C
 string at 04AC218C

No need to allocate 50 million strings; 5 is enough to get the idea. As you can see, calling new String(_T("foo")) five or a zillion times generates a new object each time, whereas calling new String(S"foo") loads the same object each time. So the real reason the 50-million-string test takes so much longer with C++ literals is that the runtime has to allocate a lot of memory and perform garbage collection.

All of this may already be more than you bargained for, but just for fun I wrote yet another modification, StrTime3, that forces a garbage collection before allocating the strings, then displays the amount of memory "thought to be allocated" (as the documentation for GC::GetTotalMemory describes it):

GC::Collect();
GC::WaitForPendingFinalizers();
// allocate strings
printf("TotalMemory = %d\n", GC::GetTotalMemory(false));
And here are the results:
C:\>StrTime3 50000000
Test 1: Allocate 50000000 strings using LPCTSTR
 TotalMemory = 190952, time: 33147 msec
Test 2: Allocate 50000000 strings using .NET S literal
 TotalMemory = 43484, time: 110 msec

As you can see, 50 million managed Strings take up a lot less space—especially when there's really only one! Even with TCHARs, the garbage collector uses only 190,952 bytes. Advanced mathematical reasoning suggests that 50 million strings can't fit in 191KB, so it's safe to conclude that the Framework has made a few trips to the dumpster.

In the end I'm afraid all my test programs have failed to answer your question—how much faster are managed String literals? Without a timer with microsecond or nanosecond granularity, it's impossible to tell. But at least my investigations will help you understand what's going on under the hood, and you can see why ldstr is significantly more efficient than newobj.

So which type of literal should you use? The truth is, a user probably will never notice the difference. But it's best practice to use S literals wherever you call a function that takes a String. Note that whereas in MFC you can use LPCTSTR and CString interchangeably, you can't pass a String to C/C++ functions that require TCHARs. That's because managed strings are always Unicode (wide-character) and there's no automatic conversion from String to wchar_t. For that, you need the PtrToStringChars function provided in vcclr.h. The next question shows an example of this.

One final minor note on managed versus native string literals. If you examine the ILDASM code in Figure 3 and Figure 4, you'll see that while the text "This is a String literal" is plainly visible in the disassembled code, the text for the C++ literal "This is a TCHAR string" is nowhere to be found. It's hidden in the mysterious variable ?A0x1f1e2151.unnamed-global-0. Does this mean C++ literals provide more obfuscation—that is, let you hide your strings from prying eyes? Not really. The MSIL code in Figure 3 reports that the mysterious "unnamed-global-0" is "at D_0000B14C." Out of curiosity, I examined StrLit.exe in a hex viewer. Sure enough, there's the string, precisely at location 0xB14C. Figure 6 shows what the hex dump looks like.

Figure 6 The Hex Dump
Figure 6 The Hex Dump

Q I'm presented with the problem of protecting our intellectual property in an upcoming C# application. I understand that all code can be disassembled with ILDASM, which makes it easy for someone to recover our mathematical formulas. To resolve this problem, I can write a C++ DLL and use DllImport to import my functions in C#, but I prefer to use managed C++ to write a managed __gc class so I can expose both the properties and methods. When I do this and compile my DLL, I can still use ILDASM to disassemble it, which defeats what I'm trying to do. I thought the C++ compiler would generate native machine code, not MSIL. What's going on?

Matt Hess

A When you compile a C++ program with /clr, the compiler treats all functions as managed and compiles them to MSIL by default (except in a few situations, which I'll discuss shortly). Structs and classes can be managed or unmanaged, depending on your use of __gc. If you want to compile a particular function to native code (perhaps in order to hide its implementation, or for performance reasons), you can use #pragma unmanaged:

   #pragma unmanaged
   // this fn and all subsequent ones are compiled
   // to native code
   void func(/* args */)
   {
      •••
   }
   #pragma managed

The #pragma managed directive switches back to managed mode. Figure 7 shows a simple C++ program I wrote with two functions, PrintFive1 and PrintFive2, that each print a string five times. The first function is native and the second managed. If you compile with /FAs and look at the .asm file produced, you'll see that the for loop in PrintFive1 compiles to something like this:

; for (int i=0; i<5; i++) {
   mov DWORD PTR _i$1662[ebp], 0
   jmp SHORT $L1663
   •••
; etc... (more native assembly)
Whereas the same for loop in PrintFive2 compiles like so:
; for (int i=0; i<5; i++) {
   ldc.i.0 0 ; i32 0x0
   stloc.0   ; _i$1669
   •••
; etc... (more MSIL)

Figure 7 Managed and Unmanaged Code (Close Figure 7)
////////////////////////////////////////////////////////////////
// MSDN Magazine — July 2004
// If this code works, it was written by Paul DiLascia.
// If not, I don't know who wrote it.
// Compiles with Visual Studio .NET 2003 on Windows XP. Tab size=3.

// To see what #pragma unmanaged does, compile this file with
//
//   cl /FAs /clr mixed.cpp
//
// and look at the .asm generated. You can see that unmanaged functions
// generate native machine code whereas managed ones generate MSIL.
// By default, when you use /clr, all functions are managed.
//
#include <stdio.h>
#include <vcclr.h>
typedef wchar_t WCHAR;
#using <mscorlib.dll>
using namespace System;

#pragma unmanaged
// This function will be compiled to native machine language
void PrintFive1(const WCHAR* msg)
{
   for (int i=0; i<5; i++) {
      wprintf(msg);
   }
}
#pragma managed

// This function and main will be compiled to MSIL
void PrintFive2(const WCHAR* msg)
{
   for (int i=0; i<5; i++) {
      wprintf(msg);
   }
}

int main()
{
   String* s2 = new String(S"Hello, world\n");
   const WCHAR __pin* pstr = PtrToStringChars(s2);
   PrintFive1(pstr);
   PrintFive2(pstr);
   return 0;
}

I said all functions are managed by default when you use /clr, but that's not quite true. While MSIL is general enough to express most C/C++ constructs, it can't express them all. There are a few situations that force a function to be compiled natively, even without #pragma unmanaged. They tend to be obvious: for example, if your function has __asm blocks, setjmp/longjmp instructions, or a variable parameter list (varargs). Since MSIL can't handle these constructs, the compiler automatically compiles such functions to native code.

You can only use #pragma unmanaged to compile pure C++ functions—that is, functions that don't use managed types. And you can't use #pragma unmanaged to compile the methods of a managed __gc class (that wouldn't make sense.) So if you want to use C++ to hide your algorithms, you have to implement them as pure C++ functions with native C/C++ types, which your __gc class methods can call. You must convert any managed parameters to unmanaged types before calling your implementation:

__gc class Widget {
public:
  MyMethod(String *s)
  {
    // convert managed types to C++ types
    const WCHAR __pin* pstr = PtrToStringChars(s);
    // call unmanaged C++ function (in #pragma unmanaged block)
    DoSecretAlgorithm(pstr);
  }
};

Managed C++ is the only Microsoft language that lets you combine native and MSIL code in the same file, which is one reason it's so powerful! Another way managed C++ mixes code is through linking. For example, whenever you call C library functions like printf or memset, they get statically linked from the appropriate libraries. When you use /clr, it automatically implies /MT to ensure your program links with the multithreaded versions of the C runtime libraries. The CLR requires them because the garbage collector runs finalizers in a separate thread.

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at http://www.dilascia.com.