 MSDN Magazine March 2005

MSDN Magazine March 2005

Web Services: Increase Your App's Reach Using WSDL to Combine Multiple Web Services

The very tools that have helped drive the growing adoption of Web services, and the enabling abstractions that they provide, can often prevent developers from peeking behind the curtains at the XML standards that make up the Web services stack. This article will offer a solution that enables type sharing between proxies created for complementary Web services, while at the same time providing an opportunity to examine the Web Services Description Language (WSDL) and its interaction with the Web services tools you know and love.

Gerrard Lindsay What is BPEL4WS?: Build Better Business Processes with Web Services in BizTalk Server 2004

In this article the author focuses on one critically important Web services specification that has been largely overlooked: the Business Process Execution Language for Web Services (BPEL4WS, or BPEL). He describes why BPEL is important and what it offers you if you are implementing Web services today or planning to in the future. Concrete examples using BizTalk Server 2004 are presented in the article.

Jon Fancey ASP.NET: Combine Web and Windows Services to Run Your ASP.NET Code at Scheduled Intervals

If you want to schedule ASP.NET tasks, one solution is to use a Web service to provide an interface to your ASP.NET application and build a Windows service that calls to it at scheduled intervals. Thus the ASP.NET application doesn't have to own the scheduling logic. Here the author shows how to schedule your ASP.NET tasks using a Windows service to initiate the Web service call because Windows services can start themselves when Windows boots up.

Andrew Needleman WSE 2.0: Give Your Web Services Consumers the Exact XML They Need to Succeed

Web services use XML to communicate with each other. But sometimes the XML your service emits is not the same XML that another service is expecting. If you want to avoid the problems that this kind of situation can cause, you need to get good control over your XML serialization. Here author Chris Dix shows you exactly how to tackle this problem.

Chris Dix Security: Manipulate Privileges in Managed Code Reliably, Securely, and Efficiently

When the author was faced with implementing support for changing a security descriptor on an object, he noticed there was not support for that operation in .NET. So he devised two solutions to the problem: the first, simpler one, is tailored to the .NET Framework 1.1 and can be used today. The second solution incorporates several advanced features available only in the .NET Framework 2.0. Both are presented here.

Mark Novak Go Global: Make the .NET World a Friendlier Place with the Many Faces of the CultureInfo Class

CultureInfo is a complex .NET class whose objects are used for resource loading, formatting, parsing, casing, sorting, and other conventions that change as the language, location, or writing system is changed. It can be tricky to use correctly in every situation. Here Michael Kaplan explains some common scenarios in which you will use CultureInfo and provides information about the behavior, best practices, and consequences of a wrong decision.

Michael Kaplan Editor's Note: There’s A Word For That…

This month in MSDN Magazine, we are looking at the world of Web services. But before we tell you all about this month's issue, we have a topic we would like to address. In fact, it's our civic duty to do so.

New Stuff: Resources for Your Developer Toolbox

XMLSpy 2005 by Altova is an XML development environment for modeling, editing, debugging, and transforming all XML technologies. It implements the W3C's new XSLT 2. 0 and XPath 2. 0 specs, and provides native support for XQuery 1.

Marnie Hutcheson Web Q&A: Data Integrity Over HTTP, IIS Credentials, DataGrids, and More

Edited by Nancy Michell Advanced Basics: Doing Async the Easy Way

If you've been following Ted Pattison's excellent series of Basic Instincts columns on multithreading and asynchronous behavior, you should by now be an expert on handling the issues involved in working with multiple threads in Windows®-based apps.

Ken Getz Cutting Edge: Moving DataGrid Rows Up and Down

Imagine opening your Inbox one morning and finding a message that reads "Dear Mr. DataGrid, I urgently need an ASP.NET DataGrid that lets my users move rows on the client. You're my last hope. Will you please help me?"

Dino Esposito Test Run: Automate Your ASP.NET Web Services Testing

It's no exaggeration to say that Web services are revolutionizing application-to-application communication. Web services are already being used extensively in corporate intranet environments and are making their way into commercial use, too.

James McCaffrey Inside MSDN: Using InfoPath as a Reporting Solution

I love the part of my job that lets me write code and develop software. In addition to those responsibilities, however, I manage a development organization that's working on a number of large strategic projects.

Larry W. Jordan Jr. Security Briefs: Access Control List Editing in .NET

Access control lists (ACLs) can be complex beasts, and user interfaces for editing them are incredibly tricky to implement properly. That's why I was really excited when Windows® 2000 shipped with a programmable ACL editor, shown in Figure 1.

Keith Brown Bugslayer: Mini Dump Snapshots and the New SOS

In debugging some large Microsoft® . NET Framework-based ap-plications over the last few months, I've been spending more time looking at mini dumps than at live processes. This is mainly because in those large applications problems surface when the apps are running in production and not on test systems.

John Robbins C++ at Work: Making Static Links Keyboard-Capable, Launching URLs from Your App

Paul DiLascia { End Bracket }: To Confirm is Useless, to Undo Divine

The common user interface technique of confirmation, popping a dialog box into the user's face and asking, "Are you really REALLY sure you want to do that?" is evil. It's unfriendly, it's distracting, and it's completely ineffective.

David S. Platt


Web Services: Increase Your App's Reach Using WSDL to Combine Multiple Web Services

Web Services

Increase Your App's Reach Using WSDL to Combine Multiple Web Services

Gerrard Lindsay
This article discusses:
  • The mysteries of XML namespaces and the Web Services Description Language
  • Sharing types between multiple Web services without having to write client code
  • The WSDL.exe tool
This article uses the following technologies:
The .NET Framework, C#, XML, and Web services
Code download available at: WSDL.exe (147KB)
Contents
XML Namespaces versus CLR Namespaces
What is a Binding Anyway?
The Design of the Solution
Create WeatherServices WSDL
Implementing the CalculateChanceOfRain Service
Implementing the GetWeather Service
The WeatherServices Client App
Type Sharing in the .NET Framework 2.0
Conclusion

E
nterprise solutions typically aggregate information from myriad internal applications and external sources. Web services have gained rapid adoption as a method to easily and reliably consume the varied data required by these solutions. Situations inevitably arise in which a single solution requires the consumption of multiple complementary Web services, which often share company-specific or domain-specific XML types. Unfortunately, the very tools that have helped to drive the growing adoption of Web services, and the enabling abstractions that they provide, can often prevent developers from peeking behind the curtains at the XML standards that make up the Web services stack. This article will offer a solution that enables type sharing between proxies created for complementary Web services, while at the same time providing an opportunity to examine the Web Services Description Language (WSDL) and its interaction with the Web services tools we know and love.

The first time I tried to create two complementary Web services that would share a common XML type, I ran into an unpleasant surprise. Each of the proxies that had been created in my client application had a separate object representation for the common message, in a separate C# namespace. I had to copy each of the properties manually from an instance of one class to an instance of the other, which seemed an unnecessary artifact of the current implementation of the tools.

I eventually stumbled onto a technique that allows you to expose a single Web service contract that encapsulates multiple Web service endpoints through the creation of a WSDL file with multiple bindings. This explicitly defines the relationship between the services, and also has the benefit of generating proxies that can share the managed classes that represent the common messages.

It's possible to share types defined by multiple contracts in the same namespace by manually editing tool-generated proxies after the fact. However, combining the Web service bindings into a single WSDL file allows the Web service implementer to define the shared relationship rather than placing the burden on the developers creating clients for the services. This technique also saves you from having to recreate those changes to the proxy class each time you change the service contract during development.

XML Namespaces versus CLR Namespaces

Before proceeding it's important to make sure that you understand the concept of namespaces, and the difference between namespaces in the context of XML and namespaces in the context of managed code. Both types of namespaces serve a similar purpose: to help developers avoid name collisions. They allow for a single group of items to include multiple items with identical names, but different meanings. This allows you to use the name "Order" to represent both a customer's order and also the sequence of an item in her shopping cart. In that way, namespaces act like the surnames used to distinguish "John Smith" from "John Brown." Let's take a look at this by examining the XML namespace shown in Figure 1.

Figure 1 XML Namespace Example (Close Figure 1)
<CustomerInfo
    xmlns:o="http://www.example.com/OrderService"
    xmlns:s="http://www.example.com/ShoppingCartService ">

    <o:Orders>
        <o:Order orderNumber="1">
        <!-- other order information would go here -->
        </o:Order>
    </o:Orders>

    <s:ShoppingCart>
        <s:Item>
            <s:Order>1</s:Order>
        </s:Item>
    </s:ShoppingCart>

</CustomerInfo>

In Figure 1, the Order element in a schema created by the company that uses the http://www.example.com/OrderService namespace can live in harmony with the Order element in a schema created by the company that uses the http://www.example.com/ShoppingCartService namespace. The URIs used as XML namespaces are often, but not always, in the form of URLs. However, there is no requirement (and it's very often not the case) that those URLs actually point to a schema file or to any other actual resource.

To keep XML files from being unbearably verbose, one of the criticisms of the XML format in general, namespace prefixes can be used as a shorthand for the actual URI reference that defines a namespace. When the namespace prefix is appended to the local name of an element in a namespace, with the help of a semicolon you get the qualified name for the element. Here's an example:

<namespace prefix> + ":" + <local name> = <qualified name>

"o" + ":" + "Order" = "o:Order"

Similarly, namespaces in the managed world separate classes with identical names. A great example of this is that the Windows® Forms library contains a class called System.Windows.Forms.Button, and the ASP.NET library contains a class called System.Web.UI.WebControls.Button without any problems. Although my example uses classes in different assemblies, it's important for you to keep in mind that it's also possible to have multiple classes with the same name in a single assembly as long as they both reside in different namespaces.

What is a Binding Anyway?

Before I delve into the solution, let's take a moment for a quick refresher on the moving parts that make up a WSDL file. I'll focus on a very simple document-literal Web service that operates over HTTP. There are lots of options, such as SoapHeaders and different transports and various encodings because WSDL is intended to be a very extensible standard. The simple WSDL file will describe a service whose body contains a custom XML document, the document part, defined in the service using XML Schema Definition Language (XSD), the literal part. If you'd like more detailed information on RPC/encoded Web services, WSDL, SOAP, or the other XML standards that make up the Web services stack, pay a visit to the MSDN Web Services Developer Center. Meanwhile, I'll give you a whirlwind tour of service contracts.

WSDL is a standard for describing the wire format of the messages that are carried by SOAP envelopes. A WSDL file is an XML document that acts as a contract for a Web service. It can be used by consumers of that Web service as a guide for creating and validating the XML payloads delivered to and received from that service. Let's take a look at the skeleton of a WSDL file, as shown in Figure 2, to get a better idea of its components. Keep in mind that this isn't a valid service contract. I've simplified it for this example by removing certain attributes and namespace declarations so that I can focus on the most interesting parts.

Figure 2 Simplified WSDL File (Close Figure 2)
<wsdl:definitions
   xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
   xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
   xmlns:xsd="http://www.w3.org/2001/XMLSchema"
   xmlns:my="http://example.com/supersimplewebservice"
   targetNamespace="http://example.com/supersimplewebservice"
   >

   <wsdl:types>
      <xsd:schema>
         <xsd:element name="RequestMessageElement"
            type="xsd:string" />
         <xsd:element name="ResponseMessageElement"
            type="xsd:string" />
      </xsd:schema>
   </wsdl:types>

   <wsdl:message name="RequestMessage">
      <part name="RequestMessagePart"
         element="my:RequestMessageElement" />
   </wsdl:message>

   <wsdl:message name="ResponseMessage">
      <part name="ResponseMessagePart"
         element="my:ResponseMessageElement" />
   </wsdl:message>

   <wsdl:portType name="PortTypeName">
      <wsdl:operation name="GetWeather">
         <wsdl:input message="my:RequestMessage" />
         <wsdl:output message="my:ResponseMessage" />
      </wsdl:operation>
   </wsdl:portType>

   <wsdl:binding name="MyServiceBindingSoap">
      <soap:binding transport="http://schemas.xmlsoap.org/soap/http"
          style="document" />

         <wsdl:operation name="GetWeather">
            <soap:operation soapAction="" style="document" />
            <wsdl:input>
               <soap:body use="literal" />
            </wsdl:input>
            <wsdl:output>
               <soap:body use="literal" />
            </wsdl:output>
         </wsdl:operation>
     </wsdl:binding>
   <wsdl:service name="SuperSimpleService">
      <port name="StockQuotePort" binding="tns:StockQuoteBinding">
        <soap:address
          location="http://example.com/supersimplewebservice.asmx"/>
      </port>
   <wsdl:service>
</wsdl:definitions>

The definitions element is the root element of the WSDL file and contains the rest of the elements that I'll examine. As the root element, it also includes the namespace declarations for the WSDL namespace, the SOAP namespace, the XSD namespace, and any custom namespaces that are used by elements defined in the WSDL file or imported schemas. In the example file, the http://example.com/supersimplewebservice namespace is declared with the "my" namespace prefix.

The next element I'll examine is types, which is similar to an inline XSD. It defines the structure of the XML elements that make up the messages described by the WSDL file, in this case RequestMessageElement and ResponseMessageElement. A WSDL designer also has the option of defining the types in an external schema file, which can be imported into the service contract using import (not shown in the sample file). I use imported schemas with message-focused contract-first design, as you'll see when taking a closer look at the solution.

That brings me to the message element. Each message element describes a distinct message which will be used by the bindings defined in the WSDL file. The message element acts primarily as a container for part elements. Each part has an element attribute that specifies either a base XSD type such as string, float, or one of the elements defined in the types section of the WSDL file. While it's possible to include multiple part elements within a single message, the majority of the WSDL files that you'll encounter will only include a single part.

The portType element acts as a container for a group of operations and the messages that are sent or received during the course of each operation. Operations can be thought of as defined exchanges of messages and are modeled as instances of the operation element. If you think in terms of object-oriented programming, you can also see a vague resemblance between operations and abstract method signatures.

An operation element can have one or zero input, output, or fault elements. If you've ever taken a look at a WSDL file you've probably seen the elements that specify the inward and outward bound messages that an operation accepts, but I've yet to see a fault element in the wild. Although each of the operation's child elements is optional, at least one input or output message must be defined. In practice, most Web services include at least an input message. Each of these child elements contains a message attribute, which specifies the name of one of the message elements defined earlier in the file.

Now let's take a look at the binding element, or more precisely, the wsdl:binding element. All of the elements I've previously discussed have defined the messages in an abstract sense that may or may not directly correspond to the way that the messages are represented as they travel across the wire. The binding defines the concrete format of the messages by detailing the actual format of the messages, the protocol used during the interactions, and the network transport used to exchange the messages. The binding element's type attribute specifies the port type to which it is linked. Its operation child element specifies the particular operation within the port type for which the binding is intended, as well as the input, output, or fault messages to be used.

Each binding also contains extensibility elements, which are used to specify additional information specific to the protocol used by the binding. Since my sample service uses SOAP 1.1 as its protocol, it includes a child soap:binding element. Remember that the two binding elements have the same local name, but are, in fact, different elements because of the namespaces they are in. The soap:binding element's transport attribute from the example service specifies that the service will use HTTP rather than SMTP or TCP/IP as its network transport. The soap:binding element's style attribute declares the default body style used by the operations specified in the wsdl:binding as either document or rpc (in this case, document). The other binding extensibility element in the sample WSDL file is the soap:operation element, which specifies the SOAPAction header expected by services that implement the binding. It also provides the ability to override the default style specified by the soap:binding element.

The final element to look at is service. Each service element corresponds to a specific Web service endpoint and defines the name and set of ports exposed by that endpoint. The service element also contains an extensibility element, which in this case is soap:address. The soap:address element's location attribute specifies the URI of the endpoint itself. When you start building real WSDL files, you may notice that they do not include a service element as they are intended to be reused by several endpoints.

The Design of the Solution

At this point, you have enough of a foundation to look at the technique I described earlier. Yasser Shohoud wrote about message-focused, contract-first design of Web services in his great article "Place XML Message Design Ahead of Schema Planning to Improve Web Service Interoperability" in the December 2002 issue of MSDN®Magazine. In the article, Yasser walks you through the steps of creating a Web service that returns information about the current weather using a message-first design methodology. To summarize, he starts by defining the XML messages that will be exchanged with his service. He then uses that XML structure to define an XML schema and then an abstract service description. That abstract service description is used to create an implementation of the service.

Using message-focused, contract-first design allows the Web service designer to define the exact structure of the messages that will travel over the wire, rather than designing and implementing the object representations and letting the tools decide on the wire level representation. There are probably a large number of Web services in production whose architects have never given a thought to what their services's messages look like on the wire, generally because they have been implemented and tested on a single platform and only consumed through proxies. If those developers were forced to dive below the abstraction provided by their development tools to solve an esoteric issue, or if they needed to make use of the powerful tools available to XML developers, they might suddenly be surprised to confront less than ideal XML representations of their messages.

Imagine you are part of a different development team asked to build a new Web service called CalculateChanceOfRain, which will be complementary to the GetWeather service implemented by Yasser's team. The new Web service will use one of the elements defined in the original service's schema, but it will return a float value representing the likelihood of rain based on the current weather. Your service will be hosted on its own server in a different datacenter, and it will use different back-end systems than the GetWeather service. In fact, you could easily assume that the new service is being developed by a completely standalone partner organization. You'll have no access to the source code or other implementation details for the original service, just the schema and WSDL files. Figure 3 displays the sequence of operations in a single user interaction with the two services.

Figure 3 Overview of a Single User Transaction
Figure 3 Overview of a Single User Transaction

As you can see, the two service implementations form a "virtual" aggregate service from the client's point of view. If you were tasked with designing the new solution, your first thought might be to create a brand new service contract (WSDL file) for the new service that would import the schema defined for the GetWeather service. Unfortunately, using that technique will mean that the current .NET tools will treat the two Web services as completely distinct entities. When Visual Studio® .NET or WSDL.exe is used to create proxies for the two Web services, each proxy will have its own object representation of the "CurrentWeather" message.

The classes will be identical and will be serialized and deserialized into the identical XML structure. In fact, you could use an XmlSerializer to serialize an instance of one class into a document, and then deserialize that same document successfully into an instance of the other class. However, from the runtime's perspective the two classes will be separate managed classes in separate namespaces that just happen to have the same name. That being the case, the object instance returned from a call to the GetWeather proxy object's method can't be passed to the method of the CalculateChanceOfRain proxy. They aren't the same class in the managed world, any more than System.Windows.Forms.Button and System.Web.UI.WebControls.Button are the same class.

This means that you're left with either copying values between instances of the different classes or manually editing the proxy code generated by a tool. Copying the data between fields probably wouldn't make much of a performance difference if your messages aren't huge, but it's definitely an extra maintenance task that I'd prefer to avoid. If you decided to change the proxies manually, you'd have to remove one of the two classes that represent the common XML message, and to recode the proxy whose representation was removed to use the other proxy's class. Remember, those changes would have to be reimplemented each time you created a new Web reference to the services. In my mind, neither of these solutions is particularly ideal because they place the burden of enforcing the conceptual sharing of messages between the services on the client developer.

By creating a single interface WSDL with a binding for each of the implementation services you can gain a level of encapsulation that explicitly defines the heretofore implicit relationship between the two service implementations. Instead of just having a GetWeather service and a CalculateChanceOfRain service, you can expose a virtual WeatherServices service that encompasses both sets of behavior. You also allow the Web service tools you use to create a single class representation of the data types common to both services. Let's take a look at creating an aggregate service description containing bindings for both of the services.

Create WeatherServices WSDL

The first step towards creating a new aggregate service description is to decide on the messages that will be passed to the new CalculateChanceOfRain service. As I've previously discussed, the message passed as input to the new Web service will be the same CurrentWeather element returned by Yasser's GetWeather service. Let's take a look at the sample XML in the original article:

<CurrentWeather xmlns="http://learnxmlws.com/Weather">
    <Conditions>Sunny</Conditions>
    <IconUrl>http://www.LearnXmlws.com/images/sunny.gif</IconUrl>
    <Humidity>0.41</Humidity>
    <Barometer>30.18</Barometer>
    <FahrenheitTemperature>75</FahrenheitTemperature>
    <CelsiusTemperature>23.89</CelsiusTemperature>
</CurrentWeather>

As you can see, the CurrentWeather element contains child elements that include a textual description of the current conditions, the temperature in both Celsius and Fahrenheit, and other pieces of information about the weather in a certain area. The new CalculateChanceOfRain service will ostensibly use this information about the weather to forecast the likelihood of rain in that area. The percent chance of rain will be returned to the caller as a simple element containing a float value. You'll need to create a new schema to define that output message, which you'll call CalculateChanceOfRainResponse. The contents of the new RainServiceMessages schema are shown in the following:

<xs:schema
    xmlns:mstns="http://somecompanydotcom/RainServices"
    xmlns="http://somecompanydotcom/RainServices"
    targetNamespace="http://somecompanydotcom/RainServices"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
    elementFormDefault="qualified">
  <xs:element name="CalculateChanceOfRainResponse" type="xs:float" />
</xs:schema>

Now that you've defined your message, you can create the new aggregate service interface. Rather than creating a new WSDL file from scratch, you're going to modify the WSDL file created by the team responsible for the GetWeather service. Figure 4 shows the contents of the file in its current form.

Figure 4 GetWeather Service WSDL (Close Figure 4)
<?xml version="1.0" encoding="utf-8" ?>
<definitions
  xmlns="http://schemas.xmlsoap.org/wsdl/"
  targetNamespace="http://learnxmlws.com/Weather"
  xmlns:s0="http://learnxmlws.com/Weather"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/">
    <import
      namespace="http://learnxmlws.com/Weather"
      location="ServiceMessages.xsd" />
    <types/>
    <message name="RequestMessage">
      <part name="ZipCode" element="s0:WeatherRequest" />
    </message>
    <message name="ResponseMessage">
      <part name="Weather" element="s0:CurrentWeather" />
    </message>
    <portType name="PortTypeName">
    <operation name="GetWeather">
      <input message="s0:RequestMessage" />
      <output message="s0:ResponseMessage" />
    </operation>
    </portType>
    <binding name="WeatherInterface" type="s0:PortTypeName">
      <soap:binding
        transport="http://schemas.xmlsoap.org/soap/http"
        style="document" />
        <operation name="GetWeather">
          <soap:operation soapAction="" style="document" />
          <input>
            <soap:body use="literal" />
          </input>
          <output>
            <soap:body use="literal" />
          </output>
      </operation>
    </binding>
</definitions>

To create a new aggregate service interface, you need to copy the contents of the WeatherInterface.wsdl to a new file I'll call Aggregate_WeatherInterface.wsdl. Then you need to add the namespace of your new RainServiceMessages schema to the definitions element in the new WSDL file. Then add a request message that uses the CurrentWeather element as its data part, and a response message that uses the new CalculateChanceOfRainResponse element. You also need to add a port and binding for the new service to the WSDL file. Figure 5 shows the completed Aggregate_WeatherInterface service description.

Figure 5 Completed Aggregate WSDL (Close Figure 5)
<?xml version="1.0" encoding="utf-8"?>
<definitions xmlns:http="http://schemas.xmlsoap.org/wsdl/http/"
  xmlns:soap="http://schemas.xmlsoap.org/wsdl/soap/"
  xmlns:i1="http://somecompanydotcom/RainServices"
  xmlns:s="http://www.w3.org/2001/XMLSchema"
  xmlns:soapenc="http://schemas.xmlsoap.org/soap/encoding/"
  xmlns:i0="http://learnxmlws.com/Weather"
  xmlns:tm="http://microsoft.com/wsdl/mime/textMatching/"
  xmlns:mime="http://schemas.xmlsoap.org/wsdl/mime/"
  targetNamespace="http://learnxmlws.com/Weather"
  xmlns="http://schemas.xmlsoap.org/wsdl/">
  <import namespace="http://learnxmlws.com/Weather"
    location="ServiceMessages.xsd" />
  <import namespace="http://somecompanydotcom/RainServices"
    location="RainServiceMessages.xsd" />
  <types />
  <message name="RequestMessage">
    <part name="ZipCode" element="i0:WeatherRequest" />
  </message>
  <message name="ResponseMessage">
    <part name="Weather" element="i0:CurrentWeather" />
  </message>
  <message name="CalculateChanceOfRainRequestMessage">
    <part name="Weather" element="i0:CurrentWeather" />
  </message>
  <message name="CalculateChanceOfRainResponseMessage">
    <part name="Weather" element="i1:CalculateChanceOfRainResponse" />
  </message>
  <portType name="PortTypeName">
    <operation name="GetWeather">
      <input message="i0:RequestMessage" />
      <output message="i0:ResponseMessage" />
    </operation>
  </portType>
  <portType name="CalculateChanceOfRainPortType">
    <operation name="CalculateChanceOfRain">
      <input message="i0:CalculateChanceOfRainRequestMessage" />
      <output message="i0:CalculateChanceOfRainResponseMessage" />
    </operation>
  </portType>
  <binding name="WeatherInterface" type="i0:PortTypeName">
    <soap:binding transport="http://schemas.xmlsoap.org/soap/http"
      style="document" />
    <operation name="GetWeather">
      <soap:operation soapAction="" style="document" />
      <input>
        <soap:body use="literal" />
      </input>
      <output>
        <soap:body use="literal" />
      </output>
    </operation>
  </binding>
  <binding name="CalculateChanceOfRainInterface"
    type="i0:CalculateChanceOfRainPortType">
    <soap:binding transport="http://schemas.xmlsoap.org/soap/http"
      style="document" />
    <operation name="CalculateChanceOfRain">
      <soap:operation soapAction="" style="document" />
      <input>
        <soap:body use="literal" />
      </input>
      <output>
        <soap:body use="literal" />
      </output>
    </operation>
  </binding>
</definitions>

Once it's complete, you'll need to move the WSDL file to a location where it will be accessible to your clients. When you tell client developers about the Web service, or when you publish it using services like UDDI, you'll point users to the location of this static WSDL file instead of using the dynamically generated file created by accessing the ASMX URL with "?WSDL" on your query string. Of course, like all good Web service citizens you've been saving that dynamic file as a static file once you go into production, right? Of course you have. It's not a contract if you're dynamically generating it using reflection. You could also have the dynamically generated WSDL point at your static WSDL using the mechanism that you will find described at Building XML Web Services Using Industry Standardized WSDLs.

Implementing the CalculateChanceOfRain Service

Now that the aggregate service description is complete, you're ready to implement the CalculateChanceOfRain Web service. The first step in that process will be to create an abstract stub for the service implementation using the wsdl.exe tool, just as Yasser did while implementing the GetWeather service. Here's the command line I used to create my CalculateChanceOfRain stub:

wsdl /server /o:RainServiceStub.cs
  http://localhost/WeatherService/aggregate_weatherinterface.wsdl

If you examine the Web service stub file that is generated, you'll notice that the file includes two abstract Web service classes. Each corresponds to one of the bindings included in the aggregate Web service description file. You should manually remove the GetWeather binding implementation from the stub file since you're only using this class to implement your CalculateChanceOfRain Web service, but that's more for aesthetics than any real technical motivation. You'll need to keep the definition of the CalculateWeather class as you don't have access to the source code defining the class in the GetWeather service project in your scenario. Figure 6 shows the modified Web service stub file.

Figure 6 Modified RainServiceStub Code (Close Figure 6)
[System.Web.Services.WebServiceBindingAttribute(
    Name="CalculateChanceOfRainInterface",
    Namespace="http://learnxmlws.com/Weather")]
public abstract class CalculateChanceOfRainInterface :
    System.Web.Services.WebService
{
    [System.Web.Services.WebMethodAttribute()]
    [System.Web.Services.Protocols.SoapDocumentMethodAttribute("",
         Use=System.Web.Services.Description.SoapBindingUse.Literal,
         ParameterStyle=
             System.Web.Services.Protocols.SoapParameterStyle.Bare)]
    [return: System.Xml.Serialization.XmlElementAttribute(
         "CalculateChanceOfRainResponse",
         Namespace="http://somecompanydotcom/RainServices")]
    public abstract System.Single CalculateChanceOfRain(
        [System.Xml.Serialization.XmlElementAttribute(
            Namespace="http://learnxmlws.com/Weather")]
        CurrentWeather CurrentWeather);
}

[System.Xml.Serialization.XmlTypeAttribute(
    Namespace="http://learnxmlws.com/Weather")]
public class CurrentWeather
{
    public string Conditions;
    public string IconUrl;
    public System.Single Humidity;
    public System.Single Barometer;
    public System.Single FahrenheitTemperature;
    public System.Single CelsiusTemperature;
}

Now you need to create a service implementation that will derive from the Web service stub class. The first step is to create a new Web service project. The project in the sample code included with this article is called "RainService." Add a Web service named MyRainService to the project, and open its codebehind file. Figure 7 shows the code in the MyRainService Web service codebehind file after it has been modified.

Figure 7 Completed MyRainService Code (Close Figure 7)
[WebService(Namespace = "http://somecompanydotcom/RainServices")]
[WebServiceBinding(Name="CalculateChanceOfRainInterface",
     Namespace="http://learnxmlws.com/Weather",
     Location="http://localhost/WeatherService/
        aggregate_weatherinterface.wsdl")]
public class MyRainService : CalculateChanceOfRainInterface
{
    public MyRainService()
    {
        // This call is required by the ASP.NET Web Services Designer
        InitializeComponent();
    }

    [WebMethod]
    [SoapDocumentMethod("", ParameterStyle=SoapParameterStyle.Bare)]
    [return: XmlElement ("CalculateChanceOfRainResponse",
        Namespace="http://somecompanydotcom/RainServices")]
    public override float CalculateChanceOfRain(
        [System.Xml.Serialization.XmlElementAttribute(
            Namespace="http://learnxmlws.com/Weather")]
        CurrentWeather CurrentWeather)
    {
        float result = 0;

        if (CurrentWeather.Conditions == "Rainy") result = 100;
        else
        {
            Random rnd = new Random(DateTime.Now.Millisecond);
            result = (float)rnd.Next(101);
        }

        return result;
    }
}

Notice that I've applied a WebServiceAttribute to the class in order to set the namespace that will be used by the service. This isn't strictly necessary, as the ASP.NET Web service infrastructure will default to using the tempuri.org namespace, but it's a good idea for a production Web service. I've also applied a WebServiceBindingAttribute to the class to indicate that the binding for the Web service is defined in the aggregate service description that was created earlier. This is the crucial attribute; it specifies that the service is going to use the CalculateChanceOfRainInterface binding defined in the aggregate WSDL file.

Given my complete lack of knowledge when it comes to meteorology as well as my lack of friends named Storm or Chase, the implementation of the CalculateChanceOfRain method simply returns a random float value between zero and one hundred. More accurate weather forecasting logic is left as an exercise for the reader. That said, the success rate of this algorithm would probably compare favorably with that of many of the local weather forecasters I've watched on TV.

Implementing the GetWeather Service

Since your scenario is fictional, you'll need to create a sample implementation of the GetWeather service in order to test your solution. The steps are almost identical to those you followed to implement the previous Web service, so I won't cover them in detail again. Suffice it to say that you'll again use the WSDL.EXE tool to generate a Web service stub and then create a new class file to hold the service implementation. Figure 8 shows the completed implementation of the Web service.

Figure 8 Completed WeatherService Code (Close Figure 8)
[WebService(Namespace = "http://learnxmlws.com/WeatherService")]
[WebServiceBinding(Name="WeatherInterface",
     Namespace="http://learnxmlws.com/Weather",
     Location=
         "http://localhost/WeatherService/WeatherInterface.wsdl")]
public class WeatherService : WeatherInterface
{
    [WebMethod]
    [SoapDocumentMethod("",Binding="WeatherInterface",
        ParameterStyle=SoapParameterStyle.Bare)]
    [return: XmlElement("CurrentWeather",
        Namespace="http://learnxmlws.com/Weather")]
    public override CurrentWeather GetWeather(
        [System.Xml.Serialization.XmlElementAttribute(
            Namespace="http://learnxmlws.com/Weather")]
        string WeatherRequest)
    {
        //ignore the zipcode specified
        Random rnd = new Random();

        CurrentWeather result = new CurrentWeather();
        result.Barometer = (float)rnd.Next(101);
        result.CelsiusTemperature = (float)rnd.Next(78);
        result.Conditions = (rnd.Next(2) == 1) ? "Rainy" : "Sunny";
        result.Humidity = (float)rnd.Next(101);

        return result;
    }
}

As I've said, I have very little knowledge of the mysteries of meteorology, and I don't have access to weather-detecting equipment. In lieu of a contract with a weather information provider, myGetWeather method will ignore the zip code specified by the caller. Instead, the service returns a CurrentWeather instance filled with random values. It isn't important since the CalculateChanceOfRain service ignores the message sent to it anyway.

The WeatherServices Client App

At this point, you're ready to test your new aggregate Web service. To do so, create a simple Windows Forms application called WeatherClient which will act as a client for the Web services. It will look like the form in Figure 3 (without the background image).

Now, let's take a look at what's necessary for the implementation. First, you need to add some controls to your main form to allow the user to interact with the application. Add a textbox to the main form and name it "Zipcode," then add a button to the form, and name it "SubmitButton." Finally, add a label to display the results of your calculation and name it "DisplayLabel."

Next, use Visual Studio .NET to add a reference to the aggregate service description. Remember to add the reference to the static WSDL file that was created, not to the ASMX file's dynamically created WSDL. The aggregate service description doesn't include service elements that specify the endpoints for the service implementations, so you'll have to manually set the URL properties of your proxy instances before invoking the Web services.

Like many .NET XML Web service developers, the first thing I do after adding a Web reference is set the Web reference's URL Behavior property to Dynamic. When this property is set, the proxy class determines the URL of the Web service endpoint by examining a setting in the application's configuration file so you don't have to have the URL hardcoded into the class itself. Unfortunately, this doesn't work with the multibinding file you've created, so you'll have to create the configuration entries manually and set the URL properties for your proxy instances.

Now you're ready to add the client code to the application. Figure 9 shows the important section of the code in the form.

Figure 9 Client Code (Close Figure 9)
private void SubmitButton_Click(object sender, System.EventArgs e)
{
    //let the user know we're working
    this.Cursor = Cursors.WaitCursor;
    updateStatus("Getting weather!", false);

    //call the GetWeather method asynchronously
    weatherservices.WeatherInterface weatherService =
        new WeatherClient.weatherservices.WeatherInterface();

    weatherService.Url =
        ConfigurationSettings.AppSettings["weatherServiceUrl"];

    weatherService.BeginGetWeather(ZipCode.Text,
        new AsyncCallback(this.getWeatherResult), weatherService);
}

private void getWeatherResult(IAsyncResult ar)
{
    //we're returned from our call to GetWeather!
    updateStatus("Getting chance of rain!", false);

    //let's get the result of our call to GetWeather
    weatherservices.WeatherInterface weatherService =
       (weatherservices.WeatherInterface)ar.AsyncState;

    weatherservices.CurrentWeather currentWeather =
       weatherService.EndGetWeather(ar);

    //now we need to call the GetWeather method asynchronously
    weatherservices.CalculateChanceOfRainInterface rainCalculator =
       new WeatherClient.weatherservices.CalculateChanceOfRainInterface();

    rainCalculator.Url =
       ConfigurationSettings.AppSettings["rainServiceUrl"];

    rainCalculator.BeginCalculateChanceOfRain(currentWeather,
       new AsyncCallback(this.getCalculateChanceOfRainResult),
       rainCalculator);
}

private void getCalculateChanceOfRainResult(IAsyncResult ar)
{
    weatherservices.CalculateChanceOfRainInterface rainCalculator =
        (weatherservices.CalculateChanceOfRainInterface)ar.AsyncState;

    float chanceOfRain = rainCalculator.EndCalculateChanceOfRain(ar);

    string resultMessage =
        string.Format("There is a {0}% chance of rain.", chanceOfRain);

    updateStatus(resultMessage, true);
}

// a delegate that is used by updateStatus to invoke itself recursively
private delegate void UpdateStatusDelegate(
    string textToDisplay, bool changeCursorToDefault);

private void updateStatus(string text, bool changeCursorToDefault)
{
    if (DisplayLabel.InvokeRequired)
    {
        DisplayLabel.Invoke(new UpdateStatusDelegate(this.updateStatus),
            new object[] { text, changeCursorToDefault});
        return;
    }

    DisplayLabel.Visible = true;
    DisplayLabel.Text = text;
    if (changeCursorToDefault) this.Cursor = Cursors.Default;
}

In the event handler for the button's Click event, initiate a call to the GetWeather service using the Web service proxy class. In order to keep the user interface responsive, use the Web service proxy's implementation of the Asynchronous Method Invocation Design Pattern. This causes the Web service request to execute on a thread from the ThreadPool, rather than on the user interface thread which needs to handle posted windows messages in order to keep the UI responsive. The getWeatherResult method is specified as the AsyncCallback delegate for your asynchronous call to the GetWeather method. The proxy is the third parameter.

When the GetWeather Web service call is completed, the getWeatherResult method executes. The getWeatherResult method converts the AsyncState property of the IAsyncResult object provided as its argument into an instance of CurrentWeather. It then uses the CurrentWeather instance as the argument for a call to the CalculateChanceOfRain Web service. The getCalculateChanceOfRainResult method is specified as the AsyncCallback delegate for your asynchronous call to the GetWeather method.

When the getCalculateChanceOfRainResult method executes, the updateStatus method is called to display the chance of rain to the user by updating the Text property of the DisplayLabel label. As you are probably aware, Windows Forms controls are not thread safe and their methods should only be called from the user interface thread. When the asynchronous responses from the Web services are received on a ThreadPool worker thread, the DisplayLabel's InvokeRequired property will be true. The method then invokes itself recursively through DisplayLabel.Invoke so that the assignment of the label's text property occurs on the user interface thread.

Type Sharing in the .NET Framework 2.0

You'll be happy to know that the type sharing story is greatly improved in the forthcoming release of the Microsoft® .NET Framework 2.0. The next version of the WSDL.EXE tool makes type sharing between services simpler with the addition of the sharetypes command-line argument. By providing multiple WSDL URLs to the tool, and by adding the sharetypes flag to your arguments, you signal to the tool that you would like any common types to be shared between the generated proxies. Common types are elements with the same local name and namespace, defined in the same schema file. Here's an example command line that could be used to generate a Web service client stub that can be used to call two Web services that share types:

wsdl /sharetypes http://localhost/service1.wsdl
      http://localhost/service2.wsdl

The created file will contain classes for each Web service's binding, and a single class for each of the types that are shared between the services. In fact, the created file is almost identical to the file created when a developer using the .NET Framework 1.x adds a reference to the aggregate WSDL file. Obviously, the Web services team paid attention to the Web service development community which has been clamoring for this new feature.

If you're testing the .NET Framework Beta 1 release, you may notice that the new functionality isn't currently exposed in the development environment. Rest assured, the same capability will be available from Visual Studio 2005 in later betas.

There is a caveat if you are considering making use of WSDL files with multiple bindings, and your services also make use of SoapHeaders. There is a known issue with client proxy generation in that specific scenario where only the first binding's SoapHeaderAttribute parameters are correct. A simple work around is explained in the Knowledge Base article "BUG: Incorrect Proxy Is Generated When WSDL File Contains Multiple Bindings". I haven't touched on SoapHeaders in this article, but you should be aware of the issue in case you stumble into it on one of your future projects.

Conclusion

Obviously, the encapsulation accomplished here is leaky. The client application is well aware that there are two separate bindings as there are two different proxy classes. They are also well aware that the two services live in separate locations, since the client configures the URLs of the endpoints used to access the services. At the same time, there is nothing to indicate to the client application that the two services were implemented separately, at different times, by different teams. You also have the benefit of being able to share types between the two Web services in the client code, which eliminates a lot of copying of data from one structure to another identical structure in a different namespace.

Due to the incredible ease of creating and consuming XML Web services using tools available in the .NET Framework, many developers and architects never take the time to familiarize themselves with the protocols that underlie the Web services stack. It couldn't be easier to write a normal method, apply a WebMethodAttribute to it, and work exclusively with autogenerated proxy objects. However, having an understanding of the underlying protocols, and how the tools work with those protocols, can expand the options available to you as you design your solutions. Hopefully this article will inspire you to dedicate some time to looking behind the curtains, even if you're already focused on using the sharetypes functionality in the .NET Framework 2.0.

Gerrard Lindsay is an application development consultant with Microsoft Services in the East Region. This is his first article, so please make sure all flames are written in simple language. Gerrard can be contacted at glindsay@microsoft.com.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


What is BPEL4WS?: Build Better Business Processes with Web Services in BizTalk Server 2004

What is BPEL4WS?

Build Better Business Processes with Web Services in BizTalk Server 2004

Jon Fancey
This article discusses:
  • Easier service orientation with BizTalk Server 2004
  • Orchestrating Web services as business processes
  • Business Process Execution Language for Web Services for flexible orchestration
This article uses the following technologies:
XML, Web Services, BizTalk Server 2004
Code download available at: BPEL4WS.exe (158KB)
Contents
From Object-Orientation to Service-Orientation
The New Lingua Franca
Boundaries
Conclusion

T
he potential of the service-oriented enterprise is huge. Service-oriented architecture (SOA) will simplify the building, managing, and maintaining of distributed systems, in part because the technologies you will use to build these new systems are standards-based, thanks to the collaborative efforts of companies like IBM BEA, and Microsoft. Working together has enabled them to establish many Web services specifications, including WS-Security, WS-Addressing, and WS-Policy.

In this article I will focus on one Web services specification that is critical, yet has been largely overlooked: the Business Process Execution Language for Web Services (BPEL4WS, or BPEL).

I will describe why BPEL is important and what it offers you if you are implementing Web services today or planning to in the future. I'll also present concrete examples using BizTalk® Server 2004. I'll discuss the various advantages of BPEL throughout the article, but for a quick look at some of the most important ones, see the sidebar "Benefits of BPEL." For background on all the great benefits of SOA, take a look at Aaron Skonnard's Service Station column in the February issue of MSDN®Magazine.

From Object-Orientation to Service-Orientation

With the ability to easily integrate with and invoke Web services comes the problem of the coordination and management of messages as they are routed from service to service. As time passes, it will become increasingly unlikely that any interesting business processes will execute in a single message exchange. Traditionally, implementing business logic meant writing code (C++ or C#, for example). With BizTalk Server, however, many common requirements can be realized without coding a single line. It seems reasonable to code manually when dealing with a few services, but this becomes more problematic as the number and complexity of services increases. Enter BPEL—the first step towards abstracting the collaboration and sequencing logic out of platform-dependent code and into a formal process definition based on XML, WSDL, and XML Schema.

Successful SOA implementations require reusable logic and service autonomy. In order to achieve them, you need to start thinking of applications as the collaboration of message exchanges. Of course, when you begin exposing application functionality as services, you relinquish a certain amount of control because you no longer know how your services are being employed or even who is using them. I'll take a look at how you can come to grips with that fact later on in this article.

If I were to describe business orchestration in musical terms, the orchestra would be made up of Web services, BPEL would be the sheet music, and the composer would be the coordinator or business process manager (BPM). Designing an application architecture that fits this description using ubiquitous standards in such a way that the specifications themselves do not intrude on the process is the great challenge in Web services today. BizTalk Server is the solution to that problem. Let's look at how BizTalk Server uses Web services, BPEL, and Web Services Enhancements (WSE) as parts of a BPM solution.

The New Lingua Franca

As I mentioned, BPEL is one of a number of technologies that provides a rich Web services infrastructure. Each of the specifications (some still waiting for vendor implementations and consequently widespread adoption) addresses a different area of concern. BPEL was submitted to OASIS last year, where it is currently undergoing finalization (see www.oasis-open.org). However, you can take advantage of a growing number of major specifications right now in your orchestrations and Web services by using WSE and BizTalk Server in tandem.

BPEL gives Web services architects the kind of flexibility that the open database connectivity standard (ODBC) gave database architects. That standard enabled uniform data access regardless of the actual database used. BPEL also enables a process to be defined in one environment and executed in another. Another advantage is that it clarifies business processes and enables the designers of the business processes to define the orchestrations themselves (more on this shortly).

There are a couple of different methods by which the BPEL specification can be implemented. For example, you could run the BPEL through an interpreter the way a scripting language is executed, or you could convert or compile it into a more efficient format prior to execution.

BizTalk Server takes the second approach and realizes a number of benefits from doing so. BizTalk Server enables the import of BPEL 1.1-compliant processes and converts them to XLANG/s, the Microsoft process definition language. XLANG/s has been supported in all previous versions of BizTalk Server, thus it provides important backward compatibility. XLANG/s also provides many of the common facilities required to implement processes such as mapping, conversion, and transformation functions. Currently the BPEL standard only allows such functionality to be exposed as additional services. This can incur performance overhead without runtime optimization.

BizTalk Server goes even further than merely converting to XLANG/s by actually compiling the converted XLANG/s into .NET assemblies prior to execution, resulting in a five-fold performance improvement over the previous version of BizTalk Server.

Perhaps the main reason BPEL has received less attention than other specifications is that most vendors (including Microsoft) do not expect it to be created the same way you create WSDL or XSD—using an XML editor. Instead, tooling in BizTalk Server provides a rich graphical process editor to create orchestrations—the Orchestration Designer.

Support for interoperability in BizTalk Server enables BPEL documents to be imported and turned into orchestration (.odx) files. An orchestration can then be viewed and edited graphically in the designer and even exported back to BPEL if required. This allows companies or business units to share a business process quickly and easily, even if they run on completely different hardware and operating systems. This can make enterprises much more agile, particularly where changing a process could result in taking advantage of a brand new market opportunity.

By focusing on interoperability, BizTalk Server embraces BPEL without being constrained by it. Consider, for example, that the major processing requirement in messaging systems is transformation—the conversion from one format or structure to another. BPEL can't do this in the standard, but BizTalk Server can.

To give you an idea of what's going on behind the scenes, take a look at the code in Figure 1. It's important to bear in mind that BPEL is a substrate that binds pre-existing Web services together and is not intended to be used to implement the services themselves. This is one of the reasons that the feature set in BPEL is constrained to simple expressions and constructs. BPEL allows you to form new Web services through recombination and enables the assembly of complex business processes.

Figure 1 Sample BPEL Process—Adding Two Numbers (Close Figure 1)
<?xml version="1.0"?>
<process xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/"
    xmlns:q2="http://tempuri.org/Arithmetic"
    xmlns:q1="http://tempuri.org"
    xmlns:xsd="http://www.w3.org/2001/XMLSchema" name="Example1.Add"
    targetNamespace="http://tempuri.org"
    xmlns="http://schemas.xmlsoap.org/ws/2003/03/business-process/">
    <partnerLinks>
        <partnerLink name="Arithmetic"
            partnerLinkType="q1:ArithmeticType" myRole="portRole" />
        <partnerLink name="InternalArithmeticService"
            partnerLinkType="q1:InternalArithmeticServiceType"
            partnerRole="portRole" />
    </partnerLinks>
    <variables>
        <variable name="AddRequest" messageType=
            "q1:__messagetype_Example1_ArithmeticSchema_AddMessageIn"/>
        <variable name="AddResponse" messageType=
            "q1:__messagetype_Example1_ArithmeticSchema_AddMessageOut"/>
        <variable name="InternalAddRequest" messageType="q2:AddSoapIn" />
        <variable name="InternalAddResponse"
            messageType="q2:AddSoapOut" />
     </variables>
     <sequence>
         <receive partnerLink="Arithmetic" portType="q1:ArithmeticPT"
             operation="AddTwoNumbers" variable="AddRequest"
             createInstance="yes" />
         <assign>
             <copy>
                 <from variable="AddRequest"
                     part="part" query="/*[local-name()='AddMessageIn' and
                         namespace-uri()='http://Example1
                         .Arithmetic']/*[local-name()='X' and
                         namespace-uri()='']" />
                 <to variable="InternalAddRequest" part="A"
                     query="/*[local-name()='int' and
                         namespace-uri()='http://tempuri.org']" />
             </copy>
             <copy>
                 <from variable="AddRequest" part="part"
                     query="/*[local-name()='AddMessageIn' and namespace-
                         uri()='http://Example1
                         .Arithmetic']/* [local-name()='Y' and
                         namespace-uri()='']" />
                 <to variable="InternalAddRequest" part="B"
                     query="/*[local-name()='int' and namespace-
                         uri()='http://tempuri.org']" />
             </copy>
         </assign>
         <invoke partnerLink="InternalArithmeticService"
             portType="q2:AddSoap" operation="Add"
             inputVariable="InternalAddRequest"
             outputVariable="InternalAddResponse" />
         <assign>
             <copy>
                 <from variable="InternalAddResponse" part="AddResult" />
                 <to variable="AddResponse" part="part" query="/*[local-
                     name()='AddMessageOut' and namespace-
                     uri()='http://Example1
                     .Arithmetic']/*[local-
                     name()='Z' and namespace-uri()='']" />
             </copy>
         </assign>
         <reply partnerLink="Arithmetic" portType="q1:ArithmeticPT"
             operation="AddTwoNumbers" variable="AddResponse" />
     </sequence>
</process>
Figure 2 Relationship Between BPEL, WSDL, and XML Schema
Figure 2 Relationship Between BPEL, WSDL, and XML Schema

A BPEL definition generally requires two other document types, WSDL and XML schema, and BizTalk Server orchestrations are no exception to this. Figure 2 shows the relationship between them. BPEL extends WSDL to both provide and consume Web services in an abstract way. This enables one of the great strengths of WSDL, namely the ability to separate the abstract message and port information required at composition time from the physical binding and address details required at invocation time. The WSDL and XSD for this example are given in Figure 3 and Figure 4.

Figure 4 XSD for Adding Two Numbers (Close Figure 4)
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://Example1.Arithmetic"
  xmlns:b="http://schemas.microsoft.com/BizTalk/2003"
  targetNamespace="http://Example1.Arithmetic"
  xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="AddMessageIn">
    <xs:annotation>
      <xs:appinfo>
        <b:properties xmlns:b="http://schemas.microsoft.com/BizTalk/
            2003">
          <b:property distinguished="true" xpath="/*[local-
            name()='AddMessageIn' and namespace-
            uri()='http://Example1.Arithmetic']/*[local-
            name()='X' and namespace-uri()='']" />
          <b:property distinguished="true" xpath="/*[local-
            name()='AddMessageIn' and namespace-
            uri()='http://Example1.Arithmetic']/*[local-
            name()='Y' and namespace-uri()='']" />
        </b:properties>
      </xs:appinfo>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:element name="X" type="xs:int" />
        <xs:element name="Y" type="xs:int" />
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="AddMessageOut">
    <xs:annotation>
      <xs:appinfo>
        <b:properties>
          <b:property distinguished="true" xpath="/*[local-
            name()='AddMessageOut' and namespace-
            uri()='http://Example1.Arithmetic']/*[local-
            name()='Z' and namespace-uri()='']" />
        </b:properties>
      </xs:appinfo>
    </xs:annotation>
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Z" type="xs:integer" />
      </xs:sequence>
    </xs:complexType>
Figure 3 WSDL for Adding Two Numbers (Close Figure 3)
<?xml version="1.0"?>
<definitions xmlns:tns="http://tempuri.org"
  xmlns:xsd="http://www.w3.org/2001/XMLSchema"
  xmlns:q2="http://Example1.Arithmetic"
  xmlns:q1="http://tempuri.org/WebService2/Service1"
  xmlns:bpel="http://schemas.xmlsoap.org/ws/2003/03/business-process/"
  xmlns:plnk="http://schemas.xmlsoap.org/ws/2003/05/partner-link/"
  targetNamespace="http://tempuri.org" name="Example1"
  xmlns="http://schemas.xmlsoap.org/wsdl/">
  <import namespace="http://Example1.Arithmetic"
    location="Arithmetic.xsd" />
  <types>
  <xsd:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
    xmlns="http://Example1.Arithmetic"
    xmlns:b="http://schemas.microsoft.com/BizTalk/2003"
    targetNamespace="http://Example1.Arithmetic">
    <xs:element name="AddMessageIn">
      <xs:annotation>
        <xs:appinfo>
          <b:properties xmlns:b=
            "http://schemas.microsoft.com/BizTalk/2003">
            <b:property distinguished="true" xpath=
              "/*[local-name()='AddMessageIn' and namespace-
              uri()='http://Example1.
              Arithmetic']/*[local-name()='X' and namespace-
              uri()='']" />
            <b:property distinguished="true" xpath="/*[local-
              name()='AddMessageIn' and namespace-
              uri()='http://Example1.
              Arithmetic']/*[local-name()='Y' and
              namespace-uri()='']" />
          </b:properties>
        </xs:appinfo>
      </xs:annotation>
      <xs:complexType>
        <xs:sequence>
          <xs:element name="X" type="xs:int" />
          <xs:element name="Y" type="xs:int" />
        </xs:sequence>
      </xs:complexType>
    </xs:element>
    <xs:element name="AddMessageOut">
      <xs:annotation>
        <xs:appinfo>
          <b:properties>
            <b:property distinguished="true" xpath="/*[local-
              name()='AddMessageOut' and namespace-
              uri()='http://Example1.
              Arithmetic']/*[local-name()='Z' and
              namespace-uri()='']" />
          </b:properties>
        </xs:appinfo>
      </xs:annotation>
      <xs:complexType>
        <xs:sequence>
          <xs:element name="Z" type="xs:integer" />
        </xs:sequence>
      </xs:complexType>
    </xs:element>
  </xsd:schema>
  </types>
  <message name="__messagetype_Example1_ArithmeticSchema_AddMessageIn">
    <part name="part" element="q2:AddMessageIn" />
  </message>
  <message name="__messagetype_Example1_ArithmeticSchema_AddMessageOut">
    <part name="part" element="q2:AddMessageOut" />
  </message>
  <portType name="ArithmeticPT">
    <operation name="AddTwoNumbers">
      <input message=
        "tns:__messagetype_Example1_ArithmeticSchema_AddMessageIn" />
      <output message=
        "tns:__messagetype_Example1_ArithmeticSchema_AddMessageOut" />
    </operation>
  </portType>
  <plnk:partnerLinkType name="ArithmeticType">
    <plnk:role name="portRole">
      <plnk:portType name="tns:ArithmeticPT" />
    </plnk:role>
  </plnk:partnerLinkType>
  <plnk:partnerLinkType name="InternalArithmeticServiceType">
    <plnk:role name="portRole">
      <plnk:portType name="q1:Service1Soap" />
    </plnk:role>
  </plnk:partnerLinkType>
</definitions>

A cursory examination of the BPEL vocabulary in Figure 1 reveals many familiar concepts such as scoped variables and type-compatible assignments. BPEL also provides logical constructs such as conditionals, loops, and parallel processing. The BPEL in Figure 1 needs little explanation. The basic constructs are termed activities in BPEL, the full list for which is provided in Figure 5. Because of the flexibility an XML-based representation provides, activities can be combined and nested in interesting ways. This process is particularly suited to a graphical design environment.

Figure 5 BPEL Activities (Close Figure 5)

BPEL Activity | BizTalk Server 2004 Shape
<receive> | Receive
<reply > | Send
<sequence> | [Default flow]
<assign> | Message Assignment
<while> | Loop
<switch > | Decide
<invoke> | Send/Receive
<scope> | Scope
<flow> | Parallel Actions
<pick> | Listen
<empty> | {Not implemented}
<throw> | Throw Exception
<compensate> | Compensate
<terminate> | Terminate
<wait> | Delay/Suspend

Many BPEL features relate to timing and synchronization. The loosely coupled, highly latent world of Web services requires a much more casual attitude to sequence and time than do regular applications. To see what I mean, think about how an orchestration runs. Waiting for a message is a passive activity and something in the orchestration will almost certainly have to wait for the appropriate message to appear. BizTalk Server deals with messages by implementing subscriptions. An orchestration can subscribe to a particular message type, and all messages received go into a persistent store called the message box, implemented using SQL Server™. The message box is examined and the subscribed orchestrations are executed by the runtime. This results in orchestration instances being created, each running with its own state. This is important for reasons I'll discuss shortly.Benefits of BPEL

  • Cross-vendor support
  • Enables process interoperability
  • XML-based language
  • Powerful control flows allowing arbitrary nesting
  • Currently (BPEL 1.1) based on WSDL 1.1, XML Schema 1.0, XPath 1.0 and WS-Addressing
  • Sync and async messaging
  • Uses callbacks and correlation sets to compose services
  • Transactions
  • Atomic and long-running support using correlation sets
  • Exception handling
  • Compensation and fault processing

The next activity I want to discuss is the invoke activity. It enables an orchestration to call a Web service to help fulfill the implementation of the process. This functionality is present in BizTalk Server with the Send and Receive shapes. In the example in Figure 1, I am simply invoking a Web service that will add two numbers and return the sum. The expected WSDL definitions are referenced to provide the necessary binding and context. This case is a little contrived, but it does illustrate the ability to decouple externally exposed services from changes to internally defined services.

For example, this internal service could be replaced with another one or the schema changed to alter the message structure without affecting the caller in any way. Such encapsulation is important in order to achieve reuse of existing services as components of a larger orchestration. A public Web service for a bank that enables the transfer of funds between accounts will consist of internal invocation and collaboration of a number of different systems internally. Exposing these internal systems APIs to the outside world often leads to brittleness and can be highly insecure.

Instead, an orchestration can be used to provide a facade to bridge the outside world and your internal systems in a loosely coupled way. This is a common requirement in enterprise application integration (EAI) scenarios and typically employs content-based routing. Bear in mind that orchestrations in BizTalk Server do not have to be exposed as Web services. They can be run by a variety of means using different configurations. An orchestration's trigger may be an XML document dropped via FTP into a watched file system folder, for example.

It should be apparent by now that a messaging system has a large number of requirements that are not directly related to the execution of any one particular service. Because these requirements cut across all services, they are driving the creation of BPM software infrastructure. BizTalk Server is both a BPM and integration product. It consists of around 1.8 million lines of managed code. This gives a good indication of the size of a BPM application—you don't want to start writing one yourself.

If you're going to build business processes using existing services, you need to have those services in place first. But that's not the reality for most companies today, so it's critical to make your services generic and reusable, rather than services targeted to a particular consumer. Adopting an interface-first design approach makes this much easier to achieve.

The visibility of the actual running business processes represents a huge step forward and with BizTalk Server you can create a new process by linking together available "shapes" in the graphical editor. These shapes map naturally to BPEL activities already described and allow a process to be modeled by dragging and dropping these activities onto the designer surface. Developers can do this using Visual Studio® .NET 2003, and business analysts can use Visio® to design orchestrations. As you can see in Figure 6, the receive shape (labeled ReceiveMessage) is synonymous with the BPEL Receive activity. Similarly, the send shape (labeled ReplyMessage) maps to the Reply activity in BPEL. The goal is that each type of user has the best tools for their own familiar environment yet they share the same artifacts.

Figure 6 Orchestration in BizTalk Server 2004
Figure 6 Orchestration in BizTalk Server 2004

Now let's consider a more complex scenario. One of the assumptions made explicit with distributed service-based systems is that they will occasionally fail. By their nature, timeouts can occur due to network latency or the endpoint being unavailable, for example. Messages may not be delivered, may be delivered multiple times, or may be delivered in an unexpected order. These characteristics make transactions problematic as the traditional approach of using atomicity to guarantee consistency and isolation through a two-phase commit scheme (such as that provided by the Microsoft Distributed Transaction Coordinator, or MS DTC) cannot be employed. With the WS-Transaction specification there will be a comparable solution, but even then there are many cases when transactions spanning organizational boundaries are not appropriate. A different approach taken in BPEL is based on the notion of compensation.

Compensation enables long-running transactions (LRTs) to be managed so that if failure occurs, a defined compensatory action will be invoked. One example is an airline reservation system. In booking a ticket, you may have to provide payment before your seat on the flight can be confirmed. This process could take several seconds. If the seat cannot be confirmed, the payment must be refunded so that it appears that the transaction never took place. LRTs are the mechanism by which Web services can achieve this state. Because of this, the term LRT is somewhat of a misnomer as it isn't really a transaction at all. Each action and compensatory action taken is distinct from the others. BizTalk Server implements this scheme and logically ties this unrelated activity coordination together to manage it and attempt to prevent inconsistencies. Note, though, that this is not a fail-safe mechanism because it does not preclude the possibility that inconsistencies could occur. It is also important to realize that this is not a failing of BizTalk Server itself; rather it is due to the pragmatic approach taken in dealing with failures within a loosely coupled distributed system.

Another problem Web services introduce is the lack of shared knowledge or state. This should be viewed as a strength because it reduces any dependency or unnecessary knowledge by one component of another. It does, however, require a different approach to solving a perennial problem. For example, an LRT may consist of a fire-and-forget style of asynchronous message being sent. It may be expected, though, that at some point in the future the service to which the message was directed will respond in a callback-style pattern or "solicit-response." When this occurs, the original orchestration must correlate the inbound message with the original outbound one. BPEL provides support for this with correlation sets. Correlation sets in BizTalk Server are based on the notion of specific process instances, as I mentioned earlier.

You must mark an orchestration in BizTalk Server as able to be activated in order to indicate that it can be the start of a new process instantiation. From that point on, if an activity is executed that causes the orchestration to suspend execution, it can be resumed later when the callback comes in. BizTalk Server supports this BPEL feature by allowing the receipt of a message and correlating it back to the original in-flight suspended orchestration instance. As the callback message is likely to differ from the original one that initiated the orchestration, BizTalk Server also provides the facility to alias correlation information. This allows unique data items to be identified across different message types in order for the BizTalk Server engine to associate them. This is achieved through property promotion, that is, the process of annotating the schema type being used with the relationships between different elements or complex types with the same meaning.

For example, a customer ID may be present in all messages but is unique to a particular customer. Because of this uniqueness property, it can be used to correlate different messages to a single orchestration instance even though in one message type it may be called CustID while in another message type, it may refer to the same entity called Customer.

BPEL contains other examples of synchronization activities all supported in BizTalk Server. One of the more interesting is the pick activity, which BizTalk Server provides via the Listen shape (see Figure 5). This activity allows the selection of a logic path based on the message type received. An orchestration that can receive one of several different messages can use this activity to wait until the first message is received and continue down a particular path based on its type. This is useful when you are dealing with a number of partners using different message structures but the semantics of the actual business process being created are the same for each.

Boundaries

As I've said, BPEL should be viewed purely as an orchestration language, and this is where the interoperability benefits lie. Because of this, BPEL imposes some restrictions on the expressions allowed in conditional activities and assignments. In order to discourage its use in implementing the actual business logic of a service the specification actually mandates that only simple expressions are allowed. This is because its stated purpose is to control the execution of the business logic components, namely the Web services.

Consequently, a BPEL orchestration cannot accomplish much by itself. A BPEL orchestration cannot actually implement a business process. Instead, it acts as a coordinator by passing messages from system to system, employing other services along the way. The orchestration only knows how to call, consume, and manage the calling of the other Web services and nothing more. BPEL is also just a specification, not a product.

For it to be useful, it has to exist as an implementation on a specific platform and that means implementation-specific code will be written to integrate with and extend it. This is likely to change as the specification process continues on to recommendation status, with the possibility that concessions will be made to provide for some of the common requirements in a standard, interoperable way. These current limitations make it possible to define orchestrations in BizTalk Server that cannot be expressed in BPEL. Although this is unavoidable, the extensibility BPEL allows will likely lead to greater improvements in this area over the next few years, and in particular, will lead to greater interoperability.

Conclusion

BPEL is first and foremost an orchestration language. It is enjoying widespread industry support and adoption by providing a full-feature set to define business process interactions. BPEL's major promise is that the creation of abstract and executable schemes can be defined as business process and run in any compliant engine. While today a single organization may not need to run BPM engines from multiple vendors, many organizations are moving towards implementing connected business processes across extended enterprises. In these situations especially, the ability to specify those aspects of a shared process in a technology-neutral manner is very advantageous.

The push is on to make Web services work both inside and outside of the enterprise. Many core specifications are now complete with implementations emerging in WSE technology today and the new set of .NET technologies for building and running connected systems (known as "Indigo") tomorrow. BPEL will almost certainly become central to complex business orchestration, but with a clear technology roadmap in place you can be sure that your Web services investment today will reap rewards for years to come. BizTalk Server unlocks the door to this interconnected world and enables you to do more with less.

Jon Fancey runs an IT consultancy in the UK that specializes in enterprise-class software design and implementation. You can reach him at msdn@jonfancey.com or read his blog at www.jonfancey.com.


ASP.NET: Combine Web and Windows Services to Run Your ASP.NET Code at Scheduled Intervals

ASP.NET

Combine Web and Windows Services to Run Your ASP.NET Code at Scheduled Intervals

Andrew Needleman
This article discusses:
  • Scheduling the running of ASP.NET code
  • N-tier architecture design
  • Web services fundamentals
  • Windows services fundamentals
This article uses the following technologies:
C#, .NET, ASP.NET, Windows
Code download available at: SchedulingASPNETCode.exe (173KB)
Contents
The Application Structure
Adding Scheduled Jobs to the Application
Building Your Web Service
Building the Windows Service
Expanding the Flow Layer to Handle Scheduled Jobs
Job Timing Information
Retrieving and Setting Job Timing Information
Calling the Job Timing Stored Procedures
Working with Scheduled Jobs
Conclusion

S
uppose you've written a great n-tier application in ASP.NET and you want to extend it to perform scheduled tasks, such as sending e-mail to selected users in the database every two hours, or regularly analyzing the data in the ASP.NET cache for application health monitoring. You don't want to throw away your object model from your ASP.NET application or create too many dependencies between a separate scheduler and the ASP.NET application, so how can you avoid this but still have these apps work together?

In .NET Framework-based applications, timers are frequently used to perform activities at scheduled intervals, so using one would seem to be an appropriate solution. You could start a timer from the Application_Start handler in Global.asax to run your scheduled tasks. Unfortunately, this solution is not robust through application domain, process, or system restarts because a request must be made to the application to start the timer. ASP.NET is a passive programming paradigm that only responds to HTTP requests, so a process or user input must call the code for it to run.

A better solution is to use a Web service to provide an interface to your ASP.NET application and to build a Windows® service that calls to it at scheduled intervals. This way the ASP.NET application doesn't have to own the scheduling logic and only needs to be concerned with performing tasks it is already capable of performing. And since a Web service can run in the same application context as the rest of your ASP.NET application, it can execute in the same context that your existing code expects.

I'll be using a Windows service to initiate the Web service call because Windows services can start themselves when Windows boots up. So even if the server is restarted, the application will be able to start itself. This restart capability makes a Windows service a more robust solution for the task than a typical Windows-based application. It's also the reason why Windows services are used for many background processes (such as IIS).

In this article, I'll demonstrate how to do this while creating the smallest number of dependencies between your scheduling application and your ASP.NET application. The solution involves simplifying the scheduling application that initiates the ASP.NET job. In the scheduling application, there will be no logic called that is specific to the ASP.NET application except for the Web service endpoint that it calls. The Windows service will use an app.config file to store both the URL of the Web service and the interval that the Windows service should wait between calls to the Web service. By storing these two settings in the Windows service's app.config file, you can change them without having to recompile the Windows service. If you need to change the application's behavior when it is called, you can just change the logic in the ASP.NET application; however, you won't have to change the scheduling application's code. This means that the scheduling application will be isolated from changes in the ASP.NET application.

Note that this solution is based on the premise that there are some tasks that should only be executed in the context of a running ASP.NET application. If this is not a requirement for your tasks, you should strongly consider referencing the ASP.NET application's business logic assembly directly from your Windows service and bypassing the ASP.NET process to fire the tasks.

The Application Structure

A typical ASP.NET application is built with a series of independent layers that perform specific functions. In my particular example, I have database access classes, business logic classes, business flow classes, and ASP.NET pages that serve as the entry point to these layers (see Figure 1).

Figure 1 The Plan
Figure 1 The Plan

The ASP.NET pages are merely used to display and retrieve data. They are an interface into and out of the business flow classes which actually coordinate all of the work. The flow classes call the business logic classes in the proper order to complete a particular transaction, such as ordering a widget. For example, the flow class could first call the business logic to check inventory, then to order the widget, and finally to decrease the inventory to the proper level.

The business logic classes decide how to call the database access classes and process that result if necessary to get a final result that you can use for other operations. For example, business logic would be used to calculate the total price including the tax for a particular state. First you may need to retrieve the tax rate for that state and base prices from the database using the data access classes, then multiply them to find the total tax on each item.

The database access classes hold the logic to connect to the database and to return a resultset in a format such as a DataSet, DataTable, or a DataReader that can be consumed by the higher layers. These classes merely retrieve data from the database and update it according to the information they are fed; they do not process the result. For example, they may retrieve the tax rate for a particular state, but they would not calculate the total tax on the order.

The Microsoft® Data Access Application Building Block simplifies the data access classes by providing easier ways to communicate with the database and stored procedures (for the download, see Data Access Application Block). For example, you can make a call to its SQLHelper object's FillDataSet method to fill a DataSet from the output of a stored procedure using one line of code. Typically, you would have to write the code to create at least the DataAdapter and a command object, which would take at least four lines of code.

The Data Access Application Block connects to the stored procedures that are in the database. The stored procedures provide the SQL code that is needed to access and modify data in the database.

Adding Scheduled Jobs to the Application

An ASP.NET Web service will provide you with an interface into the existing ASP.NET application that holds the task logic. This will serve as the broker between it and the Windows service that calls the ASP.NET application into action. A Windows service will then call the ASP.NET application at scheduled intervals. By building an ASP.NET Web service in the existing ASP.NET application, the business objects and logic that were already created for the ASP.NET application can be reused in the scheduled jobs. Figure 2 shows the details of the application flow from the client Windows service application through the Web service initiating that run on the server, all the way through the execution of each scheduled task.

Figure 2 Run Scheduled Jobs (Close Figure 2)

As you can see in Figure 3, the process will require some modifications to the standard layering previously depicted. The Windows service will wake up the ASP.NET Web service at a specified interval. The ASP.NET Web service will then call a method in the Web application's flow layer that will actually determine which scheduled jobs should be run and will then run them. Once the basic solution is implemented, you'll use the client-side app.config file to determine the intervals at which the Windows service calls the Web service. Next, you'll add the functionality needed by the business flow layer in order to loop through and run jobs. You n-tier gurus out there will be much more interested in the flow tier than the remaining ones, so I'll save the database table, database stored procedure, data access code, and business logic for last.

Figure 3 Additions to App
Figure 3 Additions to App

Finally, add the code to the existing layers of the application from the bottom (the database table level) to the middle (the business logic layer) in order to support the job functionality used by the flow layer.

Building Your Web Service

To build the Web service, first add the JobRun ASP.NET Web service to the ASP.NET application within the same layer as your existing ASP.NET code. Make sure that your ASP.NET project has a reference to the business logic, flow, and data access projects. Next, to create the RunJob Web service method in the JobRun Web service, the Web service method will need to call the flow layer's function that runs the proper jobs. This means that the RunJob method can start out as simply as this:

[WebMethod]
public void RunJob()
{
    Flow.JobFlow jf = new Flow.JobFlow();
    jf.RunAllActiveJobs();
}

Use the RunJob function to create an instance of the JobFlow class (which is in the flow layer) and call its RunAllActiveJobs function. The RunAllActiveJobs of the JobFlow function does all the real work in coordinating the running of the jobs, while the RunJob function merely serves as an entry point into the sequence.

Note that this code does not prevent jobs from running on more than one thread at a time, which could happen if the Windows service scheduled tasks too frequently (faster than they could be run) or if some other application invoked the entry point. If the method is not thread safe and allows multiple threads through it at the same time, it may cause problems with the results of these jobs. For example, if job X sent an e-mail to Mary Smith, but hadn't yet updated the database when job Y queried the database to do its e-mails, then Mary could receive two e-mails.

To synchronize access to the function, I'll use the Mutex class from the System.Threading namespace:

private static Mutex mut = new Mutex(false, "JobSchedulerMutex");
Mutex provides for cross-process synchronization, so this will prevent multiple runs at the same time even if two different ASP.NET worker processes are involved. Now, let's change the RunJob method to use the Mutex to ensure that no other job is running before starting the jobs.

As you can see in the RunJob function in Figure 4, you call the WaitOne function of the Mutex to make this thread wait until it is the only one before executing. The ReleaseMutex function is then called to indicate that you are finished with the code that needs to run only in one thread. Of course, blocking here may not be the correct solution. You might choose to return immediately if another thread is already executing jobs, in which case you could specify a short timeout to the WaitOne method, and immediately return from RunJob if the mutex couldn't be acquired.Put all of the main actions of the function in a try-finally block so that ReleaseMutex is called even if an unexpected exception in the RunAllActiveJobs function causes the RunJob function to exit.

Figure 4 RunJob Web Service (Close Figure 4)
[WebMethod]
public bool RunJob()
{
    bool ranJob = false;
    mut.WaitOne();
    try
    {
        Flow.JobFlow jf = new Flow.JobFlow();
        jf.RunAllActiveJobs();
        ranJob = true;
    }
    finally
    {
        mut.ReleaseMutex();
    }
    return ranJob;
}

You'll want to secure your Web service using some form of authentication and authorization, possibly using Windows security, to ensure that no one runs the service without proper authorization, but I won't go into the details of that in this article.

Now that you have the Web service built so that you can call it from another app, let's build the Windows service that will use it.

Building the Windows Service

Start by creating a new Windows service project in another instance of Visual Studio® .NET and name it InvokingASPNetService.cs. Make sure that this service will start properly by adding a Main method as follows:

public static void Main()
{
    ServiceBase.Run(new InvokingASPNetService());
}
Now add using statements for the following namespaces:
using System.Configuration;
using System.Globalization;
Add an installer for the service by right-clicking the design surface of the InvokingASPNetService.cs and selecting Add Installer. You should change the created serviceInstaller1's StartType property to Automatic so that the Windows service starts when Windows boots. Set the ServiceName property of the serviceInstaller1 to InvokingASPNetService so it will be appropriately named in your Services Manager, and then change the serviceProcessInstaller1 Account property to Local Service.

The third step is to create a Web reference to the InvokingASPNetService Web service and then name it JobRunWebService. Change the JobRunWebService URL Behavior property to be Dynamic in order to have Visual Studio .NET automatically augment the app.config with your Web reference's URL. The proxy class generated will look to this configuration file for the Web service's URL, thus allowing you to point the Windows service at a different endpoint without recompiling.

Fourth, create a method in the Windows service to run the Web service every time it is called. The method will look like this:

private void RunCommands()
{
    JobRunWebService.JobRunInterval objJob =
        new JobRunWebService.JobRunInterval();
    objJob.RunJob();
}
As you can see, you'll declare the Web service proxy and create it just like any other .NET object. Then, call the Web service's RunJob method in order to run the jobs on the remote Web server. Note that neither step is different from using a local class even though you are using a Web service.

Fifth, you'll need to call the RunCommands function in the Windows service. You should call this method at a set interval of time based on how often you would like to run the jobs on the remote server. Use a System.Timers.Timer object to ensure that the RunCommands function runs at the proper intervals. The Timer's Elapsed event will allow you to trigger any function that you specify after each interval has elapsed. (Note that interval length is specified in the Interval property.) You'll use the triggered function to call the RunCommands function so you can automate this feature. By default, this timer class only triggers an event the first time that the timer expires, so you need to ensure that it repeatedly resets itself every time by setting its AutoReset property to true.

You should declare it at the service level, so that any function of the service can reference it:

private Timer timer;
Next, create a function that will initialize the timer and set all of its relevant values:
private void InitializeTimer()
{
    if (timer == null)
    {
        timer = new Timer();
        timer.AutoReset = true;
        timer.Interval  = 60000 * Convert.ToDouble(
            ConfigurationSettings.AppSettings["IntervalMinutes"]);
        timer.Elapsed += new ElapsedEventHandler(timer_Elapsed);
    }
}

To allow for the configuration interval to be changed without recompiling the application, I've stored the interval in the app.config file so that the InitializeTimer method can access it using ConfigurationSettings.AppSettings instead of having it hardcoded, as shown in the following:

<add key="IntervalMinutes" value="5" />
Make sure that the timer calls the timer_Elapsed function to handle the Elapsed event when the timer runs out. The timer_Elapsed method is very simple and calls the RunCommands function that was just built, as shown here:
private void timer_Elapsed(object source,System.Timers.ElapsedEventArgs e)
{
    RunCommands();
}
Finally, you have to install the Windows service using the installutil command. The easiest way is to open the Visual Studio .NET command prompt, navigate to the service's directory, and run the installutil utility, specifying your assembly as the parameter.

Expanding the Flow Layer to Handle Scheduled Jobs

It is important to expand the flow layer to handle the needs of running scheduled jobs (assuming the jobs differ enough that they need to be coded rather than merely parameterized). This involves collecting all jobs from the database where the next start time in the database has passed and running them individually. Within the flow layer, you will create a base class called Job to provide all of the functionality that is common between jobs. This includes a mechanism to initialize and retrieve the JobID, a common method (RunSingleJob) to run the job and set the next time to run in the database after a successful run, and an overridable method (PerformRunJob) to be customized for each individual job.

The flow layer will also need to have job-specific classes built for each job it performs. These will inherit from the base Job class and will override the PerformRunJob function of the Job class to customize the execution of that particular job. You'll also need a factory class (JobFactory) to create and initialize the JobID of the correct Job class. The static CreateJob function will create the appropriate job based on the JobID passed into it. Finally, the flow layer will need to be able to determine which jobs need to run, loop through them, and run them. This is what the JobFlow class will provide through its RunAllActiveJobs method.

First, let's create the Job base class in the flow layer project, which will be the parent of each individual job class. The core of the Job abstract base class is shown in Figure 5. It allows the initialization and retrieval of its JobID, as well as ensuring that the database is updated if the job is run successfully. The JobID will not change for a given job after it is created, so you must ensure that after initialization the set function will not change the value. The JobFactory class that creates each Job class will set its JobID value.

Figure 5 Job Abstract Class (Close Figure 5)
protected bool isInitialized = false;
protected int mJobID;
public int JobID
{
    get { return mJobID; }
    set
    {
        if (!isInitialized)
        {
            mJobID = value;
            isInitialized = true;
        }
        else throw new InvalidOperationException("JobID already set.");
    }
}

public void RunSingleJob()
{
    if (isInitialized)
    {
        PerformRunJob();
        RecordJobSuccess();
    }
}

protected abstract void PerformRunJob();

protected void RecordJobSuccess()
{
    JobLogic jl = new JobLogic();
    jl.UpdateJobDone(JobID);
}

The RunSingleJob function determines that this job's JobID has been initialized, runs the job (PerformRunJob), and updates the database after successful runs with the RecordJobSuccess method. The isInitialized variable is used to make sure that each job has its JobID initialized before running the job. The PerformRunJob abstract method is implemented by derived Job classes and holds the actual logic for the task.

After a job's implementation (PerformRunJob method) runs successfully, the base class calls the RecordJobSuccess function, which uses the UpdateJobDone method of the Business Logic layer's JobLogic class to record the time that it ran in the database as well as the next scheduled time to run. I will create the JobLogic class of the Business Logic layer later.

The Job class provides both the ability to initialize the JobID variable and to update the database upon success with the next run time. Plus, you only have to override one function with class-specific code. This allows you to create the child classes of the Job class. To do so, you need to create two classes that will run a particular type of job and inherit from the Job class to obtain the rest of their functionality. Create a JobRunTest class and a JobEmailUsers class and make sure that each one inherits from the Job class, as shown in the following:

public class JobRunTests : Job
Now, override the PerformRunJob method for both classes as follows (using the JobRunTest class as a sample):
protected override void PerformRunJob()
{
    ///Do RunTest specific logic here
}
Place your job-specific logic inside this method. The rest of the code that runs the jobs and updates the next run time in the database is inherited from the Job base class. Your jobs will combine calls to the existing Business Logic classes in order to run complex processes. Now that you have the sample jobs, let's look at how to create these jobs using the JobFactory object.

The JobFactory class is used to create the corresponding child Job class for each JobID. The JobFactory class takes a JobID variable in its static CreateJob function and returns the appropriate Job subclass. Figure 6 shows the code in the JobFactory.

Figure 6 Job Factory (Close Figure 6)
public static Job CreateJob(int currentJobID)
{
    Job myJob;
    switch(currentJobID)
    {
        case 1:
            myJob = new JobEmailUsers();
            break;
        case 2:
            myJob = new JobRunTest();
            break;
        default:
            return null;
    }
    myJob.JobID = currentJobID;
    return myJob;
}

The CreateJob function takes a currentJobID and uses it in a case statement to determine which child class of the Job class should be returned. It then initializes the current JobID and returns the Job-derived class. Now that you have the Job base class, its job-specific children, and a way to select which class to create, you can look at how to pull it all together using the JobFlow class.

To create a class called JobFlow that will gather and execute the appropriate jobs, add a function called "RunAllActiveJobs" to loop through each job that you need to run and call their individual RunSingleJob functions. You'll need the RunAllActiveJobs function to grab a list of the jobs that are due to run from the database through the business layer, data access layer, and stored procedures, and then run them using their respective RunSingleJob functions. The following code shows how the RunAllActiveJobs method of the JobFlow class accomplishes these goals:

JobLogic jl = new JobLogic();
DataSet jobsActiveData = jl.GetAllActiveJobs();
foreach (DataRow jobsActive in jobsActiveData.Tables[0].Rows)
{
    int currentJobID = Convert.ToInt32(jobsActive["JobID"]);
    Job myJob = JobFactory.CreateJob(currentJobID);
    myJob.RunSingleJob();
}

Basically, you would store the jobs in the database with information on the last time that they ran as well as the interval that the code should wait between runs. The jobs that need to be run are then retrieved through the JobLogic class of the BusinessLogic layer with the GetAllActiveJobs method. Each active job's ID is used to get a Job object, whose RunSingleJob method can be used to execute the task as previously described.

Job Timing Information

Determining which scheduled jobs should be run means that you need to store basic information about them such as the interval between runs, the last time that they ran, and the next time that they should run. In order to do this, create a job table in a SQL Server database (see Figure 7).

Figure 7 Job Table (Close Figure 7)

Column |  | Datatype
 | JobID | int identity
 | JobTitle | varchar(500)
 | JobInterval | datetime
 | DateLastJobRan | datetime
 | DateNextJobStart | datetime

The JobID column holds the unique identifier for each job in the job table. The JobTitle column contains the job name so that you can determine which job is being run. The JobInterval column holds the interval between jobs. This is the date and time interval greater than 1/1/1900 that should be added to the current time after a job succeeds to calculate when the next job should be run. For example, a value of 1/2/1901 in the JobInterval field would mean that one year and one day would be added to the time that the job last ran.

The DateLastJobRan column contains a datetime value for the date and time that the job last ran. The last column, DateNextJobStart, contains the next time that the job should run. While this column should be a computed column which is equal to JobInterval plus DateLastJobRan, you can understand the application layers more vividly if you set this up as a regular datetime column.

Retrieving and Setting Job Timing Information

To retrieve and set job timing information through the new stored procedures in the SQL Server database, the stored procedures must find all of the jobs in the database that need to be run by the application, update a single job's information in the database to indicate that it has run, and set the next job-run date for that job. Each job has a DateNextJobStart column in the database that indicates the date and time at which the job should run. If the current date and time is past that of the DateNextJobStart column, then the job should be run in the process. The stored procedure that selects the jobs that should be run is shown here:

   CREATE PROCEDURE
     dbo.Job_SelectJobs_NextJobStartBefore
@DateNextJobRunStartBefore datetime
AS
SELECT * FROM JOB WHERE DateNextJobStart < @DateNextJobRunStartBefore

This selects all of the columns of the Job table for the jobs that have a DateNextJobStart value that is before (less than) that of the @DateNextJobRunStartBefore DateTime parameter. To find which jobs should run, simply pass in the current date and time through the stored procedure's parameter. Now that you can select the jobs that need to run, you can switch to building the procedure to update them after they run.

The stored procedure that updates the database with a single job's last run date and next run date is as follows:

CREATE PROCEDURE dbo.Job_Update_StartEnd_CalcNext
@JobID int,
@DateLastJobRan datetime
AS
UPDATE JOB
    SET
    DateLastJobRan = @DateLastJobRan,
    DateNextJobStart = @DateLastJobRan + JobInterval
WHERE
    JobID = @JobID
This procedure updates the job that is identified by @JobID with a new DateLastJobRan and calculates the DateNextJobStart value by adding the JobInterval to the @DateLastJobRan that was passed in. This procedure should only run after the job referenced in @JobID is run and should be called with the @DateLastJobRan parameter equal to the date and time that the job ran last.

Calling the Job Timing Stored Procedures

You can extend the data access layer to call the job timing stored procedures by adding a new class called JobAccess. The role of functions in the data access layer is to translate the parameters passed to it by the business layer into a stored procedure database query and return the result to the business layer. The parameters in the data access layer's functions will mirror those of the stored procedures that they access because they do not perform any Business Logic on the values.

You'll be accessing the database through Microsoft Data Application Building Block's SQLHelper class. This class contains functionality that simplifies data access code, making your code more concise and readable.

To change the data access layer to run the scheduled jobs, first add a JobAccess class to the existing data access layer to hold the functions that are needed to schedule jobs. Next, create a function in the JobAccess class that returns a DataSet of the jobs that need to be run through calling the Job_SelectJobs_NextJobStartBefore stored procedure. You'll also need to create a function in the JobAccess class to call the Job_Update_StartEnd_CalcNext stored procedure without returning a result.

First add the JobAccess class to the data access layer. Then, edit the JobAccess class to add the following "using" statements:

using System.Data;
using System.Data.SqlClient;
using Microsoft.ApplicationBlocks.Data;

Let's look now at how to add the SelectJobsBeforeDate function, which retrieves the list of jobs that need to be run. Here is the signature of the SQLHelper's ExecuteDataset function:

public static DataSet
ExecuteDataset(
    string connectionString, string spName,
    params object[] parameterValues)

The following is the SelectJobsBeforeDate function, which uses ExecuteDataset to invoke the Job_Update_StartEnd_CalcNext stored procedure, returning a DataSet of the results:

public DataSet SelectJobsBeforeDate(DateTime beforeDate)
{
    return SqlHelper.ExecuteDataset(
        ConnectionInfo.connectionString,
        "Job_SelectJobs_NextJobStartBefore, myparams);
        new object[]{new SqlParameter("BeforeDate", beforeDate)});
}

After jobs have been run, you'll need to execute the stored procedure which updates status information about the jobs. The method that accomplishes this, UpdateJob, will use the SQLHelper class's ExecuteNonQuery method. Here is the signature:

public static int ExecuteNonQuery(
    string connectionString, string spName, params object[]
    parameterValues)
The UpdateJob method can be written as follows:
public void UpdateJob(int jobID, DateTime dateLastJobRan)
{
    string connStr = ConnectionInfo.connectionString;
    string spName = "Job_Update_StartEnd_CalcNext";
    SqlParameter myparam1 = new SqlParameter("JobID", jobID);
    SqlParameter myparam2 = new
        SqlParameter("DateLastJobRan",dateLastJobRan);
    object[] myparams = {myparam1, myparam2};
    SqlHelper.ExecuteNonQuery(connStr, spName, myparams);
}

The UpdateJob function in the JobAccess class is supposed to mirror the parameters that are passed to the stored procedure that it uses. Therefore, the UpdateJob function has a jobID parameter and a dateLastJobRan parameter with the same datatypes as those in the Job_Update_StartEnd_CalcNext stored procedure. Using the jobID and the dateLastJobRan parameters, you can create the two SqlParameters, put them in the myparams object array, and use the ExecuteNonQuery function to execute the stored procedure. Now that you've created the JobAccess class, you need to create the final layer of classes to bridge the gap between the flow layer and the data access layer.

Working with Scheduled Jobs

The final layer that must be modified to work with scheduled jobs is the Business Logic layer, which I'll call JobLogic. This class will perform basic logic on the variables between the flow layer and the data access layer.

First, add the JobLogic class to the DataAccess layer using the following statements::

using System.Data;
using ScheduledWebService.DataAccess;
Second, build the GetAllActiveJobs function of the JobLogic class to find all of the jobs that still need to be run at or before the current time, as shown here:
public DataSet GetAllActiveJobs()
{
    JobAccess ja = new JobAccess();
    return ja.SelectJobsBeforeDate(DateTime.Now);
}
The GetAllActiveJobs function creates an instance of the JobAccess class and calls its SelectJobsBeforeDate with the parameter value of the current date. GetAllActiveJobs picks the current date to pass to this function, so you can find out which jobs were scheduled to run before the current time.

Lastly, create the UpdateJobDone function of the JobLogic class to update the database to indicate that the job specified was just completed, as shown here:

public void UpdateJobDone(int jobID)
{
    JobAccess ja = new JobAccess();
    ja.UpdateJob(jobID, DateTime.Now);
}
This function creates an instance of the JobAccess class and calls its UpdateJob method. It passes along the jobID parameter and then uses the current date for the dateLastJobRan parameter. You pass the current date and time to the UpdateJob function because it is the time at which the job completed successfully.

Conclusion

Extending your ASP.NET application with automated tasks allows you to program events explicitly rather than waiting for a request to execute code. You can harness this power to perform a variety of tasks from running complex calculations to creating and sending reports to executives on a regular schedule. Such tasks can reuse both your existing logic and the objects in your ASP.NET layers, decreasing development time and improving maintainability. You can also expand the jobs that this scheduler starts without changing the Windows service that initiates it.

Note that there are many variations to what I've discussed in this article. For example, rather than creating a custom Windows service to act as the scheduler, you could use something as straightforward as the Windows Task Scheduler, which is quite robust and implements much of the capabilities discussed here. That said, the creation of Windows services has been vastly simplified by the .NET Framework, so they should be reconsidered as an option even if you have previously found them too difficult to use. Similarly, Web services are a great way for apps to expose functionality to other apps and will continue to be valuable in that regard.

Andrew Needleman is managing partner of Claricode, a consultancy based near Boston that specializes in architecting and developing n-tier Web apps in .NET. He has trained hundreds of developers in C#, the .NET Framework, and Visual Basic .NET. Andrew can be reached at Andrew@claricode.com.


WSE 2.0: Give Your Web Services Consumers the Exact XML They Need to Succeed

WSE 2.0

Give Your Web Services Consumers the Exact XML They Need to Succeed

Chris Dix
This article discusses:
  • Web Services Enhancements (WSE) Service Pack 2
  • Web services and SOAP messages
  • How the .NET Framework versions 1.x and 2.0 serialize messages
  • Getting your messages serialized the way you want
This article uses the following technologies:
XML, Web services, and Visual Basic
Code download available at: WSE20.exe (141KB)
Contents
EndpointReference
SoapEnvelope
The Big Easy
XmlSerializer
When Good Messages Go Bad
The Decorative Way
The Backdoor Way
The WSE Way
OpenElement
Practice Makes Perfect

W
eb services are first and foremost about messages. No matter what platform you are using, what industry you are building services for, or what levels of complexity are available in your Web services stack, it all comes back to messages. When you are developing services or their corresponding clients, you are creating, transporting, and processing XML messages. Because tools try to make that process easy, sometimes you might (conveniently) forget what it's really all about.

No matter how good your Web services tools are, there will inevitably be those moments when something doesn't work, and you find yourself brought back down to earth and staring at the content of your messages. The more sophisticated the service, the more complex the messages, and the more likely that you will find yourself in this situation. It is at that moment that you want more from your tools than just a good way of tracing messages. Developers building sophisticated Web services will inevitably need to take control over the content and structure of their messages.

The Microsoft® .NET Framework has always provided good support for XML, the serialization of objects as XML, and support for Web services in general. For support of the latest Web services specifications, like WS-Addressing or WS-Security, however, developers must use the Web Services Enhancements (WSE) for .NET. The latest version of WSE (WSE 2.0) provides its own mechanism for custom message handling, in addition to supporting the hooks already in place in .NET. This article will discuss each of the methods available to developers for custom message serialization and will cover in detail the method used in WSE 2.0.

WSE 2.0 makes it possible for developers to build Web services and related client applications that possess, as the name suggests, enhanced features that the rest of the current .NET Web services infrastructure does not provide. For example, WSE provides message-level security through an implementation of the WS-Security specification, message routing through WS-Addressing, and service policy control through WS-Policy. While WSE gives developers the ability to use all of these enhanced features, at its core WSE 2.0 uses a simple messaging model that does not change, regardless of how many enhancements you take advantage of.

EndpointReference

The EndpointReference class is the WSE implementation of the WS-Addressing concept of an endpoint reference, which describes a Web service in terms of its address (both logical and physical) and additional properties. An EndpointReference class represents a Web service endpoint, which represents the place the messages should be sent. This could be either an intermediary or the ultimate destination of the message, but in any event it is used throughout WSE to indicate the target of a message.

To create an EndpointReference, you may provide a URI that defines the address of the service. In the following lines of code you can see how to create an EndpointReference using an instance of the Uri class in Visual Basic® .NET:

Dim destUri As Uri = New Uri("http://www.example.com/MyService.asmx")
Dim destination As EndpointReference = New EndpointReference(destUri)
The EndpointReference is used throughout WSE 2.0 to indicate where a client should send messages, or where a service endpoint will be listening for them. The use of the EndpointReference instead of just a URI is significant for two reasons. First, it allows you to define an endpoint in terms of both a logical and physical address, something that was not possible before in .NET Web services. Second, the EndpointReference class is significant because it is an excellent example of WSE's custom serialization hooks at work (as you'll see later).

For a more complete discussion of the enhanced features of WSE 2.0, such as WS-Security, take a look at Aaron Skonnard's article from the August 2004 issue of MSDN®Magazine, "What's New in WSE 2.0". If you are looking for more information on EndpointReference and the move towards WS-Addressing in WSE 2.0, you can also take a look at Aaron's article on WS-Addressing and WSE 2.0, "Moving From WS-Routing to WS-Addressing Using WSE 2.0".

Web Services Enhancements provides two levels of messaging objects, allowing developers to work at either a low (raw XML) or high (objects as messages) level. Whichever level of messaging you choose to do with WSE, each individual message is contained in an instance of the SoapEnvelope class.

SoapEnvelope

The SoapEnvelope class allows direct access to the SOAP message in its raw form. For developers new to SOAP or Web services, the SOAP envelope is a wrapper around the message payload that can also contain one or more header blocks. One very interesting feature of SoapEnvelope is that it derives from the System.Xml.XmlDocument class. This means that WSE allows you to manipulate your raw messages in exactly the same way you are probably familiar with: using the properties and methods of the XmlDocument class. Because the message is already loaded into an XmlDocument, you can also do things like execute XPath expressions or test the namespaces used in the message using the same methods you would use on any other XML document:

' Load a message from a file, using XmlDocument.LoadXml
Dim envelope As SoapEnvelope = New SoapEnvelope()
envelope.LoadXml( "c:\soapmessages\soap.xml" )

For those developers with a true passion for XML, or those who feel they must have complete control over their messages, WSE 2.0 provides a low-level API for sending and receiving messages. There are two objects that are used to handle messages at this level: SoapSender and SoapReceiver. Both of these objects work with messages as raw XML via the SoapEnvelope class.

Once you have a message to send in the form of a SoapEnvelope, you can use the SoapSender class provided by WSE 2.0 to send the message. To listen for SOAP messages at a particular endpoint, the client can also derive from the SoapReceiver class. The following code illustrates a brief example of how a Web service client can be created using these two objects:

Dim destUri As Uri = New Uri("http://www.example.com/MyService.asmx")
Dim destination As EndpointReference = New EndpointReference(destUri)

Dim sender As SoapSender = new SoapSender(destination)

' Fill the contents of the envelope, including any addressing
Dim envelope As SoapEnvelope
sender.Send(envelope)

It would seem that any discussion of custom serialization of messages in WSE would end here. If you can construct or process your messages as raw XML documents, what more could you ask? For one thing, you could ask to not have that required of you. A great deal of work has gone into making it possible for developers to work with Web services in a way that is familiar to them—as objects. While this may hide some of the details from developers, it also provides a more productive environment for many developers to work in. In addition, if your messages are being treated as raw XML documents, you have done nothing to create a description of your Web service—in other words, Web Services Description Language (WSDL) and associated schema. Without a service description, the contract between service and client is open ended and able to accept any XML as a message.

Not everyone loves XML enough to build their messages by hand, and the WSE team appreciates that fact. For those developers who feel more comfortable using the .NET Framework to convert between their objects and XML messages (the model that developers who use ASP.NET Web services will find familiar), WSE 2.0 also provides objects to build clients and services. These are the SoapClient and SoapService classes, respectively. Like SoapSender and SoapReceiver, these classes are responsible for sending and receiving messages, but they take it a step further. These classes combine features that make it easier for developers to use SOAP messages in the request-response model that is commonly used when working with Web services over HTTP.

Consider the service defined in Figure 1 which derives from SoapService and which most likely looks more familiar to developers than the earlier example that used SoapSender. Internally, SoapService is using both a SoapReceiver and a SoapSender to make the request-response message exchange pattern happen, but it does much more than that. You will notice that there is no longer a SoapEnvelope in sight. Instead, you are working with objects as both parameters and return type.

Figure 1 SoapService at Work (Close Figure 1)
Imports System.Xml
Imports System.Xml.Schema
Imports System.Xml.Serialization

Imports Microsoft.Web.Services2
Imports Microsoft.Web.Services2.Addressing
Imports Microsoft.Web.Services2.Messaging
Imports Microsoft.Web.Services2.Xml

Public Class MyWseService
   Inherits SoapService
      <SoapMethod("http://tempuri.org/LookupEmployee")> _
      Public Function LookupEmployee(name As String) As MyEmployee

          Dim myEmployee As MyEmployee = New MyEmployee()
          ' Find the employee information and populate ...
          Return myEmployee

      End Function
End Class

It's nice that WSE 2.0 provides this familiar model, but how does it work? How does WSE translate from an employee class I have defined to an XML message and back again? The answer is the same for both WSE and ASP.NET Web services: XML serialization.

The Big Easy

Whether you are creating Web service applications with ASP.NET or with WSE, XML serialization is used to convert objects into XML messages and back again. The System.Xml.Serialization namespace of the .NET Framework provides a number of classes that handle the object-to-XML conversion process. These objects are not just available to the .NET internals; you can use them in any application, and it helps to have a level of familiarity with them since they are used on your behalf every time you consume or create a Web service with .NET.

In general, the serialization of objects to XML (and vice versa) is taken care of automatically by these objects. The key object in the XML serialization process is the aptly named XmlSerializer.

XmlSerializer

The XmlSerializer class is responsible for converting between XML and instances of objects. XmlSerializer will convert an instance of a class to an XML element of a given name and namespace. It then uses reflection to examine all the public data members of a class, and it maps each of those to a child element of the original element. Figure 2 shows a simple class definition and its corresponding XML serialized structure once it has passed through the XmlSerializer.

Figure 2 MyEmployee Before and After (Close Figure 2)

MyEmployee Class
Public Class MyEmployee
    Public Name As String
    Public DOB As DateTime
    Public IQ As Integer
End Class
MyEmployee.xml
<MyEmployee>
    <Name>John Q. Public</Name>
    <DOB>7/4/1976</DOB>
    <IQ>150</IQ>
</MyEmployee>

If you want to use an XmlSerializer with your SoapEnvelope, you can. The following code sheds some light on what is really happening under the covers of the Web services stacks:

Dim xs as System.Xml.Serialization.XmlSerializer
xs = New XmlSerializer(GetType(MyEmployee))

Dim envelope As SoapEnvelope = New SoapEnvelope()
' Envelope gets populated from the Web service call

Dim nodeReader as XmlNodeReader = _
    new XmlNodeReader(envelope.Body.FirstChild)

employee = xs.Deserialize(nodeReader) As MyEmployee
Since your SoapEnvelope is an XmlDocument, you can construct any number of XmlReader or XmlWriter objects on top of it and pass them to the Serialize or Deserialize methods, respectively. Now that you have a better idea about how XML messages and objects are related, let's take a look at some more complex messages and see where custom serialization becomes necessary.

When Good Messages Go Bad

The default behavior of XML serialization through XmlSerializer can take you through many cases, but sometimes you need to take more control over the way in which your objects are serialized. The class provided here illustrates two potential problems commonly encountered when working with XML serialization of objects:

Public Class MyService
    Public States As StringCollection = New StringCollection()
    Public ServiceUri As Uri
End Class

In the first case, you have a StringCollection that represent names or abbreviations of the states (for example, North Carolina). There are a number of ways that you could serialize such a list but which one should the XmlSerializer choose? In this case, you would need to provide more information. The second issue is a bit trickier because one of the members of the class is of the System.Uri type, and the System.Uri class has no default constructor (a public constructor that accepts no parameters). Classes that have no such constructor cannot be deserialized from XML because the XmlSerializer class does not know how to instantiate one, and if it can't be deserialized, XmlSerializer will also refuse to serialize it. Because of this, the XmlSerializer would also fail to serialize the MyService class because ServiceUri happens to be a public member.

The default behavior of XML serialization can take you through most cases, but sometimes you need to take more control over the way in which objects are serialized. In addition to handling messages as raw XML, as you saw previously, there are three other methods at your disposal that you can use to shape the content of your Web services messages: attributes, IXmlSerializable, and IXmlElement.Implementing IXmlElement

While WSE provides the OpenElement class and other similar classes to use for custom serialization, it is the IXmlElement interface that is the hook. If you want, you can implement IXmlElement in your own classes and achieve much the same results as if you had derived from OpenElement. If you do choose to implement IXmlElement yourself instead of deriving from a WSE base class, you must remember to provide a special constructor for your object.

This special constructor must take as a parameter an instance of the XmlElement class, the same parameter that you would pass to the IXmlElement.LoadXml method. This constructor can be nothing more than a pass-through to the LoadXml method, as shown here:

Public Class MyMessageClass
    Implements IXmlElement

    ' IXmlElement implementation goes here...

    Public Sub New( element As XmlElement )

        Me.LoadXml( element )

    End Sub

End Class

Without this constructor, your message objects will appear to be serialized to XML successfully, but they will fail when WSE attempts to deserialize them. The reason for this is that the WSE code assumes that message objects that implement IXmlElement also implement this special constructor (since all the WSE classes do).

Instead of creating the object with a default empty constructor and calling LoadXml, it simply constructs the object using the XmlElement. The thing to remember here is to derive from a WSE class like OpenElement, or provide a special constructor in your own IXmlElement implementation.

The Decorative Way

There are times, of course, when you need to step in and control the behavior of the serialization process. The first method that a developer can use to control the XML serialization process is through attributes. The System.Xml.Serialization namespace provides a number of attributes that can be used to change normal serialization behavior. Figure 3 lists a few of the more common attributes and the effect that each of them has on the serialization process. The full list is available in the documentation for the System.Xml.Serialization namespace.

Figure 3 Sample of XML Serialization Attributes (Close Figure 3)

Name | Used On | Effect
XmlRootAttribute | Public class declarations | Sets the element name and namespace used by a root element
XmlIgnoreAttribute | Public properties and fields | Serialization skips the member entirely
XmlAttributeAttribute | Public field, property, parameter, or return value | Serializes the member as an attribute instead of as an element
XmlArrayAttribute | Public field, property, parameter, or return value that returns an array of complex objects | Serializes the array as an XML array

Whenever a class that uses one or more of these attributes is serialized, the XmlSerializer class uses reflection to determine what attributes are being used and how they should be interpreted. In the class shown here, I have added attributes to the previously defined class in order to declaratively shape the XML serialization, as shown in Figure 4.

Figure 4 Adding Attributes (Close Figure 4)
<XmlRoot(ElementName:="MyService", Namespace:="http://tempuri.org")> _
Public Class MyService

    <XmlArray(ElementName := "States", _
         Namespace := "http://tempuri.org/states/", _
         IsNullable := True)> _
    Public States As StringCollection = New StringCollection()

    <XmlIgnore()> _
    Public ServiceUri As Uri

End Class

The XmlRootAttribute describes the MyService class as a whole, the XmlArrayAttribute describes how the States collection should be serialized as a list of items, and the XmlIgnoreAttribute states that the Uri member ServiceUri should be skipped completely during serialization. With those changes, the class can now be successfully passed to XmlSerializer. The XML that is created when this object is serialized looks like this:

<MyService>
    <States>
        <Item>NC</Item>
        <Item>NV</Item>
        <Item>VA</Item>
    </States>
</MyService>

Although using the attributes to shape XML messages is straightforward, it cannot fix every problem. Most obviously, there is still no serialized information about the Uri member, which may or may not be a problem depending on the class design. On the other hand, suppose you did not like how the States list was serialized. You may still be unable to get what you want from attributes. For example, there are no attributes available that can tell the XmlSerializer to write an array of strings as an XML type of xsd:list, as shown in the following:

<MyService>
    <States xmlns="http://tempuri.org">NC NV VA</States>
</MyService>

This is a common enough construct in XML and one that is well suited to representing a list of items, but that does not require the overhead of an element for each item. For that reason, elements of the xsd:list type appear in many of the messages used for the latest Web services specifications, like WS-Discovery. In order to format the list of strings as an xsd:list, you would need to take more control over the XML serialization process.

The Backdoor Way

Another way to take more control is by implementing the IXmlSerializable interface, which is a mechanism for overriding the XML serialization process:

Public Interface IXmlSerializable
    Sub WriteXml(writer As System.Xml.XmlWriter)
    Sub ReadXml(reader As System.Xml.XmlReader)
    Function GetSchema() As System.Xml.Schema.XmlSchema
End Interface
The IXmlSerializable interface has three methods: two for serialization and one for schema generation. The WriteXml and ReadXml methods are invoked whenever the object in question is being serialized or deserialized. These methods take an XmlWriter instance and XmlReader instance respectively, types which are probably familiar to developers who have been working with XML in the .NET Framework. Just as you could use familiar XmlDocument methods when working at the SoapEnvelope level, you can rely on methods like XmlWriter.WriteStartElement in order to create your message. The GetSchema method returns an instance of System.Xml.Schema.XmlSchema, the .NET type that represents an XML Schema document. This method is called whenever schema information is needed about the object, such as during the construction of WSDL for the Web service.

Using this method takes care of some of the issues you might encounter with XML serialization. For example, if your object implements IXmlSerializable, you can provide public properties and members that are non-serializable data types without generating exceptions from XmlSerializer. This would allow you to take care of the Uri member that you could not serialize earlier.

Figure 5 shows an example of how you can serialize a collection of strings as an xsd:list type. This code is written to serialize a generic list, but the SerializableList class can serve as a base class, and by initializing the ListName and ListNamespace members in the constructor, you can further customize your serialization.

Figure 5 Serializing a String Collection as an xsd:list Using IXmlSerializable (Close Figure 5)
' This class serves as an example of how you could use
' the IXmlSerializable interface to serialize a StringCollection as an
' xsd:list.  This technique will not work for Web services in the
' .NET Framework 1.1 and WSE 2.0 because of the issues with
' combining schemas in the service description, but it could still be
' used in other situations where XML serialization is used in .NET.

Public Class SerializableList
    Implements IXmlSerializable

    Public Sub New()
        Me.ListName = "SomeList"
        Me.ListNamespace = "http://www.thoughtpost.com/2004/xml"
    End Sub

    Public Sub New(ByVal ListName As String, _
      ByVal ListNamespace As String)
        Me.ListName = ListName
        Me.ListNamespace = ListNamespace
    End Sub

    Public Items As StringCollection = New StringCollection
    Public ListName As String = String.Empty
    Public ListNamespace As String = String.Empty

    Public Function GetSchema() As System.Xml.Schema.XmlSchema +
      Implements System.Xml.Serialization.IXmlSerializable.GetSchema

        Dim xsd As XmlSchema = New XmlSchema

        xsd = New System.Xml.Schema.XmlSchema
        xsd.Namespaces.Add("xsd", "http://www.w3.org/2001/XMLSchema")
        xsd.Namespaces.Add("xsi", _
            "http://www.w3.org/2001/XMLSchema-instance")
        xsd.Namespaces.Add("tns", Me.ListNamespace)
        xsd.Id = Me.ListNamespace
        xsd.TargetNamespace = Me.ListNamespace

        Dim elem As XmlSchemaElement = New XmlSchemaElement
        elem.Name = Me.ListName

        Dim stype As XmlSchemaSimpleTypeList = New XmlSchemaSimpleTypeList
        elem.SchemaTypeName = New System.Xml.XmlQualifiedName( _
            "list", "http://www.w3.org/2001/XMLSchema")
        xsd.Items.Add(elem)

        Return xsd

    End Function

    Public Sub ReadXml(ByVal reader As System.Xml.XmlReader) _
      Implements System.Xml.Serialization.IXmlSerializable.ReadXml

        Me.ListName = reader.Name
        Me.ListNamespace = reader.NamespaceURI

        Dim data As String
        data = reader.ReadElementString()

        If ((data <> Nothing) AndAlso (data.Length > 0)) Then

            Dim delims(1) As Char
            delims(0) = " "

            Dim items() As String
            items = data.Split(delims)

            For Each s As String In items
                If ((s <> Nothing) AndAlso (s.Length > 0)) Then
                    Me.Items.Add(s)
                End If
            Next s

        End If
    End Sub

    Public Sub WriteXml(ByVal writer As System.Xml.XmlWriter) _
      Implements System.Xml.Serialization.IXmlSerializable.WriteXml

        Dim data As StringBuilder = New StringBuilder()

        For i As Integer = 0 To Items.Count
            data.Append(Items(i) & " ")
        Next

        writer.WriteElementString(Me.ListName, Me.ListNamespace, _
            data.ToString().Trim())

    End Sub
End Class

It would seem that implementing the three methods of IXmlSerializable would take care of every issue developers might have with their messages, but there are a couple of problems that prevent that from being the case. Originally, the IXmlSerializable interface was not recommended for use outside of the .NET Framework classes. The documentation stated that it was meant for internal use only, and so developers who chose to use it knew that it was a technical risk. Since then, it has become clear that its use will be supported in the .NET Framework 2.0, so you can use it to do custom serialization without worrying about the interface changing in the next version. However, you will likely encounter problems with its use in the current version of .NET, especially when it comes to working with Web services.

Such a problem can be reproduced by trying to use an object that implements IXmlSerializable with WSE 2.0. The problem lies in the way that XML serialization is used by a Web services stack like WSE. In order for a message to be used by the service, it must be described as part of the description of the service itself, namely its WSDL document. The WSDL documents contain embedded schema information about messages, and this is where the problem appears. Through IXmlSerializable, an object can only return a complete schema (not a portion) through the GetSchema method, and the Framework has problems with combining schemas when generating the service description. If your message object is of a different XML namespace than the service, duplicate schema information is generated in the subsequent WSDL file (one for the correct XML namespace, and one for the service). If your message object is of the same XML namespace as the service, then the WSDL generation fails because the document can't contain the same namespace twice. In either of these cases, you do not get the behavior that you are looking for.

It is worth mentioning that in the .NET Framework 2.0, this problem is solved by allowing the object to return an element of a schema instead of an entire schema document, which makes it possible to merge schemas during the WSDL generation step. For more information on this change and on IXmlSerializable in the .NET Framework 2.0, see the MSDN online article "New Features for Web Service Developers in Beta 1 of the .NET Framework 2.0".

The WSE Way

WSE 2.0 introduces its own mechanism for custom serialization: the IXmlElement interface. This interface is defined in the Microsoft.Web.Services2.Xml namespace, and by implementing this interface, an object can override the XML serialization process.

Like IXmlSerializable, the IXmlElement interface defines methods for creating an object from XML, as well as for retrieving an object's state as XML. A definition of the IXmlElement interface is shown in the following code snippet:

Public Interface Microsoft.Web.Services2.Xml.IXmlElement
    Function GetXml(ByVal document As XmlDocument) As XmlElement
    Sub LoadXml(ByVal element As XmlElement)
End Interface

In fact, the IXmlElement interface is implemented by a number of WSE 2.0 objects. Objects that have a complex structure and need to be represented in a message implement IXmlElement in order to handle the serialization process in a custom manner. The best example of one of these objects is the EndpointReference class, which I discussed earlier in the article, and is used throughout the entire object model. If I try to define a Web service method that returns an EndpointReference, or even create my own message object that contains a member of the EndpointReference type, the service will fail. The reason for this is that EndpointReference does not have a constructor that takes no parameters. If a class has no empty constructor, then it cannot be serialized by the Framework because it does not know how to create one. So how does WSE manage to get EndpointReference instances serialized in your messages? It first checks to see if an object implements IXmlElement before passing it to an instance of XmlSerializer. If it does implement the interface, then it uses the LoadXml and GetXml methods to handle serialization; if it doesn't, then the object goes the same route as all of the methods, which is through XmlSerializer.

It may look as though IXmlElement is the answer to handling complex messages within WSE, but there are still two problems. First, if you read the WSE 2.0 documentation, IXmlElement is described in much the same way as IXmlSerializable in that it supports the .NET Framework infrastructure and is not intended to be used directly from your code. Second, there is no method in IXmlElement for providing schema information about your custom object. Where IXmlSerializable had the GetSchema method, IXmlElement has nothing. In some cases, you could probably live without describing your object in schema, but remember that you are dealing with Web services in this case. Without schema information, it is difficult for the XML serialization model to determine how your object should be represented in the WSDL that describes the service. Luckily, the WSE team provided a set of classes in the Microsoft.Web.Services2.Xml namespace that solve these two problems and can be used to build complex message objects. The OpenElement class is a good example of one of these support classes, and it can serve as a base class for complex messages.

OpenElement

So how does deriving from OpenElement solve the schema generation problem? Any classes that derive from OpenElement (or the other similar classes in the WSE Microsoft.Web.Services2.Xml namespace) generate schema with an open content model. For those not familiar with XML Schema, an open content model indicates that the XML type in question can accept a number of additional elements or attributes beyond what is strictly described in the schema. Required items are still required, but it is not an error if additional information exists. This opens a sort of back door to the validation process, saying that this type will accept anything (or some subset of anything, depending on what wildcards you use). It is then up to your custom serialization code to decide what to do with this additional content, if anything. OpenElement is an abstract class; because it derives from IXmlElement but does not implement the serialization methods, the derived class must do this.

This technique gives developers a nice middle ground in terms of flexibility versus structure. Objects that have a complex structure, one that is difficult to automatically serialize to XML, can handle their own serialization, describe as much of their structure as they reasonably can, and still describe themselves using an open content model to allow for additional items that are hard to describe. A nice side effect of this approach is the support this technique provides for versioning of message information. By describing the message with an open content model, a service can allow the message structure to grow somewhat over time without requiring changes to the service contract. The ability to process that additional content through the IXmlElement methods comes in handy at that point.

Figure 6 shows an example of the earlier message class, now implemented by deriving from OpenElement. This class now handles all of the custom serialization issues. The Uri is replaced with an EndpointReference, but the structure is inherently the same, as are the issues involved. This class is included in a set of sample programs where WSE custom serialization methods are used to send and receive complex messages. You can download the entire set of samples from the MSDN Magazine Web site, where the code is available in both C# and Visual Basic .NET. (See the sidebar Implementing IXmlElement.)

Figure 6 MobileService Sample Message Class (Close Figure 6)
' MobileService, the complex message returned by our sample services,
' uses a combination of XML attributes (ex. XmlRootAttribute,
' XmlIgnoreAttribute, etc.) and WSE's IXmlElement hooks to generate a
' complex message.

<XmlRoot(ElementName:="MobileService", _
    Namespace:="http://www.thoughtpost.com/2004/MSDN/WSEXML")> _
    Public Class MobileService
    Inherits OpenElementElement

    Public Sub New()
    End Sub

    Public Sub New(ByVal element As XmlElement)
        Me.LoadXml(element)
    End Sub

    ' Rename Name to LocationName
    <XmlElement(ElementName:="LocationName")> _
    Public Name As String

    ' Skip the EndpointReference; XML serialization can't handle it
    ' because of the lack of an empty constructor
    <XmlIgnore()> _
    Public EndpointLocation As EndpointReference = Nothing

    ' Our additional IXmlElement implementor.
    ' WSE drills down, so messages can implement IXmlElement, but their
    ' members can as well (see sample code).
    Public States As StateList = New StateList

    Public Overrides Function GetXml(ByVal document As _
        System.Xml.XmlDocument) As System.Xml.XmlElement

        Dim element As XmlElement
        element = document.CreateElement("MobileService", _
            "http://www.thoughtpost.com/2004/MSDN/WSEXML")

        Dim nameelem As XmlElement
        nameelem = document.CreateElement("LocationName", _
            "http://www.thoughtpost.com/2004/MSDN/WSEXML")
        nameelem.InnerText = Me.Name
        element.AppendChild(nameelem)

        ' It's easy to include members in the serialization process if
        ' they too implement IXmlElement, such as EndpointReference
        Dim endpoint As XmlElement
        endpoint = Me.EndpointLocation.GetXml(document)

        element.AppendChild(endpoint)

        Dim stateelem As XmlElement
        stateelem = Me.States.GetXml(document)

        element.AppendChild(stateelem)

        Return element

    End Function

    Public Overrides Sub LoadXml(ByVal element As System.Xml.XmlElement)

        Me.Name = ""
        Me.EndpointLocation = Nothing
        Me.States.Items.Clear()

        Dim node As XmlNode
        For Each node In element.ChildNodes
            Select Case (node.LocalName)
                Case "EndpointReference"

                    Dim endpointelem As XmlElement = node
                    Me.EndpointLocation = _
                        New EndpointReference(endpointelem)

                Case "LocationName"
                    Me.Name = node.InnerText

                Case "States"
                    Dim stateelem As XmlElement = node
                    Me.States.LoadXml(node)

            End Select
        Next
    End Sub
End Class

Practice Makes Perfect

You probably won't need to take control of the serialization process for every message you send, but it helps to be familiar with the hooks when you do encounter messages that WSE 2.0 and the XML serialization process just don't approve of. Some of these techniques can also be used outside of Web services.

For example, you can use attributes or IXmlSerializable to control how your objects should be serialized in XML format to a database or file. The new interfaces and classes in the WSE 2.0 XML namespace provide support only within WSE Web services, but inside that domain, the support they provide is flexible enough to support the latest Web service specifications. If this technique is good enough to do that, it is good enough to handle just about any SOAP message your application can cook up to send out over the wire to any other Web service or application.

Chris Dix is the founder of Thoughtpost, a consulting and training company in Charlotte, NC, where he focuses on the .NET Framework and Web services. Chris is a frequent speaker at industry conferences, and he is the co-author of several books, including Professional XML Web Services (Wrox Press, 2001). You can contact Chris at chris@thoughtpost.com.


Security: Manipulate Privileges in Managed Code Reliably, Securely, and Efficiently

Security

Manipulate Privileges in Managed Code Reliably, Securely, and Efficiently

Mark Novak
Parts of this article are based on a prerelease version of the .NET Framework 2.0. All information pertaining to those sections is subject to change.
This article discusses:
  • Manipulating token privileges from managed code
  • Application domain isolation
  • Performance and kernel transitions
  • Constrained execution regions
This article uses the following technologies:
Security, C#, and the .NET Framework
Code download available at: TokenPrivileges.exe (140KB)
Contents
What Are Privileges?
Privileges in Unmanaged Code
Design Overview
Challenge 1: Living Inside Application Domains
Challenge 2: Changing Multiple Privileges
Challenge 3: Performance
Challenge 4: Deterministic Execution
Challenge 5: Safe Encapsulation of Windows NT Handles
Improving the Privilege Class

L
ike many developers of managed code, I frequently interact with system internals. One day, while working on the managed access control list (ACL) class library for the Microsoft® .NET Framework 2.0 (see my article in the November 2004 issue of MSDN®Magazine), I had to implement support for changing a security descriptor on an object. In some situations, this operation requires enabling Take Ownership and Security privileges. I poked around and discovered that there was no existing support in the .NET Framework for performing this task.

Simple enough, I thought—a P/Invoke of the AdjustTokenPrivileges API, and I'd be on my way. Little did I know how complex this task would become. I quickly discarded my original implementation due to several critical flaws, and about a dozen iterations later, I bring you the story of one of the coolest system programming quests in my recent memory.

This article contains two solutions to the problem: the first, simpler one, is tailored to the .NET Framework 1.1 and can be used today. The second solution incorporates several advanced features available only in the .NET Framework 2.0. Regardless of which version of the Framework you are developing for, and irrespective of whether you want to understand the many technical complexities of the implementation, the solution presented here is going to be a valuable addition to your programming toolbox.

What Are Privileges?

MSDN defines a privilege as the right of an account, such as a user or group account, to perform one of several system-related operations on the local computer, such as shutting down the system, loading device drivers, or changing the system time. There are important differences between privileges and access permissions or rights. In some cases, privileges can be thought of as a way to circumvent security protections placed on various operations by the operating system. In others, they allow one to perform system tasks that are not related to manipulating individual objects. For instance, a file's ACL may disallow a particular account (FABRIKAM\Joe) the right to read the file, yet Joe may still be able to read the file if he possesses the Backup privilege. Similarly, the Restore privilege will allow Joe write access to a file, even if the file's ACL forbids it.

The other aspect of privileges that makes them stand out is the way they are assigned. Whereas group membership is decided by either Active Directory® (and computed on domain controllers) or local Security Accounts Manager (SAM), privileges are always computed and assigned to the user by the local system, according to the local machine's policy (of course, group policy may be used to push privilege assignment onto the local machine in order to change the local policy). Finally, unlike security descriptors which are attached to individual objects, privileges are global in scope—they are part of security tokens, and apply universally to anything a user does on the machine he or she is logged on to.

To better understand how privileges work, consider the case of user FABRIKAM\Joe accessing a file on the system. In my example, Joe requests WRITE_OWNER (take ownership) access to the file. The access check algorithm will first establish the caller's security token. For that, it will attempt to open the thread token. Here, you may encounter one of two conditions: either the current thread is impersonating (meaning the thread has a security token associated with it) or it is not. By default, threads do not have security tokens associated with them, and the system proceeds to open the process token, an operation that is unlikely to fail.

The algorithm for checking the security descriptor will first attempt to grant all the rights it can, based on the currently enabled privileges in the security token. The WRITE_OWNER right we are interested in here will be granted if the Take Ownership privilege is enabled. WRITE_OWNER may also be granted if there is an access control entry (ACE) in the discretionary access control list (DACL) that grants the right explicitly. This (simplified) view of the access check algorithm is illustrated in Figure 1.

Note the clause "currently enabled privileges." It is not enough for a privilege to merely be present in the security token: it must also be in the "enabled" state for it to take effect. Whenever a privilege is present, it can be enabled or disabled at any time, so this mechanism is used as a safeguard against accidentally using an existing privilege to perform some action, a classic example of "running with least privilege."

In order to perform a privileged operation, your first step is to run code that enables the necessary privilege, remembering the initial state as it was before the privilege was enabled. Then you run the code that requires the privilege you just set, then revert the privilege back to its initial state.

Figure 1 Checking for Privilege
Figure 1 Checking for Privilege

Note that the last step here is not disabling the privilege but reverting it. This is because the privilege could have already been enabled, either by default or by an earlier explicit action and it would be bad form to disable it if so.

In my example, if Joe possessed (and remembered to enable) the Take Ownership privilege prior to opening the file for WRITE_OWNER access, the access check would succeed regardless of what the file's ACL had to say on the matter.

Privileges in Unmanaged Code

At its core, the task of changing the state of a privilege on a token has little to do with managed code; it is a sequence of system API calls. Understanding the required APIs and the proper invocation protocol is the first step toward creating a managed code implementation of this functionality.

All privileges have human-readable names. For instance the Take Ownership privilege is usually referenced by its name: SE_TAKE_OWNERSHIP_ NAME or SeTakeOwnershipPrivilege. However, in order to make a call to the API which manipulates token privileges—AdjustTokenPrivileges—the privilege must first be converted to its corresponding locally unique identifier (LUID). This is done with a call to a special API: LookupPrivilegeValue. The LUID corresponding to a particular privilege name does not change for as long as a system is up, so it makes sense to cache the LUID value if you are going to touch the privilege more than once.

Before AdjustTokenPrivileges can be called, you must obtain a security token that will become the target of manipulation. For reasons having to do with application domain isolation that I'll describe later, it's not proper to modify the process token, although the Windows® API does allow it. Therefore, you start by trying to obtain the thread token with a call to OpenThreadToken. Due to the Windows NT® security architecture, threads normally run without a security token, defaulting it to the process, so the initial call to OpenThreadToken will likely fail with ERROR_NO_TOKEN. In this case, the right thing to do in order to avoid messing with the process state is to place a copy of the process's security token on the thread, an operation called impersonation. There are several ways to do this (you can call ImpersonateSelf or ImpersonateLoggedOnUser, for instance), but for reasons described later in a section that deals with performance issues, I'll take a different approach. Open the process token with OpenProcessToken, duplicate this token handle with DuplicateTokenEx, and use the output of the latter to complete the impersonation with a call to SetThreadToken. After this step, the thread is impersonating and you can complete the operation by calling the API designed for changing the privileges on a token—AdjustTokenPrivileges. This process is illustrated in Figure 2.

Figure 2 API Calls
Figure 2 API Calls

As you can see, the call to LookupPrivilegeValue converts the privilege name into a LUID, after which OpenThreadToken checks whether the thread is impersonating (though usually it isn't). OpenProcessToken gets the process security token (requesting DUPLICATE_HANDLE access) and DuplicateTokenEx duplicates the process token handle, obtaining another impersonation token with rights sufficient for subsequent operations (impersonate, query, and adjust token privileges). The SetThreadToken method sets the copy of the process handle on the thread, such that the thread is now impersonating. Finally, AdjustTokenPrivileges takes as its parameter the new state of the privilege, and returns the initial state.

You must be careful to remember this initial state, otherwise it would be impossible to revert to it once the privilege is no longer required. Once the user is ready to revert the privilege, AdjustTokenPrivileges can be called again, this time passing as the new state what was returned earlier as the initial state of the privilege. If you started by impersonating, the thread can be reverted to its original impersonation state by calling RevertToSelf.

Things are pretty simple so far. If the logic was confined to unmanaged code, you'd be finished by now and there would be no need for an entire article on the subject. The real fun starts when porting these ideas to managed code.

Design Overview

The class for manipulating privileges that I'm going to develop here will have the public interface illustrated in Figure 3. Once instantiated, Privilege objects are used as shown here:

Privilege debugPrivilege = new Privilege(Privilege.Debug);
  try
{
    debugPrivilege.Enable();
    // perform tasks requiring the privilege (the payload) ...
} catch {
    debugPrivilege.Revert();
    throw;
} finally {
    debugPrivilege.Revert();
}
The call to the Revert method is placed inside the finally block as a precaution to remember to revert the privilege despite any exception that might be thrown by the code following the Enable method. Forgetting to revert a privilege is a serious programming error. Any call to Enable or Disable must be followed by a call to Revert. The design makes it illegal to call Disable immediately after Enable, or vice versa. A call to Revert can be skipped only if the NeedToRevert property returns false. For reasons that will become apparent later, methods of the Privilege class must be invoked on the same thread that the Privilege object was created on.

Figure 3 Public Interface of the Privilege Class (Close Figure 3)
public delegate void PrivilegedCallback(object state);

[ComVisible(false)]
public sealed class Privilege
{
    public const string Debug = "SeDebugPrivilege";
    public const string Security = "SeSecurityPrivilege";
    public const string TakeOwnership = "SeTakeOwnershipPrivilege";
    ... // continue with constants for all known privileges

    public Privilege(string name)
    public void Enable()
    public void Disable()
    public void Revert()
    public bool NeedToRevert { get; }

    public static void RunWithPrivilege(
        string privilege, bool enabled, PrivilegedCallback callback, object state)
}

A careful reader will have noticed another call to Revert inside a catch block. The reason for this is to avoid a security vulnerability known as a luring attack. If the payload throws an exception, the exception is going to propagate up the callstack twice—once in search of an exception filter, and the second with an actual stack unwind. This is the same two-pass exception handling mechanism that Win32® structured exception handling (SEH) follows. You can't write a user exception filter in C# today, but you can in Visual Basic®, Visual C++®, or in plain Microsoft intermediate language (MSIL). The code inside the exception filter could then run with elevated privilege, which is not intended behavior and can be damaging in certain situations. This is why a very broad catch block is inserted after the try block. This catch block is guaranteed to catch all exceptions, stopping the luring attack. Inside this catch block, I revert the privilege, and rethrow the exception.

In my implementation, all calls to Windows APIs are wrapped by an internal class called NativeMethods. You will see references to this class in subsequent code snippets.

In order to implement the functionality correctly, you must face a number of challenges, most of which are unique to managed code programming. I'll describe these challenges and their solutions in the following sections.

Challenge 1: Living Inside Application Domains

The common language runtime (CLR) uses application domains to provide isolation between applications. You can find an excellent overview of application domains at Using Application Domains, so I won't repeat it here. Suffice it to say that extreme care must be taken to maintain this isolation. Operations that an application performs should in no way affect the operation of other applications running inside its sister application domains.

This is therefore the first hurdle you must overcome in order to manipulate privileges safely in managed code. In unmanaged code applications, even multithreaded ones, it is sometimes okay to enable the privilege on the process token, thereby affecting all threads inside the process. For instance, a backup application might enable the Backup privilege globally, so all of its threads can then open files regardless of their ACLs. This approach clearly does not work for some managed apps. Multiple application domains can be hosted inside a single Windows process, so whatever happens to the process token affects every application domain inside that process. Of course, this isn't unique to managed applications. When developing native applications, you wouldn't tamper with the process state if you suspected that your application could be co-hosted with other applications inside the same Windows NT process, as is, for example, possible with Windows NT services.

The Privilege class takes several precautions in order to be a good app domain citizen. One such precaution limits privilege manipulation to the thread on which the privilege is required. The privilege code remembers the thread on which the object was constructed, and forces all subsequent operations to take place on the same thread. The Thread.CurrentThread property is helpful in this instance. This is why when you read the code, you will see a private member defined as:

private readonly Thread currentThread = Thread.CurrentThread;
All subsequent attempts to call methods of a Privilege object begin with a check against this cached thread instance:
// All privilege operations must take place on the same thread
if (!this.currentThread.Equals(Thread.CurrentThread))
{
    throw new InvalidOperationException(
        "This operation must be performed on the same " +
        "thread that created this instance.");
}

As an aside, this mechanism works even if there is a user-mode scheduler in place, and if scheduling takes place on fibers instead of threads. Thread ID equality will continue to work even if your code is preempted and resumed on a different fiber.

The sequence of steps that lets you avoid modifying the security token of the hosting process has been outlined in the previous section. Figure 4 shows what it looks like in managed code (several details were stripped or modified in this sample; see the full solution in the code download for complete implementation).

Figure 4 Maintaining Application Domain Isolation (Close Figure 4)
bool success = true;
if (false == NativeMethods.OpenThreadToken(
    NativeMethods.GetCurrentThread(),
    TokenAccessLevels.Query | TokenAccessLevels.AdjustPrivileges,
    true, ref this.threadHandle ))
{
    error = Marshal.GetLastWin32Error();
    if (error != NativeMethods.ERROR_NO_TOKEN)
    {
        success = false;
    }
    if (success == true)
    {
        error = 0;
        if (false == NativeMethods.OpenProcessToken(
            NativeMethods.GetCurrentProcess(),
            TokenAccessLevels.Duplicate, ref processHandle))
        {
             error = Marshal.GetLastWin32Error();
             success = false;
        }
    }
    if (success == true)
    {
        if (false == NativeMethods.DuplicateTokenEx(
            processHandle,
            TokenAccessLevels.Impersonate | TokenAccessLevels.Query |
                TokenAccessLevels.AdjustPrivileges,
            IntPtr.Zero, SecurityImpersonationLevel.Impersonation,
            TokenType.Impersonation, ref this.threadHandle))
        {
             error = Marshal.GetLastWin32Error();
             success = false;
        }
    }
    if (success == true)
    {
        if (false == NativeMethods.SetThreadToken(
            IntPtr.Zero, this.threadHandle))
        {
            error = Marshal.GetLastWin32Error();
            success = false;
        }
    }
    if (success == true)
    {
        this.isImpersonating = true;
    }
}

When the time comes to revert the privilege, one of two code paths is taken, depending on the initial state of the thread at the time of creation. If the thread was impersonating to begin with, the privilege can be reverted by a call to AdjustTokenPrivileges, passing what was originally returned as the initial state of the privilege as its new state. If the thread was not impersonating when the privilege was enabled, the privilege can be reverted to its initial state (that of defaulting its security token to the process) by simply removing the thread token. This can be accomplished by calling RevertToSelf. In this particular case, calling AdjustTokenPrivileges prior to RevertToSelf is a legal, but unnecessary, step.

In either case, the process token remains unaffected by the manipulations you perform, so sister application domains will maintain their proper degree of isolation.

Challenge 2: Changing Multiple Privileges

An easy caveat to overlook when designing privilege manipulation code is the possibility that the state of multiple privilege objects will be manipulated on the same thread at the same time. For instance, if you want to change an owner and a system access control list (SACL) on an object, you'll want to open the file handle for both WRITE_OWNER and ACCESS_SYSTEM_SECURITY access. In order to do this, the caller may have to enable both Take Ownership and Security privilege at the same time. The algorithm developed so far looks like it handles it just fine (here you assume that the thread is not impersonating to begin with).

Let's start with enabling Privilege A. You must first impersonate, enable the privilege, and remember the previous state of the privilege (A-before) so that the privilege can later be reverted. Now that the thread is already impersonating, to enable Privilege B, you simply enable the privilege and remember the previous state of the privilege (B-before).

After all operations have been performed with the privileges enabled, they now need to be reverted. To revert Privilege B, revert the privilege to B-before, but do not revert to self, since the thread was already impersonating before Privilege B was enabled. To disable Privilege A, there is no need to revert the privilege, since reverting to self will take any changes to the thread token with it, including changes made to privileges.

The callers of the Privilege class cannot be reasonably expected to always properly nest their privilege manipulations as described. A caller can make a tiny mistake, or have a legitimate need, leading them to revert Privilege A before Privilege B. If this happens, the logic unravels. When Privilege A is reverted, the reversion to self will cause all changes to the thread token to be lost, including those for Privilege B.

Clearly, the algorithm needs tweaking. Multiple instances of the Privilege class should all cooperate, as long as they are all instantiated on the same thread. It should not matter whether they are instantiated within the same function call, or in different places in the same callstack. A facility that comes in handy in this case is known to Windows system developers as Thread Local Storage (TLS), also known to managed code programmers as Local Data Store (LDS). TLS is data that is available exclusively to all code running inside a particular thread. Under this arrangement, every thread has its own value of the data. This is different from having global or static data, where all threads share the same value. TLS is going to be helpful in this instance because you can use it to keep track of the state of privilege manipulation on a given thread.

My implementation introduces a private class inside the Privilege class called TlsContents. Instances of this class are disposable, reference-counted wrappers around a token handle, which keep track of whether the token handle was placed on the thread via impersonation. The internal workings of this class are worth discussing, in particular the manner in which the enclosing Privilege class makes use of it. The technique boils down to this: for as long as any privileges are being manipulated on a thread, there is going to be a local data store slot allocated with a TlsContents instance inside of it. This instance is going to be reference-counted with as many references as there are privileges being changed on the thread.

Before a privilege can be toggled on a thread, a TlsContents instance is created and a reference to it is placed inside the local data store slot inside that thread. The TlsContents constructor performs the necessary steps to ensure that the thread is impersonating by executing the code in Figure 4. It is not until all privileges are reverted that the TlsContents instance is disposed, taking away the thread token with it, if appropriate.

Armed with this functionality, the privileges can now be enabled and reverted in any succession. Before a privilege can be toggled, you must make use of TLS, as shown in Figure 5.

Figure 5 Making Use of Thread Local Storage (Close Figure 5)
// Private static member that contains the local data store slot
private static LocalDataStoreSlot tlsSlot = Thread.AllocateDataSlot();

// Retrieve TLS state
this.tlsContents = Thread.GetData( tlsSlot ) as TlsContents;

if (this.tlsContents == null)
{
    // no privileges touched yet on this thread
    this.tlsContents = new TlsContents();
    Thread.SetData(tlsSlot, this.tlsContents);
}
else
{
    // increment the reference count of the existing instance
    this.tlsContents.IncrementReferenceCount();
}

The TokenHandle member of the TlsContents instance can, at any point, be used to reference the thread token, as is illustrated in the following call:

NativeMethods.AdjustTokenPrivileges(
    this.tlsContents.TokenHandle, false, ref newState,
    (uint)Marshal.SizeOf(previousState),
    ref previousState, ref previousSize)
When the time comes to revert the privilege, reference counting is used to determine whether it is time to revert from impersonation:
if (this.tlsContents != null)
{
    if (0 == this.tlsContents.DecrementReferenceCount())
    {
        this.tlsContents = null;
        Thread.SetData(tlsSlot, null);
    }
}
The DecrementReferenceCount method will do the necessary cleanup work when the reference count reaches 0; the caller needs to simply null out the thread local store value.

Challenge 3: Performance

You should keep in mind that privilege manipulation is a very expensive process. In fact, doing it too frequently can adversely affect the performance of the entire application, so in some cases it makes sense to have a thread that runs with the privilege enabled, and enqueue the tasks that require the privilege to that thread. It is instructive to go through a series of steps involved in enabling a privilege. To begin, let's consider the most computationally significant of steps involved: kernel transitions. These are API calls that perform their work by switching from user mode to kernel mode and back. Kernel transitions are so expensive (requiring thousands upon thousands of instructions) that they completely dwarf all other processing of the Privilege class. Ultimately, how fast a privilege can be enabled depends on little else but the number of kernel transitions involved.

For instance, one of the early attempts at the Privilege class implementation achieved impersonation by making the following sequence of Windows API calls: OpenThreadToken, ImpersonateSelf, OpenThreadToken, and AdjustTokenPrivileges. Ideally, the operation could be completed simply by opening the thread token handle and then adjusting the token privileges, resulting in two transitions. However, as noted earlier, this process will usually fail. As such, after opening the thread token, the ImpersonateSelf Windows API was used. This function first opens the process token handle, duplicates the process token, and then sets the copy of the handle on the thread. It then closes the process token and closes the thread token. This sequence results in an additional five transitions, bringing the total to eight transitions for the entire operation.

This number can be improved upon by addressing some inefficiencies. The first inefficiency comes from opening (and then closing) the process token every time a privilege is toggled. The process security token is not going to change and can be cached. The second inefficiency comes from ImpersonateSelf, which sets a token on a thread, just to close it. After ImpersonateSelf returns, the thread token needs to be opened again (at a cost of an extra kernel transition). The optimized solution opens the process token once and then caches it, as shown in the code in Figure 6.

Figure 6 Caching the Process Token (Close Figure 6)
static IntPtr processHandle = IntPtr.Zero;
static readonly object syncRoot = new object();

...

if (processHandle == IntPtr.Zero)
{
    lock (syncRoot)
    {
        if (processHandle == IntPtr.Zero)
        {
            if (false == NativeMethods.OpenProcessToken(
                NativeMethods.GetCurrentProcess(),
                TokenAccessLevels.Duplicate, ref processHandle))
            {
                error = Marshal.GetLastWin32Error();
                success = false;
            }
        }
    }
}

The "check, lock, check again" code (known as a "double-checked lock") you see in Figure 6 is a standard implementation pattern for thread-safe setting of write-once static or global variables. (Note that this pattern is unsafe in unmanaged C++ due to the lack of a strong memory model in the language. The CLR uses a strong memory model on all platforms, so this pattern is safe.) Having the process handle cached makes all subsequent privilege operations cheaper by two kernel transitions, saving one open and one close call—clearly a good thing.

With the process handle in hand, the rest of the impersonation operation can be optimized by opening a thread token (as mentioned already, this will likely fail, but you have to make sure), duplicating the process token with DuplicateTokenEx, and setting the duplicate token on the thread with SetThreadToken.

An experienced system programmer will have noticed that DuplicateTokenEx was called instead of DuplicateToken. The reason for this stems from the fact that the duplicated handle will be used for two purposes—impersonation (setting it on the thread) and adjusting the token privileges by calling the AdjustTokenPrivileges API. DuplicateTokenEx is an API that allows you to specify the rights with which the handle is to be opened. This is different from DuplicateToken, which defaults to Query and Impersonate rights only.

The new sequence of API calls, after caching the process token, takes only four kernel transitions (open, duplicate, set, adjust) instead of the original eight, so it is going to be almost twice as fast. Not a bad payback for a little attention to kernel transitions!

For the revert operation, the steps (assuming revert to self without readjusting the privileges) involve calling RevertToSelf (amounts to setting a NULL token handle on the current thread) and closing the handle. As you can see, revert is cheaper, requiring only two operations, which could have been three if you didn't understand what you were doing by calling AdjustTokenPrivileges prior to RevertToSelf.

This is not the end of our bag of performance optimizing tricks, however. Another, simpler performance gain comes from the mapping of privilege names to the corresponding LUIDs. The privilege class internally maintains a static mapping of privilege names to their corresponding LUIDs. Since LUID values corresponding to each privilege name remain the same for as long as a system is up, the LUIDs can be cached in order to improve the performance of subsequent lookups. This caching can result in a noticeable performance improvement because each call to the LookupPrivilegeValue API results in a Local Procedure Call (LPC) into the Local Security Authority process—a cost best avoided. The Privilege class uses a HybridDictionary data structure for doing this job. This data structure has exactly the same public interface as the Hashtable class, but with significantly better performance characteristics when dealing with small data sets. The implementation of this logic comes down to nothing more than performing locking and lookup correctly. A slightly abbreviated version of this logic is illustrated by the code in Figure 7.

Figure 7 Caching Privilege LUIDs (Close Figure 7)
private static HybridDictionary privileges = new HybridDictionary();
private static HybridDictionary luids = new HybridDictionary();
private static ReaderWriterLock privilegeLock = new ReaderWriterLock();

private static NativeMethods.LUID LuidFromPrivilege(string privilege)
{
    NativeMethods.LUID luid;

    // Look up the privilege LUID inside the cache
    try
    {
        privilegeLock.AcquireReaderLock(Timeout.Infinite);

        if (luids.Contains(privilege))
        {
            luid = (NativeMethods.LUID)luids[privilege];
            privilegeLock.ReleaseReaderLock();
        }
        else
        {
            // LUID not found! Look it up, but do it outside of any lock.
            privilegeLock.ReleaseReaderLock();
            luid.LowPart = 0;
            luid.HighPart = 0;
            if (false == NativeMethods.LookupPrivilegeValue(
                null, privilege, ref luid))
            {
                int error = Marshal.GetLastWin32Error();
                ... // throw appropriate exception based on the error
                    // code
            }
            privilegeLock.AcquireWriterLock(Timeout.Infinite);
        }
    }
    finally
    {
        if (privilegeLock.IsReaderLockHeld)
        {
            privilegeLock.ReleaseReaderLock();
        }
        if (privilegeLock.IsWriterLockHeld)
        {
            // check the cache again! After all, we are in
            // a multithreaded world and things could have changed
            if (!luids.Contains(privilege))
            {
                luids[privilege] = luid;
                // map both ways to enable better error reporting
                privileges[luid] = privilege;
            }
            privilegeLock.ReleaseWriterLock();
        }
    }
    return luid;
}

A careful reader will have noticed something strange about how the cache is updated: you obtain the writer lock inside the try block, yet the cache is updated inside the finally block. The reasons for this will be explained in a later section dealing with writing reliable code. When using the .NET Framework 1.1, this trick is not really necessary, but the code stays this way to make it as close as possible to the version written for version 2.0 of the Framework.

This completes the version of the Privilege class tailored to the .NET Framework 1.1. The remainder of this article deals with features available only in .NET Framework 2.0. Even if you are developing for the .NET Framework 1.x, you should continue reading. There are some features in the .NET Framework 2.0 that provide guarantees for writing deterministic, reliable code that .NET Framework 1.x cannot possibly match. If you discover that those guarantees are important to your code, you may have to postpone developing Privilege code until .NET Framework 2.0 is available.

Challenge 4: Deterministic Execution

This section deals with a very thorny aspect of managed code programming—writing reliable code. Authoring reliable code can be a daunting task. There is an entire class of "exceptional" conditions that developers generally do not expect in the normal code execution path. These exceptional conditions include thread aborts, out of memory conditions, and stack overflow exceptions (unmanaged code has equivalents to many of these problems; stack overflows and allocation failures aren't unique to managed development and are difficult to deal with correctly). In managed code, these conditions generate what are usually called asynchronous or out of band exceptions. How can the code make any guarantees regarding its behavior under those types of conditions?

It turns out that making the privilege manipulation logic deterministic, even in the face of those catastrophic exceptions, is very important. You have seen thus far that the process of manipulating privileges is something that is done in several computationally expensive and error-prone steps. This sequence of steps, once initiated, must run to completion. Failure to do so can leave the thread in an inconsistent state (like impersonating when it shouldn't be). This sort of challenge could not be addressed using facilities available in the .NET Framework 1.x. Fortunately, there are new features of the .NET Framework 2.0 called Reliability Contracts and Constrained Execution Regions and they are going to be helpful to the Privilege class.

What follows is a brief introduction to constrained execution regions (CERs), focusing on the subset of CER functionality that the Privilege class uses. Within the .NET Framework 2.0 is ReliabilityContractAttribute, a custom attribute that defines a contract specifying the reliability guarantees that a method makes in the face of exceptional conditions. In essence, this attribute enables reliability guarantees to be both documented and discoverable via a programming model. Constrained execution regions help implement code that satisfies the contract specified by the ReliabilityContractAttribute by moving CLR-induced failures to either before or after a block of code that executes under some constraints. CLR-induced failures include running out of memory while JITing code, as well as aborting a thread. The reliability contract can be stated in one of several ways, as shown in Figure 8.

Figure 8 Reliability Contract (Close Figure 8)

In order to declare that | Use the attribute
A method can potentially corrupt all state (instance, process, and so on) in the face of exceptional conditions | [ReliabilityContract(Consistency.MayCorruptProcess, Cer.None)]
A method can potentially corrupt shared state within the app domain (for example, static variables) in the face of exceptional conditions | [ReliabilityContract(Consistency.MayCorruptAppDomain, Cer.None)]
A method can potentially corrupt instance state in the face of exceptional conditions | [ReliabilityContract(Consistency.MayCorruptInstance, Cer.None)]
A method will not corrupt state but may fail in the face of exceptional conditions | [ReliabilityContract(Consistency.WillNotCorruptState, Cer.MayFail)]
A method will not corrupt state and will not fail even in the face of exceptional conditions (modulo failures due to invalid arguments) | [ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]

How are these guarantees implemented? In the first three cases, the developer does nothing special to the code—he inspects the code and simply documents the behavior should exceptional conditions occur. If state transitions in a method affect just the instance, the developer should go with option three; if manipulations are more extensive, options one or two are best.

Methods of the privilege class may fail for a variety of reasons (the caller may not possess the privilege, and there could be a variety of other operating-system level failures). They should not, however, have the luxury of corrupting the system state. Once a privilege manipulation is initiated, the following three guarantees must be made:

  • Either a privilege will be placed in its desired state, or it won't. No legal intermediate states exist.
  • In case a privilege is placed in its desired state, the instance of the Privilege class must be placed in a consistent state that would allow a Revert operation to be executed.
  • If something happens along the way and the privilege can not be toggled, the thread token and the Privilege instance must be reverted exactly to the state in which the operation was initiated. Failure to do so will leave the thread in an inconsistent state—a problem that is more severe than corrupting just a Privilege instance.

Thus, the correct reliability contract for Privilege methods is the same as the fourth option in Figure 8, using a ReliabilityContractAttribute with Consistency.WillNotCorruptState and Cer.MayFail. You will see several methods in the .NET Framework 2.0 version of the Privilege class decorated in this manner.

So much for decorating methods. How about actually implementing the guarantees stated by the reliability contract? This is where constrained execution region functionality comes into play. The code is made reliable in the face of asynchronous exceptions by instructing the runtime to perform a sequence of steps known internally as "eagerly preparing" the compiled code.

This means that the CLR has already done all the work necessary to run your code. As an example, a method may need to be just-in-time (JIT)-compiled and may call a method in another assembly. This may require loading another assembly, running a few class constructors, then compiling your callee as well as your calling method. Eagerly preparing code ensures that the failures the CLR may introduce to your code are hoisted to the point at which you do the eager preparation. Asynchronous exceptions are then delayed over catch and finally blocks. This guarantees that catch and finally blocks will execute to completion in case of asynchronous system exceptions. All this is achieved via the implementation pattern (see Figure 9).

Figure 9 Programming with Constrained Execution Regions (Close Figure 9)
using System.Runtime.CompilerServices;

[ReliabilityContract(Consistency.WillNotCorruptState, Cer.MayFail)]
public void SomeMethod()
{
    RuntimeHelpers.PrepareConstrainedRegions();
    try
    {
        // Implementation goes here
    }
    catch( /* any necessary catch blocks */ )
    {
    }
    finally
    {
        // put in the back-out code here to ensure consistency
        // the code is guaranteed to run, ensuring reliability
    }
}

Before the try block is entered, a special method, RuntimeHelpers.PrepareConstrainedRegions, is invoked. This instructs the runtime to perform any operations necessary to ensure that the following catch and finally blocks will be allowed to execute. Pretty simple for such a complex piece of functionality, isn't it? It gets more interesting still. In cases in which the code undertakes a nontrivial sequence of steps, making the back-out code too complex, the trick that is shown in Figure 10 can be employed.

Figure 10 Making Code Uninterruptible (Close Figure 10)
RuntimeHelpers.PrepareConstrainedRegions();
try
{
    // The payload is entirely in the finally block.
    // This is how we ensure that the code will not be
    // interrupted by catastrophic exceptions.
}
finally
{
    try
    {
        // payload goes here
    }
    catch( /* any necessary catch blocks */ )
    {
    }
    finally
    {
    }
}

The arrangement in Figure 10, strange as it may look, is exactly what you will see inside the version of the Privilege class that works within the .NET Framework 2.0. And now you know why!

Challenge 5: Safe Encapsulation of Windows NT Handles

Constrained execution regions are used extensively in the .NET Framework 2.0. One very common area where they are applied is in dealing with handle objects. In the case of the Privilege class, let's turn our attention to token handles.

Token handles can be tricky to work with correctly. In the .NET Framework 1.x, all handle values were manipulated as IntPtr objects (essentially, thinly wrapped numerical values). This approach is inherently unsafe and insecure for a variety of reasons, which led to the development of the SafeHandle family of classes in the .NET Framework 2.0.

The SafeHandle class in the .NET Framework 2.0 was designed with one important goal in mind—to make sure that the CLR never leaks an operating system handle. These handle objects are created in a noninterruptible way and released even in catastrophic failure scenarios. They can be passed to unmanaged code. They are also reference counted to prevent security problems related to a phenomenon known as handle recycling. The SafeHandle class is therefore very important. The CLR team believes that without this functionality, it is impossible to write reliable managed code.

In the .NET Framework 1.1, handles were opened using a line of code that looks something like this:

IntPtr myFileHandle = NativeMethods.FindFirstFile(fullPath, data);
This line will leak a file handle if, for example, a ThreadAbortException is thrown before the handle is stored in the myFileHandle local variable. Additionally, even if the value was stored in a managed object, there would be no guarantee that the finalizer on that object would run. SafeHandle guarantees finalization and is safe in the face of catastrophic exceptions. The interop layer in the .NET Framework 2.0 recognizes P/Invoke methods that return SafeHandle subclasses (as return values or as an out parameter) as a special case of P/Invoke and takes the appropriate action to ensure the handle's identifier is wrapped successfully in the SafeHandle object.

Special handling of SafeHandle subclasses does not end there. Every time a SafeHandle instance is passed to the unmanaged side, the interop layer will increment its reference count. When the call is completed, the ref count will be decremented. This reference count is separate from the lifetime of the object in the garbage collection (GC) heap, but instead refers to the number of threads that may be currently using a SafeHandle.

This reference-counting feature was introduced to eliminate a security problem related to handle reuse (also known as handle recycling). Consider this scenario: two threads are executing the same semi-trusted code that accesses some publicly available file. One thread gets an OS handle with value x. Before any read is performed, this thread is context switched out. The second thread gets the same handle, performs some read, and then closes the file. Without ref counting, when the file is closed, handle is considered invalid and available for recycling. The third thread that executes fully trusted code opens some private file, gets a handle back from the OS, and this handle has the same value x but now points to different file. When the first thread is started again, it will have an access to something it shouldn't. Since SafeHandle subclasses are ref counted when passed to unmanaged code, they won't be released back to the OS via a CloseHandle-like method until their ref count reaches zero.

The Privilege class wraps the Windows NT token handles with the SafeTokenHandle class illustrated in Figure 11. This class derives from SafeHandleZeroOrMinusOneIsInvalid, a lengthy but descriptive name of a SafeHandle subclass that parallels the behavior of Windows NT handles (whose invalid values are 0 and -1). This implementation is boilerplate and you can use it in all instances that call for the creation of SafeHandle subclasses. The only significant difference between different types of handles is the name of the API that has to be called in order to close the handle (in this case, CloseHandle is called).

Figure 11 SafeTokenHandle Class (Close Figure 11)
internal sealed class SafeTokenHandle :
    SafeHandleZeroOrMinusOneIsInvalid
{
    private SafeTokenHandle() : base (true) {}

    internal SafeTokenHandle(IntPtr handle) : base (true)
    {
        SetHandle(handle);
    }

    internal static SafeTokenHandle InvalidHandle
    {
        get { return new SafeTokenHandle(IntPtr.Zero); }
    }

    [DllImport("kernel32", SetLastError=true),
     SuppressUnmanagedCodeSecurity,
     ReliabilityContract(Consistency.WillNotCorruptState, Cer.Success)]
    private static extern bool CloseHandle(IntPtr handle);

    protected override bool ReleaseHandle()
    {
        return CloseHandle(handle);
    }
}

The .NET Framework 2.0 version of the Privilege class uses SafeTokenHandle in every place where the .NET Framework 1.1 version uses IntPtr. You will see this if you examine the code. Switching to safe handles is really that simple.

Improving the Privilege Class

The Privilege class has one additional feature specifically for invoking privileged operations in a correct and safe manner. This is achieved via a special PrivilegedCallback delegate and a corresponding RunWithPrivilege method. The method is illustrated in the code in Figure 12.

Figure 12 Using Privileges with Delegates (Close Figure 12)
public delegate void PrivilegedCallback(object state);

public static void RunWithPrivilege(
    string privilege, bool enabled, PrivilegedCallback callback, object state)
{
    if (callback == null) throw new ArgumentNullException("callback");

    Privilege p = new Privilege(privilege);
    RuntimeHelpers.PrepareConstrainedRegions();
    try
    {
        if (enabled) p.Enable();
        else p.Disable();

        callback(state);
    }
    catch
    {
        p.Revert();
        throw;
    }
    finally
    {
        p.Revert();
    }
}

As you can see, the RunWithPrivilege method does all the necessary setup work by creating a privilege instance, preparing a constrained region, and structuring the code in a way that enables or disables the privilege as necessary, and then invoking the delegate. All the while, privilege reversal is assured, and luring attacks are prevented by strategic placement of the catch block. If you do not use RunWithPrivilege, make sure that the code that invokes privileged operations is structured in exactly the same way.

As you can see, privilege tweaking is not for the faint of heart. The logic necessary to do it correctly touches on a number of advanced managed code and system programming techniques, such as constrained execution regions and safe handles. Some of the tricks necessary to protect your code are not available until the .NET Framework 2.0.

Like most software, though, my implementation is not perfect. For instance, CLR hosting processes other than Windows may find operations on thread tokens offensive and prevent them through the very low-level CLR hosting interfaces. Under such hosting environments, calls like ImpersonateSelf or SetThreadToken have no effect and the thread token is not changed. This restriction can be circumvented by calling methods in the WindowsIdentity class, but that is beyond the scope of this article.

Another great improvement to my implementation would be to supplement it with a dedicated code access security (CAS) permission. As coded, the Privilege class requires unmanaged code permissions in order to execute because of its heavy reliance on the Windows API. Decorating this class with demands for a dedicated TogglePrivilege permission would allow you to create a version of the class that can live inside "allow partially trusted callers" assemblies. Ideally, such CAS permission would have the granularity of individual permission names, rather than a blanket permission to enable anything in the caller token's process.

All in all, however, this code is likely to save you countless hours of design, implementation, and debugging work. The full implementation for both the .NET Framework 1.x and the .NET Framework 2.0 are available for download from the MSDN Magazine Web site. In the meantime, it would be a good idea for you to review your existing codebase for calls to the AdjustTokenPrivileges API in order to make sure all the pitfalls that I discussed in this article have been handled.

Mark Novak is a Security Development Lead at Microsoft. Mark has been with Microsoft since 1995. After contributing to several releases of Microsoft Exchange Server, he has spent the last five years concentrating on software security. This is his second article for MSDN Magazine. Mark can be contacted at markpu@microsoft.com.


Go Global: Make the .NET World a Friendlier Place with the Many Faces of the CultureInfo Class

Go Global

Make the .NET World a Friendlier Place with the Many Faces of the CultureInfo Class

Michael Kaplan
This article discusses:
  • What CultureInfo can do for you
  • Collation, casing, formatting, and resource loading
  • CultureInfo changes in the .NET Framework 2.0
This article uses the following technologies:
.NET Framework and C#
Contents
Getting CultureInfo from Properties
CultureInfo from Initial Settings
Getting CultureInfo from Input Methods
Creating New CultureInfo Objects
Uppercasing and Lowercasing
Collation
Formatting and Parsing
Resource Loading
Encoding
Input Languages
Recent Changes
Custom Cultures and Beyond

O
ne of the most widely used classes in the Microsoft® .NET Framework is CultureInfo, a class whose objects are used for resource loading, formatting, parsing, casing, sorting, and other conventions that change as the language, location, or writing system is changed. It's a relatively complex class and can be tricky to use correctly in every situation.

In this article I'll walk through some of these scenarios and provide enough information about the behavior, best practices, and consequences of a wrong decision to allow you to make the right choices for your use of CultureInfo and its related classes in the System.Globalization namespace in your future projects.

It all starts with the creation of the object, and there are many ways to get a CultureInfo object. You can use the cultures available through the built-in CultureInfo.CurrentCulture, CultureInfo.CurrentUICulture, or CultureInfo.InvariantCulture properties. In addition, you can use a built-in CultureInfo based on selected or installed input methods. You can use an application-created CultureInfo instance, or alternatively, you can use no culture at all.

Getting CultureInfo from Properties

The CultureInfo instance that is returned from the CultureInfo.CurrentCulture property is based on the locale the user selected in Windows® Regional Options (shown in Figure 1). This is called the "user locale" for programmers, and "language" for Standards and Formats in Windows XP and Windows Server™ 2003. This value can be changed in .NET at the thread level, but it does not change while the application is running when the language in Regional Options is changed by the user. In fact, even changes to individual settings are not detected unless the CultureInfo.ClearCachedData method is called.

Figure 1 Windows Regional Options as CurrentCulture
Figure 1 Windows Regional Options as CurrentCulture

Unfortunately, there is no specific event that is raised when the user changes settings in Regional Options. There is an event raised when the current input language has changed, but not when the general settings change. However, Windows broadcasts the WM_SETTINGSCHANGE message any time this setting is changed by a user. The code shown in Figure 2 will help an application created in the .NET Framework respond to changes in this context. It listens for the appropriate message, namely, WM_SETTINGCHANGE with the string "intl" in the LPARAM. If that message is passed, the code makes the appropriate changes either to the whole locale or to the individual settings that may have changed. It is important to use methods such as these if you want a .NET-based application to respond to changes that the user has made through the control panel.

Figure 2 Responding to Regional Options Changes (Close Figure 2)
private const int WM_SETTINGCHANGE = 0x001A;

[DllImport("kernel32.dll", ExactSpelling=true)]
private static extern int GetUserDefaultLCID();

CultureInfo m_ciOld = new CultureInfo(GetUserDefaultLCID());

protected override void WndProc(ref Message m) {
  switch(m.Msg)
  {
    // change in a systemwide or policy setting
    case WM_SETTINGCHANGE:

      if(m.LParam != IntPtr.Zero) {
        int localeCur = GetUserDefaultLCID();
        string val = Marshal.PtrToStringAuto(m.LParam);

        if(val == "intl") {
          // change in locale settings
          Thread thread = Thread.CurrentThread;

          if(thread.CurrentCulture.LCID != localeCur() &&
            thread.CurrentCulture.LCID == m_ciOld.LCID) {
            // user default locale has changed — so
            // change the current culture.
            thread.CurrentCulture = new CultureInfo(localeCur);
          }
          else
          {
            // Some individual setting has changed — so
            // clear the cached data to pick up that change.
            thread.CurrentCulture.ClearCachedData();
        }

        m_ciOld = new CultureInfo(localeCur);
      }
    }
    break;
  }

  base.WndProc(ref m);
}

When deciding whether to use the CurrentCulture or another option, keep in mind that the CurrentCulture setting is the user's preference for how information such as date, time, and number formatting and sorting are presented.

CultureInfo from Initial Settings

The CurrentUICulture object's initial setting is based on the interface language of Windows. When Windows has built-in Multi-lingual User Interface (MUI) functionality, this choice can be set by the user; otherwise it is the language into which Windows is localized. As the application developer, you are the provider of the user interface and whatever language or languages you want to provide. You can even choose to provide your own user interface for language selection if you want your users to be able to change to one of the available languages. In most cases, it's only appropriate to use the CurrentUICulture when you're choosing a language for your user interface. Although usually it will be the same value as the CurrentCulture, sometimes it will differ. This should always be respected when you have the appropriate user interface language available.

Getting CultureInfo from Input Methods

An input language is a culture/keyboard layout pair that determines how the physical keys on a keyboard map to characters in a language. The System.Windows.Forms.InputLanguage class provides the following:

  • Read-only access to a collection containing the full list of input languages installed on the computer, available through the InstalledInputLanguages property.
  • Read-only access to the default input language, available through the DefaultInputLanguage property. This will be the initial input language for any new thread or process.
  • Read-write access to the input language for the current thread, available through the CurrentInputLanguage property.

In addition, Windows Forms provides two events, InputLanguageChanging and InputLanguageChanged, both of which provide an InputLanguage object that has within it a CultureInfo. That CultureInfo can be useful for any operation in which the language the user is typing might be relevant. This might include storing information about the language or choosing an appropriate dictionary or thesaurus.

There are many times that you may need an object that does not change based on user settings or changes in application settings. It is for this reason that the InvariantCulture was created and exposed for developers.

As important as it can be to adhere to one of the cultures I've mentioned, there are times that no culture (even the invariant one) is an appropriate choice. Some of these scenarios are discussed in the later section about collation. It is important to recognize when such a decision is needed.

Creating New CultureInfo Objects

Although most applications do not need it, there are times where the ability to simply create a CultureInfo object that is not based on any particular user settings can be very useful. Some of these scenarios include enumerating available CultureInfo choices, checking the behavior of an application with other culture settings, reproducing bugs that have been reported with specific cultures, and creating the objects to assign to the current culture, or current UI culture, based on some externally provided information (such as in HTTP requests to an ASP.NET page). Now that you know how to get CultureInfo, the following sections will explain operations you can perform using CultureInfo objects.

Uppercasing and Lowercasing

Although casing is arguably the easiest operation, it is nevertheless one that has caused a great many bugs due to misunderstandings. Casing is the operation that moves text between uppercase and lowercase. It gets tricky because there are two languages (Turkish and Azeri) that use Turkic casing rules. As depicted in Figure 3, in those two cultures, #1 and #2 are cased variations of each other, as are #3 and #4. In every other culture, #1 and #4 are cased variants of each other.

Figure 3 Turkic Casing Rules (Close Figure 3)

Number | Letter | Unicode Code Point | Character Name
1 | I | U+0049 | LATIN CAPITAL LETTER I
2 | ı | U+0131 | LATIN SMALL LETTER DOTLESS I
3 | İ | U+0130 | LATIN CAPITAL LETTER I WITH DOT ABOVE
4 | i | U+0069 | LATIN SMALL LETTER I

If you perform an uppercasing or lowercasing operation using Turkic rules and then later try to compare results without them, your code can run into serious problems trying to identify strings.

In deciding whether to use the Turkic casing rules, it is important to look at the specific scenario. Obviously, a Turkish or Azeri user would expect strings displayed for them to follow the Turkic rules when casing is performed, and just as obviously, operations involving file names and registry keys should never use them. It is important to distinguish between what is visible to the user and what needs to be consistent with an external source or with anyone else running the application.

No other cultures have alternate casing rules, and I don't know of any other languages that do. However, if you are able to properly work with this one exceptional case, then you will be able to handle any that come up in the future. Properly using CurrentCulture to handle Turkish will allow any such future culture to work as users would expect.

Collation

Although I'm discussing collation in terms of CultureInfo, all of the relevant methods come out of System.Globalization.CompareInfo. However, since the only way to get a CompareInfo is to either use the one hanging off of a CultureInfo or to pass a culture name or ID to the static CompareInfo.GetCompareInfo method, it's easier to talk about how to choose the collation method in terms of cultures.

Collating, or sorting, is simply the ordering of items. It is one of the most fundamental features and everyone expects it to work properly. Ideally, it should be entirely transparent. When users click the top of a list view column that looks like Windows Explorer, they assume that the column will be sorted according to their cultural and language expectations.

Luckily, the bulk of the work has been done , and you just have to select the appropriate culture/settings. There are three possibilities that together handle virtually every conceivable scenario: CurrentCulture, InvariantCulture, and CompareOptions.Ordinal.

CurrentCulture is used any time data is presented to the user and you want that data to be ordered or compared intuitively. It is the default setting, but it's often better to put it in explicitly, as doing so will prevent static analysis warnings from tools like FxCop and will make you aware that you have considered all the options.

InvariantCulture is used any time you still want proper results from a linguistic standpoint, but you need results that do not change their order when user settings do. It is an uncommon requirement for ordering; it's much more common in comparisons.Parse versus ParseExact

There are two parsing methods you can use to parse your strings: Parse and ParseExact. The Parse method's functionality is rooted in COM (which was itself rooted to older versions of Visual Basic) and conversions from string to date happened no matter what the cost. The risks of improper parsing are an unfortunate side effect, one that is visible to people who have to work with both dd/mm/yy and mm/dd/yy dates. The DateTime.Parse method in the Microsoft .NET Framework has goals much like its predecessors, but unfortunately it suffers from some of the same problems. The code is slower since the extra checking takes time, and there will always be some new format that is not properly detected. In those older products, you may remember, the behavior was sometimes disparagingly referred to as "evil date parsing."

DateTime.ParseExact, on the other hand, takes the exact formats specified in the DateTimeFormatInfo object and uses them and nothing else. There is no forgiveness for data that does not match, and the issue of whether or not gratuitous spaces should be forgiven makes for some interesting arguments in certain hallways at Microsoft. Simply put, its goal is more along the lines of "here is the format; here are the strings in that format. Just do the job." This makes it faster and more exact as a semantic, and as such, it is much better suited if the flexibility of DateTime.Parse is not desired.

CompareOptions.Ordinal is an ordinal flag (as is the new OrdinalIgnoreCase flag added in the .NET Framework 2.0) and is used for unchanging binary comparisons of strings. This is the type of comparison that is not only the fastest but also the best if characters that have no weight in sorting (such as directional formatting characters used in bidirectional rendering) must be given some weight rather than being ignored.

There is also a possible fourth requirement which concerns making decisions about file names and whether they are equal. Unfortunately, none of the three methods I just mentioned will give a complete answer for that question, in either managed or unmanaged code. The only way to get a correct answer is to attempt to create a file and trap the exception if it already exists.

The final features I'll discuss include resource loading, encoding based on locales, and input methods. You can think of these as more understandable features, or features less likely to be broken in ways that are difficult to decipher.

Formatting and Parsing

When it comes to parsing, you may want to consider using ParseExact instead of Parse, when you can, to protect your own code. Flexibility is great when you need it, but when you don't, its better not to risk the problems associated with it. The sidebar "Parse versus ParseExact" explains the differences between the two methods.

Of all of the uses of cultures, the most difficult ones to manage and that cause the most difficulty are the formatting and parsing functions. They are ideally converse operations, which is why they are lumped together here. There are two situations in which parsing and formatting are not converse operations. The first is when a custom format is used to convert a number, date, or time to a string, and the identical formatting string is not used to parse that string and retrieve the data. The second is when formatting strings are not the default and Parse is used, rather than ParseExact, since the heuristics in Parse can, at times, read information improperly.

The heart of this support is the IFormatProvider interface, which many of the classes in the System.Globalization namespace support (including CultureInfo, NumberFormatInfo, and DateTimeFormatInfo). This interface has one method—GetFormat—which the parsing and formatting code can use to acquire the correct format information. Most of the formatting and parsing methods have at least one override that accepts an IFormatProvider.

The fact that a single interface is used can be a bad thing. The code has to be able to handle the case where a NumberFormatInfo is passed to a DateTime.Parse, which it does by ignoring the parameter. (In current versions of FxCop, however, this also silences the warning about providing an IFormatProvider.) The same issue can be seen with formatting and with a DateTimeFormatInfo being passed to the number formatting and parsing methods. I usually suggest passing a CultureInfo, since the more specialized DateTimeFormatInfo and NumberFormatInfo classes can always be extracted from it unless you have specifically customized an object and are sure you are passing the right one. With any luck, future versions of FxCop will warn you when you pass a parameter that is nothing but a slower version of NULL (slower because it has to check if it's one of two other types first).

Resource Loading

For apps that do the work to support other languages, the simple rule of using the CurrentUICulture as a default is what makes this item easier to deal with. If you localize either an ASP.NET app or a Windows Forms app, the resource loading methods will use this setting as their default. You can, of course, override that default by passing a separate CultureInfo that uses a different language.

Make sure to set the CurrentUICulture to the user's expectations. Whether it is using the Windows UI language, the Web Server's impression of the user based on the HTTP_ACCEPT_LANGUAGE header, or some interface choice that you provide in your application will completely depend on your application. One good rule is to use the same model as Windows MUI—provide a list of languages that an application supports using the native name of each language (CultureInfo.NativeName). This amazingly simple plan is really common sense, for if a user does not understand how to read a language name, why would they want to switch their interface to that language?

Encoding

Encoding is supported through the System.Text.Encoding class and the various classes that support its interfaces. This topic is worthy of many articles, but the support that is culture-specific is limited to the ANSI, OEM, Mac, and EBCDIC code pages of each language. In the real world, when you need to convert anything to or from a legacy code page, you must know what the code page is rather than guessing based on a specific culture. The utility of a culture-based encoding plan is therefore quite limited. It is best to look at encodings in a code page-based way, using the exact code page conversion in which the data is encoded.

Input Languages

Support for input languages (otherwise known as input methods) is provided through the InputLanguage class. The class is limited, since through the .NET Framework you can only use or change to languages that the user has already installed. Every InputLanguage object has a Handle property on it. Although handles do not always have a specific size (32 bits on a 32-bit platform and 64 bits on a 64-bit platform), the InputLanguage Handle only ever uses 32 bits, and the first half of that number is actually an LCID value. The culture attached to the InputLanguage is actually created from that LCID.

In an application, the culture can be quite useful any time language-specific choices, such as the way to use tools that do spelling or grammar checking, must be made. Unmanaged applications like Microsoft Word use the LCID half of the input language handle for this very purpose. They tag portions of the document to be in the appropriate language, based on the language under which the input method was placed.

If you follow this route, you may want to also have logic to work with characters based on the characters entered rather than purely by the language choice. This will avoid a hazard that even Word does not handle well, such as when a user puts a keyboard under an entirely unrelated language (like an Arabic keyboard under the Hungarian language). Within the .NET Framework, there is no way to retrieve the keyboard language, but you can use P/Invoke to do the work for you, as shown in Figure 4.

Figure 4 Retrieving the Keyboard Language (Close Figure 4)
[DllImport("user32.dll")]
private static extern bool GetKeyboardLayoutName(StringBuilder pwszKLID);
private const int KL_NAMELENGTH = 9;

private CultureInfo CultureOfCurrentLayout() {
    StringBuilder sb = new StringBuilder(KL_NAMELENGTH);

    if(GetKeyboardLayoutName(sbKLID)) {
        int klid = int.Parse(
           sbKLID.ToString().Substring(KL_NAMELENGTH - 1),
           NumberStyles.AllowHexSpecifier, CultureInfo.InvariantCulture);

        // strip all but the bottom half of the number
        klid &= 0xffff;

        return new CultureInfo(klid, false);
    }

    return(null);
}

This function will return a CultureInfo representing the language, region, and script for which the keyboard was designed.

Recent Changes

Windows-Only Cultures The list of supported cultures in the .NET Framework is more extensive than it is in various versions of Windows. Starting with Windows XP SP2, however, the reverse happened when 25 new locales were added. This causes a huge problem for both the CurrentCulture/CurrentUICulture properties and the InputLanguage class, which were always able to simply create a culture using the locale identifier (LCID) value from Windows.

The solution to this problem, which will ship with the release of the .NET Framework 2.0, is to synthesize a CultureInfo object any time one attempts to create a culture that is unavailable in the Framework, but available in Windows. All of the principles discussed in prior sections will allow your applications to work well with these "Windows-only" CultureInfo objects.

The New, Improved CultureTypes Enumeration In the .NET Framework 1.x, the CultureTypes enumeration was defined as:

[Flags]
public enum CultureTypes
{
    NeutralCultures        = 0x0001,
    SpecificCultures       = 0x0002,
    InstalledWin32Cultures = 0x0004,
    AllCultures            = NeutralCultures | SpecificCultures |
                             InstalledWin32Cultures,
}
This does not leave much room for the addition of new types of cultures (Windows-only, Custom, and Replacement). Therefore, the updated CultureTypes enumeration that will ship with the .NET Framework 2.0 is defined in Figure 5.

Figure 5 Updated CultureTypes Enumeration (Close Figure 5)
public enum CultureTypes
{
    NeutralCultures         = 0x0001,  // Neutral cultures such as "en",
                                       // "de", and "zh".
    SpecificCultures        = 0x0002,  // Non-neutral cultures such as
                                       // "en-us", and "zh-tw".
    InstalledWin32Cultures  = 0x0004,  // Win32-installed cultures in the
                                       // system that exist in the
                                       // Framework, too.

    AllCultures             = NeutralCultures | SpecificCultures |
                              InstalledWin32Cultures,

    UserCustomCulture       = 0x0008,  // User-defined custom culture
    ReplacementCultures     = 0x0010,  // User-defined replacement
                                       // custom culture.
    WindowsOnlyCultures     = 0x0020,  // Culture exists in Win32 but
                                       // not in the Framework.
    FrameworkCultures       = 0x0040,  // Language tag matches a culture
                                       // that ships with the Framework.
}

The updates not only make room for the new culture types but they also allow for future growth. Existing code that uses AllCultures will continue to work, but new code should use the new CultureTypes enumeration members. These members are enumeration values and flags and, as such, can be used in combination. For example, you can have UserCustomCultures that have either the NeutralCultures or SpecificCultures flag associated with them. When calling CultureInfo.GetCultures, you should pass the most restrictive type to cover your requirements.

Custom Cultures and Beyond

The ability to create custom cultures, also new in the .NET Framework 2.0, is accomplished by creating a new culture with the CultureAndRegionInfoBuilder class. Following the best practices I've suggested in this article will ensure that your managed application is much more likely to work properly with custom cultures!

CultureInfo is a rich and powerful class, encompassing almost every common aspect of how users interface with computers and applications. With the great power that this class provides it is important that you make sure you're making the right choices when deciding what CultureInfo to use in each situation. It is often that little bit of extra thought that can maximize both the efficiency and the effectiveness of CultureInfo usage.

Michael Kaplan is a technical lead at Microsoft, working on both Windows and the .NET Framework, particularly on collation, keyboards, locales, and Unicode support. He is the developer/owner of MSKLC, the Microsoft Keyboard Layout Creator, written in C#. He can be reached at blogs.msdn.com/michkap.


Editor's Note: There’s A Word For That...

Editor's Note
There’s A Word For That...

This month in MSDN Magazine, we are looking at the world of Web services. But before we tell you all about this month's issue, we have a topic we would like to address. In fact, it's our civic duty to do so.

Here in the magazine's offices, we editors have more than a passing familiarity with the world of the programmer. It's this experience that gives us the right—nay, duty—to speak out. We realize that a lecture may make us seem "square" or "uncool" in the eyes of some developers, but it's time to take our position of self-appointed responsibility and use it for everyone's benefit. Our request is simple and to the point: please, for the love of all that's good in this world, stop making up words.

We realize that invented verbiage is not a new phenomenon in the world of the developer. Some new words may, in limited circumstances, be okay. In fact, there are three classifications of invented English that we run across, two of which are acceptable.

  1. Words that are invented or adapted to describe new programming techniques. When you talk about boxing and unboxing a value, it's actually the adaptation of an existing word to describe something for which there was no previous word. Similarly, the world of design patterns is rife with descriptive words like "factory" that make sense in their new surroundings.
  2. Words that are created as slang. True, these can sound forced and cloying. But when cyberspace has worn you out, you can always go back to your life in meatspace—it's colorfully descriptive, not a buzzkill at all.
  3. Words that represent the wholesale butchering of the English language. This is the problem we're addressing here.

We are asking for your help in eradicating words that have been invented for no good reason. Sometimes, it's too late to do anything about them. Look at the word "canonicalize," for instance. It is used to mean "to create the canonical form" of something, like a URL (as in InternetCanonicalizeUrl from the WinINet API). It's not English; it was invented because someone didn't know that there was already a perfectly adequate word for this process: "canonize." However, once this non-word has been created, the rules of the language suddenly apply again, so the process of "canonicalizing" something is "canonicalization" instead of "canonization."

More recently, we've seen the word "performant" start its crawl into the everyday vocabulary of devspace. It is used to mean "highly performing." It's also not a word. When something provides information, it's informative. It's not "informant." The word "performant," if it existed, would be a noun—not an adjective. But it doesn't exist, so if you do see it in print, remember that it's not really there.

Any readers who have made it this far are probably rolling their eyes now, thinking to themselves, "Why are they being such sticklers here? Isn't the language a wonderful, evolving thing?" Yes, our language is evolving. As there is a need for new words, new words enter the language. But making up new words is just as bad as using fancy words in place of short ones. Why say "This project's goals are orthogonal to the company's needs"? Admit it—if you were at home, you'd just say "different from" or "at odds with."

Or should we just give up and start inventing our own language? We could always be goaled to introduce our own efficiencisms into code samples for the purposing of improvating the downloadicant experience for our readers. Then we can get back to work with the manuscriptive process, making sure that the articles in the magazine are exactitudinally microcoded for perfectant reading. What's the problem? You can figure out what those words all mean!

Now that we're all in agreement, let's talk Web services. This month's issue brings a spring bouquet of articles that will give you some cool ideas for enhancing your Web services. If you want to combine the data from more than one Web service for maximant efficiencisms in your code, Gerrard Lindsay explains how WSDL is the way to do it. If you want fine-grained control over the XML of which your Web service is emittive, Chris Dix takes a look at message control with WSE 2.0. We even take a look at BPEL4WS and how it is interactionate with BizTalk Server 2004.

So polish up your Web services, drop us a line , and stop making up words. We are behoovant of you.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Mark Boulter, Israel Burman, Tom Christian, Rebecca Dias, Kedar Dubhashi, Brian Grunkemeyer, Jonathan Keljo, Martyn Lovell, John Nisi, Polita Paulus, Riyaz Pishori, Matt Powell, Mark Novak, Eric Richards, Scott Robinson, Yasser Shohoud, Chandu Thota, and Cathy Wissink.

Active Directory, ActiveX, BizTalk, InfoPath, JScript, Microsoft, MSDN, MS-DOS, SharePoint, Visio, Visual Basic, Visual C++, Visual C#, Visual Studio, Windows, Windows NT, Win32, and Windows Server are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


New Stuff: Resources for Your Developer Toolbox

New Stuff
Resources for Your Developer Toolbox
Marnie Hutcheson

Contents
Transform XML Technologies
Identify the Location of Your Visitors
All About PE Files
Multipurpose Add-Ins
Printing Documents as PDF Files
A Guide to Real World Web Services

Transform XML Technologies

XMLSpy 2005 by Altova is an XML development environment for modeling, editing, debugging, and transforming all XML technologies. It implements the W3C's new XSLT 2.0 and XPath 2.0 specs, and provides native support for XQuery 1.0. It also automatically generates code in C++, C#, or Java-language class files based on data elements defined in a schema. If I devoted this whole column to XMLSpy 2005, I still couldn't list all its features.

XMLSpy 2005 lets you tackle Web services development on all of the major Web services platforms. You can integrate it into Visual Studio® .NET 2002 and 2003; then you can use the views and functionality of XMLSpy 2005 right from the Visual Studio .NET toolbar. You can also access many of the powerful XMLSpy 2005 functions programmatically using the COM or Java-language systems integration API.

In addition, XMLSpy 2005 includes SchemaAgent, a graphical client/server schema administration tool that allows you to view a collection of schemas as a pool of resources. You can access schema files stored in any standard repository, and all changes are automatically saved back to the repository and reflected on the other clients in the workgroup.

The XML Schema Design/WSDL View, shown in the accompanying screen shot, lets you perform visual design and editing for your XML interfaces. The Enhanced Grid View gives you a spreadsheet style tool for performing XML editing operations on the document as a whole, instead of line-by-line. There is also a SOAP debugger built right in.

Price: XMLSpy 2005 Professional Edition starts at $499 for a single-user license, XMLSpy 2005 Enterprise Edition starts at $999, and XMLSpy 2005 Home Edition is available for free.

www.altova.com

Identify the Location of Your Visitors

IP2Location.com, from Hexa Software Development Center, provides Internet-based geo-location software and services for the e-commerce industry. IP2Location is a database that identifies the geographical location of IP addresses being used in Internet address space. You can use it to identify the geographic location of your Web site visitors by their IP addresses.

Knowing the location of your visitors in real time opens the door to automatically offering localized, customized content and advertising, improving the user experience and increasing satisfaction along with click-throughs.

But, most importantly for e-business Web sites, the IP2Location database provides the information that will let you identify suspicious Internet orders, reduce charge-backs, avoid orders from high-risk countries, display native currency on order forms, and calculate correct tax fees based on country and regions.

The complete IP2Location database contains more than 2.5 million records for all IP addresses. The company reports that it has over 95 percent matching accuracy at the country level. There are a couple of instances where it won't help you much, specifically when dealing with subscribers of ISPs like AOL, who issue all their users IP addresses out of Reston, VA regardless of where they live.

Price:$499 per year; the database is available via download with free monthly updates.

www.ip2location.com

All About PE Files

When I found this handy tool, I wasn't looking for a disassembler, but I was searching for a tool that could reset time and date stamps on a group of files. One of my servers had been hacked and the dates on several files were modified in their root kit to be several years old, so that they would not show up as "new" files during my routine server sweeps.

The old dates on the files got the security guys thinking that the server had been hacked years earlier, but the fact that the server was only six months old didn't convince them otherwise, so I decided that I needed to prove my point.

PE Explorer, by Heaventools Software, can reset time and date stamps, a relatively minor feature of this product as it turns out. It also gives you a clear view of portable executable (PE) file structure and all of the resources in the file. It tells you just about every little detail you could possibly want to know about a PE file. Once you get inside, you can analyze and optimize the file structure, diagnose problems, make changes, repair or replace resources, and track down spyware, even if you do not have the source code and even if its original developers are long gone.

Besides being an effective resource editor, PE Explorer also gives you an API function syntax lookup, dependency scanner, section editor, and an easy-to-use disassembler for generating annotated code dumps.

Price: Personal license $129. Single-user business license $199.

www.heaventools.com

Multipurpose Add-Ins

Both CodeSMART 2005 for Visual Basic 6.0 and CodeSMART 2003 for Visual Studio .NET, released by AXTools, are multipurpose add-ins that have long lists of features that will boost your productivity and help you write cleaner code faster. Both install into their respective IDEs to become an integrated part of the development environment.

The standard features that are more or less common to both offerings include auto-text completion for blocks of code, tools and templates to help you create complex interfaces, objects, exception handlers, and formatter classes.

CodeSMART 2005 for Visual Basic® 6.0 has several distinct features that give you improved ways to view and understand your code. You get the CodeSMART Project Explorer, with its expandable code, emphasizing color, type and scope distinction, customizable sorting, and more. There is also a "logical view" in Project Explorer that uses virtual folders.

The Designer Explorer shows you a tree view of the control hierarchy in your project and gives you a roadmap of the layout, making it easy to control navigation. You can create shortcuts in the Workbench window that will let you navigate to functions, properties, and classes. Code Flow view lets you follow the logic structure inside your methods.

Price: Single-user license of CodeSMART for Visual Basic 6.0 or Visual Studio .NET is $189; combined cost is $299.

www.axtools.com

Printing Documents as PDF Files

Print2PDF by Software602 Inc is a server-based printer driver that produces version 1.3 or 1.4 PDF files that retain the formatting and appearance of the original and can be viewed using the free Adobe Acrobat Reader. Network users download the Print2PDF network print driver and "print" any document as a PDF file from any application that has a print command.

The program supports 40-bit or 128-bit RC4 encryption and two levels of password protection. Print2PDF has a "Stamp" feature that allows users to set their own watermark, such as a corporate logo or signature, directly into the document. The program supports both portrait and landscape orientations, multiple paper sizes, document resolution from 96 to 600 DPI, and multiple image compression options to ensure top print quality.

Price: Print2PDF 4.0 Server Edition is $299.95 for up to 25 users, $899.95 for 26 to 100 users, and $1,999.95 for an unlimited user license.

www.software602.com

A Guide to Real World Web Services

Real World Web Services, by Will Iverson (O'Reilly), guides developers through the building of eight Web applications. The heart of the book is a series of projects that demonstrate the use and integration of Google, Amazon, eBay, PayPal, FedEx, and many other Web services.

The book documents how to add functionality such as automating listings for auctions, dynamically calculating shipping fees, automatically sending faxes to suppliers, using an aggregator to pull data from multiple news and Web service feeds into a single format, and monitoring the latest blog discussions and Google searches to keep Web site visitors current with topics of interest. For each example application, Iverson provides an overview, architecture description, and working code examples.

Price: $29.95, 207 pages.

www.oreilly.com

All prices were confirmed at press time and are subject to change.

Send your New Stuff to newstuff@microsoft.com.

Marnie Hutcheson is president of Internet Development Associates (Ideva), a firm in Ocala, Florida, that specializes in Internet and intranet Web application design and development. She has published technical papers and books on various computing topics. You can reach her at marnie@ideva.com.


Web Q&amp;A: Data Integrity Over HTTP, IIS Credentials, DataGrids, and More

Web Q&A
Data Integrity Over HTTP, IIS Credentials, DataGrids, and More
Edited by Nancy Michell

Q If I establish a Web connection using HttpWebRequest and then read data from HttpWebResponse.GetResponseStream, I know it will fail if the Internet connection is lost, but do I need to be concerned about data packets being lost if the reading of data from HttpWebResponse.GetResponseStream finished without error? Does the underlying transport/protocol take care of retrying lost data packet and other problems?

A Yes, HTTP is built on TCP/IP, which handles all of the retries and data integrity for you. That said, there are a variety of ways this could legally fail to deliver the file. For instance, if the server is using HTTP 1.0 and uses a "Content-Close" header instead of a "Content-Length" header, the socket could be closed before the file was completely sent. The HttpWebRequest object won't throw an error because the server said "I'll close this when I'm done," and the connection had been closed, so the operation was entirely legal.

Q How can I run an application under specific credentials on the server side of an ASP.NET application? I tried the runas command, but it requires the user to key in the password at run time.

A If you're using Windows Server™ 2003 with IIS 6.0, you can create a new application pool in IIS with different credentials, and then set the virtual directory for your ASP.NET application to use that application pool. Here are the steps you should follow:

  1. Right-click Application Pool folder, select New.
  2. Walk through the wizard.
  3. After the application pool has been created, right-click it and then select Properties.
  4. Click the Identity tab and change the account.
  5. Assign the app pool to an application by right-clicking the folder in IIS and selecting Properties.
  6. Click the Directory/Home Directory Tab, change the AppPool dropdown to the application pool you created. The application will now run under the new application pool as the identity that you specified.

If you are using IIS 5.0, you can use impersonation in the Web.config to run the application as a specific user:

<identity impersonate="true" userName="domain\user" password="<pass>" />
You could then use aspnet_setreg.exe to encrypt the username and password and put them in the registry, as you can read in Knowledge Base article 329290 ("How To Use the ASP.NET Utility to Encrypt Credentials and Session State Connection Strings"). You can also programmatically impersonate a user using the LogonUser function exported from advapi32.dll. For an example of how to do that in your ASP.NET application, see Knowledge Base article 306158 ("How To Implement Impersonation in an ASP.NET Application"), particularly the section on impersonating a specific user in code.

In IIS 5.0, if you want to run all applications as a specific user, you can change the userName and Password attributes of the <processModel> tag in the machine.config.

Q I am creating a DataGrid on a form and on page load I'm populating it by dynamically building template columns using a stored procedure. Everything works fine and I get the DataGrid. Now I am writing an update method which is the handler for the update button click. As soon as I enter this method, the value of DataGrid1.Items.Count becomes 0 and the DataSet says that the referenced object has a value of nothing. Why is the DataGrid emptied on a postback?

A The DataGrid control does not persist dynamically added columns across posts. If you decide that you want to add columns on the fly, you will have to recreate them each time the page is loaded and then rebind to the data source.

Q I am using data binding with a DataGrid. How can I get the index of a particular row in the DataTable's row collection?

A The Microsoft® .NET Framework 2.0 will have a property called IndexOf on DataRowCollection which gives the index of the row in the DataTable, as shown here:

dt.Rows.IndexOf(row);
The IndexOf function exposed in the .NET Framework 2.0 performs well even for very large DataSets. However, if you're using the .NET Framework 1.x, then you'll have to resort to getting the index by brute force logic: Loop through all the rows in the row collection and find the row in the table which matches your row.
public int GetIndex(DataRow findRow) {
    for (int i = 0; i < _dt.Rows.Count; i++) {
        DataRow row = _dt.Rows[i];
        if (row == findRow) {
            return i;
        }
    }
    return -1;
}
Be careful not to call the brute force function too often in your code, especially for larger DataSets, because of the performance implications of looping through all the rows in the table.

Q I am implementing a custom ADO.NET provider and so I would like to know which of the well-known ADO.NET interfaces is responsible for returning the list of tables and views in the underlying data store?

A In the ADO.NET 2.0 API, there are metadata discovery APIs. You can start with Connection.GetSchema. Unfortunately, in the .NET Framework 1.x, ADO.NET didn't include support for that in providers. If you choose to invent something to expose this functionality, you may want to stay in line with the ADO.NET 2.0 API so that you have a smooth upgrade path in the future. In the .NET Framework 2.0, you can use the new GetSchema methods. For more information on this, you can take a look at ADO.NET 2.0 new GetSchema method.

For the .NET Framework 1.x, you can use the native OLE DB schema information with GetOledbSchemaTable, so if you are using the System.Data.OleDb ADO.NET classes, try the technique described at Retrieving database independent schema information.

Q How can I know when to use available data providers versus developing database-agnostic data access?

A There are some great articles comparing when you can use each data provider and how you can develop database agnostic data access. For example, you should take a look at ".NET Framework Data Providers" along with "INFO: Roadmap for .NET Data Providers".

That document explains that Data Access Objects (DAO) and the Jet database engine minimize the differences between database systems so that you can port an application from one database to another with very few changes. With RDO and ADO you don't have the Jet intermediary, so you get better performance and more server-specific functionality. However, this can make it harder to port an application between databases.

ADO.NET provides better performance but it includes separate classes for each .NET data provider. You can, however, use standard interfaces and isolate initialization code into "factory" functions so that you change less code. Plus, the DataSet object represents a central object for data binding and remoting that is independent of the data provider.

For further information, see "Writing Common Code for .NET Data Providers", as well as "HOW TO: Use Base Classes to Reduce Code Forking with Managed Providers in Visual Basic .NET" (the C# version of the same article is available at HOW TO: Use Base Classes to Reduce Code Forking with Managed Providers in Visual C# .NET).

Q I want to issue a call (such as a WinHttp call) that can send an HTTP GET to retrieve a Web page using JScript®. Is it possible to use JScript .NET in this situation?

A JScript .NET is the JScript language compiled to target the common language runtime (CLR). It uses the .NET base class library that contains the HttpWebRequest class and other classes to do this. However, such code would not work as embedded script in current script hosts like Microsoft Internet Explorer. If you want to use regular JScript, you can use the XMLHTTP component on the client side, and the ServerXMLHTTP component on the server.

Q In Internet Explorer, if the user navigates to the previous page using the Back button, is there any way to locate where in window.history the user is currently? Is it possible to get the URL? I am trying to handle session variables when multiple windows are opened. I keep a copy of session variables for each window and attempt to maintain the mapping. But this strategy does not work if the user opens a new window using Control-N and then goes backwards in the new window to perform some task. I am trying to keep the user from going back past the page on which the new window was opened.

A First of all, you can't read history information. You can only use window.history.back and window.history.forward to navigate backward and forward. Window.history.length will tell you how many elements are in the history list. Allowing access to the URL would be a huge security issue. Consider what a hostile Web page would do with that information. The URL history contains all kinds of private data—not just what Web sites you visit, but URLs often contain information about products you've looked at, what your user name is on various Web sites, what queries you sent recently, and lots more private data.

If you read the MSDN® docs for IomHistory, the property type for IHTMLWindow2::history, you'll see only the following exposed to automation through the dispatch interface:

  • back(*pvargdistance : VARIANT)
  • forward()
  • go(*pvargdistance : VARIANT)
  • get_length(*p : short)
If you need to know the referrer URL, you can usually get it by passing it back from the server.

Most browsers will pass a referrer URL to a server in response to a request (though there are many cases where it won't, or it'll pass something intentionally wrong). On the server you can pick up that URL and perform any logic you need, or pass it back to the client as part of the content. There's no guarantee you'll get it though.

If you want to get page history from your own site, you might consider storing page history in cookies on the client. For example, every time a page loads, add its name or some kind of ID to a cookie. Just keep in mind that in the multiple-window situation you mentioned, the cookies will be the same for both windows. Modifications you make to one window will happen in the other one, too. To be safe in cases in which the user uses multiple windows, you can consider using state logic. This issue is one of biggest challenges of Web development.

Q I have a JavaScript file that has a number of useful functions that I'm trying to incorporate into a new ASP.NET site. However, I get errors wherever WScript is used in the various functions. The following line produces a "WScript is undefined" error:

var _LogFile = WScript.StdOut;
My search for documentation revealed that WScript usage requires Cscript.exe to be the host. I would really like to avoid rewriting the script functions so I need to know if there's any way to incorporate it in the ASP.NET site as is? I'm using JavaScript on the client:
<script language="JavaScript" src="CommonScripts.js">
  document.write('<BR/><B> Current Date = ' & _logCurDate & '<B><BR/>');
</script>
CommonScripts.js is the file that has all the functions I want to use (with _logCurDate being one of them).

A If you're using client-side script, then the question really doesn't involve ASP.NET. You would see the same results/limitations if you were to use a simple static HTML file. WScript is an object provided by the Windows Scripting Host and is not applicable inside of Internet Explorer or on the server side.

You are not likely find a way to simply retrofit the existing scripts to a different object model. You will need to find the functionality provided in whichever object model is provided by your script engine and replace those function calls.

If it's client-side JavaScript, then you need to look at the objects like document and window. If it is server-side ASP with JavaScript, look at the Request/Response objects (additionally, you can instantiate the FileSystemObject). If you are using ASP.NET, then use Request/Response/Page, and so on.

Got a question? Send questions and comments to webqa@microsoft.com.

Thanks to the following Microsoft developers for their technical expertise: Yasir Alvi, Gaurav Bansal, Mike Barrett, Kawarjit Bedi, Todd Carter, Hitesh Chaudhary, Saurabh Dasgupta, Lale Divringi, David Ebbo, Chris Gariepy, Mick Hakobyan, Koce Ivanov, Ronghua Jin, Eric Lawrence, Erik Leaseburg, Xin (Sheen) Li, Eric Lippert, David Lovell, Richie Meyer, Satvir Randhawa, Angel Saenz-Badillos, Kurt Schenk, Fyodor Soikin, Heath Stewart, and Charles Torre.


Advanced Basics: Doing Async the Easy Way

Advanced Basics
Doing Async the Easy Way
Ken Getz

Code download available at: AdvancedBasics0503.exe (152KB)
Contents
Single-Thread Conflicts
Using a Second Thread
Using Invoke to Avoid Thread Contention
Introducing BackgroundWorker
Canceling the Search

If you've been following Ted Pattison's excellent series of Basic Instincts columns on multithreading and asynchronous behavior, you should by now be an expert on handling the issues involved in working with multiple threads in Windows®-based apps. Recently, a friend, who evidently hadn't read Ted's columns, sent me a Windows-based application that performed a very lengthy file search operation that locked the user interface while executing. Users couldn't interact with, move, or resize the form while the main activity was running, so they weren't happy. He wanted to know what he could do to make the app more responsive. The answer, of course, is to perform the work in a background thread.

I recommended Ted's series of columns to my friend, but couldn't resist fixing things up on my own in the meantime. The application and how you can safely perform a background operation with as little code as possible—both in Visual Basic® .NET 2003 and in Visual Basic 2005—is this month's topic.

Single-Thread Conflicts

If you've downloaded the sample associated with this column, try running the Async2003 example in Visual Studio® .NET 2003. On the sample form, choose a drive that contains many files, enter some parameters other than *.* if you like, select the Search Subfolders option, and then click Simple Search. While the search is running, try interacting with the form, perhaps attempting to move or resize it. Try clicking Cancel Search. I dare you. You can't, right? Because the form is performing the search in the main thread, the user interface is blocked while the search is proceeding. Once the search has been completed, the listbox on the form should contain the full list of found files as shown in Figure 1.

Figure 1 File Search Application
Figure 1 File Search Application

The simple search creates an instance of the FileSearch class (included in the sample project), sets some of its properties, adds an event handler for its FileFound event, and calls the FileSearch object's Execute method:

Dim fs As New FileSearch(Me.cboDrives.Text, Me.txtFileSpec.Text, _
   Me.chkSearchSubfolders.Checked)
AddHandler fs.FileFound, AddressOf EventHandler
fs.Execute()

The FileSearch.Execute method sets up internal parameters and then calls the private FileSearch.Search method, which calls itself recursively looking for matches against the requested file specification. In Figure 2, the Globals.Cancelled property is a Shared Boolean value in the Globals class—clicking the Cancel button (if you ever get a chance) sets this value to True. Because forms in Windows-based applications are single-threaded, all the code called from the form runs in a single thread, and prevents user interface interactions while it's running.

Figure 2 Searching for Files (Close Figure 2)
Private Sub Search(ByVal Path As String)
  Dim diLocal As New DirectoryInfo(Path)
  Dim di As DirectoryInfo

  If Globals.Cancelled Then
    Exit Sub
  End If

  Try
    ' Handle subfolders, if required.
    If Me.SearchSubfolders Then
      For Each di In diLocal.GetDirectories
        If (File.GetAttributes(di.FullName) And _
          FileAttributes.ReparsePoint) <> _
          FileAttributes.ReparsePoint Then
            Search(di.FullName)
        End If
      Next
    End If

    ' Search for matching file names.
    Dim fi As FileInfo
    Dim afi() As FileInfo = diLocal.GetFiles(Me.FileSpec)

    For Each fi In afi
      If Globals.Cancelled Then
        Exit For
      End If
      OnFileFound(fi)
    Next
    alFiles.AddRange(afi)

  Catch ex As UnauthorizedAccessException
    ' Don't do anything at all, just quietly get out. This means you
    ' weren't meant to be in this folder. All other exceptions will
    ' bubble back out to the caller.
  End Try
End Sub

Using a Second Thread

The second button, labeled Incorrect Async, attempts to fix the thread blocking issue by running the FileSearch.Execute method in a background thread. The code adds this behavior by first creating a delegate type with a signature that matches that of FileSearch.Execute so that it can refer to the FileSearch.Execute method:

Private Delegate Function ExecuteDelegate() As ArrayList
Instead of calling the FileSearch.Execute method directly, the second version of the code creates an instance of the ExecuteDelegate type containing a reference to the Execute method, and calls the BeginInvoke method of the delegate to run the code on a secondary thread, as shown in the following:
Dim ed As New ExecuteDelegate(AddressOf fs.Execute)
AddHandler fs.FileFound, AddressOf EventHandler
ed.BeginInvoke(AddressOf HandleCallback, ed)
The HandleCallback procedure retrieves the original delegate instance from the state information passed to the procedure, and calls the EndInvoke method:
' After the async search has completed, run this code. If you call
' BeginInvoke, you generally must call EndInvoke as well.
Dim ed As ExecuteDelegate = DirectCast(ar.AsyncState, ExecuteDelegate)
Dim al As ArrayList = ed.EndInvoke(ar)
' You could use the ArrayList here, if you like.

Try running the sample again, clicking Incorrect Async, and you'll see that the form's user interface remains responsive while the search is running. Clicking Cancel Search now cancels the operation, and you can move or resize the form as the code is running. In addition, the form includes an event handler for the Form.Closing event, ensuring that if you attempt to close the form, the code sets the Globals.Cancelled flag first. This avoids any unpleasantness that might occur when the .NET runtime attempts to call the form's HandleCallback procedure after the form has been removed from memory.

Figure 3 Using Background Threads
Figure 3 Using Background Threads

Unfortunately, although this solution appears to solve the problem, it breaks a cardinal rule of Windows-based applications and multithreaded programming: "Thou shalt not interact with a control's properties from a thread other than the one that created the control." (The Form class inherits from the Control class, so the same issues apply when referring to properties of a form.) The corollary to the prime directive is that every procedure runs in the thread from which it was called. Because the EventHandler procedure is called from code running in a secondary thread, it too runs in the secondary thread. The EventHandler procedure updates the form in several ways, breaking the all-important rule. You can verify this transgression by examining the contents of the output window in the sample application. Debugging code within the sample writes the current thread ID into the window, and you can see that the active thread while writing to the form is different from the form's thread. Figure 3 shows output after running the sample application.

Using Invoke to Avoid Thread Contention

Although the demonstrated breach of protocol may not show any symptoms in the sample application, it can cause problems for you in real applications when multiple threads contend for a single resource (such as a property of a control or of the form itself). The accepted solution in Visual Basic .NET 2003 is to use the form's Invoke method inside the event handler (which will always run on a secondary thread in this example because it's called from the secondary thread) to perform a thread switch back to the form's thread. The Invoke method takes care of the problem. As described in the documentation, the method "executes the specified delegate on the thread that owns the control's underlying window handle."

You can run a third example that demonstrates this behavior by clicking Better Async on the form. This version calls the event handler named EventHandlerAsync each time a file is found. In this handler, instead of calling the UpdateDisplay method directly, the event handler uses a delegate type created in the form's class to invoke the UpdateDisplay method indirectly, in the form's thread:

' In the form's class:
Private Delegate Sub UpdateDisplayDelegate( _
    ByVal Text As String, ByVal Value As Long)

' In the EventHandlerAsync procedure:
' UpdateDisplay(e.FileFound.FullName, e.Counter)
Dim atd As New UpdateDisplayDelegate(AddressOf UpdateDisplay)
Me.Invoke(atd, New Object() {e.FileFound.FullName, e.Counter})

To observe this safer behavior, run the sample again, and view the debugging information in the output window. Figure 4 shows sample output, demonstrating that it's possible, with enough effort, to execute code called from a background thread without breaking the rules.

Figure 4 Executing Code from a Background Thread
Figure 4 Executing Code from a Background Thread

As an aside, note that updating the UI for every file found, as I'm doing here, can cause the UI to be flooded if files are found frequently enough. This would happen if files are found too quickly for the UI to process them effectively. You can overcome this situation in your own applications by batching the updates.

Obviously, 1,000 words on the topic of Windows Forms and asynchronous behavior barely begin to do justice to this complex topic. I've presented the .Microsoft® NET Framework 1.x version of the sample merely to provide the backdrop for what's to come in the .NET Framework 2.0—the BackgroundWorker component.

Introducing BackgroundWorker

The BackgroundWorker component is an implementation of a new event-based pattern, also exposed by Web service proxies and System.Net.WebClient in the .NET Framework 2.0. BackgroundWorker makes it possible to handle asynchronous operations on Windows Forms without requiring you to lose sleep about cross-thread disasters lurking in the distance and without requiring delegate creation and invocation. This new component exposes an event and property model that makes it easy for any developer to take advantage of asynchronous features.

Of course, there's no free lunch—using the BackgroundWorker component requires you to work the way the component wants you to work, and forces you to rethink your application in order to fit into its very narrow view of the world. You will never have the same flexibility using the BackgroundWorker component that you would have if you created and managed your own threads. On the other hand, you don't have the intricacies, difficulties, and potential potholes, either. And the resulting code is often much cleaner.

Converting the previous example so that it takes advantage of the new BackgroundWorker component was relatively simple. After doing the research into how the BackgroundWorker component operated, the conversion took about 10 to 15 minutes. It also allowed me to remove large chunks of the original code.

I started by opening the project in design mode. From the Components tab on the Toolbox window, I dragged an instance of the BackgroundWorker component to the form. For this sample, I renamed the component so its Name property was bgw. If you examine the control in Design view, you'll find two important properties in the Properties window: WorkerReportsProgress and WorkerSupportsCancellation. The sample sets both of the properties programmatically. Without setting these properties to True either in the Properties window or in code, a user won't be able to cancel the background process and the app won't be able to report on the status of the process.

To get the BackgroundWorker component "started," simply call its RunWorkerAsync method, like this:

Dim fs As New FileSearch(Me.cboDrives.Text, Me.txtFileSpec.Text, _
    Me.chkSearchSubfolders.Checked)

' Start the background process, in a separate thread.
bgw.RunWorkerAsync(fs)
The RunWorkerAsync method, normally run in the form's thread, causes the component to grab a background thread from the thread pool, and then triggers the BackgroundWorker's DoWork event handler, which runs in that background thread. In this procedure you add your code that does work asynchronously, in the background thread. The DoWork event handler might include code like that shown in Figure 5. (This isn't the exact code used in the demo because the Execute method needs to be altered to support cancellation—more on that later.)

Figure 5 DoWork Event Handler (Close Figure 5)
Private Sub bgw_DoWork(ByVal sender As System.Object, _
  ByVal e As System.ComponentModel.DoWorkEventArgs) Handles bgw.DoWork

  'This method will run on a thread other than the UI thread.
  'Be sure not to manipulate any Windows Forms controls created
  'on the UI thread from this method.

  ' Retrieve the FileSearch object passed into the procedure,
  ' and then set up its event handler:
  Dim fs As FileSearch = CType(e.Argument, FileSearch)
  AddHandler fs.FileFound, AddressOf EventHandler

  ' Execute the search:
  fs.Execute()
End Sub

Because this procedure's code runs in a different thread than the form's thread, it's imperative that you don't interact with any member of the form or its controls from this code.

When the work has been completed, BackgroundWorker raises its RunWorkerCompleted event, allowing you to clean up. This procedure runs in the same thread as the form, so you can again access user interface elements with impunity. Your event handler might look like the following code snippet (again, the final code will be modified to support cancellation):

Private Sub bgw_RunWorkerCompleted(ByVal sender As Object, _
  ByVal e As System.ComponentModel.RunWorkerCompletedEventArgs) _
  Handles bgw.RunWorkerCompleted

  ' The background process has completed. This procedure
  ' runs in the same thread as the host form.

  Me.lblResults.Text = "Search complete"
End Sub

Of course, you haven't yet seen code that actually does something once you've found a file. In the previous example, the code added the file name to the listbox on the form in code that was called from the FileSearch.FileFound event handler. The BackgroundWorker component allows for the same behavior, but instead of doing the work directly (as in the earlier "bad" example) or creating your own delegate and calling the form's Invoke method to run it, the BackgroundWorker component handles this thread switch in an elegant fashion. The BackgroundWorker component allows you to call its ReportProgress method from the background thread, which triggers its ProgressChanged event handler back in the form's thread.

You don't have to handle the thread switch yourself, using the delegate/Invoke method; instead, call ReportProgress, and the component does the rest. There's one important caveat, however: the component only raises the ProgressChanged event if you have set the component's WorkerReportsProgress property to True. Without setting that property, the component will never call your event-handling code, so make sure to set it.

When you call the ReportProgress method, you supply an integer between 0 and 100 that indicates the percentage of the background activity that has completed. You can also supply any object as a second parameter, allowing you to pass state information to the event handler. Both the percent and your own object (if supplied) get passed to the ProgressChanged event handler as properties of the ProgressChangedEventArgs parameter passed to the procedure. The properties are named ProgressPercentage and UserState, respectively, and your event handler can use them in any way it requires. The sample application calls the ReportProgress method from its FileSearch.FileFound event handler:

Private Sub EventHandler( _
  ByVal sender As Object, ByVal e As FileFoundEventArgs)

  ' A file was found. Report the progress, triggering the
  ' BackgroundWorker.ProgressChanged event:
  Dim intPercent As Integer = _
    CInt((e.Counter - pbStatus.Minimum) Mod pbStatus.Maximum)
  bgw.ReportProgress(intPercent, e)
End Sub
The sample application uses the ProgressChanged event to update the display as each file is found (see Figure 6).

Figure 6 Using ProgressChanged to Update the Display (Close Figure 6)
Private Sub bgw_ProgressChanged(ByVal sender As Object, _
  ByVal e As System.ComponentModel.ProgressChangedEventArgs) _
    Handles bgw.ProgressChanged
  ' The BackgroundWorker.ReportProgress method was called.
  ' This procedure runs in the same thread as the host form.

  ' Retrieve the FileFoundEventArgs reference, and display the file name,
  ' the number of found files, and the current progress percentage:
  Dim state As FileFoundEventArgs = _
    CType(e.UserState, FileFoundEventArgs)
  UpdateDisplay(state.FileFound.FullName, _
    state.Counter, e.ProgressPercentage)
End Sub

The UpdateDisplay procedure simply takes the information and updates the form's content, as shown here:

Public Sub UpdateDisplay( _
  ByVal Item As String, ByVal Counter As Long, ByVal Value As Integer)

  ' Add the item to the list box, update the status bar,
  ' and display the number of found files in the label control.
  lstResults.Items.Add(Item)
  pbStatus.Value = Value
  lblCounter.Text = String.Format("{0} file(s) found", Counter)
  Me.Update()
End Sub
As you can see, it takes no extra effort to get asynchronous behavior—there's no need to create a delegate type or to invoke a delegate instance. However, there is still one unresolved issue: how do you cancel the search? In theory, this is easy.

Canceling the Search

Once you set the WorkerSupportsCancellation property of the BackgroundWorker component to True, you can cancel the asynchronous operation, calling the BackgroundWorker's CancelAsync method like so:

Private Sub btnCancel_Click(ByVal sender As System.Object, _
  ByVal e As System.EventArgs) Handles btnCancel.Click
  ' Cancel the async operation:
  bgw.CancelAsync()
End Sub
Just as with the WorkerReportsProgress method, unless you explicitly set the WorkerSupportsCancellation property to True, you'll never successfully cancel the operation. (In this version of the sample, the Form.Closing event handler also calls the CancelAsync method, ensuring that closing the form cancels the asynchronous operation if it's running.)

By calling the CancelAsync method, all you've really done is signal to the component that you want to cancel any pending operations. It's up to your code to stop doing its work, by checking the CancellationPending property of the BackgroundWorker component. Once you find that this flag is set, you can decide whether or not to actually suspend your operation. You must explicitly set the Cancel property of the DoWorkEventArgs object passed into the DoWork event handler to indicate that you want to cancel. Normally, you would simply include code like this to cancel the operation from within the DoWork event handler:

If bgw.CancellationPending Then
  e.Cancel = True
  ' You could perform some cleanup before
  ' leaving, or just exit here:
  Exit Sub
End If

In my sample, however, this simple solution won't work. All the activity in the file search occurs in the FileSearch class and is triggered by the call to fs.Execute from inside the BackgroundWorker's DoWork event handler. One of the drawbacks to using the BackgroundWorker component is that it's difficult to extract the working code from the user interface—that is, in order to cancel the operation, you must interact with the BackgroundWorker component itself (retrieving the CancellationPending property) and the DoWorkEventArgs object passed to the DoWork event handler (setting the Cancel property). Because the FileSearch class is independent of the user interface, the code in this example needs to be modified so that the FileSearch class can "see" the BackgroundWorker component and the DoWorkEventArgs object.

To solve this problem, I modified the FileSearch.Execute method to take as its parameters the two necessary object references. The procedure stores the references in class-level variables for use during the search operation. The FileSearch class adds the two class-level variables shown in the following:

' Keep track of BackgroundWorker and DoWorkEventArgs,
' so this code can cancel or report progress:
Private bgw As System.ComponentModel.BackgroundWorker
Private eventArgs As System.ComponentModel.DoWorkEventArgs
The FileSearch.Execute method takes the values and stores them in the class variables, as shown here:
Public Function Execute( _
  ByVal backGroundWorker As System.ComponentModel.BackgroundWorker, _
  ByVal e As System.ComponentModel.DoWorkEventArgs) As ArrayList

  ' Store the reference to the BackgroundWorker and the
  ' DoWorkEventArgs objects:
  bgw = backGroundWorker
  eventArgs = e

  ' Search for matching files.
  Search(LookIn)
  Return alFiles
End Function

Finally, the FileSearch.Search method uses the class-level variables to determine whether it should cancel its search, and indicates to the BackgroundWorker component that it has quit (see the code in Figure 7).

Figure 7 Should It Cancel? (Close Figure 7)
Private Sub Search(ByVal Path As String)
  Dim diLocal As New DirectoryInfo(Path)
  Dim di As DirectoryInfo

  ' Cancellation pending? Cancel and get out.
  If bgw.CancellationPending Then
    eventArgs.Cancel = True
    Exit Sub
  End If

  Try
    ' Handle subfolders, if required.
    If Me.SearchSubfolders Then
      For Each di In diLocal.GetDirectories
        If (File.GetAttributes(di.FullName) And _
          FileAttributes.ReparsePoint) <> _
          FileAttributes.ReparsePoint Then
            Search(di.FullName)
        End If
      Next
    End If

    ' Search for matching file names.
    Dim fi As FileInfo
    Dim afi() As FileInfo = diLocal.GetFiles(Me.FileSpec)

    For Each fi In afi
      ' Cancellation pending? Cancel and get out.
      If bgw.CancellationPending Then
        eventArgs.Cancel = True
        Exit For
      End If
      OnFileFound(fi)
    Next
    alFiles.AddRange(afi)

  Catch ex As UnauthorizedAccessException
    ' Don't do anything at all, just quietly get out. This means you
    ' weren't meant to be in this folder. All other exceptions will
    ' bubble back out to the caller.
  End Try
End Sub

Although it seems that using the BackgroundWorker component takes a lot of effort, it really doesn't. This particular example showcases most of the features of the component, but you don't need to use them all in every application you write. One thing's for sure: if you're creating Windows-based applications that require asynchronous behavior, and you don't need explicit control over the background operation, using the BackgroundWorker component can save you time and relieve you from the effort of creating your own delegates and invoking them. It doesn't really get much easier than that.

Send your questions and comments for Ken to basics@microsoft.com.

Ken Getz is a senior consultant with MCW Technologies. He is coauthor of ASP.NET Developer's JumpStart (Addison-Wesley, 2002), Access Developer's Handbook (Sybex, 2001), and VBA Developer's Handbook, 2nd Edition (Sybex, 2001). Reach him at keng@mcwtech.com.


Cutting Edge: Moving DataGrid Rows Up and Down

Cutting Edge
Moving DataGrid Rows Up and Down
Dino Esposito

Code download available at: CuttingEdge0503.exe (136KB)
Contents
The RowOver DHTML Behavior
So What's Wrong with RowOver?
The Mover Behavior
The TableGrid Control
Under the Hood of the Table Rendering
That Darn PostBack
ASP.NET 2.0 Considerations
Note on the Sample Code

Imagine opening your Inbox one morning and finding a message that reads "Dear Mr. DataGrid, I urgently need an ASP.NET DataGrid that lets my users move rows on the client. You're my last hope. Will you please help me?"

Wouldn't you do your best to answer such a plea? When I got that very message, I pointed its author to my January 2004 column where I discuss how to enhance an ASP.NET DataGrid to drag columns and sort displayed rows on the client. That solution was built atop DHTML behaviors. Although internally it's pure script, a DHTML behavior is really more than that. It adds object orientation to plain JavaScript and lets you expose properties and methods to drive virtually any client-side HTML element.

However, when my needy programmer friend was still unable to solve his problem, it was time for Mr. DataGrid to save the day (yep, just like a cartoon superhero). Thus was born this month's column—not merely an account of how the specific problem was solved, but also a sneak preview of the control development process as it will work in ASP.NET 2.0.

The RowOver DHTML Behavior

The RowOver.htc file is part of the Microsoft® Internet Explorer DHTML Behavior Library, a collection of free components for advanced client functionalities (rowover Behavior). Specifically designed for Internet Explorer 5.0 and later, behaviors add interactive effects to Web pages and provide a means for script encapsulation and code reuse. Applying a behavior to an HTML element is as easy as attaching a style. In its original version, the RowOver.htc behavior adds alternate row shading and highlighting for table elements:

<table Style="behavior:url(rowover.htc)"
       ID="grid" Selectable="true">
...
</table>

The Selectable attribute refers to a public property exposed by the RowOver behavior. A behavior can be seamlessly attached to ASP.NET controls too, and the following lines of code adds one to the output of a DataGrid:

<asp:DataGrid Runat="server" ID="grid"
     Style="BEHAVIOR:url(rowover.htc)"
     Selectable="true" />

In Figure 1, the ordinary output of the DataGrid is extended with row highlighting and selection. The row under the mouse is tracked as you move across the table and is displayed with a different set of graphical attributes. In addition, when you click on a highlighted row, the row is tracked as selected. It is important to note that all this occurs on the client with no postback to the server.

Figure 1 Ordinary DataGrid
Figure 1 Ordinary DataGrid

While all of this looks great, once you start testing the feature more thoroughly you will discover a number of issues with the RowOver behavior that prevent it from addressing the problem at hand.

So What's Wrong with RowOver?

The row highlighting feature selects all table rows found, including caption, head, and foot rows, and there's no way to get the index of the currently selected row. In addition, the RowOver behavior is not designed to move rows up and down after the user clicks on client buttons, but my friend's ultimate goal was to get a list of rows that could be ordered (moved up and down) using client-side buttons and then posted to the server.

The solution is to build a custom DataGrid control and implement IPostBackDataHandler. But before I get started on this, I'll first enhance the original RowOver.htc behavior to make it support row movements. Next, I'll build a DataGrid control that returns a better-formed table with THEAD, TBODY, and TFOOT tags. This custom control will implement IPostBackDataHandler and carry data back and forth using its own hidden field. Let's start by going over the behavior enhancements.

The Mover Behavior

My new Mover.htc behavior is different from RowOver.htc on several fronts. Mover.htc has none of the original code that does row shading. If you want alternate row coloring you'd be better off using DataGrid styles today. (The original RowOver source code dates back to late 1999, before ASP.NET was born.)

Internally, RowOver.htc tracks the currently selected row and stores a reference to it in an internal variable. However, the behavior doesn't have a public variable to read the index of this row:

<PROPERTY NAME="SelectedIndex" />
This is just what the new SelectedIndex property does. Using this property you can write the following event handler for the button that was shown in Figure 1:
function DoSelect() {
   alert("Selected Row: " + grid.SelectedIndex);
}

Mover.htc also features two new methods to move the selected row one position up and down:

<METHOD NAME="MoveUp"   />
<METHOD NAME="MoveDown" />
Figure 2 shows the source code for these methods. It is interesting to note the richness of the DHTML object model. The DHTML Table object includes a method that moves rows from a starting to an ending position, as shown here:
element.moveRow(moveFrom, moveTo);

Figure 2 MoveUp and MoveDown Methods on the DHTML Behavior (Close Figure 2)
//+----------------------------------------------------------------------
//  Function:       InitComponent
//  Description:    Called during the initialization of the behavior.
//                  Caches some internal-use variables
//  Arguments:      none
//  Returns:        nothing
//-----------------------------------------------------------------------

function InitComponent()
{
    tBody = element.tBodies[0];
    bodyRowCount = tBody.rows.length-1;

    nFirstBodyIndex = tBody.rows[0].rowIndex;
    nLastBodyIndex = tBody.rows[bodyRowCount].rowIndex;

    SelectedIndex = -1;
}

//+----------------------------------------------------------------------
//  Function:       MoveUp
//  Description:    Move the selected row one position up
//  Arguments:      none
//  Returns:        nothing
//-----------------------------------------------------------------------

function MoveUp()
{
    if (!eval(Selectable)) return false;
    if (oSelectRow == null) return false;

    // Determine starting and ending position
    var moveFrom = oSelectRow.rowIndex;
    var moveTo = oSelectRow.rowIndex-1;

    // Should not move out of <tbody> rows
    if (moveTo < nFirstBodyIndex)
       moveTo = nLastBodyIndex;

    // Move
    element.moveRow(moveFrom, moveTo);
}

//+----------------------------------------------------------------------
//  Function:       MoveDown
//  Description:    Move the selected row one position down
//  Arguments:      none
//  Returns:        nothing
//-----------------------------------------------------------------------

function MoveDown()
{
    if (!eval(Selectable)) return false;
    if (oSelectRow == null) return false;

    // Determine starting and ending position
    var moveFrom = oSelectRow.rowIndex;
    var moveTo = oSelectRow.rowIndex+1;

    // Should not move out of <tbody> rows
    if (moveTo > nLastBodyIndex)
       moveTo = nFirstBodyIndex;

    // Move
    element.moveRow(moveFrom, moveTo);
}

To get the index of the selected row, you use the rowIndex property. You should be aware that rowIndex returns the absolute index of the row in the table element; it is not a value relative to the body. In order to get the relative index, I use a couple of internal helper variables, as you can see in the code.

Now that I have a fully functional behavior to move rows up and down, I just need to create a slightly modified DataGrid control to create the client table. Mover.htc works with or without a TBODY tag, but having that tag in place automatically limits highlighting and selecting the body rows.

The TableGrid Control

The new control is named TableGrid and overrides the rendering mechanism of the grid to insert THEAD, TBODY, and TFOOT tags. The control automatically binds to the Mover.htc behavior and exports a new Selectable property to activate client-side selection from the server. Figure 3 shows the implementation of the Selectable Boolean property.

Figure 3 Selectable Property on the Custom Control (Close Figure 3)
namespace Msdn
{
    public class TableGrid : System.Web.UI.WebControls.DataGrid
    {
        // **************************************************************
        // PROPERTY: Selectable
        // Enables the mover.htc component to work on the grid's output
        public bool Selectable
        {
            get
            {
                object o = ViewState["Selectable"];
                if (o == null)
                    return true;
                return (bool) o;
            }
            set
            {
                ViewState["Selectable"] = value;
                Attributes["Selectable"] = value.ToString().ToLower();
                if (value)
                    Style["behavior"] = "url(mover.htc)";
                else Style.Remove("behavior");
            }
        }
        •••
    }
}

As I mentioned, THEAD, TFOOT, and TBODY tags are important because the HTC detects their presence and automatically adjusts its runtime behavior. The issue is that the DataGrid control is not designed to output those tags. In the January 2004 column, I faced the same problem while building a drag and drop DataGrid. In that case, I first had the control output the standard markup to an in-memory buffer and then parsed the text to inject any missing tags. Although functional, this approach is neither elegant nor particularly efficient. The primary drawback lies in the fact that you need to both hold in memory and process the entire output of the DataGrid, which can be a large chunk of text and which could limit scalability under certain conditions. Note that the default DataGrid is much faster because it writes directly to the output buffer. In this context, page output caching can help alleviate your scalability troubles at least for the first request of the page, that is, non-postback requests.

Is there a way to avoid parsing the whole markup for the DataGrid? My understanding is that this is impossible unless you completely replace the rendering engine of the DataGrid. Let's take a look at this engine. The most interesting thing that is going on here is that the DataGrid control doesn't have its own rendering engine; it simply relies on the rendering engine of its constituent controls. The DataGrid is a composite control based on a table that works by building a collection of rows and cells. Next, it iteratively calls to these controls to render out to HTML. In the end, stating that the ASP.NET 1.1 DataGrid control doesn't support the TBODY tag is technically incorrect because the DataGrid doesn't even attempt to render any HTML markup.

The point is that the Table control (and related controls like TableRow) in ASP.NET 1.1 don't support those tags. All things considered, I still believe that parsing up the generated markup is a reasonable way out in ASP.NET 1.1. If you foresee major scalability issues with the approach outlined here (and also implemented in the January 2004 drag and drop control) then you might also want to try output filters.

An output filter is a custom stream object that you attach to the page's output buffer to post-process any outgoing markup. In this way, you process the full markup of the page in a single shot, apply your changes and, best of all, you don't need to use custom controls and or modify existing pages.

An alternative way to support the table section tags, and one that's arguably more elegant, is to derive from Table and override the RenderContents method in order to output section tags. Then you would override DataGrid's CreateControlHierarchy method to create your own Table type and override CreateItem to create your own DataGridItem type. This is a much more complicated solution because you have to override CreateControlHierarchy, but it's more efficient at run time. That said, the solution I've proposed works and, for most situations, it's perfectly adequate.

The following code snippet shows the overridden Render method of the custom DataGrid control:

protected override void Render(HtmlTextWriter writer)
{
   string markup = GetDefaultMarkup();
   markup = FixMarkup(markup);
   writer.Write(markup);
}
For more details, take a look at the code available for download at the link at the top of this article.

Under the Hood of the Table Rendering

To better understand the DataGrid's rendering machinery, it's useful to review how the DataGrid's implementation has changed after the hotfix discussed in Knowledge Base article 823030 ("FIX: DataGrid Made Compliant with Section 508 of the Rehabilitation Act Amendments of 1998"). In ASP.NET 1.1, the DataGrid's markup is not compliant with Section 508 of the Rehabilitation Act Amendments of 1998, a U.S. law regarding accessibility standards. To resolve this problem, Microsoft made a hotfix rollup package available, as explained in the aforementioned article. In short, the act states that data tables that contain two or more rows or columns must clearly identify row and column headers. This means that, at the very minimum, the header row(s) of an HTML table must define its cells through the TH tag instead of the ordinary TD tag.

This change has been implemented by slightly modifying the DataGrid's protected virtual method InitializeItem:

protected virtual void InitializeItem(
    DataGridItem item, DataGridColumn[] columns)
Any method marked as protected and virtual can be overridden in a custom control that inherits from the class. InitializeItem checks the type of the item being added to the grid; if the header is being initialized, it adds TableHeaderCell objects to the Cells collection instead of TableCell objects. The code in Figure 4 shows the method before and after the fix. TableHeaderCell inherits from TableCell and replaces the TD tag with TH:
public TableHeaderCell() : base(HtmlTextWriterTag.Th)
{
}

Figure 4 How the DataGrid Distinguishes TD and TH Cells (Close Figure 4)

Original ASP.NET 1.1 Code
protected virtual void InitializeItem(
    DataGridItem item, DataGridColumn[] columns) {
      TableCellCollection cells = item.Cells;
      for (int i=0; i<columns.Length; i++)
      {
            TableCell cell = new TableCell();
            columns[i].InitializeCell(cell, i, item.ItemType);
            cells.Add(cell);
      }
}
Code Fixed as for KB823030 (Same Code in ASP.NET 2.0)
protected virtual void InitializeItem(
    DataGridItem item, DataGridColumn[] columns) {
    TableCellCollection cells = item.Cells;
    for (int i=0; i<columns.Length; i++)
    {
        TableCell cell;
        if ((item.ItemType == ListItemType.Header) &&
             UseAccessibleHeader)
        {
            // Renders out as TH
            cell = new TableHeaderCell();

            // Cell provides header information for its column
            cell.Attributes["scope"] = "col";
        }
        else
        {
            cell = new TableCell();
        }
        columns[i].InitializeCell(cell, i, item.ItemType);
        cells.Add(cell);
    }
}

Unfortunately, InitializeItem can't be used to add extra wrapper tags like TBODY and THEAD. TBODY and similar tags don't simply replace a tag; rather they divide the contents of a table into logical sections. To correctly implement TBODY, THEAD, and TFOOT you must intervene in the process that builds a <table> out of the DataGrid contents. This would require you to rewrite the code that renders the Table class and add new members to the DataGrid to add section information to each added row!

The good news is that the rendering of tables changes significantly in ASP.NET 2.0. The TableRow class boasts a new property named TableSection to indicate which section the row belongs to. Possible values are defined in the TableRowSection enum: TableBody, TableHeader, TableFooter. When the Table control renders out to HTML, it checks the section of the row and inserts TBODY, THEAD, and TFOOT as appropriate.

The bad news, as far as ASP.NET 2.0 Beta 1 is concerned, is that neither the DataGrid nor the GridView reflect these infrastructure changes. In other words, DataGrid and GridView in Beta 1 don't output rich tables (those that include THEAD, TFOOT, and TBODY) but merely HTML tables (that have only TABLE, TR, TD). However, if you override the CreateItem method of both (protected virtual method), you can make them output THEAD, TFOOT, and TBODY. This was impossible with DataGrids in ASP.NET 1.1. By default, the CreateItem method creates and returns an instance of the control's row class—DataGridItem for DataGrid and GridViewRow for GridView. Both classes inherit from TableRow but neither sets the TableSection property accordingly. Figure 5 shows a quick fix for the GridView.

Figure 5 Set Table Section (Close Figure 5)
protected override GridViewRow CreateRow(
   int rowIndex,
   int dataSourceIndex,
   DataControlRowType rowType,
   DataControlRowState rowState)
{
   GridViewRow row;
   row = base.CreateRow(rowIndex, dataSourceIndex, rowType, rowState);

   switch (rowType) {
    case DataControlRowType.Header:
       row.TableSection = TableRowSection.TableHeader;
       return row;
       ••• // also deal with pager rows
   }
}

To sum it up, having TBODY, THEAD, and TFOOT tags in the grid output is not strictly required. However, these tags simplify the development of advanced and rich client-side functionalities to make the grid more interactive and pleasant to use.

That Darn PostBack

The new Mover.htc behavior, when applied to a new DataGrid control that exports grouping tags (the TableGrid control), allows you to present a table of rows that can be moved up and down, as in Figure 6. This would be just what's needed if it weren't for postbacks. When you refresh the sample page in Figure 6, all of your changes will be lost. Obviously you need to make client-side changes persistent across postbacks and, more importantly, expose the new order of rows to the server-side code.

Figure 6 Mover Behavior in Action on Sample Page
Figure 6 Mover Behavior in Action on Sample Page

The typical way for a server-side control to receive values from the client is through input fields, including hidden fields. Whenever an HTTP request is posted, the ASP.NET runtime matches the names of the posted fields to server-side controls. If a match is found, the runtime looks for the IPostBackDataHandler interface on the server-side control and invokes its methods. In light of this, the simplest solution you can implement is to create a hidden field with the same name as the grid control and make the control implement the IPostBackDataHandler interface. Let's modify the Render method of the TableGrid control to register a hidden field:

Page.RegisterHiddenField(ClientID, "");

The IPostBackDataHandler interface consists of two methods—LoadPostData and RaisePostDataChangedEvent, as you can see in Figure 7. LoadPostData is loaded to give the control a chance to update its state based on posted values. RaisePostDataChangedEvent, in turn, looks at the return value of the other method and fires a server-side event to notify of any occurred changes.

Figure 7 Implementing the IPostBackDataHandler (Close Figure 7)
// **********************************************************************
// METHOD: LoadPostData
// Process the posted values (non-empty if the order changed)
public virtual bool LoadPostData(string postDataKey,
    NameValueCollection postCollection)
{
    // Skip if no values was posted (field empty)
    string postedRowsOrder =
        Convert.ToString(postCollection[postDataKey]);
    if (postedRowsOrder.Length==0)
        return false;

    // New order posted (will fire a server-side event)
    _rowsOrder = postedRowsOrder;

    return true;
}

// **********************************************************************
//  METHOD: RaisePostDataChangedEvent
//  Fires the RowsOrderChanged server-side event
public virtual void RaisePostDataChangedEvent()
{
    // Fire a server-side event to let developers update the data source
    RowsOrderChangedEventArgs e =
        new RowsOrderChangedEventArgs(_rowsOrder);
    OnRowsOrderChanged(e);
}

In this implementation of the interface, you check for the control's posted value, that is, what the behavior wrote in the hidden field. The contents of the hidden field will provide you with the user's activity on the client with the rows of the table. I'll return to this topic in a moment, but for now let's just assume that this content—a string—represents the order of rows as they are modified on the client. If the string is empty, then it means that the user didn't reorder any rows. In this case, LoadPostData returns false and no server event is ever fired. If a new order has been set, the contents of the hidden field are stored internally and passed as an argument to a custom server event—RowsOrderChanged. The page author will use this value to update the application's state accordingly. Note that it's the developer's responsibility to save the new order to the cache or to a database so that the grid will reflect the new order through ordinary data binding.

In Figure 7, RaisePostDataChangedEvent triggers a custom event defined in the following manner:

public delegate void RowsOrderChangedEventHandler(
   object sender, RowsOrderChangedEventArgs e);
public event RowsOrderChangedEventHandler RowsOrderChanged;
The RowsOrderChangedEventArgs data structure defines the arguments being passed to the page's event handler. The class extends EventArgs by adding an array of integers:
public class RowsOrderChangedEventArgs : EventArgs
{
   public int[] RowsOrder;
   •••
}
Why is it so? The structure of this class depends mostly on the format chosen to stream data to the hidden field. To close the circle, some changes to the behavior are in order. Whenever a row is moved up or down, the resulting order should be persisted to the hidden field. Each row must be uniquely identified through an invariant ID that represents the original position. This ID is dynamically assigned when the behavior loads up. Each displayed body row gets a progressive number:
for(i=0; i<bodyRowCount; i++) {
   var nnm = tBody.rows[i].attributes;
   var namedItem = document.createAttribute("pos");
   namedItem.value = i.toString();
   nnm.setNamedItem(namedItem);
}

When the row has been moved, these IDs are composed to a pipe-separated string to reflect the new order. For example, if you move row #2 one position up in a table that contains three rows, the strings written to the hidden field looks like "2|1|3", reflecting the fact that the second row is now the first. Figure 8 illustrates the code to stream the row order to the hidden field.

Figure 8 Persisting Changes to the Hidden Field (Close Figure 8)
//+----------------------------------------------------------------------
//  Function:       SaveChanges
//  Description:    Serializes the new order of rows to the hidden field.
//                  This function is called from within MoveUp and
//                  MoveDown
//  Arguments:      none
//  Returns:        nothing
//-----------------------------------------------------------------------

function SaveChanges()
{
   // Loop through the body rows and create a pipe-separated string
   var chrSep = "|";
   var sTmp = "";

   for(i=0; i<bodyRowCount; i++)
   {
      sTmp += tBody.rows[i].attributes["pos"].value;
      if (i<bodyRowCount-1)
         sTmp += chrSep;
   }

   // Save the string down to the hidden field (same name as the element)

   // Go up in the hierarchy until the FORM is found
   var obj = element.parentElement;
   while (obj.tagName.toLowerCase() != "form")
   {
    obj = obj.parentElement;
    if (obj == null)
        return;
   }

   var hiddenField = element.id;    // same ID of the server-side control
   if (hiddenField != null)
      obj[hiddenField].value = sTmp;
}

A similar string is passed to the server with each postback and is received by the LoadPostData method. From here, the string makes its way to RaisePostDataChangedEvent and then to the constructor of the RowsOrderChangedEventArgs class, as in Figure 7. The pipe-separated string is then split to an array of integers and served to the code in charge of handling the RowsOrderChanged event.

This code takes the responsibility of persisting the row order as required by the application. You can, for instance, store the new order to a database or to an object in the ASP.NET cache. The sample application updates the cache entry from where the binding mechanism retrieves the data to bind to the grid:

private void BindData()
{
   DataTable table = (DataTable)Cache["MyData"];
   grid.DataSource = table;
   grid.DataBind();
}

How can you apply the new row order to a DataTable object? This is mostly up to you; however, the approach I've used here entails cloning the existing DataTable object and filling it with the same rows bound in the order suggested by the integer array as shown in Figure 9.

Figure 9 Applying Row Changes to the Bound Data Source (Close Figure 9)
private void grid_RowsOrderChanged(object sender,
    Msdn.RowsOrderChangedEventArgs e)
{
    // e.RowsOrder is an int-array with the new order

    // Update the data in the cache and rebinds
    int [] newPositions = e.RowsOrder;
    DataTable table = (DataTable) Cache["MyData"];
    DataTable newTable = table.Clone();
    for(int i=0; i<newPositions.Length; i++)
    {
       int posToImport = newPositions[i];
       DataRow oldRow = table.Rows[posToImport];
       newTable.ImportRow(oldRow);
    }

    table.Rows.Clear();
    table.Columns.Clear();
    table.Dispose();

    Cache["MyData"] = newTable;
}

ASP.NET 2.0 Considerations

The sample code that accompanies this column is for ASP.NET 1.1 and requires that you make available the Mover.htc file along with the assembly that contains the TableGrid control. In the context of a real-world application this is a non-issue. However, the fact that Mover.htc is just a small text file doesn't prevent you from researching a better approach to wrapping everything up. This is an instance of a more general problem—packaging resources in ASP.NET assemblies. In ASP.NET 1.1, the problem can be addressed through a custom HTTP handler that retrieves and downloads the resource, setting the proper MIME type. Using an HTTP handler requires you to enter a new line in the Web.config file and likely a new entry in the IIS metabase.

In ASP.NET 2.0, a new system HTTP handler (named WebResource.axd) and the WebResource attribute let you retrieve resources from assemblies. WebResource is an assembly-level attribute through which you mark embedded resources as URL-accessible. You first add the Mover.htc file to the project as an embedded resource and then insert the following code to the AssemblyInfo file of the project:

[assembly: WebResource("Mover.htc", "text/javascript")]

To programmatically access the resource (that is, to compile the Style property of the TableGrid control) you use a new method on the Page class—the GetWebResourceUrl method.

In this example, I used a DataGrid-based control, the TableGrid control, which is probably overkill if all that you need is to let users reorder some data-bound rows. The DataGrid, in fact, contains paging, sorting, and editing capabilities that are of no use in this scenario. The rub lies in the fact that in ASP.NET 1.1, building a data-bound table-like control is significantly harder than you may think at first. For this reason, many developers just end up with a customized version of the DataGrid control. Fortunately, this aspect of Web programming is going to change with ASP.NET 2.0.

In ASP.NET 2.0, creating custom data-bound controls is just as easy as picking up the right base class—DataBoundControl, CompositeDataBoundControl, BaseDataBoundControl, or HierarchicalDataBoundControl. These base classes incorporate most of the boring and repetitive boilerplate code (and best practices) you had to learn and manually code in ASP.NET 1.1. What's left out of these extremely useful and well-designed classes is exposed through virtual methods. Finally, creating custom collections for binding and advanced properties is child's play thanks to generics. In ASP.NET 2.0 you don't have to rack your brains to work with data-bound controls.

Note on the Sample Code

When I first tested the MoveGridRows.aspx sample page, I noticed it was suffering from a common software disease—page refresh-itis. Imagine you move row #2 one position up and post values to the server. The grid is updated and redisplayed bound to a new data object that reflects the new order. Looks great. Now try hitting F5 or refresh the page from the toolbar. The most recent request is reissued and the last move (row #2 moved one position up) is repeated. A second, and likely unexpected, movement is performed. This may, or may not, be a problem for you. If it is, though, you're experiencing page refresh-itis.

In a recent article in the ASP.NET Developer Center, I discussed an HTTP module and a new Page class that inform you if the postback results from a regular submit or a browser's page refresh (see Build Your ASP.NET Pages on a Richer Bedrock). Once you have downloaded the code, make the codebehind class of the ASPX page inherit from Msdn.Page, and use the IsPageRefresh property to check against page refreshes. In addition, enable the RefreshModule HTTP module in the Web.config file.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a Wintellect instructor and consultant based in Italy. Author of Programming ASP.NET and his newest book Introducing ASP.NET 2.0 (both from Microsoft Press), he spends most of his time teaching classes on ASP.NET and ADO.NET and speaking at conferences. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Test Run: Automate Your ASP.NET Web Services Testing

Test Run
Automate Your ASP.NET Web Services Testing
James McCaffrey

Code download available at: TestRun0503.exe (146KB)
Contents
The Web Service and the Web App
The Test Automation
Alternative Approaches
Conclusion

It's no exaggeration to say that Web services are revolutionizing application-to-application communication. Web services are already being used extensively in corporate intranet environments and are making their way into commercial use, too. But because Web services are relatively new, techniques to test Web services programmatically are not widely known. In this column I'll show you a technique to quickly write test automation that verifies Web service functionality. The best way to show you what I mean is with two screenshots. Figure 1 shows a simple but representative ASP.NET Web application that uses a Web service.

Figure 1 Web Application that Uses a Web Service
Figure 1 Web Application that Uses a Web Service

In this case the Web application uses a Web service that searches a proprietary SQL database of hypothetical products. Searching product names for the letter "i" returns four products. Notice the application is rather crude (no formatting of the product price data for example) just as it would be in a pre-production environment—I'm usually testing a system while it's still in development rather than after it's been released.

Now suppose you want to test the underlying Web service component of this system. Manually testing the service through any of its three interfaces (the Web client, the Web application, or the Web service) would be error-prone and inefficient—not to mention just plain boring. A better approach is to use the powerful capabilities of the .NET environment to programmatically send input to the Web service, capture the response from the service, and then examine the response for an expected value. The console application shown in Figure 2 does exactly this.

Figure 2 Testing the Web Service
Figure 2 Testing the Web Service

Behind the scenes, the test automation reads test case inputs and expected values, sends each input to the Web service under test, grabs the response from the service, compares the actual response value to the expected value to determine a pass or fail result, and writes the result to an XML file. In Figure 2 test case 003 corresponds to manually searching for products with "i" in their name, as shown in Figure 1. The expected result in this test is the number of items returned from the corresponding search.

In the following sections I'll show you the ASP.NET Web service that I'll be testing, briefly examine the ASP.NET Web application that uses the service, and describe in detail the test automation that tests the service. I'll conclude by discussing how you can adapt and extend the code presented here to meet your own needs in a production environment. I think you'll find the ability to write test automation quickly for Web services interesting and a valuable addition to your skill set.

The Web Service and the Web App

Let's examine the underlying Web service. One of the most common uses of Web services is to expose proprietary data to another application. That's what my Web service does, so let's look at its SQL Server™ database. Using a short T-SQL script I created a database test bed, dbProducts, of product information. The database contains a single table, tblProducts, which has columns for a product ID, a product name, and a product unit price. Figure 3 shows the contents of the products table displayed using the SQL Query Analyzer program.

Figure 3 SQL Test Data
Figure 3 SQL Test Data

When testing any system with a SQL back end, you normally want to create a dedicated database test bed rather than relying on the development or production databases. Here I just have five rows of back-end test data; in a production test environment you will have many more. Database dbProducts also has a stored procedure, usp_GetProducts, that retrieves product information and a SQL login, webServiceLogin, that can access dbProducts and has permission to execute the usp_GetProducts stored procedure. You can see the entire SQL script I used in the code that accompanies this column.

Now that you've seen the database, let's look at the Web service that exposes the data. Creating a Web service with Visual Studio® .NET is very easy because almost all of the plumbing code is generated for you automatically. After launching Visual Studio I created a new ASP.NET Web service project. I decided to use C# as my language, but the test automation I'll present is independent from the Web service language, so I could have used any .NET-compliant language. After creating the project I added a namespace import so that I could use the SqlDataAdapter class without having to fully qualify it, as shown here:

   using System.Data.SqlClient;
Next I added a Web method, GetProducts, to the ProductsService class in the TheWebService namespace (see Figure 4).

Figure 4 GetProducts Web Method (Close Figure 4)
[WebMethod]
public DataSet GetProducts(string filter)
{
  try
  {
    using(SqlConnection sc = new SqlConnection(
      ConnectionSettings.AppSettings["ConnStr"]))
    {
        SqlCommand cmd = new SqlCommand("usp_GetProducts", sc);
        cmd.CommandType = CommandType.StoredProcedure;
        cmd.Parameters.Add("@filter", SqlDbType.VarChar, 35);
        cmd.Parameters["@filter"].Direction = ParameterDirection.Input;
        cmd.Parameters["@filter"].Value = filter;
        SqlDataAdapter sda = new SqlDataAdapter(cmd);
        DataSet ds = new DataSet();
        sda.Fill(ds);
        return ds;
    }
  }
  catch
  {
    return null;
  }
}

Frequently, the terms Web service and Web method are used interchangeably, but an ASP.NET Web service is really a collection of one or more WebMethods. My single WebMethod accepts a string as an input filter, connects to database dbProducts, and returns a DataSet of product information where input filter is contained in the product name.

Notice that GetProducts retrieves data from the dbProducts database into the DataSet using a SQL stored procedure called usp_GetProducts. My Web service could have retrieved data from dbProducts using an embedded T-SQL statement like this one:

cmd.CommandType = CommandType.Text;
cmd.CommandText =
  "select * from tblProducts where prodname like '%' + @filter + '%'";
However, using a stored procedure instead will generally provide better performance and increased security.

ASP.NET provides a very neat way to test simple ASP.NET Web services manually. If you press the F5 key to start the debugger, Visual Studio .NET will launch Microsoft® Internet Explorer and load a page that lists the methods in the Web service. If you click on a Web method, you'll go to a page that allows you to enter arguments for the method manually, invoke the method, and then view the results (see Figure 5).

Figure 5 Manual Testing in Visual Studio .NET
Figure 5 Manual Testing in Visual Studio .NET

The ability to test Web services manually through Visual Studio .NET is very useful during development, but for thorough testing I use an automated system like the one in this column. Before I move on to the test automation program, let's take a quick look at the ASP.NET Web application that uses the Web service.

If you look at Figure 1 you'll see that the Web application has a TextBox control where the user enters a product name, a DataGrid control where product data is displayed, and a Label control where miscellaneous messages are displayed. Some of the productivity-enhancing features in .NET are client-side proxies for Web services that make using a WebMethod just like calling a local method. Figure 6 shows the key code that calls the Web service.

Figure 6 Call Web Service Method (Close Figure 6)
private void Button1_Click(object sender, System.EventArgs e)
{
  try
  {
    WebReference.ProductsService ps = new WebReference.ProductsService();
    string filter = TextBox1.Text.Trim();
    DataSet ds = ps.GetProducts(filter);
    DataGrid1.DataSource = ds;
    DataGrid1.DataBind();
    Label3.Text = ds.Tables["Table"].Rows.Count + " items found";
  }
  catch(Exception ex)
  {
    Label3.Text = ex.Message;
  }
}
Figure 7 Calling a Web Service Through Multiple Layers
Figure 7 Calling a Web Service Through Multiple Layers

After adding a Web Reference to my service (which I created in a namespace called "WebReference" instead of accepting the default namespace "localhost"), I instantiated the ProductsService Web service proxy class just as I would any other ordinary class. Now I can call the WebMethod just like I would an ordinary instance method. You can think of the WebMethod component of a system under test as being very much like a local method. And as you'll see in a moment, despite their loosely coupled, distributed nature, testing Web service methods is similar in many respects to traditional API testing. See Figure 7 for a generalized schematic of a typical system that calls a Web service.

The Test Automation

Now let's examine the test case input data and test automation harness that generated the test run that was shown in Figure 2. I decided to store my test case input data in an XML file (see Figure 8). I could have used SQL Server or a text file to store my test case input data, but because Web services are built on XML, it was a natural choice.

Figure 8 Test Case Input Data (Close Figure 8)
<?xml version="1.0" encoding="utf-8" ?>
<cases>
  <testcase>
    <id>001</id>
    <input>widget</input>
    <expected count="1" clue="widget" />
  </testcase>
  <testcase>
    <id>002</id>
    <input>wiidget</input>
    <expected count="0" clue="" />
  </testcase>
  <testcase>
    <id>003</id>
    <input>i</input>
    <expected count="4" clue="widget" />
  </testcase>
  <testcase comment="deliberate failure">
    <id>004</id>
    <input>idg</input>
    <expected count="0" clue="" />
  </testcase>
  <testcase>
    <id>005</id>
    <input>w</input>
    <expected count="3" clue="widget" />
  </testcase>
</cases>

The test case input data is self explanatory except for the <expected /> element. Each expected element has a count attribute, which holds the number of rows the resulting DataSet object should have for the specified input. For example, in test case 001 if the input is "widget", then the method GetProducts should return a DataSet with exactly 1 row. The clue element is additional information that the test automation can use to determine if the correct rows are returned. I'll discuss this in more detail in the next section. Notice I have deliberately included a test case that will fail so that I can make sure my test automation doesn't have a logic error that makes all test cases pass.

The overall test harness plan is very simple but has been used successfully on several medium-sized software projects. In pseudocode the plan is as follows:

read test case data from XML into memory
loop
  fetch a single test case node
  parse out test case input and expected values
  send input argument to GetProducts() Web method
  retrieve the DataSet return value
  if DataSet is consistent with expected values
    write "pass" to XML result node
  else
    write "fail" to XML result node
end loop
write all result data from memory to XML

I implemented this plan using a C# console application. I could just as easily have used another .NET-targeted language like Visual Basic® .NET, and the code I present here will also work in a Windows®-based application or an ASP.NET Web application. The test harness is surprisingly short and is shown in Figure 9. Let's go over it in detail so that you'll be able to modify it to meet your own particular needs.

Figure 9 Test Automation Code (Close Figure 9)
static void Main(string[] args)
{
  if (args.Length != 1)
  {
    Console.WriteLine("usage: test.exe testCases.xml");
    return;
  }

  try
  {
    string id, input, expected, clue;
    DateTime startTime = DateTime.Now;
    Console.WriteLine("\nStart Web Service Test Automation");

    XmlDocument results = new XmlDocument();
    XmlDeclaration xdeclare =
      results.CreateXmlDeclaration("1.0", null, null);
    results.AppendChild(xdeclare); // declaration
    XmlElement root = results.CreateElement("results"); // root element
    results.AppendChild(root); // root
    XmlElement result;

    XmlDocument tests = new XmlDocument();
    tests.Load(args[0]); // path to XML test case data is first arg
    XmlNodeList xnl = tests.SelectNodes("/cases/testcase");
    foreach (XmlNode tcn in xnl)
    {
      id = tcn.ChildNodes.Item(0).InnerText;
      input = tcn.ChildNodes.Item(1).InnerText;
      expected =
        tcn.ChildNodes.Item(2).Attributes.GetNamedItem("count").Value;
      clue = tcn.ChildNodes.Item(2).Attributes.GetNamedItem("clue").Value;

      WebReference.ProductsService ps =
        new WebReference.ProductsService();
      DataSet ds = new DataSet();
      ds = ps.GetProducts(input);

      result = results.CreateElement("result");
      result.SetAttribute("id", id);
      result.SetAttribute("input", input);
      result.SetAttribute("expected", expected);

      result.InnerText = IsConsistent(ds, expected, clue) ?
        " Pass " : " *FAIL* ";

      root.AppendChild(result);
    } // foreach test case node
    results.Save("results.xml");
    Console.WriteLine("End Test Automation");
    DateTime endTime = DateTime.Now;
    TimeSpan ts = endTime - startTime;
    Console.WriteLine(
      "Elapsed time = " + ts.Milliseconds + " milliseconds");
  } // try
  catch(Exception ex)
  {
    Console.WriteLine(ex.Message);
  }
}

After adding a Web reference to the ProductsService service and adding a using statement for the System.Xml and System.Data namespaces, I start by declaring the four local string variables and one DateTime object that my test harness will use:

string id, input, expected, clue;
DateTime startTime = DateTime.Now;
Console.WriteLine("\nStart Web Service Test Automation");

The id, input, expected, and clue variables will hold information parsed from the XML test case input file. I get the start time so I can measure the total elapsed time of the test automation. It's often a good idea to time your test automation because an unusually long or short test run should be investigated. Timing Web service test automation is especially useful because there are generally several network connections involved and they are a potential source of trouble.

The next few lines of code prepare an in-memory data store to hold my test case results:

XmlDocument results = new XmlDocument();
XmlDeclaration xdeclare = results.CreateXmlDeclaration("1.0", null, null);
results.AppendChild(xdeclare);
XmlElement root = results.CreateElement("results");
results.AppendChild(root);
XmlElement result;
The object referenced by results is an XmlDocument from the System.Xml namespace. Just as with my test case input data, I could have written my test results data to a SQL Server database or a text file but I prefer XML here. After creating the document, I write an XML header declaration into it and then I create and append the required XML root element. Finally, I create an XML element named <result> to hold the pass or fail result of each test case. The next part of the test harness does most of the work:
XmlDocument tests = new XmlDocument();
tests.Load(args[0]);
XmlNodeList xnl = tests.SelectNodes("/cases/testcase");
foreach (XmlNode tcn in xnl)
{
  // execute test case and store test result
}
results.Save("results.xml");
Console.WriteLine("End Test Automation");
DateTime endTime = DateTime.Now;
TimeSpan ts = endTime - startTime;
Console.WriteLine("Elapsed time = " + ts.Milliseconds + " milliseconds");

I start by creating a second XmlDocument object, this one to hold the test case input. I use the XmlDocument.Load method to read my test case input file, whose path is provided to the test application is the first command-line argument, into memory. Next I grab all of the XML testcase nodes into an XmlNodeList object using XmlDocument.SelectNodes. Recall that each node looks like the following:

<testcase>
  <id>xxx</id>
  <input>xxx</input>
  <expected count="x" clue="xxx" />
</testcase>

I iterate through the list of test case input nodes with a foreach loop. Because an XmlNodeList object is a collection, I could have also iterated with a for loop like so:

for (int i = 0; i < xnl.Count; ++i) { ... }

Inside the loop I send a test case input to the GetProducts Web method, capture the resulting DataSet object, use the expected value and clue to determine if the DataSet is correct, and write a pass or fail element into the test case result XmlDocument. After all test case input nodes have been processed, I use the XmlDocument.Save method to write the in-memory results to an external file. I finish the test run by determining and displaying the total elapsed time for the text cases.

Inside the foreach loop I begin by using the ChildNodes property to fetch the test case ID, test case input argument, an expected number of rows in the resulting DataSet, and a "clue" that will help me determine if the DataSet holds the correct data in its rows:

id = tcn.ChildNodes.Item(0).InnerText;
input = tcn.ChildNodes.Item(1).InnerText;
expected =
  tcn.ChildNodes.Item(2).Attributes.GetNamedItem("count").Value;
clue = tcn.ChildNodes.Item(2).Attributes.GetNamedItem("clue").Value;

This logic assumes that each test case input node has the exact same element and node structure so that even if I don't want to have a clue for a particular test case, I've still got to include one as an empty string. (If you want, you can add logic to this part of test harness to first test whether the various attributes actually exist in order to handle variable-structure test case input.) Once I have the test case inputs, calling the Web service is easy:

WebReference.ProductsService ps = new WebReference.ProductsService();
DataSet ds = ps.GetProducts(input);

I instantiate the ProductService Web service proxy just as I would any other local object that has a parameterless constructor. Of course, I had to add a Web Reference to the service into my test harness. Then I call the GetProducts WebMethod just like a normal method and save the return value into a DataSet object. Under the covers, the client proxy object makes a SOAP request to the server where ASP.NET parses that request and invokes the appropriate WebMethod with the specified arguments. With the results returned to the client, I write test case data into the XML test result element I created earlier:

result = results.CreateElement("result");
result.SetAttribute("id", id);
result.SetAttribute("input", input);
result.SetAttribute("expected", expected);
This will produce an element that looks like the following:
<result id="xxx" input="xxx" expected="xxx"></result>
Because XML is so flexible, there are many alternative formats I could have used. The last few lines inside the test case input-controlled loop determine whether the test case passes or fails and adds that result to the test case result node:
result.InnerText = IsConsistent(ds, expected, clue)?
  " Pass " : " *FAIL* ";
root.AppendChild(result);

I call a helper method IsConsistent to determine if each test case passes or fails. The IsConsistent method examines the result DataSet ds and checks to see if it has the correct number of rows and if the clue string is in the first row. The code for IsConsistent is shown here:

static bool IsConsistent(DataSet ds, string expected, string clue)
{
  if (ds.Tables["Table"].Rows.Count != int.Parse(expected)) return false;

  if (clue != null && clue.Length > 0)
  {
    if (ds.Tables["Table"].Rows[0]["prodname"].
      ToString().IndexOf(clue) < 0)
      return false;
  }
  return true;
}

If the expected number of rows in the result DataSet doesn't equal the expected number of rows then the test case fails. Next I check to see if a clue string is contained in the first row of the result DataSet. This does not guarantee that the result DataSet is exactly correct, but it does give me confidence that the result DataSet is probably correct. I'll discuss other possible approaches next.

Alternative Approaches

You can modify and extend the Web service test automation technique I've presented here in many ways. This test automation harness connects to the Web service under test by adding a Web reference in Visual Studio .NET. Some of my colleagues prefer to generate this plumbing code from the command line using the wsdl.exe tool. Wsdl.exe is part of the .NET Framework SDK and can be used to generate code that you can copy and paste into the test automation harness or simply add the generated file to your project. The advantage of using wsdl.exe is that you can fine-tune your Web service connection plumbing if necessary without worrying that Visual Studio .NET will regenerate the file and overwrite your changes.

The technique I've presented here reads all test case input data into an in-memory XmlDocument object. Another approach is to use the XmlTextReader class to read one XML test case input node at a time. Using an XmlDocument is simpler but doesn't scale well if you are expecting to have many hundreds of thousands of test cases. Using an XmlTextReader is slightly more complicated but scales well to a large number of test cases.

When I present the Web service test automation technique in this column to experienced testers, one of the first things they want to discuss is the IsConsistent method. Recall that IsConsistent accepts a DataSet, an expected number of rows, and a "clue" string to determine if the DataSet is correct or not. The logic in IsConsistent does not guarantee that the actual result DataSet is exactly the same as an expected result DataSet. You can argue that it is better to write a helper method IsEqual(DataSet actual, DataSet expected) that does a deep comparison of two DataSet objects.

This idea is valid, but there are two problems. First, DataSet objects can be very complex, so determining if two of them are exactly the same is often very difficult. A major theme of the technique presented here is that the test automation should be quick and easy to write. The second problem with an IsEqual method is that because of the DataSet complexity you can easily end up with huge test case input data. Ultimately how you choose to compare actual and expected results will depend upon the details of your particular production environment.

Let me remind you that this technique is just one small part of Web services testing. You also need to test your Web services through any clients of the service, and you should thoroughly and separately test any database functionality on which the service relies. Furthermore, the technique presented here only tests Web service functionality. You'll need to test for performance, security, load capacity, and all the other types of software testing. One of the advantages of using software test automation is that it takes care of the time-consuming mundane testing that has to be done, which frees up time for you to test tricky and unusual product scenarios.

Note that I've only included minimal error checking in the Web service, the Web application, and the test automation. In a production environment you'll want to use the try/catch/finally mechanism liberally to prevent your automation from stopping in mid-run. In general your test automation will be running unattended and you can expect to find problems. Also, I only have a few rows of data in the test bed database and only a few test cases—in a production environment you'll want to add lots more data.

Conclusion

One of the reasons ASP.NET Web services have quickly gained acceptance by the software development community is that the programming model allows you to instantiate Web services and call WebMethods just like you would local objects and methods. As you've seen here, a consequence of this consistency is that testing Web service methods is much like traditional API testing.

The use of Web services is certainly going to increase. The release of Visual Studio 2005 will include the .NET Framework 2.0, which will deliver significant performance, productivity, and security enhancements to ASP.NET Web services. My colleagues and I have successfully used the test automation techniques presented here with Visual Studio 2005 so you'll be ready for it, too.

Send your questions and comments for James to testrun@microsoft.com.

James McCaffrey works for Volt Information Sciences Inc., where he manages technical training for software engineers working at Microsoft. He has worked on several Microsoft products including Internet Explorer and MSN Search. James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Inside MSDN: Using InfoPath as a Reporting Solution

Inside MSDN
Using InfoPath as a Reporting Solution
Larry W. Jordan Jr.

Code download available at: InsideMSDN0503.exe (355KB)
Contents
The Employee Data List
The Project Data List
The Status Reporting Form
InfoPath Merge Reporting
Form Design Elements
Charting
Conclusion

I love the part of my job that lets me write code and develop software. In addition to those responsibilities, however, I manage a development organization that's working on a number of large strategic projects. As you know, one of the challenges faced in managing a team is finding the right balance between getting the information I need to be an effective manager and avoiding the kind of intrusion on team members that can inhibit their work. To get my information without intruding, my team and I built a reporting solution that each team member can use to report project status. The application rolls up all of the data reported into multiple views that I can then analyze to get a sense of how all the various parts of the projects are progressing.

I based the solution on Microsoft® Windows® SharePoint® Services and on Microsoft InfoPath® 2003 SP1 because they're easy to use and deploy. Employees create, edit, and view reports in InfoPath, with SharePoint serving as a data and report repository. InfoPath makes it simple for the individual teams to deploy various project status forms to SharePoint sites. As a bonus, this allows us to maintain the security and privacy of the status data because SharePoint provides domain and user-level security. Plus, it is easy to administer. SharePoint also offers many data storage and reporting options, and InfoPath simplifies integration with Web services, SQL Server™, and Visual Studio®.

One of my main goals was to have the solution integrate across the development, test, and program management teams of MSDN® and TechNet. This required that the solution be maintainable by people who don't work in Visual Studio every day and it required that updates and changes to employee and project data could be made by anyone on the team.

I also wanted to collect a number of metrics to use in resource allocation decisions, including active development projects, developer and tester activity in the defect tracking system, check-in e-mails to the change management systems (source control), and discussions with the leads on my team.

I also had the following requirements:

  • The app should be written in managed code.
  • It should integrate our product defect and change tracking system metrics.
  • It should use the charting capabilities of the Office Web Components.
  • It should post the submissions to both a SharePoint site and a Web service interfacing to a SQL database.
  • The overall organizational reports should be generated from the database and implemented using SQL Reporting Services.

The Employee Data List

Knowing up front what data will be harvested helps to drive future decisions, so the first step in authoring the solution was to arrive at a schema to be used for the individual status reports. Figure 1 shows the status reporting schema. This schema can then be laid out in a simple user interface, like the one shown in Figure 2.

Figure 2 Simple UI for Status Report Based on Schema (Close Figure 2)

Figure 1 Schema
Figure 1 Schema

Before I dig further into the InfoPath development, let's do some of the preparatory work to build and configure the SharePoint site and the lookup lists that will be used by the InfoPath client. In Office 2003 SP1, InfoPath has excellent integration with the lists provided by SharePoint and it makes cross-team maintenance and security very easy.

First, you must create the Employee Data list, which centralizes the lookup that will be used to assign the employee's manager, department, and team. On your SharePoint site, create a Custom List in Datasheet View, name the list "EmployeeData," and customize it to include the following columns:

Column | Type | Required
Name | Single line of text | Yes
Alias | Single line of text | Yes
Manager | Single line of text | Yes
ManagerAlias | Single line of text | Yes
Department | Single line of text | Yes
IsManager | Yes/No | Yes

Add in some sample data and designate one person in the group as the manager. This is done by having each person's Manager and ManagerAlias fields contain the manager's Name and Alias. The manager's IsManager field can then be set to Yes.

The Project Data List

The Project Data list will be used for the assignment of the projects an employee is working on and the allocation of time spent on various activities. First, create a Custom List in Datasheet View, name the list "ProjectData", and then modify it to include the following columns:

Column | Type | Required
Activity-ProjectName | Single line of text | Yes
Initiative | Single line of text | Yes
Commitment | Single line of text | Yes

Now you can add some sample data for projects, aligned to Initiatives and Commitments, as shown in Figure 3. This is the way we look at project resource allocations in my work group. You will want to organize your project list according to the way you want to track your own group's data.

Figure 3 The Project Data List (Close Figure 3)

The Status Reporting Form

With the lists in SharePoint configured appropriately, you can now focus on wiring up InfoPath to work with this list data. The status reporting form itself is quite minimal and is designed to be filled out quickly.

There are several steps I take when I deploy a form to a new SharePoint site for a specific team. Since I have the standards for the schemas of the EmployeeData and ProjectData lists, I merely have to point the form to the location of those SharePoint lists:

  1. Open up the Tools | Data Connections options in the InfoPath Designer.
  2. Select "Add" (or select "Modify" if the connection already exists) to take you through the wizard to connect to the SharePoint list.
  3. Set the form to point to your top-level site that hosts the ProjectData list and click Next.
  4. Select the ProjectData item from the list and click Next.
  5. Highlight the values that you will use in your form (Initiative, ActivityProjectName, or Commitment) and click Next.
  6. To finish, indicate the name of the ProjectData data source (which I'll keep as ProjectData so my XML code works without modifications), tell InfoPath to "Automatically retrieve data when form is opened," and click Finish.

I can think of many opportunities to make this programmatic and configurable so that the step of modifying the form does not need to happen, but in my experience, the five minutes it takes to deploy a new instance outweighs the abstraction investment.

You will need to repeat steps 2-6 to associate the EmployeeData list with the form, but with a change in step 5 to indicate the schema of the EmployeeData.

Now that you have modified the form to pull from your SharePoint site as the source for the lists, you need to set up the data filter to use for the lookup of the EmployeeData. Remember that the EmployeeData list schema includes the employee's department. This value is used to set the header on the form and to filter the list of employees based on the specific department. In order for the list to work, I use the filtering capabilities built into the InfoPath designer. Again, there is another opportunity to make this more programmatic by setting the department to a field value on a hidden view or the like, but for the purpose of this column I want you to see how I'm doing it.

In the InfoPath designer, highlight and right-click the Name dropdown list and select "DropDown List Box Properties" from the context menu. Select the Option button next to the "Entries" textbox and bring up the Data selection for "Select Field or Group." You should note that the Name field is highlighted. Click the Filter Data button and set the formula to filter on the "Development" department. Select Modify, then choose "Type text..." from the dropdown, and enter "Development."

Now that you have your data sources for Employee and Project lookups configured, you will need to add some code to facilitate the population of the fixed fields such as Manager Alias, Department, and so forth.

The code in Figure 4 is associated with the Name DropDown list that runs when you change the employee name. The code looks up the employee in the associated EmployeeData data source and fills in the fields in the XDocument DOM.

Figure 4 Look Up Employee (Close Figure 4)
// The following function handler is created by Microsoft Office InfoPath.
// Do not modify the type or number of arguments.
[InfoPathEventHandler(
 MatchPath="/sr:Team/sr:EmployeeReport/sr:Employee/sr:Name",
 EventType=InfoPathEventType.OnAfterChange)]
public void Employee_Name_OnAfterChange(DataDOMEvent e)
{
    if (e.IsUndoRedo)
    {
        // An undo or redo operation has occurred; the DOM is read-only.
        return;
    }

    try
     {
        string sEmployeeName = e.NewValue.ToString();
        if (sEmployeeName != null && sEmployeeName != string.Empty)
        {
            // -- load the XML Data for EmployeeData
            xmlEmployeeDataDoc.LoadXml(
                thisXDocument.DataObjects["EmployeeData"].DOM.xml);

            // -- set EmployeeData namespace managers
            XmlNamespaceManager nsEmployeeDataMgr =
                new XmlNamespaceManager(xmlEmployeeDataDoc.NameTable);
            string dfNamespace = "http://schemas.microsoft.com/office" +
                "/infopath/2003/dataFormSolution"
            nsEmployeeDataMgr.AddNamespace("dfs", dfNamespace);
            nsEmployeeDataMgr.AddNamespace("dsf", dfNamespace);

            XmlNode xmlEmployeeDataNode =
                xmlEmployeeDataDoc.SelectSingleNode(
                    "/dfs:myFields/dfs:dataFields/" +
                    "dsf:EmployeeData[@Name=\"" + sEmployeeName + "\"]",
                    nsEmployeeDataMgr);

            // -- set to the form xml and update the data
            IXMLDOMNode xmlNodeEmployee = thisXDocument.DOM
                .selectSingleNode("//sr:Employee");
            IXMLDOMNode xmlNodeReportDate = thisXDocument.DOM
                .selectSingleNode("//sr:ReportDate");

            xmlNodeEmployee.selectSingleNode("//sr:EmailAddress").text =
                xmlEmployeeDataNode.SelectSingleNode("@Alias").Value;

            xmlNodeEmployee.selectSingleNode(
                "//sr:Manager/sr:EmailAddress").text =
                xmlEmployeeDataNode.SelectSingleNode(
                "@ManagerAlias").Value;

            xmlNodeEmployee.selectSingleNode(
                "//sr:Department").text =
                xmlEmployeeDataNode.SelectSingleNode(
                "@Department").Value;

            xmlNodeEmployee.selectSingleNode("//sr:IsManager").text =
                xmlEmployeeDataNode.SelectSingleNode("@IsManager").Value;

            // -- Set Focus to Item element under Accomplishments section
            IXMLDOMNode xmlNodeAccomplishment = thisXDocument.DOM
                .selectSingleNode("//sr:Accomplishments/sr:Item");
            if (xmlNodeAccomplishment != null)
            {
                thisXDocument.View.SelectText(
                    xmlNodeAccomplishment, Type.Missing);
            }

            // -- update global filename
            SetFileName();
        }
    }
    catch (Exception ex)
    {
        thisXDocument.UI.Alert(ex.Message + " Inner: " +
            (ex.InnerException != null ? ex.InnerException.Message : "");
    }
}

There is an additional function to set the field associated with the file name and this is used when the user submits the form so that you can save the file in a specific format and parse the value on the parallel submission to the back-end Web service (see Figure 5).

Figure 5 Set the File Name (Close Figure 5)
void SetFileName()
{
    try
    {
        // file name
        sFileName = ActiveDate.Year.ToString() + "_";

        // -- set working variables
        int iYear = ActiveDate.Year;
        int iMonth = ActiveDate.Month;
        int iDay = ActiveDate.Day;

        if (iMonth < 10) sFileName += "0" + iMonth.ToString() + "_";
        else sFileName += iMonth.ToString() + "_";

        if (iDay < 10) sFileName += "0" + iDay.ToString() + "_";
        else sFileName += iDay.ToString() + "_";

        // -- set file name
        IXMLDOMNode xmlNodeEmployee = thisXDocument.DOM.selectSingleNode(
            "/sr:Team/sr:EmployeeReport");
        if (xmlNodeEmployee != null)
        {
            sFileName += xmlNodeEmployee.selectSingleNode(
                "//sr:EmailAddress").text;
            sFileName += ".xml";
            xmlNodeEmployee.attributes.getNamedItem(
                "my:Filename").text = sFileName;
        }
    }
    catch (Exception ex)
    {
        thisXDocument.UI.Alert(ex.Message + " Inner: " +
            (ex.InnerException != null ? ex.InnerException.Message : "");
    }
}

If you compile the form and preview it, you'll get a screen that looks like Figure 6.

Figure 6 Status Report Integrated with SharePoint Data (Close Figure 6)

InfoPath Merge Reporting

InfoPath works well with custom merge forms and rollups. Up to this point I have focused on the individual status reporting which is useful and easy to open, fill out, and submit. However, viewing these forms individually is inefficient when I want to see what the 27 developers on my team are working on or when I want to see trends and other views of the data.

Rollup reporting encapsulates the schema associated with the Individual Reports under the EmployeeReport node, which is configured as a Group and set to be a repeating node, as shown in Figure 7.

Figure 7 Rollup Reporting (Close Figure 7)

The final design incorporates all of the additional fields that you need for capturing attributes such as number of days in the working week, number of employee working days (the sum of the employee working days times the number of working days in the week), number of people on vacation, and so on. These attributes provide the metrics and the charts that graph out the resource allocation by projects.

Form Design Elements

The first action taken when rolling up the form is the selection of the Department (the radio button list in the upper-right corner of Figure 7). This selection is the filter value that is used by the code to determine rollups and aggregations for the specific team.

The next interesting section is the summary area towards the top of the report. I can bring up the metrics for number of workdays and total workdays, and then highlight for the managers the Risks, which indicate any actions or support the people on the team may need to complete their work.

InfoPath provides an OnAfterImport event method that you can customize for handling actions after the merge of the forms has occurred. This is where all the power of the implementation comes in. For example, the merge form bubbles up the Risks in each individual report by calling into the node collection and appending it to the Risks node under the main Team node for the form. The code in Figure 8 is part of a larger loop that works through each individual report. It acts upon the data and sets flags, looks up the active bugs in the source defect system, checks the activity for the developer in the source configuration system, and so on.

Figure 8 Bubbling Up Risks (Close Figure 8)
// -- append in the summary of risks
foreach (IXMLDOMNode xmlNodeEmpRisk in xmlNodeEmp.selectNodes(
    "//sr:EmployeeReport[sr:Employee/sr:EmailAddress=\"" +
    sEmailName + "\"]/sr:Risks/sr:Item"))
{
    if (xmlNodeEmpRisk!= null)
    {
        string sRiskText = xmlNodeEmpRisk.text.Trim();
        if (sRiskText != string.Empty)
        {
            string sNSUri = thisXDocument.DOM.selectSingleNode(
                "/sr:Team/my:Risks").namespaceURI;
            IXMLDOMNode xmlNode = thisXDocument.DOM.createNode(
                1, "my:RiskItem", sNSUri);
            xmlNode.text = sName + ": [" + sRiskText + "]";
            thisXDocument.DOM.selectSingleNode("/sr:Team/my:Risks")
                .appendChild(xmlNode);
        }
    }
}

Charting

Next, let's take a look at the capabilities I built into the merge application. Fill out reports for each member of your team and submit it to the SharePoint site. Then open the merge application and aggregate the four reports. When completed, you'll get the presentation of the risks.

Take a look at some of the charts you can generate from this aggregation. You can use the Office Web Components charting ActiveX® control within the merge application.

Actually implementing the charts is a little more difficult than you might think, so I'll present some of the tricks I used during the development phase. The first step is to add the Office Charting Control to your toolbox. From the Control View pane in the InfoPath designer, click the option at the bottom to "Add or Remove Custom Controls..." Then click the Add button on the dialog and select the "Microsoft Office Chart 11.0" control from the list. The next screen will ask if you want to include the *.cab file for the ActiveX controls download. Since I expect the controls to already be installed in our enterprise, I pass on this option.

The next dialog asks for the property of the ActiveX control that you want to bind to your element. There is only one property that you can bind to without custom wrapping the control, so you must choose the XMLData property.

Bypass the next screen, which asks you to enable or disable properties. Then, the next option in the wizard asks you to select the type of element to bind to the XMLData property of the control; select Text. Then select Finish and close the Add Control dialog box. The control pane will now include a custom control at the bottom that you can drop onto your form.

When you drop it onto the form, it will prompt you to select a field from your form's schema to bind to. I have created text elements within my application specifically dedicated to the XML associated with the view that I want the chart to show. I recommend you create another view and work on that surface to design and test your chart.

In my example "Employee By Initiatives" chart, I used an element of the team schema called XMLInitiativeChartDataStackedByEmployee. This element is populated during the OnAfterImport event, which fires after the merge of the individual reports I am rolling up into the full report.

The process works like so:

  1. The data is merged and the calculation of working days and initiative assignments are completed.
  2. The code then fires a method to extract the project resources nodes for each person. It creates a new XML document that simplifies the nodes and creates the proper hierarchy.
  3. An XSLT is then applied that transforms the data into the schema that is expected by the Office Charting Control.

That all sounds fine, except that the schema associated with the XMLData property for the control is not documented! In order to test and model the data, I created text fields that exposed the XMLInitiative ChartDataStackedByEmployee element on the form, copied the XML, and worked through the data transforms using a C# Windows Form application to model and test. (This tool is included in the code download accompanying this column, which is available from the MSDN®Magazine Web site). The transforms are then loaded into InfoPath as resource files and accessed through the DOM of the application.

Figure 9 shows an example of the code called to do the transform for the chart. Keep in mind the code is minimal compared to the effort spent in modeling and designing the XSLT files.

Figure 9 Transform Data to be Shown in a Chart (Close Figure 9)
private void ShowChartInitiativeStackedByIndividual()
{
    // -- load the XML data file.
    try
    {
        // -- get the xslt xml
        XmlDocument xsltInitiativeChart = new XmlDocument();
        xsltInitiativeChart.LoadXml(thisXDocument.DataObjects[
            "ChartInitiativeStackedByIndividual"].DOM.xml);

        // -- create the XslTransform and load the stylesheet.
        XslTransform xslt = new XslTransform();
        xslt.Load(xsltInitiativeChart.CreateNavigator(), null, null);

        // -- get the doc loaded
        XmlDocument xmlXMLXFormProjectData = new XmlDocument();
        xmlXMLXFormProjectData.LoadXml(thisXDocument.DOM
            .selectSingleNode("/sr:Team/my:XMLXFormProjectData").text);

        // -- Transform the file.
        MemoryStream xmlOut = new MemoryStream();
        xslt.Transform(xmlXMLXFormProjectData, null, xmlOut, null);
        xmlOut.Position = 0;
        using(StreamReader sr = new StreamReader(xmlOut))
        {
            thisXDocument.DOM.selectSingleNode(
                "/sr:Team/my:XMLInitiativeChartDataStackedByEmployee")
                .text = sr.ReadToEnd();
        }
    }
    catch (XsltException ex)
    {
        thisXDocument.UI.Alert(ex.InnerException != null ?
            ex.InnerException.Message : ex.Message);
    }
    catch (Exception ex)
    {
        thisXDocument.UI.Alert(ex.Message);
    }
}

This view lets you examine the resource allocation by developer for each of your Initiatives within the organization by displaying a bar graph. Using the toolbar on the charting control, you can transpose the data to look at it from another perspective based on developers allocated to each project (see Figure 10).

Figure 10 Resource Allocation by Developer (Close Figure 10)

Conclusion

InfoPath is a rapid application tool with an incredibly rich design surface and powerful development capabilities. The application I built here shows two sides of InfoPath development (the forms capabilities and the custom merge) and shows the range of possibilities that each approach provides.

I did not cover the integration of the Web services and SQL Reporting Services as that would considerably increase the scope of the column, but the design and benefits should be easy to grasp. The advantage of adding that functionality to the solution is that you get a back end for cross-department reporting integration.

Send your questions and comments to insdmsdn@microsoft.com.

Larry W. Jordan Jr. manages the MSDN/TechNet development team. He has been a speaker at many Microsoft conferences over the years in areas pertaining to Microsoft.com architectures and services.


Security Briefs: Access Control List Editing in .NET

Security Briefs
Access Control List Editing in .NET
Keith Brown

Code download available at: SecurityBriefs0503.exe (171KB)
Contents
Exploring ISecurityInformation
GetAccessRights and Friends
Runtime Modeling
The Managed Adapter
Memory Management
Conclusion

Access control lists (ACLs) can be complex beasts, and user interfaces for editing them are incredibly tricky to implement properly. That's why I was really excited when Windows® 2000 shipped with a programmable ACL editor, shown in Figure 1. Technically this is more than an ACL editor; it also allows you to edit the owner of an object and both the discretionary and system access control lists (DACLs and SACLs, respectively). But I'll call it "ACL editor" here rather than the more technically correct "Security Descriptor Editor."

Figure 1 Permissions Editor Property Sheet
Figure 1 Permissions Editor Property Sheet

If you have been reading Security Briefs since its inception in 1999, you may remember a column I wrote in May of that year that demonstrated the ACL editor. Until now, you had to be a C++ programmer to use this feature in your programs, but I've recently completed a cool little adapter that allows you to use the ACL editor from C#, Visual Basic® .NET, or any other managed language.

There are a number of reasons why you might want to do this. If you're building a tool for deploying software, you often need to set ACLs on files. And it's often impossible to know ahead of time what those ACLs should look like: that's something only the system administrator will know when she's deploying your software. An ACL editor is a great way to gather that information for use in a later automated deployment.

Another reason is that you might want to view and edit ACLs for objects that don't yet have an ACL editor built into Windows. The sample application that comes with this column solves this problem for services. And finally, a few of you might use private ACLs for your own objects. In this case, an ACL editor is one of the key features in your application!

In order to add the ACL editor to your program, you must implement an unmanaged interface called ISecurityInformation. This is not a .NET-friendly interface by any means. In fact, it's not even a COM-friendly interface, as I'll discuss later. But using my adapter, you can implement a managed version of this interface. With this adapter, you can add basic ACL editing to your program in approximately a half an hour.

Before you start, however, you need to know how ISecurityInformation works and how to implement its five or six core methods because the managed version you'll implement with my adapter is a direct mapping of the corresponding methods. The types have been made friendlier to the managed world, but the concepts are all the same. So let's first look at ISecurityInformation in detail.

Exploring ISecurityInformation

The model behind the ACL editor GUI is really quite straightforward. Your job is to implement an interface that describes what it is the GUI is supposed to be representing, including strings that describe what each bit in a permission mask means, as well as the features you want the editor to expose. You also must respond to queries about the current state of the security descriptor such as who the owner is and what the DACL looks like, as well as handle updates submitted through the GUI.

The code in Figure 2 shows the seven methods that make up ISecurityInformation interface. Let's walk through a typical scenario so you can see when the various methods are called and learn how to respond to each request.

Figure 2 ISecurityInformation (Close Figure 2)
interface ISecurityInformation : IUnknown {
  void GetObjectInformation(SI_OBJECT_INFO* pObjectInfo);
  void GetSecurity(SECURITY_INFORMATION si, void** ppsd,
      BOOL wantDefault);
  void SetSecurity(SECURITY_INFORMATION si, void* psd);
  void GetAccessRights(const GUID* type, DWORD flags,
      SI_ACCESS** access, ULONG* count, ULONG* defaultIndex);
  void MapGeneric(const GUID* type, UCHAR* aceFlags,
      ACCESS_MASK* mask);
  void GetInheritTypes(SI_INHERIT_TYPE** inheritTypes,
      ULONG* count);
  void PropertySheetPageCallback(HWND hwnd, UINT uMsg,
      SI_PAGE_TYPE type);
};

The first method the ACL editor calls is GetObjectInformation. This method allows you to specify a whole host of options about how you want the editor to work by filling out a data structure called SI_OBJECT_INFO:

typedef struct _SI_OBJECT_INFO {
  DWORD dwFlags;
  HINSTANCE hInstance;
  LPWSTR pszServerName;
  LPWSTR pszObjectName;
  LPWSTR pszPageTitle;
  GUID guidObjectType;
} SI_OBJECT_INFO, *PSI_OBJECT_INFO;

Note that I'm showing the Win32® version of this data structure instead of the managed type I wrote for use with the adapter. I'll stick with this convention as it'll help you find your way around the official documentation. But the mappings to my managed types are obvious, so you shouldn't have any problem using them once you've finished reading this column.

The first member of this structure is the most important, the flags. There are a dozen or so flags defined, and I'll walk you through some of the more commonly used ones.

SI_EDIT_PERMS controls whether the object's DACL should be viewed and edited. You cannot turn off this flag (its value is zero), since this is the core of the editor's functionality.

SI_EDIT_OWNER controls whether the object's owner should be viewed and edited. If you specify this, the Advanced dialog will include an Owner tab for modifying the owner of the object.

SI_EDIT_AUDITS controls whether the object's SACL should be viewed and edited. This adds an Auditing tab. Be careful about implementing this one—you should check to see if the user has SeSecurityPrivilege before specifying this flag, otherwise when the user clicks the tab and the ACL editor asks you to look up the object's SACL, you'll have to throw an exception, since without this privilege, you're not allowed to even read the SACL! It's better not to have this tab show up at all if the user doesn't have this privilege. My adapter handles this for you automatically by removing this flag if the user doesn't have the correct privilege.

Both the owner and audit flags only have meaning if you also specify SI_ADVANCED, which allows the user to get to the Advanced security dialog in the first place.

SI_CONTAINER indicates that the object being edited may have children that inherit permissions from this object. A file system folder or registry key is an example of a container, while a file is not. This flag helps the editor determine whether to show or hide features related to inheritance.

SI_READONLY is very useful. It tells the editor to allow viewing of the various parts of the security descriptor you've specified, but doesn't allow editing. The buttons for making changes will all be disabled if you set this flag.

SI_RESET displays a Default button that allows the user to reset the security settings to a default state that you can choose. This is what the second argument to GetSecurity is for; it tells you whether the ACL editor is requesting the actual security information for the object or the default settings that should be used when the Reset button is pressed. If you don't use this flag, you can ignore that parameter to GetSecurity.

SI_PAGE_TITLE allows you to control the text that shows up in the title bar via the pszPageTitle member of SI_OBJECT_INFO.

The pszObjectName argument should be set to a short name of the object, as it will be used by the editor to build the title bar text for many dialogs.

The pszServerName argument is used to determine the machine that will convert Security Identifiers (SIDs) to names and perform other security functions. This argument should be set to the machine name where the resource you're editing actually lives to enable any local user accounts and groups to be processed correctly. Typically this will be the local machine, in which case you can set this argument to NULL.

Unless you're editing Active Directory® ACLs, you'll not likely need to specify the type GUID, which only applies to directory style access control lists.

GetObjectInformation may be called several times during a session with the ACL editor. In a recent test I ran on Windows XP, this method was called three times before the GUI even popped up on screen for the first time. This can cause problems if you're not ready for it, so be consistent with the answers you give from this function (don't change your mind halfway through about what flags you're using), and be careful with memory management if you ever implement this unmanaged interface yourself. If you use the adapter presented here, I've taken care of the memory management issues for you. But more on that later.

GetAccessRights and Friends

The next method that is typically called is GetAccessRights, which is where you specify the permission masks that the object supports, along with short descriptions that the GUI will display for you. Figure 3 shows how these strings are displayed in the advanced permissions dialog. You communicate this information to the ACL editor via an array of SI_ACCESS data structures:

typedef struct _SI_ACCESS {
  const GUID* pguid;
  ACCESS_MASK mask;
  LPCWSTR pszName;
  DWORD dwFlags;
} SI_ACCESS, *PSI_ACCESS;

Figure 3 Advanced Permissions Dialog
Figure 3 Advanced Permissions Dialog

Once again, unless you're working with Active Directory ACLs, you won't need to specify a GUID here. The mask variable holds a 32-bit value with one bit set, indicating the permission being described. In a service ACL for example, 0?10 represents the permission to start the service, while 0?20 represents the permission to stop it. Along with each permission, you specify a set of flags detailing where that permission should appear, and how (if at all) it interacts with ACL inheritance schemes. I don't have room to cover all the inheritance options here, but you can read more about ACL inheritance in my online book at What is ACL Inheritance.

There are a couple of flags that you'll always need to consider, as they control where the permission shows up in the editor. You see, the ACL editor displays a concise, summarized view of an object's DACL on its main page. This summary doesn't show all the details, and many permissions are collapsed into more easily understandable categories. For example, compare Figure 1, the summary view, with Figure 3, the advanced view. Note how the advanced view details each permission explicitly, while the summary view groups whole sets of permissions together.

For a service, for example, the summary view permission called Execute is actually composed of several more detailed permissions: Start, Stop, Pause or Continue, and so forth. In the case of these summary permissions, more than one bit is set in the mask—one bit per permission that makes up the summary. For example, the mask for Execute permissions is 0x00020170, the combination of five more detailed permissions.

To specify that a permission should show up on the summary page, use the SI_ACCESS_GENERAL flag. For the details page, use the SI_ACCESS_SPECIFIC flag. On rare occasions you'll want an entry to show up in both places: the classic example is the Full Control entry. In this case you'd specify both of these flags. And just to give you a practical tip for usability, you'll definitely want to include Full Control in both places. It's really helpful in the advanced view, because when you check a box for one of these combined permission entries (like Full Control), the editor will automatically check all the permissions in the mask. If you're not sure what I mean, try to grant or deny full control using the advanced editor (see Figure 3) without having a Full Control option. You'll develop a callous on your finger clicking all those little boxes!

When you are implementing GetAccessRights, you also specify the entry in the list that represents the default permission mask. Generally you'll want to set this to the general Read permission entry. This comes into play when a user adds a new entry to the DACL. She is asked to select a user or group for the entry (in other words, to whom is she granting permissions?), and that new entry shows up in the list with whatever default permissions you specified.

The code in Figure 4 shows an example implementation, which is part of an application that allows you to edit service ACLs, so all permissions that apply to services are shown. These include Start, Stop, Change Configuration, and so on. This implementation sits behind my adapter, so it uses managed types, but the ideas are exactly the same.

Figure 4 Implementing GetAccessRights (Close Figure 4)
public AccessRights GetAccessRights(Guid objectType,
    MsdnMag.Security.Adapters.ObjectInfoFlags flags)
{
  AccessRights rights = new AccessRights();
  Access[] access = rights.Access = new Access[17];

  // these aliases should make the code more readable
  // for the printed edition (we don't get much width!)
  AccessFlags G = AccessFlags.General;
  AccessFlags S = AccessFlags.Specific;

  // summary page permissions
  access[0] = new Access(SERVICE_ALL, "Full Control", G | S);
  access[1] = new Access(SERVICE_READ, "Read", G);
  access[2] = new Access(SERVICE_WRITE, "Write", G);
  access[3] = new Access(SERVICE_EXECUTE, "Execute", G);

  // advanced page permissions
  access[4] = new Access(0x0001, "Query Configuration", S);
  access[5] = new Access(0x0002, "Change Configuration", S);
  access[6] = new Access(0x0004, "Query Status", S);
  access[7] = new Access(0x0008, "Enumerate Dependents", S);
  access[8] = new Access(0x0010, "Start", S);
  access[9] = new Access(0x0020, "Stop", S);
  access[10] = new Access(0x0040, "Pause or Continue", S);
  access[11] = new Access(0x0080, "Interrogate", S);
  access[12] = new Access(0x0100, "Send User Defined Control", S);
  access[13] = new Access(0x00010000, "Delete", S);
  access[14] = new Access(0x00020000, "Read Permissions", S);
  access[15] = new Access(0x00040000, "Change Permissions", S);
  access[16] = new Access(0x00080000, "Take Ownership", S);

  // note how I refer to access[1] as the default ("Read")
  rights.DefaultIndex = 1;

  return rights;
}

Between GetObjectInformation and GetAccessRights, you specify virtually all of the details the ACL editor needs to configure its display. For container objects, there's a third method that the ACL editor will invoke if the user drills down into the advanced permission editor. This method is called GetInheritTypes, and it's where you provide human-readable descriptions for the various inheritance options. This is the disabled listbox titled "Apply onto" that is shown in Figure 3. You'll see this in the chapter on ACL inheritance that I mentioned earlier.

Runtime Modeling

The rest of the functions you're going to see have to do with managing the actual security descriptor being edited or viewed. Let's first look at GetSecurity (refer to Figure 2). The ACL editor calls this method whenever it needs to know the owner, DACL, or SACL for the object being edited. For example, right before the ACL editor appears, it will call GetSecurity to ask for the DACL, for its initial summary display. If the user presses the Advanced button (assuming you've allowed the editor to show it), GetSecurity will be called again, as the DACL is being displayed in another form. When the user clicks the Owner tab, GetSecurity will be called yet again to get the owner. The same thing happens on the Auditing tab, so expect this function to be called several times during any editing session.

If the user makes a change and commits it by pressing Apply or OK, the ACL editor gives you the new data by calling SetSecurity. It's your job to provide a security descriptor for editing. In many of these cases you can simply forward these requests to corresponding Win32 functions such as GetNamedSecurityInfo and SetNamedSecurityInfo, as I do in my service example. My GetSecurity implementation literally asks the operating system for the requested information and passes it back to the ACL editor, and my SetSecurity implementation writes the changes directly to the service.

If you're editing a private security descriptor for a custom object of your own design, a rather advanced technique discussed in Chapter 6 of my first book, Programming Windows Security (Addison-Wesley Professional, 2000), you'll need to read and write this data via whatever store you happen to be using in your application. I've seen private security descriptors stored in database columns, binary registry keys, or encoded as Security Descriptor Description Language (SDDL) in XML files.

At run time you'll likely see another rather esoteric function being called several times: MapGeneric. This function allows the ACL editor to ask you what the four generic permissions mean for the class of object being edited. GENERIC_READ represents permissions you'd grant if you were giving read-only access to your object. GENERIC_WRITE are the additional permissions you'd grant if somebody needed basic read/write access. GENERIC_EXECUTE applies to objects that have execution semantics (processes, services, and so on) and represents whatever extra permissions you'd grant if you wanted to allow someone to execute or control the execution of the object. GENERIC_ALL is just what it sounds like—all possible permissions for the object.

These generic permissions are often granted by default to objects. For example, whenever you create a new process, SYSTEM is granted GENERIC_ALL permission to that process by default. If you want further information on generic permissions, here's another link to my online book: What Is A Permission.

The simplest way to implement MapGeneric is to fill out a data structure (called, aptly enough, GENERIC_MAPPING) that maps from the four generic permissions onto four corresponding permission masks. Then you can implement the MapGeneric method by calling the Win32 function MapGenericMask. The sample code that accompanies this column shows how to accomplish this—it's quite easy. The hard part is figuring out what the mapping is. For objects that are built into the operating system, like services, you can find the generic permission mappings in the Platform SDK documentation (which you should have if Visual Studio® .NET is installed on your machine).

For example, to find the service permissions along with their generic counterparts, search for "service GENERIC_READ" while filtering on "Platform SDK". For the actual values of each permission, you'll need to look at the corresponding Win32 header file (in this case, winsvc.h). The header file winnt.h contains many of the kernel permission definitions, along with some of the generic mappings. This is more of an art than a science right now, but believe it or not, it's gotten a lot better than it was a few years ago when you had to guess what the generic mappings for most objects were.

There's just one last function on ISecurityInformation that I have not yet covered: PropertySheetPageCallback. This function is useful if you're incorporating the property sheet into a larger dialog box and want to customize its behavior. This isn't currently possible in a Windows Forms application though, so I won't spend any time covering this method. In fact, my managed adapter just stubs this method out.

The Managed Adapter

When I first tried to use ISecurityInformation from managed code, I immediately realized that it would be a challenge. For one thing, the interface isn't specified in any type library that I can find. And you know, that's probably a really good thing because, besides the use of HRESULT and IUnknown, it really doesn't behave much like a COM interface at all. If you're one of the folks who remembers hacking up COM implementations from scratch in C++ or even ATL, you'll remember all the careful rules COM had about memory management. When someone calls your COM interface and asks for a dynamically sized string (or even an array), you allocate the memory and the code that called you frees it. And there are special functions that COM provides for doing this, although I won't drag you back to yesteryear by listing their names.

Well, ISecurityInformation breaks these rules all over the place in the name of efficiency. For example, there are all kinds of strings that you create and give to the ACL editor when it calls GetAccessRights. The ACL editor never frees these strings because ISecurityInformation was designed with the assumption that a C++ programmer would be using it and could simply pass back pointers to static strings that never needed freeing. Sadly these sorts of practices, since they violate the fundamental rules of COM, mean that COM interop simply doesn't work with ISecurityInformation. At least not without a lot of intervention on your part.

So, as I find myself doing a lot these days with version 1.1 of the .NET Framework, I dropped down into managed C++ and wrote an adapter. The idea behind this was simple: I created some easy-to-use managed types that map as directly as possible onto the types used by ISecurityInformation. I then created a new interface called ISecurityInformationManaged that uses these new types, and implemented a custom COM-callable wrapper (CCW) that does the translation.

The following code shows the new, managed interface:

 public interface ISecurityInformationManaged {
  ObjectInfo GetObjectInformation();
  AccessRights GetAccessRights(Guid type, ObjectInfoFlags flags);
  byte[] GetSecurity(SecurityInformation info, bool wantDefault);
  void SetSecurity(SecurityInformation info, byte[] sd);
  void MapGeneric(GenericAccess generic);
  InheritType[] GetInheritTypes();
};
This is the interface you will implement if you use the adapter. I think you'll find it remarkably easy to use. The only type that I didn't bother translating into managed code is the SECURITY_DESCRIPTOR, which is a very complicated beast. Fortunately, you can usually live a rich, happy life just treating this as an opaque blob, so I modeled it with a byte array. When the .NET Framework 2.0 gets a bit closer to shipping, I'd be happy to tweak this adapter to use the new System.Security.AccessControl namespace that has a very nice model of SIDs and security descriptors.

Memory Management

Writing this adapter turned out to be a somewhat complicated exercise in memory management. Because the ACL editor doesn't free most of the strings and data structures given to it, the adapter's main job is tracking dynamically allocated native memory so it can be freed after the ACL editor closes. It would be great if I could simply hand direct pointers to managed strings to the editor, but that would be a disaster as managed objects move around in the heap due to compaction during garbage collection. Each string that you give the adapter via ISecurityInformationManaged must first be marshaled (copied) into a native string, and tracked until it's no longer needed; that is, when the ACL editor closes and releases the adapter.

My current implementation is very simple. Each string that I marshal into native code is simply pushed onto a linked list (I use an STL vector at this time), so that they can be freed when the adapter is released. Seems reasonable, don't you think?

Well, after some testing, I found that this technique does not use space very efficiently, and actually leaks memory while the editor is in use. Remember how I said that the editor calls GetObjectInformation and GetAccessRights many times during a session? I'm specifying the exact same access permission descriptions each time, but I'm marshaling them again and again, and my list of strings gets longer and longer the more times these functions are called. In practice this probably won't be a problem, since the editor won't be used long enough to drain the system of enough memory to be noticeable. The adapter frees this memory when it's released as the ACL editor closes. When I get some spare cycles, I'll probably switch this over to use a map instead of a vector so that I only store any given string once.

Conclusion

As I mentioned earlier, this was mainly an exercise in memory management, and the code download that accompanies this column contains all of the source files. But this adapter should help you if you're using Visual Basic .NET or C# and were otherwise unable to add ACL editing capability to your projects.

I'd appreciate any feedback, bug reports, improvements, and so on. I will keep the most recent version of this tool available to you from Pluralsight's security sample gallery.

Send your questions or comments for Keith at briefs@microsoft.com.

Keith Brown is a co-founder of Pluralsight, specializing in developing and delivering high quality training for software developers. Keith's most recent book, The .NET Developer's Guide to Windows Security. Contact him at www.pluralsight.com/blogs/keith.


Bugslayer: Mini Dump Snapshots and the New SOS

Bugslayer
Mini Dump Snapshots and the New SOS
John Robbins

Code download available at: Bugslayer0503.exe (126KB)
Contents
Mini Dump Theory and Tools
ADPlus Usage
What's New in SOS
Wrap-Up
Tips

In debugging some large Microsoft® .NET Framework-based ap-plications over the last few months, I've been spending more time looking at mini dumps than at live processes. This is mainly because in those large applications problems surface when the apps are running in production and not on test systems. Because these mini dumps are so important, I've been surprised that many developers don't know how to create them.

In this Bugslayer installment, I will discuss the steps and tools necessary to snap mini dumps for .NET-based applications so you can debug them with the wonderful new Son of Strike (SOS) WinDBG extension DLL. With mini dumps of your .NET processes, you'll finally be able to stand a fighting chance at debugging those production-only problems.

I'll start out by discussing very briefly the type of mini dumps you'll need along with the appropriate tools. For most of the column, I'll show you the exact settings I use to get the proper mini dumps and some techniques for production mini dump snapping. Later I'll discuss some of the new features in the latest SOS Microsoft has released. I previously wrote about SOS (Bugslayer: SOS: It's Not Just an ABBA Song Anymore), but some of the new features can make tracking down tough .NET memory problems a breeze.

Mini Dump Theory and Tools

One thing that you may find confusing about mini dump snapshots is that their file size can be quite large. That's because SOS needs all the virtual memory pages marked as committed in order to do any of its magic. For a simple "Hello World" Windows® Forms application, the resulting mini dump is approximately 30MB. With ASP.NET applications, you're going to see mini dump sizes that range from 300MB to well over 1GB. While you can ask the end user to e-mail you the mini dumps, that will probably make their e-mail administrators and yours want to take mini dumps of both of you as well. At the very least, if you are going to transfer mini dump files around, compressing them with a tool like WinZip can reduce the file size by around 75 percent. Note that with the .NET Framework 2.0, you'll have the option of taking a smaller mini dump (on the order of a few hundred KB) which will allow you to use a subset of SOS commands, such as obtaining a stack trace. Some commands (like the ones related to garbage collection) will still require all committed pages, however.

For your production .NET-based applications, the best way to create mini dumps is with the old standby, ADPlus, formerly named Autodump+. The Debugging Tools for Windows package, freely downloadable at Debugging Tools for Windows, is the home for the latest version. For this column, I'll be using the October 2004 beta release 6.4.4.4 of the Debugging Tools for Windows. Every few months, there's a new release, so you should check back frequently to get the latest. (As an aside, if you are actively debugging with WinDBG or CDB/NTSD, you'll use the ".dump /ma <file>" command so that you can write an appropriate mini dump.)

ADPlus is actually a 5,000-line VBScript application that drives CDB, the console-based debugging UI on top of DBGENG.DLL. (By the way, CDB, NTSD, and WinDBG, are all different UIs on top of DBGENG.DLL so any commands I mention using one of the debuggers will work in all of the debuggers.) When you execute ADPlus, it's building up a script of debugger commands and pumping them into CDB, which does all the heavy lifting. For those of you who used ADPlus in the past, it was relatively nasty because you had to change values inside the actual VBScript file itself. Now ADPlus uses configuration files, in the ubiquitous XML format, to handle all the settings. That makes it much easier to use.

One of the big problems in production environments is that administrators get very nervous when you say you are going to install something on the server. The great news with ADPlus and its supporting files is that a simple XCOPY deployment works great. Just install the Debugging Tools for Windows on your development machine, as you'll need WinDBG to analyze the mini dumps, and then copy the installation directory to the server.

Since ADPlus uses VBScript, you'll need to have Windows Script Host (WSH) version 5.6 or later on the target machine. Run "CSCRIPT /?" at an MS-DOS® prompt to check the version installed. You can download the latest WSH at Microsoft Windows Script Downloads. Another item you'll want to change is the default script host. You should use CSCRIPT.EXE instead of WSCRIPT.EXE so that output will go to Command windows instead of to message boxes, which can hang ADPlus until you click OK. Run "CSCRIPT /H:CScript" and you'll be all set.

In ADPlus, and the Debugging Tools for Windows overall, the documentation in DEBUGGER.CHM is outstanding. You definitely need to read through everything in there to get a good handle on how to run ADPlus as well as how to use the debugging tools. Make sure to concentrate on the Debugging Tools for Windows\Extra Tools\ADPlus in and Debugging Tools for Windows\Debugging Techniques. With the background out of the way, let's turn to actually using ADPlus.

ADPlus Usage

ADPlus has two usage modes: hang and crash. Hang mode is a bit of a misnomer; it really should be called "snap" because ADPlus instructs CDB to attach noninvasively to a process or processes, write out a mini dump, and detach. A noninvasive attach suspends the process and does not actually run under a debug loop so it works on all operating systems from Windows NT® 4.0 and above. If you're familiar with the old USERDUMP.EXE program, ADPlus is the complete replacement for it.

Crash mode, on the other hand, actually attaches the CDB debugger with a native debug loop so you can have the debugger respond to exceptions and even set breakpoints. For example, if you have a case where the ASP.NET worker process is shutting down unexpectedly, you can attach ADPlus in crash mode and have it set a breakpoint on KERNEL32.DLL's ExitProcess. When that breakpoint trips, you can write out a mini dump so you can walk the stack to see what started the process termination.

For most of your debugging, you'll be using ADPlus in hang mode so you can capture dumps at various intervals to see what's going on. While you can specify command-line options to ADPlus to handle all the work, I much prefer to use configuration files. That way I can get even more control over ADPlus and more information into the output.

Figure 1 shows the hang mode configuration file I like to use with .NET-based applications. The comments in the file indicate exactly what each option does and what commands I have selected to run. If you've ever played with WinDBG, you'll see that all the commands are familiar.

Figure 1 Hang Mode Configuration File (Close Figure 1)
<!-- John Robbins - Bugslayer Column, MSDN Magazine   -->
<!-- Default HANG Options                             -->
<ADPlus>
    <Settings>
        <!-- Set the mode to HANG                     -->
        <RunMode>
                 HANG
        </RunMode>

        <!-- Snap the dumps, don't tell me about it   -->
        <Option>
                 Quiet
        </Option>

    </Settings>

    <HangActions>

        <!-- Clear out the default options            -->
        <Option>
                 Clear
        </Option>

        <!-- I like to have a dump (.dump /ma),       -->
        <!-- callstacks (~*kb250), and loaded modules -->
        <!-- (lmv) in the log every time              -->
        <Actions>
                 FullDump;
                 Stacks;
                 LoadedModules
        </Actions>

        <!-- For custom actions, I want to see all    -->
        <!-- the handle info, the managed CLR version, -->
        <!-- all managed threads, all managed         -->
        <!-- call stacks, and the heap stats for      -->
        <!-- objects/sums larger than 100 bytes.      -->
        <!-- This assumes Debugging Tools for Windows -->
        <!-- 6.4.4.4 or later.                        -->
        <CustomActions>
                 !handle 0 f;
                 .load clr10\sos;
                 !eeversion;
                 !threads;
                 ~*e!clrstack;
                 !dh -stat -min 100
        </CustomActions>

    </HangActions>

</ADPlus>

To specify the configuration file on the ADPlus command line, use the –c option. Keep in mind that you'll need to specify the complete path to the configuration file if it is in any directory other than the same one containing ADPLUS.VBS. In addition to –c, you'll also need to specify the –o and –pn (or –p).

The output directory, set with the –o, tells ADPlus where to put the output. Each time ADPlus runs, it will create a directory in the output location with the format "Hang_Mode__<date>__Time__ <time>." In that directory, it will create the dump file, the log of the entire CDB output, and a file containing the running processes on the machine. Because ADPlus ensures a unique file name every time, you can run ADPlus repeatedly with the same command-line option and not lose previous runs.

The –pn option specifies the process name you want to attach and dump. For example, if you are using IIS 5.0, you'd specify "-p ASPNET_WP.EXE" to grab your ASP.NET dumps. For IIS 6.0, you'd specify "-p W3WP.EXE". By specifying an actual process name, if there are multiple copies of that process running, ADPlus will dump all of those processes. If you want to identify only a single process, you can do so by using the –p option, which will look for the process ID.

In the case of IIS 6.0, if you want to only snap a mini dump of a particular application pool, the command "TLIST.EXE –v" will show you all running processes and their command lines. (TLIST.EXE comes as part of the Debugging Tools for Windows.) Look through the list for the different W3WP.EXE instances and their –ap command-line options to identify the application pool in which each instance is running.

The majority of my mini dump creation has been performed with the hang mode option to ADPlus. The one problem is that you have to keep manually running the same command line repeatedly to get the runs over a period of time. I have included in this month's source code distribution a program called SpawnRepeatedly that I've used to automate the task of running ADPlus over longer periods of time. You can download it from the MSDN®Magazine Web site.

There's nothing too exciting about the program itself, but I've found it very useful to run on production systems and to snap a mini dump every couple of hours. Keep in mind that the ASP.NET worker process, or any process you attach ADPlus to in hang mode, suspends the process and could possibly cause problems. In most cases, the mini dump snap should take less than a minute, but for some servers, even that could cause problems so you may want to perform some test mini dumps first to gauge the impact.

Note that the configuration file shown in Figure 1 includes actions for some things that you can do later when analyzing the dumps. If you want to minimize the time you stop the application for those occasions where time is an issue, one alternative is to use a very lean configuration file, such as one where the only option used is Clear and where the only action used is FullDump.

The second mode for running ADPlus is directed at telling ADPlus to perform some operations when the process crashes. While in hang mode the debugger is simply suspending the process; in crash mode the debugger attaches using the Win32® native debugging API. Because of the excellent scriptability of the debugging engine, this gives you the opportunity to respond to specific exceptions, conditions, or when certain functions execute.

For example, the configuration file shown in Figure 2 snaps a mini dump whenever a first-chance exception occurs. Since .NET exceptions are implemented internally with Structured Exception Handling (SEH), each time an exception is thrown in your application, you'll have the exact dump that tells you where the exception occurs. I don't run this configuration on production systems because it can slow down the application too much, but it's invaluable in testing. Be aware that this configuration will take up a huge amount of disk space. While .NET makes exceptions much more palatable than they were in Win32, they are truly for exceptional conditions and you should not be seeing them thrown frequently in your normal operation.

Figure 2 Capturing a Mini Dump on Each Throw (Close Figure 2)
<!-- John Robbins - Bugslayer Column, MSDN Magazine   -->
<!-- Default Crash Options                            -->
<ADPlus>
    <Settings>
        <!-- Set the mode to CRASH                    -->
        <RunMode>
                 CRASH
        </RunMode>
        <!-- Snap the dumps, don't tell me about it   -->
        <Option>
                 Quiet
        </Option>
    </Settings>
    <Exceptions>
        <!-- Do a unique mini dump with full heap on every      -->
        <!-- first chance exception.                  -->
        <Config>
            <Code>
                 AllExceptions
            </Code>
            <Actions1>
                 FullDump;
            </Actions1>
            <ReturnAction1>
                 GN
            </ReturnAction1>
        </Config>
    </Exceptions>
</ADPlus>

Another crash mode configuration that's quite useful is telling ADPlus to snap a mini dump when a process calls ExitProcess. With ASP.NET, this configuration can help you determine why the worker process is jumping out of memory, which is probably the sign of some sort of COM Interop problem. The trick is to tell ADPlus to set the breakpoint on KERNEL32.DLL's ExitProcess and to do a mini dump with full heap at that point.

Figure 3 shows the ExitProcess configuration. By default, ADPlus wants to write out a mini dump on each and every first-chance exception (the moment of the throw). Figure 3 shows how to turn first-chance exception logging off.

Figure 3 ExitProcess Configuration (Close Figure 3)
<!-- John Robbins - Bugslayer Column, MSDN Magazine   -->
<!-- Default Crash Options                            -->
<ADPlus>
    <Settings>
        <!-- Set the mode to CRASH                    -->
        <RunMode>
                  CRASH
        </RunMode>
        <!-- Snap the dumps, don't tell me about it   -->
        <Option>
                  Quiet
        </Option>
    </Settings>
    <Exceptions>
        <!-- Don't dump on first-chance exceptions    -->
        <Option>
                  NoDumpOnFirstChance
        </Option>
    </Exceptions>
    <Breakpoints>
        <NewBP>
            <!-- Set the breakpoint on ExitProcess    -->
            <Address>
                     kernel32!ExitProcess
            </Address>
            <!-- A normal breakpoint                  -->
            <Type>
                     BP
            </Type>
            <!-- When hit, walk the stacks and do a   -->
            <!-- mini dump with full heap.                      -->
            <Actions>
                    FullDump;
                    Stacks
            </Actions>
            <!-- After doing the actions, just        -->
            <!-- continue                             -->
            <ReturnAction>
                     G
            </ReturnAction>
        </NewBP>
    </Breakpoints>
</ADPlus>

With dump creation under your belt, I want to show how you can take advantage of some of the new SOS features to get at that hard-to-find information.

What's New in SOS

If you read my previous column on SOS, you probably saw that SOS is good at dumping out information, but you have to spend your time manually grinding across it to find the really useful sections. The latest SOS, which is now part of WinDBG, makes finding key information, such as all the objects in the Generation 2 heap, trivial. Having spent far too many hours grinding through dumped output and manually trying to find what's in what generation, I'm thrilled to have these new capabilities.

The .NET Framework has the old copy of SOS, so to load the new SOS, you'll execute the ".load clr10\sos" command. The new version is targeted at common language runtime (CLR) 1.0 and 1.1 development. If you are playing with CLR 2.0, you'll want to load the SOS version that comes with the Framework.

The first thing you need to look at is the "!help" command because it looks like all the parameters to the information commands are now documented. You will also see that there are new commands specific to ASP.NET such as "!DumpASPNETCache" (also callable with the shortened "!dac") that will dump out all the data that is in the ASP.NET cache. Since the cache is a place where you are probably holding onto objects too long, being able to see the cache in one command is a huge improvement.

The command that's gotten the most attention is the one that's the most useful, "!DumpHeap" (also shortened to "!dh"). Not only is it now able to tell you which generation an object is in, but the most noticeable improvement is that it's many orders of magnitude faster. To see the statistics of all objects in the Generation 2 heap, the magic command is "!dh –stat –gen 2". That will produce comprehensive output like that shown in Figure 4, so you can see at a glance exactly what's sticking around too long. If you want to see the individual objects, remove the –stat argument and "!dh" dumps the actual objects themselves.

Figure 4 Statistics of Objects in the Generation 2 Heap (Close Figure 4)
0:006> !dh -stat -gen 2
total 57,671 objects
Generation 2 Statistics:
    MT      Count TotalSize Class Name
0x79bf6b0c      1      12 System.IO.TextReader/NullTextReader
0x79bce950      1      12 System.Resources.FastResourceComparer
0x79bbba0c      1      12 System.Reflection.Cache.TypeNameCache
0x79bb6614      1      12 System.Security.PolicyManager
0x79bb5da8      1      12 System.Reflection.__Filters
0x79bad050      1      12 System.Security.SecurityRuntime
0x79bacb80      1      12 System.Security.CodeAccessSecurityEngine
0x79ba8a2c      1      12 System.__Filters
0x79ba8568      1      12 System.Reflection.Missing
0x79b9c8d0      1      12 System.IO.Stream/NullStream
0x0014eac8      1      12     Free
0x79c03b00      1      16 System.Security.Permissions.UIPermission
0x79bbbf14      1      16 System.Reflection.Cache.InternalCache
0x79bb9a78      1      16 System.Security.Policy.ZoneMembershipCondition
0x79bb6030      1      16 System.Security.Util.LocalSiteString
0x79bb52e4      1      16 System.Globalization.GlobalizationAssembly
0x79bab880      1      16 System.Enum/HashEntry
0x0099375c      1      16 System.Boolean[]
0x79bb3b74      1      20 System.Security.Util.DirectoryString
0x79b9fc8c      1      20 System.Text.UnicodeEncoding
0x79b9d374      1      20 System.Text.UTF8Encoding
0x79b98514      1      20 System.AppDomainSetup
0x79bce4a0      1      24 System.Globalization.TextInfo
0x79bbb4d8      2      24 System.Security.Permissions.StrongNamePublicKe...
0x79bb917c      1      24 System.LocalDataStoreMgr
0x79bac2ec      1      24 System.Security.PermissionTokenFactory
0x79b90e34      2      24 System.Object
0x79bd80d4      1      28 System.Security.Permissions.FileIOPermission
0x79bce824      1      28 System.Text.UTF8Encoding/UTF8Decoder
0x79bbc094      1      28 System.Reflection.Cache.ClearCacheHandler
0x009931d8      1      28 System.Reflection.Cache.InternalCacheItem[]
0x79bc9514      1      32 System.IO.__UnmanagedMemoryStream
0x79bb3544      1      32 System.Collections.ArrayList/SyncArrayList
0x79b95614      1      32 System.SharedStatics
0x79bef8ec      1      36 System.Security.Policy.UnionCodeGroup
0x79bcdaec      1      36 System.Resources.RuntimeResourceSet
0x79b9fed4      1      36 System.IO.BinaryReader
0x79bcf3f4      1      40 System.Runtime.Serialization.Formatters.Binary...
0x79bb4cdc      2      40 System.Globalization.CompareInfo
0x79bb2fcc      1      40 System.Security.Util.URLString
0x79bccfbc      1      44 System.Resources.ResourceManager
0x79bb55b4      1      44 System.Reflection.Module
0x79bb7800      2      48 System.Version
0x79bac190      3      48 System.Security.PermissionToken
0x79ba29d4      2      48 System.Reflection.Assembly
0x79b9f180      4      48 System.Security.Permissions.SecurityPermission
0x79bfb5a4      1      56 System.IO.StreamReader/NullStreamReader
0x79bb5eac      2      56 System.Reflection.TypeFilter
0x79baa5e0      1      56 System.Collections.Hashtable/SyncHashtable
0x79bb8384      1      60 System.Threading.Thread
0x79ba8cc8      5      60 System.Int32
0x79b94ee4      1      64 System.ExecutionEngineException
0x79b94dac      1      64 System.StackOverflowException
0x79b94c74      1      64 System.OutOfMemoryException
0x79bcd77c      1      68 System.Resources.ResourceReader
0x79b96e8c      1      80 System.AppDomain
0x79ba8b44      3      84 System.Reflection.MemberFilter
0x79baba4c      4     112 System.Security.Util.TokenBasedSet
0x79bb704c      3     132 System.Security.Policy.PolicyLevel
0x79b9e24c      5     140 System.Security.PermissionSet
0x009934a4      1     140 System.UInt64[]
0x79bba934      4     144 System.Security.NamedPermissionSet
0x79bb40e4      4     176 System.Globalization.CultureInfo
0x79bb7a94      2     240 System.Globalization.NumberFormatInfo
0x79bba56c     14     336 System.Security.Policy.StrongNameMembership...
0x009926b0      6     372 System.Int32[]
0x79ba7590     29     464 System.RuntimeType
0x79ba968c     12     624 System.Collections.Hashtable
0x0099236c     18     652 System.Char[]
0x00992970     12   1,872 System.Collections.Hashtable/bucket[]
0x79ba0884    113   3,164 System.Security.SecurityElement
0x79ba0d74    135   3,240 System.Collections.ArrayList
0x0099209c    149   6,072 System.Object[]
0x79ba13f4    413   6,608 System.Security.SecurityStringPair
0x00992f00      1   7,364 System.Reflection.Cache.TypeNameStruct[]
0x00992c3c      9  52,356 System.Byte[]
0x79b925c8  1,325 100,872 System.String
Total 2,333 objects, Total size: 186,952

Large Objects Statistics:
     MT     Count     TotalSize Class Name
Total 0 objects, Total size: 0

One undocumented feature of the "!dh" command is that you can easily look at the large object heap as well. In looking at a lot of output, I noticed some values listed as coming from Generation 3. Because the .NET garbage collector, at least according to everything I read, only has the three generations (0, 1, and 2), I was a little confused. Thinking it may be the large object heap, I manually dumped the large object heap and compared values. Sure enough, that's what I saw. To see the objects in the large object heap, use 3 as the generation like this: "!dh –stat –gen 3".

Another nifty new trick with "!dh" is the ability to specify a partial type string to search for. If you want to see just the statistics for Generation 2 on objects that have Security in the name, the command is: "!dh –stat –gen 2 –type Security". The new type option makes looking for your own objects trivial. If you haven't guessed this by now, you can mix and match command-line options to "!dh" all you want to see the heap in any way you want to slice and dice it. Finally, "!dh" works great on live debugging in addition to all those memory dumps you're about to create.

Following "!dh", the next most-used command, "!DumpObj" (also shortened to "!do"), has gotten a features boost as well. For those of you who have carpel tunnel syndrome from manually walking down object reference chains, the new –r # (recurse) switch will be much appreciated. If you want to look at an object and what it references, specify a number of objects you want to walk down and "!do" will take care of the rest. While it's a great feature, "!do" does not skip null/Nothing values. Instead, it tries to dump those objects to output, so you may see some errors in the minidump. If you see an error "Could not dump object, error returned was: 0x8007012b" just take a look at the previous CLASS value and you'll see that it is probably null/Nothing.

One of the previous biggest weaknesses of SOS was that it was impossible to look at an array. The new –v switch to "!do" now makes it trivial. Figure 5 shows a simple string array output. I've noticed that the –v switch will only dump out actual reference objects and will not display arrays with value elements.

Figure 5 Dumping an Array with SOS (Close Figure 5)
0:000> !do -v 0x00b14500
Name: System.Object[]
MethodTable 0x0098209c
EEClass 0x00982018
Size 28(0x1c) bytes
GC Generation: 0
Array: Rank 1, Type CLASS
Element Type: System.Object
Content: 3 items
--- Will only dump out valid managed objects ---
[ 0x00b14338 ]
   Name: System.String
   MethodTable 0x79b925c8
   EEClass 0x79b92914
   Size 40(0x28) bytes
   GC Generation: 0
   mdToken: 0x0200000f  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll)
   String: Windows XP
   FieldDesc*: 0x79b92978
    MT     Field    Offset          Type      Att  Value  Name
0x79b925c8 0x4000013   0x4  System.Int32  instance    11  m_arrayLength
0x79b925c8 0x4000014   0x8  System.Int32  instance    10  m_stringLength
0x79b925c8 0x4000015   0xc  System.Char   instance  0x57  m_firstChar
0x79b925c8 0x4000016     0     CLASS      shared   static Empty
      >> Domain:Value 0x0014aa70:   0x00a912b8 <<
0x79b925c8 0x4000017   0x4     CLASS      shared   static WhitespaceChars
      >> Domain:Value 0x0014aa70:   0x00a912cc <<

[ 0x00b14360 ]
   Name: System.String
   MethodTable 0x79b925c8
   EEClass 0x79b92914
   Size 40(0x28) bytes
   GC Generation: 0
   mdToken: 0x0200000f  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll)
   String: Server 2003
   FieldDesc*: 0x79b92978
    MT     Field     Offset           Type    Attr   Value  Name
0x79b925c8 0x4000013   0x4    System.Int32  instance    12  m_arrayLength
0x79b925c8 0x4000014   0x8    System.Int32  instance    11  m_stringLength
0x79b925c8 0x4000015   0xc    System.Char   instance   0x53 m_firstChar
0x79b925c8 0x4000016             CLASS      shared   static Empty
      >> Domain:Value 0x0014aa70:   0x00a912b8 <<
0x79b925c8 0x4000017   0x4       CLASS      shared   static WhitespaceChars
      >> Domain:Value 0x0014aa70:   0x00a912cc <<

[ 0x00b14388 ]
   Name: System.String
   MethodTable 0x79b925c8
   EEClass 0x79b92914
   Size 44(0x2c) bytes
   GC Generation: 0
   mdToken: 0x0200000f  (e:\winnt\microsoft.net\framework\v1.1.4322\mscorlib.dll)
   String: Windows 2000
   FieldDesc*: 0x79b92978
    MT     Field     Offset           Type    Attr  Value  Name
0x79b925c8 0x4000013   0x4    System.Int32  instance   13  m_arrayLength
0x79b925c8 0x4000014   0x8    System.Int32  instance   12  m_stringLength
0x79b925c8 0x4000015   0xc     System.Char  instance  0x57 m_firstChar
0x79b925c8 0x4000016     0          CLASS   shared  static Empty
      >> Domain:Value 0x0014aa70:   0x00a912b8 <<
0x79b925c8 0x4000017   0x4          CLASS   shared  static WhitespaceChars
      >> Domain:Value 0x0014aa70:   0x00a912cc <<

In addition to poor array handling, in the past SOS also had problems with collections, unless you had a lot of time to manually dump them with "!do". The new "!DumpCollection" (short version: "!dc") will display any collection that's derived from the ICollection interface, which means nearly everything. In the 6.4.4.4 beta release of WinDBG, the "!dc" command is not working too well, but it will be fixed in the final release.

One extremely tough problem to track down in mixed-mode applications is a GCHandle leak. The System.Runtime.InteropServices.GCHandle structure is what's used to hold onto a piece of managed memory that's passed to native code. For the most part, Interop works like a dream, but there are times when the native code no longer references the memory, and the managed side wasn't notified. At such times you have this GCHandle structure keeping your managed objects alive, and it will never be freed. This is especially nasty in the case of pinned memory since the garbage collector will have to move everything else around that location. As you can surmise, the coordination between your managed and native sides is extremely important.

To keep an eye on the GCHandle structures in use, the new "!gchandles" command reports the handle statistics and those managed objects which are being held in the GCHandles table. An example output of the command is in Figure 6. The important numbers are the Strong and Pinned Handles. Strong handles mean the memory is referenced and cannot be garbage collected. Pinned means the object is referenced and is locked in memory. Keep in mind that strong and pinned memory happens to parameters implicitly when you make COM and P/Invoke calls. You'll want to keep an eye on those two numbers and if you see them creeping up, you've got a problem.

Figure 6 Output from the !GCHandles Command (Close Figure 6)
0:000> !gchandles
Scan HandleTable 0x168320
Scan HandleTable 0x14b4b8
GC Handle Statistics:
Strong Handles: 33
Pinned Handles: 0
Async Pinned Handles: 0
Ref Count Handles: 0
Weak Long Handles: 0
Weak Short Handles: 1
Other Handles: 0
Statistics:
MT            Count   TotalSize Class Name
0x79b925c8        1          20 System.String
0x79c05374        1          24 System.IO.TextWriter/SyncTextWriter
0x79b95614        1          32 System.SharedStatics
0x79bb55b4        1          44 System.Reflection.Module
0x79bad2dc        2          48 System.Security.FrameSecurityDescriptor
0x79b94ee4        1          64 System.ExecutionEngineException
0x79b94dac        1          64 System.StackOverflowException
0x79b94c74        1          64 System.OutOfMemoryException
0x79b96e8c        1          80 System.AppDomain
0x79bb8384        2         120 System.Threading.Thread
0x79ba29d4        5         120 System.Reflection.Assembly
0x79b9e24c       10         280 System.Security.PermissionSet
0x0098209c        4      12,320 System.Object[]
0x00982c3c        3      73,652 System.Byte[]
Total 34 objects, Total size: 86,932

In addition to "!GCHandles", the new "!GCHandleLeaks" will help you track down those leaks. "!GCHandleLeaks" works by looking at the handle values in the GCHandle table and scanning all of your process memory looking for references to those objects. If it finds a reference in memory that indicates the memory is still being used, then it's not a leak. If there's no reference to the object the GCHandle structure wraps, the command reports it as a leak. There's a small possibility that there could be false positive results if the native code is holding the managed pointer in a nonstandard way such as a mask value.

The end of the "!GCHandleLeaks" output will show the handles that it believes are leaked. Those values are the actual GCHandle structures. If you take a look at the GCHandle structure with Lutz Roeder's .NET Reflector, you'll see that the structure only contains an int value. To see the actual managed object leaked, perform a debugger DD (dump DWORD) command on the value that was reported leaked; that will show the object address so you can execute the "!do" command to see it.

Another new command, "!DumpAllExceptions" (also shortened to "!dae"), can be used whenever you want to see any exceptions that are still in the GC Heap.

The final new command I want to mention is the awesome "!SaveModule" command. If you're in a situation where you're debugging (either live or a mini dump), and you need to see the source code for a module, it can sometimes be difficult. However, for a managed module, you can pass the load address of the assembly and an output name to the "!SaveModule" like this: "!SaveModule 00400000 temp.exe". That will write out the complete assembly to disk so you can use Reflector to decompile the binary! Why suffer through trying to find the right version of the source code when you've got the actual source right here?

Wrap-Up

In this column, I hope I've given you some of the tips and tricks of gathering the best mini dumps possible in a production environment. You have the ADPlus configuration files I use when debugging so you can use them as a base to start building the custom configurations you'll need for your applications. While the ADPlus documentation is quite good, I strongly recommend that you look at the actual ADPlus source code so you can figure out exactly what will happen with various options.

Finally, I'm sure you're going to love the new SOS. The fact that you can easily see the objects in a specific generation makes it massively more useful that it used to be. While it's still not as easy as debugging in the Visual Studio® .NET debugger, it is the only tool I consider for tackling those nasty production-only problems.

Tips

Tip 67 Since I'm a tools kind of guy, I would be remiss if I also didn't plug Jonathan de Halleux's outstanding MBUnit. It's fully compatible with NUnit, but offers advanced features for creating great tests such as testing against different cultures, data driven testing, and process testing. MBUnit is replacing NUnit for me.

Tip 68 While MBUnit is excellent, the other testing tool you need is from Jamie Cansdale: TestDriven.NET. It's a phenomenal Visual Studio .NET Add-In that allows you to plug in tools like MBUnit, NUnit, and even the upcoming Microsoft Team System into a single one-click testing environment. All I can say is wow!

Send your questions and comments for John to slayer@microsoft.com.

John Robbins is a cofounder of Wintellect, a software consulting, education, and development firm that specializes in programming for .NET and Windows. His latest book is Debugging Applications for Microsoft .NET and Microsoft Windows (Microsoft Press, 2003). You can contact John at www.wintellect.com.


C++ at Work: Making Static Links Keyboard-Capable, Launching URLs from Your App

C++ At Work
Making Static Links Keyboard-Capable, Launching URLs from Your App
Paul DiLascia

Code download available at: CATWork0503.exe (184KB)

Q I've been using your CStaticLink class for a couple of years now in my company's commercial app. In the March 1998 issue of MSJ, you showed how to add the hand icon to the hyperlink, but now I'm wondering about another use for the class. In Microsoft® Internet Explorer, you can use the Tab key to tab through all the hyperlinks in a Web page, and press Enter to click the link. Can I make CStaticLink do the same thing? Am I asking too much?

Tom Gifford

A If you asked me to lend you a thousand dollars, that would be asking too much—but keystrokes for CStaticLink? That's a reasonable request. Especially since as a keyboard man, I can relate. I hate reaching for the pokey old mouse (which is why I use Emacs), and I hate it when apps don't provide adequate keyboard support. So you're definitely not asking too much; on the contrary, you get the user interface gold star merit medal!

For readers just jumping in, CStaticLink is a class I first wrote way back in December 1997 to let you add Web links in forms or your About dialog. CStaticLink is based on CStatic, MFC's class for static controls. I've enhanced CStaticLink over the years as readers asked for more features, and it appears I'm about to do it again. CStaticLink handles mouse clicks to launch the URL, and does other friendly things like paint the text in the standard blue/purple unvisited/visited link colors (which you can customize), underline the text, and display the proper finger-pointing cursor when the mouse hovers over the link. CStaticLink even automatically loads the URL from your resource file by looking for a resource string with same ID as your control. If you're not using CStaticLink yet, check it out.

But now Tom has wisely asked how to make static links keyboardable (to coin a verb). There are two basic things you have to do: first, you have to make the links Tab-able. Second, you have to handle the key that navigates the link. Let's start with tabbing.

When the user presses Tab in a dialog, Windows® moves the input focus to the next control in the tab order. The focus only lands on controls that are tab stops—that is, controls that have the WS_TABSTOP style. By default, the dialog editor doesn't create static controls with WS_TABSTOP. Why should it? Static controls are, after all, static. They don't do anything. They never change and they don't interact with the user.

But now I've given CStaticLink life by allowing mouse clicks, so it's appropriate to allow tabbing. One of the cardinal rules of user interface design (pay attention, folks!) is: whatever users can do with the mouse, they should also be able to do with the keyboard. It's not only friendly to keyboard folks like me, it comes in handy for mouse people when their mouse breaks. Perhaps your user has a cordless mouse, and the batteries just died at 1:00 AM and the report is due first thing that morning—it's sure nice to know you can get the job done using only the keyboard.

So the first thing to do is make the static link a tab stop. Just add WS_TABSTOP to the style in your resource file, or select True for the control's Tabstop property in Visual Studio® .NET (see Figure 1).

Figure 1 Making the Static Link a Tab Stop
Figure 1 Making the Static Link a Tab Stop

Once you make the control a tab stop, users can tab to it when the dialog is running. The only problem now is that when they do, nothing happens. There's nothing the user can do, there's not even any visual indication that the control has focus. If you tab to an edit control, it displays a blinking cursor. If you tab to a listbox, Windows draws a dotted focus rectangle around the first item. If you tab to a button, Windows draws a focus rectangle inside the button. But CStaticLink does nothing. Nada.

So I have to add some visual cue that tells the user "you are here." For hyperlinks (<A> elements), Internet Explorer draws a focus rectangle around the link text. So why not do the same here? Especially since Windows has a handy function called (oddly enough) DrawFocusRect, which has the nifty characteristic of using an XOR raster op, so calling it twice erases the rectangle. The time to draw the focus rectangle is when your control gains focus; the time to erase is when it loses it. In code, it looks like the following:

void CStaticLink::OnSetFocus(...)
{
   DrawFocusRect();
}
void CStaticLink::OnKillFocus(...)
{
   DrawFocusRect();
}
What could be easier? This even works if your control loses focus because the user switched to another app (as opposed to tabbing to another control).

Discerning readers may be wondering: where are the arguments? The DrawFocusRect in the preceding snippet isn't the real DrawFocusRect. It's an impostor I wrote, a protected CStaticLink member function that does a bunch of setup before calling the real DrawFocusRect. Figure 2 shows the code. It's mostly the usual GDI fare—selecting fonts, converting coordinates, and so on—so I'll only highlight the interesting details.

Figure 2 DrawFocusRect.cpp (Close Figure 2)
/////////////////
// Gaining or losing focus: draw focus rectangle.
// For bitmaps, use window rect; for text, use actual text rectangle.
//
void CStaticLink::DrawFocusRect()
{
   CWnd* pParent = GetParent();
   ASSERT(pParent);

   // calculate where to draw focus rectangle, in screen coords
   CRect rc;
   DWORD dwStyle = GetStyle();
   if (dwStyle & (SS_BITMAP|SS_ICON|SS_ENHMETAFILE|SS_OWNERDRAW)) {
      GetWindowRect(&rc); // images use full window rectangle

   } else {
      // text uses text rectangle. Don't forget to select font!
      CClientDC dc(this);
      CString s;
      GetWindowText(s);
      CFont* pOldFont = dc.SelectObject(GetFont());
      rc.SetRectEmpty(); // important—DT_CALCRECT expands, so start empty
      dc.DrawText(s, &rc, DT_CALCRECT);// calculate text rect
      dc.SelectObject(pOldFont);
      ClientToScreen(&rc);             // convert to screen coords
   }

   rc.InflateRect(1,1);                // add one pixel all around
   pParent->ScreenToClient(&rc);       // convert to parent window coords
   CClientDC dcParent(pParent);        // parent window's DC
   dcParent.DrawFocusRect(&rc);        // draw it!
}

The real ::DrawFocusRect (or its MFC equivalent, CDC::DrawFocusRect) needs a rectangle, of course, as well as the device context (DC) to draw on. But which device context should I use? Normally you should only draw in your own space, not someone else's—that is, in your control's window or client DC. But this is a case where drawing outside the window is appropriate because the focus rectangle needs to be a little bigger than the control in order to look right. So CStaticLink::DrawFocusRect draws on its parent window's client DC. Drawing focus rectangles is one of the few situations where it's OK to draw directly on the screen or parent window. In general, it's OK when you're doing some kind of temporary XOR thing, like drawing an icon or transparent image while dragging it across several windows; in that case you'd probably use the screen DC. If you paint on another window's device context, the only rule is: whatever thou paintest, thou shalt also unpaintest!

Next, what rectangle should I use? The window rectangle, of course. Think again. The window rectangle is fine for bitmaps where the bitmap is the same size as the control, but what about text? With text, the control is often bigger than the text, usually wider. Who bothers to size their static controls perfectly? If you use the window rectangle, you could end up with a long rectangle enclosing a small string of text, or even bumping into another control, as shown in the top part of Figure 3. This is the sort of thing that makes you look bad. That's why for text, CStaticLink::DrawFocusRect first calls CDC::DrawText with DT_CALCRECT to calculate the rectangle that exactly encloses the text. Inflate by one pixel all around for good measure, convert to parent window client coordinates, call CDC::DrawFocusRect and—voilà! The lower part of Figure 3 shows the happy result.

Figure 3 Without DT_CALCRECT and With DT_CALCRECT
Figure 3 Without DT_CALCRECT and With DT_CALCRECT

So now the user can tab to the static link and see a focus rectangle. The last thing to do is handle keyboard input. Which key should navigate the link? Internet Explorer uses Enter, but I don't like that for two reasons. First, the accepted user interface model for pressing a button from the keyboard is to press the space bar. A hyperlink is like a button, so space makes more sense to me. This is how dialogs do it, and what Internet Explorer does for buttons in Web forms. I don't know why the Redmondtonians chose Enter to navigate links in Internet Explorer. Enter usually means "I'm done," the same as pressing OK in a dialog. Since CStaticLink is designed for use in dialogs, it shouldn't conflict with Enter. The second reason is that trapping Enter in dialogs requires more work (see my July 2000 column).

So when the user tabs to the link, pressing the space bar should navigate the link. To make it happen, you have to handle two messages. WM_CHAR is the obvious one:

void CStaticLink::OnChar(UINT nChar,...)
{
   if (nChar==VK_SPACE) {
      Navigate();
   }
}

But before your static link can even get WM_CHAR, you have to tell the dialog you're interested. Normally static controls don't get WM_CHAR messages (because they're static, remember?). There's a special message to tell the dialog you want them—WM_GETDLGCODE:

UINT CStaticLink::OnGetDlgCode()
{
   // tell dialog I want chars
   return DLGC_WANTCHARS;
}
And with that, all is done. Now users can tab to hyperlinks and press the space bar to navigate. Cool.

One final warning: be careful to choose the correct tab order for your static links. Your hyperlinks should usually come at the end of the tab order, even if they appear visually at the top of the dialog. You probably don't want your dialog to start with the focus on your ACME logo link. And if you use CStaticLink in a form/dialog with other controls, you probably don't want the Tab key to jump from an edit box to a hyperlink to another edit box or button. So my advice is to keep all your hyperlinks at the end of the tab order, unless you have good reason not to.

I wrote a fully working sample program called LinkTest that uses the new keyboard-capable CStaticLink. You can download it from the link at the top of this article.

Q I have an MFC app that calls ShellExecute to open a Web page. For example:

ShellExecute "http://www.microsoft.com"
This was working fine until I started using the Managed Extensions and suddenly it stopped working. The error code returned is 5, which is ERROR_ACCESS_DENIED in WinError.h. I don't understand why access is denied. Doesn't ShellExecute work with the Managed Extensions?

A You're among the many who have discovered this unfortunate bugaboo. Indeed, as soon as you throw the /clr switch to use Managed Extensions, ShellExecute fails if you try to use it to open a Web page. This one stumped me for a while. What could the Managed Extensions and /clr possibly have to do with ShellExecute? And why do you get an access violation?

As is so often the case in Windows, the error code provides little clue as to what's really happening. But a little hunting through the MSDN Library reveals the answer in an article called "Calling Shell Functions and Interfaces from a Multithreaded Apartment". Eons ago, ShellExecute began life humbly; all it did was let you run a program, that is, an EXE file. As Windows has become more complex, ShellExecute has grown to the point where you can "execute" almost anything—such as a disk file (open the file with the associated program), FTP protocol, or Web page—just by passing the file name or URL. This works through shell extensions and IShellExecuteHook, which is a COM interface that extends the shell by telling it how to "execute" strings passed to ShellExecute(Ex). For example, there's an HTTP protocol extension hook that handles strings that begin with "http://". The extension handler launches the default browser to the given URL.

The problem is that using /clr and the Managed Extensions forces your app into multithreading mode because the garbage collector wants to run asynchronously in a separate thread. But as the INFO article explains, many IShellExecuteHook extensions don't work in multithreaded environments because they don't have the required proxy/stub code for COM to marshal arguments and synchronize access. If you're confused, you're not alone. Suffice it to say that ShellExecute simply doesn't work for all cases in multithreaded environments.

So, if you're already using the Managed Extensions, why not use the .NET Framework Process class and Process::Start instead of ShellExecute? There's a static overload that does just what you want:

Process::Start("http://www.microsoft.com");

Alas, if you try this at home, the same thing happens. The call throws a Win32Exception which yields an even more cryptic code in its NativeErrorCode property: ERROR_SXS_KEY_NOT_FOUND. The description for this error is "The requested lookup key was not found in any active activation context." Huh?

If you poke around the documentation for Process and read all the fine print, you'll discover that Process uses something called StartInfo in order to tell it how to start the process. One of the properties in StartInfo is UseShellExecute. By default, UseShellExecute equals True, which tells the Framework to launch the process using the shell, that is, ShellExecuteEx. Okay, so try setting it to False. But then, as the fine print explains, you can only start EXEs, not file names or URLs. You're hosed either way, and going in circles.

There's even more fine print in the documentation for Process::Start, advising you that if you want to UseShellExecute, you must make sure to specify [STAThread] (single-thread apartment model) as an attribute on your app's main function:

// in C#
public class MyForm : Form {
   [STAThread]
   public static void Main(string[] args) {
       ...
   }
}

Well, that's fine for C# or even "pure" (non-MFC) C++ programs, which have their own main functions, but what if you're using MFC? In that case, main, _tmain, _tWinMain, or whatever entry point it is for your platform is buried deep inside MFC where you can't edit the source to add [STAThread]. You could use /ENTRY:MyMain and write your own [STAThread] MyMain that calls the CRT startup, but now life is getting ugly. Surely there must be a simpler way.

In fact, there is a way to force single-thread apartment (STA) model threading in MFC apps without using [STAThread]. All you have to do is call CoInitialize(NULL) before the Framework tries to call CoInitializeEx(COINIT_MULTITHREADED). Figure 4 shows a little class that does this.

Figure 4 Force STA Threading (Close Figure 4)
//////////////////
// Use this class to force STA (single-threaded apartment model)
// threading in mixed-mode apps. To use, instantiate a static
// instance in your main application module, for example, MyApp.cpp or
// anywhere where the constructor will run before the CLR.
//
class CSTAThread {
public:
    CSTAThread() {
        CoInitialize(NULL);
    }
    ~CSTAThread() {
        CoUninitialize();
    }
};

The constructor calls CoInitialize(NULL) (STA threading) and the destructor calls CoUninitialize. So all you have to do is insert an instance in your MFC program's main application module:

// This does the equivalent of the [STAThread] attribute
CSTAThread forceSTAThread;

Pretty clever. (Thanks to Redmondtonian Martyn Lovell for suggesting this to me.) In Visual C++® 2005, you'll be able to tell the linker you want STAThread on your entry point, but for now you can use CSTAThread.

One other easy way to launch URLs from your app, which works even in multithreading mode, is by using rundll32.exe, a handy program that lets you call a function in any DLL. You give it the DLL, the function, and the parameters you want to pass, and rundll32.exe calls the DLL function with your parameters. Rundll32.exe is extremely versatile. You can use it to shut down or restart Windows, create shortcuts, and launch control panel applets. There's probably even a way to make it run your dishwasher if you have the right hardware. I've seen entire Web sites devoted to all the tricks you can do with rundll32.exe; it's simply a matter of knowing which DLLs to call. You can use rundll32.exe to launch a URL from the command prompt like so:

rundll32.exe url.dll,FileProtocolHandler www.microsoft.com

The function FileProtocolHandler in url.dll does the job. With ShellExecute, it looks like this:

LPCTSTR url = _T("www.microsoft.com");
CString args;
args.Format(_T("url.dll,FileProtocolHandler %s"), url);
ShellExecute(NULL, _T("open"), _T("rundll32.exe"), args);
This works even in multithreaded apps because you're giving ShellExecute a true EXE, not a file name it has to run using shell extensions and IShellExecuteHook. The only drawback is you don't get any error code back from url.dll if it can't open the URL. For this reason, I recommend using CSTAThread instead and directly invoking the URL with ShellExecute, especially in MFC apps, which work better with apartment model threading anyway.

As a practical example of this, I updated my CStaticLink class from the previous question to use CSTAThread and ShellExecute, and wrote a managed test program, LinkTest, to prove it works in managed mode. I've included the CSTAThread class in StatLink.h. So now CStaticLink has yet another bonus feature: it works in both native apps and apps compiled with /clr and the Managed Extensions. As always, you can download the code from the MSDN Magazine Web site. Happy programming!

Send your questions and comments for Paul to cppwork@microsoft.com.

Paul DiLascia is a freelance writer, consultant, and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). Paul can be reached at www.dilascia.com.


{ End Bracket }: To Confirm is Useless, to Undo Divine

{ End Bracket }
To Confirm is Useless, to Undo Divine
David S. Platt

The common user interface technique of confirmation, popping a dialog box into the user's face and asking, "Are you really REALLY sure you want to do that?" is evil. It's unfriendly, it's distracting, and it's completely ineffective. It shouldn't exist. Anywhere. Not ever.

Have you ever, even once, said, "Whoa! I didn't want to do that. Thanks," and clicked No? Or have you seen anyone do that? I haven't. Confirmation is so vastly overused that it has, ironically, become completely useless. Because the box constantly cries "wolf!" like the shepherd boy in Aesop's fable, no one pays attention to it, even when it's warning you of something you really don't want to do. You cruise through it on auto-pilot, clicking Yes without thinking. It provides you with no safety whatsoever.

Other operations in life don't require confirmation. Your car does not ask, "Do you really want to start the engine?" when you turn the key. The supermarket clerk does not ask, "Do you really want to buy these?" when you place groceries on the register belt. Programmers constantly ask for confirmation because they think users don't understand the consequences of their commands. That may be true, given the poor quality of the user interface. But confirmation doesn't solve this problem. If the user was confused when he first gave whatever command triggered the confirmation box, he'll be even more confused when he sees it. On the contrary, it keeps programmers from having to clearly explain to the user what he's doing, and providing a way to recover when he does something that he later regrets, despite having originally confirmed it.

What if the user really is mistaken? If you put a flashlight on the register belt with a package of the wrong size batteries, wouldn't an attentive clerk ask, "Are you sure?" A good user interface should and can save us from mistakes like that by preventing the problem initially. Perhaps the Web page selling flashlights would contain a check box saying "include batteries," checked by default. Better still, the flashlight would come with batteries inside, so it'd work the instant you unwrapped it and no one would ever have to worry about buying the correct size.

Another reason that you aren't asked to confirm starting your car or buying groceries is that these operations are easy to undo. You just turn off the key or return the unwanted item. Programmers often put "undo" capability in their programs, where it's the greatest design advance since the mouse. It takes an enormous amount of effort to make this feature work so that users don't even have to think about it ("easy can be hard"), but the programmers who implement it are any user's best friends. I buy them beer whenever I meet them.

The worst confirmations are those of undoable actions, such as moving a file to the Recycle Bin. It's much more efficient to fix the relatively small number of errors that actually do occur (for example, a slip of the mouse that deleted the wrong file) than attempt to prevent them by annoying the user with a confirmation box every time (which are usually ignored out of habit). An ounce of cure is not worth five pounds of prevention.

The beauty of undo is that it allows users to explore a program. It's not always easy to understand a new program's operation from menu items and toolbar picture. With undo, a user can try different commands, knowing that he won't damage something that can't be repaired with a few keystrokes. Programmers often regard incorrect user input as the act of an idiot who should have read the %*$#% instruction manual. It isn't. It is the primary mechanism by which the human species learns. An application with undo capability recognizes and enhances the user's humanity.

If undo is implemented correctly, then there's only one destructive operation in the entire system: emptying the Recycle Bin. Some would say that this operation should have a confirmation box, as it currently does. But even here, the confirmation dialog exists only to guard against another bad design, placing the "Explore" context menu item next to "Empty Recycle Bin." One slip of the mouse, sliding down three spaces instead of two, and you get the latter instead of the former. That's bad. Emptying the Recycle Bin should have a special action used for no other purpose, perhaps clicking on it while holding down some key. Better still, the Recycle Bin should automatically delete files after some configurable period of time so you'd seldom have to empty it manually. Don't ask permission, ask forgiveness. Provide undo, not confirmation.

David S. Platt teaches programming for the .NET Framework at Harvard University and at companies all over the world. He was selected by Microsoft as one of their Software Legends, which you can read about at www.softwarelegends.com. He is the author of nine programming books, the most recent of which is The Microsoft Platform Ahead (Microsoft Press, 2004). Reach him at www.rollthunder.com.