 MSDN Magazine February 2006

MSDN Magazine February 2006

Distributed .NET: Learn The ABCs Of Programming Windows Communication Foundation

Windows Communication Foundation unifies the existing suite of .NET distributed technologies into a single programming model to improve the developer experience. This article introduces WCF so you'll be prepared to take advantage of all the new capabilities.

Aaron Skonnard UDP Delivers: Take Total Control Of Your Networking With .NET And UDP

UDP, the lesser-known sibling of TCP, is used for DNS resolution, SNMP network status, Kerberos security, digital media streaming, VoIP, and lots more. Learn how to put UDP to work for you.

Yaniv Pessach Wandering Code: Write Mobile Agents In .NET To Roam And Interact On Your Network

In artificial intelligence, an agent is a logical entity that has some level of autonomy within its environment or host. A mobile agent has the added capability that it can move between hosts. In this article Matt Neely brings mobile agents from the halls of academia to a dev shop near you.

Matt Neely WSE Security: Protect Your Web Services Through The Extensible Policy Framework In WSE 3.0

This article describes the WSE policy framework, which allows you to describe constraints and requirements a Web service must enforce. Discussions include security scenarios in WSE 3.0 and extending the framework with custom constraints and requirements.

Tomasz Janczuk Smart Clients: Build A Windows Forms Control To Consume And Render WSRP Portlets

Smart client apps use local resources, provide a rich client experience, and support intelligent install mechanisms. Web services offer powerful interoperability and integration features. Find out how to combine them to develop integrated apps that incorporate data from disconnected sources.

Carl Nolan Paste As Visual Basic: A Visual Studio Add-In That Converts C# Code To Visual Basic

Build a Visual Basic add-in that lets you copy C# code and paste it into Visual Studio as Visual Basic using the code converter of your choice. Scott Swigart shows you how.

Scott Swigart Editor's Note: Code Name Confusion

We'd like to have a word about product code names this month.

Toolbox: User-Friendly UIs, Online Data Entry Controls, and More

Microsoft® Office is one of the most well-known and most-used consumer products. For many people, their only experience with a computer is when using Office, be it checking e-mail, creating presentations, or writing a document.

Scott Mitchell Basic Instincts: Introducing ASP.NET Web Part Connections

When you begin to work with the Microsoft® . NET Framework 2. 0 and ASP. NET, you discover that the new Web Parts infrastructure adds some very powerful functionality to the underlying platform. In the September 2005 issue of MSDN®Magazine, Fritz Onion and I have an article on programming Web Parts titled "ASP.

Ted Pattison CLR Inside Out: Improving Application Startup Time

Visual Studio is a wonderful development environment, whose IntelliSense®, integrated debugging, online help, and code snippets help boost your performance as a developer. But just because you're writing code fast doesn't mean you're writing fast code.

Claudio Caldato Cutting Edge: Build Providers for Windows Forms

In ASP. NET and Windows Forms projects, some file types are treated differently than others. For example, ASPX and ASCX files are dynamically parsed and compiled to an assembly on the fly. The contents of an XML Schema Definition (XSD) file are used to create a new, strongly typed DataSet-based object at design time.

Dino Esposito Wicked Code: The SQL Site Map Provider You've Been Waiting For

Now that ASP.NET 2.0 is a shipping product, it seems appropriate to revisit an issue that tops the new features wish lists of many developers: a SQL Server™ site map provider.

Jeff Prosise Pure C++: Live Source Browsing in Visual C++

If you're wondering where Stan Lippman is, we're happy to report that he has graciously granted us the use of his column this month to talk about some of the recent work the Visual C++ team has been doing to improve developer productivity.

Boris Jabes and Ameya Limaye C++ at Work: Event Programming

Paul DiLascia { End Bracket }: Building Voice User Interfaces

The GUI is the interface of choice for scenarios when the user has a large keyboard, mouse, and display. But most cell phones have just a keypad and a small display. Drivers want to get directions without taking their eyes off the road or their hands off the wheel.

Alex Acero


Distributed .NET: Learn The ABCs Of Programming Windows Communication Foundation

Distributed .NET

Learn The ABCs Of Programming Windows Communication Foundation

Aaron Skonnard
This article is based on a prerelease version of WinFX. All information contained herein is subject to change.
This article discusses:
  • The WCF programming model
  • Defining contracts and behavior
  • Hosting a service and defining endpoints
  • Creating and configuring clients
This article uses the following technologies:
.NET Framework 2.0, WCF, WSDL, SOAP
Code download available at: WCF.exe (142KB)
Contents
WCF Programming Model
Service Contracts and Dispatch Behavior
Data Contracts
Message and Service Contracts
Implementing Service Contracts
Hosting the Service and Defining Endpoints
Choosing and Customizing Bindings
Opening the Host
Configuring Service Endpoints
Using Activation Services
Programming Clients
Configuring Client Endpoints
Generating Client Proxies
Logging Messages
Conclusion

W
indows® Communication Foundation (WCF), formerly code-named "Indigo," is about to radically change the face of distributed programming for developers using the Microsoft® .NET Framework. WCF unifies the existing suite of .NET distributed technologies into a single programming model that improves the overall developer experience through a consistent architecture, new levels of functionality and interoperability, and all the extensibility points you could want. This article introduces you to WCF programming and shows you how to get started.

As its name suggests, WCF provides the .NET Framework with a basis for writing code to communicate across components, applications, and systems. WCF was designed according to the tenets of service orientation. A service is a piece of code you interact with through messages. Services are passive. They wait for incoming messages before doing any work. Clients are the initiators. Clients send messages to services to request work.

Figure 1 Services and Endpoints
Figure 1 Services and Endpoints

Services expose one or more endpoints where messages can be sent. Each endpoint consists of an address, a binding, and a contract (see Figure 1). The address specifies where to send messages. The binding describes how to send messages. And the contract describes what the messages contain. Clients need to know this information before they can access a service.

Services can package up endpoint descriptions to share with clients, typically by using Web Services Description Language (WSDL). Then clients can use the provided service description to generate code within their environment capable of sending and receiving the proper messages (see Figure 2).

Figure 2 Sharing Endpoint Descriptions
Figure 2 Sharing Endpoint Descriptions

Windows Communication Foundation provides a new library of classes found in the System.ServiceModel namespace that bring these service-oriented concepts to life. This is what's typically referred to as the WCF programming model.

WCF Programming Model

With WCF, you're either writing services that expose endpoints or you're writing clients that interact with endpoints. Hence, endpoints are central to the WCF programming model and infrastructure. WCF models endpoints with the .NET classes and interfaces shown in Figure 3. This mapping is true whether you're writing WCF clients or services.

Figure 3 WCF Classes and Interfaces (Close Figure 3)

Element | Class or Interface
Endpoint | System.ServiceModel.ServiceEndpoint
Address | System.Uri
Binding | System.ServiceModel.Binding
Contract | Interfaces annotated with System.ServiceModel attributes

When building a WCF service, you typically start by defining a .NET interface definition to serve as the service contract. Then you implement the service contract in a .NET class, known as the service type, and configure its behavior. Next, you define the endpoints the service will expose, specifying the address, binding, and contract for each one. Finally, you host the service type in an application using the WCF hosting infrastructure. Once the service type is hosted, clients can retrieve its endpoint descriptions and begin integrating with it.

When building a WCF client, you first need the description of the target endpoint you want to access. The endpoint description can be used to dynamically create a typed proxy. WCF provides a tool named SvcUtil.exe for automating this process. Then you can write code against the typed proxy to access the service by sending the appropriate messages to the target endpoint.

Service Contracts and Dispatch Behavior

You model service contracts in .NET using traditional C# interface definitions. You can use any .NET interface as a starting point, such as the one shown here:

namespace ServiceLibrary
{
    public interface IEchoService
    {
        string Echo(string msg);
    }
}
To make this a WCF service contract, you must annotate the interface itself with [ServiceContract] and each operation you want to expose with [OperationContract]:
using System.ServiceModel;

namespace ServiceLibrary
{
    [ServiceContract(Namespace="http://example.org/echo/")]
    public interface IEchoService
    {
        [OperationContract]
        string Echo(string msg);
    }
}

These attributes influence the mapping between the worlds of .NET and SOAP. WCF uses the information found in the service contract to perform dispatching and serialization. Dispatching is the process of deciding which method to call for an incoming SOAP message. Serialization is the process of mapping between the data found in a SOAP message and the corresponding .NET objects used in the method invocation. This mapping is controlled by an operation's data contract.

WCF dispatches based on the message action. Each method in a service contract is automatically assigned an action value based on the service namespace and method name. For example, the default action for the Echo method just shown is http://example.org/echo/Echo. You can customize the action value for each method using [OperationContract]. A value of * can be used for any action when a specific match doesn't exist.

In the following example, WCF will dispatch messages with an action of urn:echo:string to the Echo method. Messages with any other action are dispatched to the EchoMessage method:

[ServiceContract(Namespace="http://example.org/echo/")]
public interface IEchoService
{
    [OperationContract(Action="urn:echo:string")]
    string Echo(string msg);

    [OperationContract(Action="*")]
    Message EchoMessage(Message msg);
}

Data Contracts

Once the target method has been determined based on the action, WCF relies on the method's data contract to perform serialization. The data contract is defined by the types used in the method signature. In the previous example, EchoMessage is generic and could be used to process a variety of incoming SOAP messages. Therefore I've used Message to model the input and output. Message is a special type used to represent all messages flowing through WCF. When using Message, WCF does not perform type-based serialization. Instead, it just gives you direct access to what's found in the SOAP message.

When Message is not used, WCF performs serialization to map between the data found in the SOAP message and the corresponding .NET objects needed to invoke the method. For example, in the case of Echo, WCF maps the SOAP payload to a .NET string. WCF provides an implicit mapping for all .NET primitive types (string, int, double, and so on).

The way WCF serializes .NET classes depends on the serialization engine in use. The default serialization engine is known as DataContract, a simplified version of XmlSerializer, the default serialization engine used in ASMX today. DataContract defines attributes for annotating class definitions to influence the serialization process. Here's an example of it in use:

[DataContract(Namespace="http://example.org/person")]
public class Person
{
    [DataMember(Name="first", Order=0)]
    public string First;
    [DataMember(Name="last", Order=1)]
    public string Last;
    [DataMember(IsRequired=false, Order=2)]
    private string id;
    ...
}
With DataContract, only fields marked with DataMember will be serialized. And you can serialize private fields. This is much different from the way XmlSerializer works. Now you can use Person in an operation contract:
[ServiceContract(Namespace="http://example.org/echo/")]
public interface IEchoService
{
    ... // previous methods omitted

    [OperationContract]
    Person EchoPerson(Person p);
}
When EchoPerson is invoked, WCF will serialize Person instances according to the DataContract attributes specified on the Person class. DataContract produces a very simple XML structure—a sequence of elements. You can control the name of each element, the order, and whether a particular element is required, but that's about it.

If you need to do anything more sophisticated, WCF lets you fall back to using XmlSerializer. You indicate your desire to use XmlSerializer by annotating the interface with [XmlSerializerFormat]:

[ServiceContract]
[XmlSerializerFormat]
public interface IEchoService { ... }
This tells WCF to use XmlSerializer for all types in the contract, according to the XmlSerializer defaults and the various System.Xml.Serialization customization attributes. This makes it much easier to move existing ASMX services forward to WCF.

WCF also supports serializing types marked with [Serializable], which allows .NET remoting types to work with WCF without change. WCF provides an implicit mapping for [Serializable] types where all public/private fields are automatically serialized.

Message and Service Contracts

All of the examples up to this point have relied on WCF to automatically map the method signature to the SOAP message. The parameter list and return type are always mapped to the SOAP body for the request and response, respectively. If you need to support headers, you can write another class that models the structure of the entire SOAP envelope for the particular operation, specifying which fields map to headers versus the body. You define this mapping with the [MessageContract] attributes:

[MessageContract]
public class EchoPersonMessage
{
    [MessageBody]
    public Person Person;
    [MessageHeader]
    public Authorization Authorization;
}

In this example, the Person field is mapped to the SOAP body while the Authorization field is mapped to a SOAP header. Now you can use EchoPersonMessage in an operation contract:

[ServiceContract]
public interface IEchoService
{
    ... // previous methods omitted

    [OperationContract]
    void EchoPerson(EchoPersonMessage msg);
}
Using MessageContract is a more advanced technique that is only necessary when you need direct control over the SOAP contract.

Implementing Service Contracts

Now you can implement the service contract by simply implementing the .NET interface in a class:

using System.ServiceModel;
namespace ServiceLibrary
{
    public class EchoService : IEchoService
    {
        public string Echo(string msg)
        {
            return msg;
        }
        ... // remaining methods omitted
    }
}
By doing this, EchoService is guaranteed to support the service contract defined by IEchoService. When using an interface to define the service contract, you don't need any contract-related attributes on the class definition. However, you may want to use [ServiceBehavior] to influence its local behavior:
using System.ServiceModel;

namespace ServiceLibrary
{
    ... // interface definition omitted

    [ServiceBehavior(
       InstanceContextMode=InstanceContextMode.Single,
       ConcurrencyMode=ConcurrencyMode.Multiple)]
    public class EchoService : IEchoService
    {
       ...

This particular example tells WCF to manage a singleton instance of the service type and to allow multithreaded access to the instance. There is also an [OperationBehavior] attribute for controlling operation-level behavior.

Behaviors influence processing within the host, but have no impact on the service contract whatsoever. Behaviors are one of the primary WCF extensibility points. Any class that implements IServiceBehavior can be applied to a service through the use of a custom attribute or configuration element.

Hosting the Service and Defining Endpoints

To use EchoService, you need to host it in a .NET-based application. The ServiceHost class gives you direct control over the WCF hosting infrastructure. You instantiate ServiceHost based on a particular service type. The following code shows how to do this in a console application:

using System;
using System.ServiceModel;
using ServiceLibrary;

class Program
{
    static void Main(string[] args)
    {
        using (ServiceHost host = new ServiceHost(
            typeof(EchoService), new Uri("http://localhost:8080/echo")))
        {
            ...

In addition to specifying the service type, you also specify the base addresses for the different transports you plan to use. In this example, I've specified a base address of http://localhost:8080/echo. This will be used as the base for any relative HTTP addresses I might specify when adding endpoints. The base HTTP address is also used as the default for retrieving the service description.

You then add the service endpoints. Again, a service may have one or more endpoints and each endpoint consists of an address, a binding, and a contract. You provide this information to your ServiceHost by calling AddServiceEndpoint. This is where you specify the service contract you defined earlier (IEchoService). For the binding, you typically choose from one of the many predefined bindings that ship with WCF and an appropriate address given the binding's transport. Here's an example:

host.AddServiceEndpoint(typeof(IEchoService),
    new BasicHttpBinding(), "svc");
host.AddServiceEndpoint(typeof(IEchoService),
    new NetTcpBinding(), "net.tcp://localhost:8081/echo/svc");

In this particular example, I've specified IEchoService as the contract for both endpoints, but with each endpoint using a different binding and address. The first endpoint uses the BasicHttpBinding and a relative HTTP address of svc (which would make its absolute address http://localhost:8080/echo/svc). The second endpoint uses the NetTcpBinding and an address of net.tcp://localhost:8081/echo/svc. Therefore, this service allows consumers to communicate with it over either HTTP or TCP using different WS-* protocols, as defined by each binding.

Choosing and Customizing Bindings

The WCF infrastructure relies heavily on endpoint definitions to control how messages are processed. In fact, WCF uses the endpoint definitions to dynamically build the appropriate message processing runtime within hosts and clients. The binding has the most significant influence on this process.

The binding controls three aspects of message communication: the suite of WS-* protocols, including WS-Security, WS-ReliableMessaging, and so on; the message encoding, such as XML 1.0, Message Transmission Optimization Mechanism (MTOM), and binary; and the transport protocol, including HTTP, TCP, and Microsoft Message Queuing (MSMQ). A given binding specifies one message encoding and one transport, but it can specify numerous WS-* protocols. Given that, there are an overwhelming number of permutations that could be used. To simplify things, WCF provides a set of predefined bindings that fit the most common use cases. Figure 4 lists some of these and describes what they embody.

Figure 4 Predefined WCF Bindings (Close Figure 4)

Class Name | Element Name | Transport | Encoding | WS-* Protocols
BasicHttpBinding | basicHttpBinding | HTTP | XML 1.0 | WS-I Basic Profile 1.1
WSHttpBinding | wsHttpBinding | HTTP | XML 1.0 | Message security, reliable sessions, and transactions
WSDualHttpBinding | wsDualHttpBinding | HTTP | XML 1.0 | Message security, reliable sessions, and transactions
NetTcpBinding | netTcpBinding | TCP | Binary | Transport security, reliable sessions, and transactions
NetNamedPipeBinding | netNamedPipeBinding | Named Pipes | Binary | Transport security, reliable sessions, and transactions
NetMsmqBinding | netMsmqBinding | MSMQ | Binary | Transport security and queue transactions

Developers who care primarily about interoperability and don't need WS-* functionality will typically use BasicHttpBinding. Developers who care about interoperability, but also need message-based security, reliable messaging, and transactions will typically choose WSHttpBinding. Developers using WCF on both sides (client and service) can choose from NetTcpBinding and NetNamedPipeBinding, which provide significant optimizations. Those building asynchronous systems can choose NetMsmqBinding. There are also a few bindings that aren't listed here, such as MsmqIntegrationBinding for integrating with existing MSMQ applications, NetPeerTcpBinding for building peer-to-peer services, and WSFederationBinding for federated security. A service can support any combination of these bindings.

Once you choose a binding and instantiate it, you can customize some of its characteristics through the properties exposed on the binding class. The following example shows how to enable transport-based security on the BasicHttpBinding:

BasicHttpBinding b = new BasicHttpBinding();
b.Security.Mode = BasicHttpSecurityMode.Transport;
b.Security.Transport.ClientCredentialType =
    HttpClientCredentialType.Basic;
host.AddServiceEndpoint(typeof(IEchoService), b, "svc");
If the predefined bindings and their customizations still don't fit your needs, you can also write a custom binding by implementing a class that derives from Binding.

Opening the Host

Now that you've configured the host with endpoint information, WCF can build the runtime needed to support your configuration. This occurs when you call Open on a particular ServiceHost:

host.Open();
Once the call to Open returns, the WCF runtime is built and ready to receive messages at the addresses specified by each endpoint. During this process, WCF generates an endpoint listener for each supplied endpoint and the code needed to support the WS-* protocols specified by the binding. (An endpoint listener is the piece of code that actually listens for incoming messages using the specified transport and address.)

You can retrieve information about the service at run time through the object model exposed by ServiceHost. This allows you to retrieve anything you might want to know about the initialized service, such as what endpoints it exposes and what endpoint listeners are currently active.

Figure 5 shows a complete example of the console application that hosts EchoService. Figure 6 shows the output. Notice there are two additional endpoint listener addresses that I didn't specify in a ServiceEndpoint. WCF automatically provided these using the base HTTP address for retrieving the service description.

Figure 5 Console App Hosting EchoService (Close Figure 5)
class Program
{
    static void Main(string[] args)
    {
        using (ServiceHost host = new ServiceHost(
            typeof(EchoService), new Uri("http://localhost:8080/echo")))
        {
            // define the service endpoints
            host.AddServiceEndpoint(typeof(IEchoService),
                new BasicHttpBinding(), "svc");
            host.AddServiceEndpoint(typeof(IEchoService),
                new NetTcpBinding(), "net.tcp://localhost:8081/echo/svc");
            host.Open();

            Console.WriteLine(
                "{0} is open and has the following endpoints:\n",
                host.Description.ServiceType);

            int i=1;
            foreach (ServiceEndpoint end in host.Description.Endpoints)
            {
                Console.WriteLine("Endpoint #{0}", i++);
                Console.WriteLine("Address: {0}",
                    end.Address.Uri.AbsoluteUri);
                Console.WriteLine("Binding: {0}",
                    end.Binding.Name);
                Console.WriteLine("Contract: {0}\n",
                    end.Contract.Name);
            }

            Console.WriteLine(
                "The following EndpointListeners are active:\n");
            foreach (EndpointListener l in host.EndpointListeners)
                Console.WriteLine(l.Listener.Uri.AbsoluteUri);

            // keep the process alive
            Console.ReadLine();
        }
    }
}
Figure 6 Output of Console App
Figure 6 Output of Console App

If you browse to http://localhost:8080/echo, you'll see the default WCF help page (see Figure 7) and if you browse to http://localhost:8080/echo?wsdl, you'll see the generated WSDL definition. The other endpoint listener (http://localhost:8080/echo/mex) knows how to speak WS-MetadataExchange. It's important to note that this sample is not using IIS in any way. WCF provides built-in integration with httpsys, which allows any application to automatically become an HTTP listener.

Figure 7 EchoService Help Page
Figure 7 EchoService Help Page

Configuring Service Endpoints

Hardcoding endpoint information into the host application is not ideal since endpoint details may change over time. It's not hard to imagine how you could store the endpoint information in a configuration file or database and read it while initializing ServiceHost. Since this is a common need, WCF automatically provides this functionality through the predefined <system.serviceModel> configuration section.

The following configuration file defines the same two endpoints that were specified by calling AddServiceEndpoint:

<configuration>
  <system.serviceModel>
    <services>
      <service type="ServiceLibrary.EchoService">
        <endpoint address="svc" binding="basicHttpBinding"
                  contract="ServiceLibrary.IEchoService"/>
        <endpoint address="net.tcp://localhost:8081/echo/svc"
                  binding="netTcpBinding"
                  contract="ServiceLibrary.IEchoService"/>
      </service>
    </services>
  </system.serviceModel>
</configuration>
If you add this configuration file to the host application shown in Figure 5 and comment out the calls to AddServiceEndpoint, you'll see the same results in the output. This increases deployment flexibility since the communication details are completely factored out of the compiled code. You can also configure bindings within <system.serviceModel>. Figure 8 shows how to configure basicHttpBinding to use transport security like I had done in code.

Figure 8 Configuring Transport Security (Close Figure 8)
<configuration>
  <system.serviceModel>
    <services>
      <service type="ServiceLibrary.EchoService">
        <endpoint
          address="https://localhost:8082/echo/svc"
          binding="basicHttpBinding"
          bindingConfiguration="MyBindingConfiguration"
          contract="ServiceLibrary.IEchoService"/>
        ...
      </service>
    </services>
    <bindings>
      <basicHttpBinding>
        <binding name="MyBindingConfiguration">
          <security mode="Transport">
            <transport clientCredentialType="Basic" />
          </security>
        </binding>
      </basicHttpBinding>
    </bindings>
  </system.serviceModel>
</configuration>

You can even define new custom bindings from scratch using the <customBinding> element, which would be equivalent to deriving a new class from Binding. When it comes to configuring endpoints, bindings, and even behaviors, anything you can do in code, you can also do through configuration.

Figure 9 Using the GUI to Define an Endpoint
Figure 9 Using the GUI to Define an Endpoint

Although IntelliSense® for the configuration file works well in Visual Studio® 2005, some developers still prefer to avoid XML altogether. The WinFX® SDK includes a tool called SvcConfigEditor, which provides a graphical interface for working with the various <system.serviceModel> settings. Figure 9 shows how to configure endpoints using this tool.

Using Activation Services

Although I've been experimenting with console applications, ServiceHost makes it easy to host WCF services in a variety of applications, including Windows Forms applications and managed Windows Services. In all of these application scenarios, you're responsible for managing the process hosting WCF. This is known as self-hosting in WCF terms.

In addition to self-hosting, WCF also makes it possible to activate services by using IIS along with traditional Web hosting techniques. You can accomplish this by using a .svc file (similar to .asmx) that specifies the service type to host:

<%@Service class="ServiceLibrary.EchoService" %>
You place this file in a virtual directory and deploy your service type to its \bin directory or the Global Assembly Cache (GAC). When using this technique you specify the service endpoint in the web.config, but you don't need to specify the address since it's implied by the location of the .svc file.
<configuration>
  <system.serviceModel>
    <services>
      <service type="ServiceLibrary.EchoService">
        <endpoint address="" binding="basicHttpBinding"
          contract="ServiceLibrary.IEchoService"/>
      </service>
      ...

Now if you browse to the .svc file, you'll notice the help page is displayed, showing that a ServiceHost was automatically created within an ASP.NET application domain. This is accomplished by an HTTP module that filters incoming .svc requests and automatically builds and opens the appropriate ServiceHost when needed.

If you install the WinFX extensions for Visual Studio 2005, you'll find a new Web Site project template called Indigo Service (this name will change). When you create a new Web site based on this template, it automatically gives you the .svc file along with the corresponding implementation class (found in App_Code). It also comes with a default endpoint configuration in web.config.

On IIS versions 5.1 and 6.0 the WCF activation model is tied to the ASP.NET pipeline, and therefore to HTTP. IIS 7.0, which is planned for release with Windows Vista™, introduces a transport-neutral activation mechanism known as Windows Activation Services (WAS). With WAS, you'll be able to leverage .svc-like activation over any transport.

Programming Clients

Programming WCF clients involves sending messages to a service endpoint according to its address, binding, and contract. To accomplish this, you first create a ServiceEndpoint representing the target endpoint. The following example shows how to create a ServiceEndpoint, based on the service's HTTP endpoint, in a client console application:

using System;
using System.ServiceModel;
using ServiceLibrary;

class Program
{
    static void Main(string[] args)
    {
        ServiceEndpoint httpEndpoint = new ServiceEndpoint(
            ContractDescription.GetContract(typeof(IEchoService)),
            new BasicHttpBinding(), new EndpointAddress(
                "http://localhost:8080/echo/svc");
This code assumes that the client has access to the same IEchoService interface definition that the service used and that the client knows what binding and address to use. Then you can create a ChannelFactory based on the ServiceEndpoint:
ChannelFactory<IEchoService> factory =
    new ChannelFactory<IEchoService>(httpEndpoint);
ChannelFactory creates typed proxies. In this case, it creates proxies of type IEchoService, which you can use to invoke the operations defined by the contract:
IEchoService svc = factory.CreateChannel();
Console.WriteLine(svc.Echo("Hello, world"));

If the service requires a customized binding, you will need to create the binding and customize it before creating ServiceEndpoint. If you want to access the TCP endpoint, simply create another ServiceEndpoint specifying the TCP endpoint details and supply it to ChannelFactory instead of the HTTP endpoint. Everything else would work the same.

The proxy shields you from the different addresses and bindings in use, allowing you to write application code against the service contract. Figure 10 shows the complete client console application.

Figure 10 EchoService Client App (Close Figure 10)
using System;
using System.ServiceModel;
using ServiceLibrary;

class Program
{
    static void Main(string[] args)
    {
        try
        {
            // define service endpoints on client
            ServiceEndpoint httpEndpoint = new ServiceEndpoint(
                ContractDescription.GetContract(
                   typeof(IEchoService)),new BasicHttpBinding(),
                new EndpointAddress("http://localhost:8080/echo/svc"));

            ServiceEndpoint tcpEndpoint= new ServiceEndpoint(
                ContractDescription.GetContract(
                   typeof(IEchoService)),
                new NetTcpBinding(), new EndpointAddress(
                   "net.tcp://localhost:8081/echo/svc"));

            IEchoService svc = null;

            // create channel factory based on HTTP endpoint
            using (ChannelFactory<IEchoService> httpFactory =
                new ChannelFactory<IEchoService>(httpEndpoint))
            {
                // create channel proxy for endpoint
                svc = httpFactory.CreateChannel();
                // invoke service operation
                Console.WriteLine("Invoking HTTP endpoint: {0}",
                    svc.Echo("Hello, world"));
            }

            // create channel factory based on TCP endpoint
            using (ChannelFactory<IEchoService> tcpFactory =
                new ChannelFactory<IEchoService>(tcpEndpoint))
            {
                // create channel proxy for endpoint
                svc = tcpFactory.CreateChannel();
                // invoke service operation
                Console.WriteLine("Invoking TCP endpoint: {0}",
                    svc.Echo("Hello, world"));
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e);
        }
    }
}

Configuring Client Endpoints

Hardcoding endpoint information into client code is also less than ideal. So WCF provides a similar configuration mechanism for specifying client endpoints within <system.serviceModel>. The following application configuration file specifies the same client endpoint details used in the code:

<configuration>
  <system.serviceModel>
    <client>
      <endpoint name="httpEndpoint"
        address="http://localhost:8080/echo/svc"
        binding="basicHttpBinding"
        contract="ServiceLibrary.IEchoService"/>
      <endpoint name="tcpEndpoint"
        address="net.tcp://localhost:8081/echo/svc"
        binding="netTcpBinding"
        contract="ServiceLibrary.IEchoService"/>
    </client>
  </system.serviceModel>
</configuration>

Notice that I've given each client endpoint a name. You can use this name in your code when creating a ChannelFactory, as illustrated in the following code:

using (ChannelFactory<IEchoService> httpFactory =
    new ChannelFactory<IEchoService>("httpEndpoint"))
{
    svc = httpFactory.CreateChannel();
    Console.WriteLine("Invoking HTTP endpoint: {0}",
        svc.Echo("Hello, world"));
}
Now when you want to create a ChannelFactory for the TCP endpoint, you can simply change the configuration name to tcpEndpoint. You can also configure bindings in the client configuration file just like I did in the service configuration file.

If you add this configuration file to the client console application shown in Figure 10, comment out the code to create the ServiceEndpoint objects, and specify the endpoint names when constructing each ChannelFactory, the result will be the same.

Generating Client Proxies

The preceding examples assumed the client had access to the same interface definition used to implement the service. This is common when .NET is used to implement both sides (in traditional .NET remoting scenarios), but that's not always the case, and it's never the case when the service wasn't implemented with the .NET Framework. The preceding examples also assumed the client knew the precise endpoint details in advance, which isn't always common either.

When clients don't have access to such information, they can discover it using the WS-MetadataExchange specification. Services share endpoint descriptions with clients through WSDL definitions. Most service platforms provide tools for generating client-side code from WSDL definitions. The WinFX SDK provides SvcUtil.exe for this purpose.

When executing SvcUtil.exe, you specify the URL of the WSDL definition and it will generate the necessary .NET code and configuration elements that define the endpoints and bindings used by the service endpoints. Figure 11 shows the output of running SvcUtil.exe against http://localhost:8080/echo. Notice that it generates two files: EchoService.cs and output.config.

Figure 11 SvcUtil.exe Output
Figure 11 SvcUtil.exe Output

EchoService.cs contains a .NET interface definition equivalent to the one the service implemented. Output.config contains the client endpoint information needed to access the service (similar to what I just wrote manually). You'll need to merge the contents of output.config with your application configuration file manually. With this in place, you can write client code using the same techniques shown in the previous examples, only using the generated interface and configuration file.

In general, the WCF client programming model is explicit about the service boundary. You create a typed channel against a specific service endpoint and then send messages through it. Although this helps emphasize the tenets of service orientation, it can be tedious to work at this level all the time. To help you, SvcUtil.exe also generates a proxy class that completely hides the ChannelFactory and ServiceEndpoint creation details. If you open EchoService.cs, you'll find the following class definition:

public partial class EchoServiceProxy :
   System.ServiceModel.ClientBase<IEchoService>, IEchoService
{
    public EchoServiceProxy()
    {
    }
    public EchoServiceProxy(string endpointConfigurationName) :
            base(endpointConfigurationName)
    {
    }
...

This proxy class can simplify your client application code. All you have to do is instantiate the proxy, specify the name of the endpoint configuration, and make method calls. Here's an example:

using (EchoServiceProxy proxy = new EchoServiceProxy("IEchoService"))
{
    // invoke service operation
    Console.WriteLine("Invoking HTTP endpoint: {0}",
        proxy.Echo("Hello, world"));
}
using (EchoServiceProxy proxy = new EchoServiceProxy("IEchoService1"))
{
    // invoke service operation
    Console.WriteLine("Invoking TCP endpoint: {0}",
        proxy.Echo("Hello, world"));
}
The generated configuration file (output.config) defines the IEchoService and IEchoService1 endpoint configurations.

Logging Messages

As you begin working with the WCF programming model, you may find it useful to trace the SOAP messages traveling between clients and services. Windows Communication Foundation provides built-in support for message logging, which can be turned on through your application configuration file.

Figure 12 SvcTraceViewer.exe
Figure 12 SvcTraceViewer.exe

SvcConfigEditor provides a Diagnostics tab where you can enable the messaging logging features. Once you enable messaging logging and rerun the app, you'll see the trace file appear in the specified location. The WinFX SDK also provides SvcTraceViewer.exe for viewing the information in a trace file (see Figure 12).

Conclusion

WCF is a new step in distributed programming for developers using the .NET Framework. If you currently build systems using any of today's .NET distributed technologies, it's time to start paying attention to WCF and the future it holds. It's only a matter of time before all .NET-targeted code related to communications will be written using WCF.

Note that all of the code samples shown in this article are based on Visual Studio 2005 and the WCF November 2005 CTP.

Aaron Skonnard is a co-founder of Pluralsight, a Microsoft .NET training provider. Aaron is the author of Pluralsight's Applied Web Services 2.0, Applied BizTalk Server 2006, and Introducing WCF courses. Aaron has spent years developing courses, speaking at conferences, and teaching professional developers. Reach him at pluralsight.com/aaron.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


UDP Delivers: Take Total Control Of Your Networking With .NET And UDP

UDP Delivers

Take Total Control Of Your Networking With .NET And UDP

Yaniv Pessach
This article discusses:
  • The pros and cons of TCP and UDP
  • Building UDP into your applications
  • Reliability protocols and security issues
  • UDP support in Windows Communication Foundation
This article uses the following technologies:
.NET Framework, C#
Code download available at: UDP.exe (136KB)
Contents
TCP Versus UDP
A Custom UDP Solution
Improving the Code
Reliability Protocols
Sample Protocol Implementation
Broadcast and Multicast
Security
Windows Communication Foundation
Conclusion

Y
ou've probably made use of Transmission Control Protocol (TCP) in your applications at some point, whether directly or indirectly—after all, HTTP runs on top of TCP, so every Web application is built atop it. But maybe you've wondered whether its brother User Datagram Protocol (UDP) might be a better fit for your intended solution.

Most Internet traffic utilizes TCP and UDP, running on top of Internet Protocol (IP), the low-level protocol used by all traffic on the Internet. While TCP is the more familiar of the two, accounting for as much as 75 percent of Internet traffic, UDP holds second place with approximately 20 percent of sent packets. All other low-level protocols combined, including raw Internet Control Message Protocol (ICMP), account for less than 5 percent of Internet traffic.

UDP is used for such important and common tasks as DNS resolution, Simple Network Management Protocol (SNMP) network status, Windows® Internet Naming Service (WINS) NetBIOS name resolution, Trivial FTP (TFTP), Kerberos security, server discovery, digital media streaming, voice over IP (VoIP) using the International Telecommunications Union (ITU) H.323 protocol, and online gaming.

Trying to design an efficient, quick, and responsive network application is hard work, so using the right tools can help a lot. Sometimes, choosing UDP as the low-level network protocol can give you the necessary flexibility to use fewer resources, support more clients, reduce latency, increase throughput, or implement services that are not otherwise practical over TCP. These benefits aren't free, however. Writing UDP code is relatively simple, but using UDP in a useful, safe, and efficient manner requires that you implement the application protocol yourself.

In this article, I will discuss the pros and cons of using UDP in your own applications. I'll take a look at the design considerations for UDP-based applications, including the details and pitfalls of implementing those applications in C#. I will also offer a preview of exposing UDP Web services through Windows Communication Foundation.

TCP Versus UDP

Since their inception, the TCP and UDP protocols have taken very different evolutionary paths. TCP is based on connections, maintains a session, and guarantees a degree of reliability and standardized flow control. UDP provides no such features and relies upon the application layer to provide such services.

UDP allows you to send a packet, with or without a checksum, and redirects the packet (multiplexes it) to a listening application based on the port number the packet was sent to. A single packet can be sent to multiple machines at once by using multicast and broadcast transmission.

With UDP, no connection is maintained—each packet sent is treated independently. UDP makes no attempt to control transmission speeds based on congestion. If a packet is lost, the application must detect and remedy the situation. If a packet arrives out of order, you're on your own again.

UDP also does not provide the security features that you can expect to find with TCP. This includes the three-way handshake that TCP uses to ensure the validity of the claimed source address.

So what can UDP do for your application that TCP cannot? To start, UDP supports multicast—sending a single packet to multiple machines. Multicast can be used to send a message even when you do not know the IP address or DNS name of the target machines. This is used, for example, to discover the servers within a local network. Using multicast can also save bandwidth. If you want to play a presentation to 20 local machines, by using multicast the server would only need to send each packet once. UDP multicast is used by UPnP and the Windows XP My Network Places feature.

UDP is also useful when the overhead of TCP is not acceptable, especially when only one request/response is needed. Establishing a TCP connection requires exchanging at least three packets (see Figure 1). If it takes 150ms for a packet to go one way from Seattle to Sydney, and if the client (the sender) makes only one request at a time from the server (the receiver), then connection establishment and a response would take at least 600ms. On the other hand, if you sent the request over UDP, it could shave 300ms off this wait. By using UDP, you also spare the server the resources it needs to manage a TCP connection, thus enabling the server to process more requests.

Figure 1 UDP and TCP Request/Response Models
Figure 1 UDP and TCP Request/Response Models

UDP can help if your application can use a different packet-loss recovery mechanism. Since TCP guarantees that data will be processed by the server in the order it was sent, packet loss on a TCP connection prevents the processing of any later data until the lost packet is received successfully. For some applications this behavior is not acceptable, but others can proceed without the missing packet. For example, the loss of one packet in a broadcast video should not cause a delay because the application should just play the next frame.

UDP can also help if you want to control or fine-tune communication parameters such as the outgoing buffer size, minimize network traffic, or use different congestion-avoidance logic than TCP provides. Specifically, TCP assumes that packet loss indicates congestion. Therefore, when packet-loss is detected, TCP slows down the rate of outgoing information on a connection. In that respect, TCP uses a "considerate" algorithm to optimize throughput. However, this can result in slow transmission rates on networks with high packet loss. In some cases, you may want to deliberately choose a less-considerate approach to achieve higher throughput.

On the other hand, TCP has distinct advantages in simplicity of implementation. Many security threats are resolved in the TCP stack so you don't have to worry about them. Also, since UDP does not have a connection, firewalls cannot easily identify and manage UDP traffic. System administrators would rather allow an outgoing TCP connection than open their firewalls.

A Custom UDP Solution

To illustrate the issues surrounding a real-world UDP implementation, I'll start by building simple UDP client and server apps modeled on an online gaming scenario. The application will send a player's information (ID, X coordinate, and Y coordinate) from the client to a server:

public class PlayerInfo
{
    public byte playerID, locationX, locationY;
}

Before sending any information on the wire, I first have to decide on a wire format and how to serialize and deserialize the message—that is, how to translate PlayerInfo into an array of bytes and then back. In this article, I use a couple of simple methods to do this (see Figure 2).

Figure 2 Serializing and Deserializing PlayerInfo (Close Figure 2)
public int ToBuffer(byte[] buffer, int pos)
{
    int newPos = pos;
    buffer[newPos++] = playerID;
    buffer[newPos++] = locationX;
    buffer[newPos++] = locationY;
    return newPos - pos;
}

public void FromBuffer(byte[] buffer, int pos)
{
    playerID = buffer[pos++];
    locationX = buffer[pos++];
    locationY = buffer[pos++];
}

The client code should initialize a Socket, serialize the data from PlayerInfo, and send the data. The data will be sent to the IP address 127.0.0.1, which is always the local computer:

const int ProtocolPort = 3001;
Socket sendSocket = new Socket(AddressFamily.InterNetwork,
    SocketType.Dgram, ProtocolType.Udp);
IPAddress sendTo = IPAddress.Parse("127.0.0.1");
EndPoint sendEndPoint = new IPEndPoint(sendTo, ProtocolPort);
byte[] buffer = new byte[PlayerInfo.MaxWireSize];
int bufferUsed = player.ToBuffer(buffer, 0);
sendSocket.SendTo(buffer, bufferUsed, SocketFlags.None,
    sendEndPoint);

Production code would also add framing—in the form of headers—to the actual data. In this example, I chose to send each PlayerInfo in a separate UDP packet.

The Socket.SendTo method sends the number of bytes specified by bufferUsed to the address in sendEndPoint. Since UDP does not implement a flow control protocol and there isn't a guaranteed response for a UDP packet, SendTo does not generally need to block. This is unlike sending with TCP, which would block until the point when too much data has been sent but not yet acknowledged by the receiver. It is safe to assume that no long delays will happen when using SendTo, which is why I did not use the asynchronous version Socket.BeginSendTo.

If you have the machine name rather than an IP address, you can modify the code to resolve the machine name to an IP address:

IPAddress FirstDnsEntry(string hostName)
{
    IPHostEntry IPHost = Dns.Resolve(hostName);
    IPAddress[] addr = IPHost.AddressList;
    if (addr.Length == 0) throw new Exception("No IP addresses");
    return addr[0];
}

Note that the DNS protocol, used in Dns.Resolve, may use either UDP or TCP. When I call FirstDnsEntry with a remote machine name, I can see a UDP packet sent from my machine to port 53. Traffic analyzers, such as Netmon.exe, can help to debug your applications by showing exactly which packets are being sent and received on your machine.

To implement the server side of this communication, I use the asynchronous Socket.BeginReceiveFrom method, which will call a delegate when a UDP packet is received (see the code in Figure 3). You have to call again if you want to receive additional packets, so a good practice to follow is to always call BeginReceiveFrom from the BeginReceiveFrom delegate.

Figure 3 Receiving a Message (Close Figure 3)
Socket receiveSocket = new Socket(AddressFamily.InterNetwork,
    SocketType.Dgram, ProtocolType.Udp);
EndPoint bindEndPoint = new IPEndPoint(IPAddress.Any, ProtocolPort);
byte[] recBuffer = new byte[PlayerInfo.MaxWireSize];
receiveSocket.Bind(bindEndPoint);
receiveSocket.BeginReceiveFrom(recBuffer, 0, recBuffer.Length,
    SocketFlags.None, ref bindEndPoint,
    new AsyncCallback(MessageReceivedCallback), (object)this);

void MessageReceivedCallback(IAsyncResult result)
{
    EndPoint remoteEndPoint = new IPEndPoint(0, 0);
    try
    {
        int bytesRead = receiveSocket.EndReceiveFrom(result,
            ref remoteEndPoint);
        player.FromBuffer(recBuffer, 0, Math.Min(recBuffer.Length,
            bytesRead));
        Console.WriteLine("ID:{0} X:{1} Y:{2}", player.playerID,
            player.locationX, player.locationY);
    }
    catch (SocketException e)
    {
        Console.WriteLine("Error: {0} {1}", e.ErrorCode, e.Message);
    }

    receiveSocket.BeginReceiveFrom(recBuffer, 0, recBuffer.Length,
        SocketFlags.None, ref bindEndPoint,
        new AsyncCallback(MessageReceivedCallback), (object)this);
}

Improving the Code

While responding to a UDP packet is normally up to the receiving application, a special case exists when there is no application listening on the port. The UDP stack then will send (in most configurations, and unless a firewall is configured to prevent it) an ICMP port unreachable response. This will be visible to the sending application if it is listening on the socket as a SocketException with an ErrorCode value of 10054 (connection reset by peer). Adding this code after socket initialization would catch the exception:

EndPoint bindEndPoint = new IPEndPoint(IPAddress.Any, 0);
sendSocket.Bind(bindEndPoint);
byte[] recBuffer = new byte[PlayerInfo.MaxWireSize];
sendSocket.BeginReceiveFrom(recBuffer, 0, recBuffer.Length,
    SocketFlags.None, ref reponseEndPoint,
    new AsyncCallback(CheckForFailuresCallback),
    (object)this);
I react to this exception in the callback:
void CheckForFailuresCallback(IAsyncResult result)
{
    EndPoint remoteEndPoint = new IPEndPoint(0, 0);
    try
    {
        int bytesRead = sendSocket.EndReceiveFrom(result,
            ref remoteEndPoint);
    }
    catch (SocketException e)
    {
        if (e.ErrorCode == 10054) serviceMissing = true;
    }
}

Incidentally, the behavior I just relied on—detecting whether any application is listening on a UDP port—can let people identify which UDP ports a machine is listening on, a process known as UDP port scanning. The same can be done for TCP, and the recommended cure is firewall configuration.

UDP is not a session-based protocol, and responses to a message, if sent at all, are treated as independent of the original message. The sender of a response must specify both the IP address and the port the response is sent to, and must use the SendTo method.

There are several ways for a sender and a receiver to agree on a return address and port. The sender IP address (as claimed on the packet) can be retrieved from the received message, along with the source port. Sending a message to the source port implies that the same socket is used by the original sender to both send the original request and listen to responses. An alternative approach is to specify the response port in the request:

bytesRead = socket.EndReceiveFrom(result, ref remoteEndPoint);
EndPoint endPointDestination =
    new IPEndPoint(((IPEndPoint)remoteEndPoint).Address,
    ((IPEndPoint)remoteEndPoint).Port);
socket2.SendTo(buffer, length, SocketFlags.None, endPointDestination);

The port a server is listening on can either be fixed (a well-known value used by a protocol and hardcoded in all clients and servers) or determined at run time based on which ports are in use. Since, in general, only one socket and one application can listen on a specific port number at a time, determining the port at run time allows multiple instances of a receiver to run on a machine. Run-time port selection is also useful when a server should send a message or messages back to clients.

To find a free port number, the receiver should bind to port 0. The actual port it is listening on can be found using the LocalEndPoint property, as shown here:

socket = new Socket(AddressFamily.InterNetwork, SocketType.Dgram,
    ProtocolType.Udp);
EndPoint endPoint = new IPEndPoint(IPAddress.Any, 0);
socket.Bind(endPoint);
listeningPort = ((IPEndPoint)(socket.LocalEndPoint)).Port;

Reliability Protocols

While the techniques I've demonstrated so far work most of the time, they don't help address the reliability issues of UDP. Packets can be lost in transit, and no corrective action will be taken. UDP will drop packets on the send side before sending them if sending happens fast enough and the internal buffers are exhausted. UDP will also drop packets upon receipt if, even momentarily, BeginReceiveFrom is not active on the socket.

The last problem is the one most easily solved. In my sample receiver code, there's a short span of time between acceptance of a packet and calling another BeginReceiveFrom. Even if this call were the first one in MessageReceivedCallback, there's a still short period when the app isn't listening. One improvement would be activating several instances of BeginReceiveFrom, each with a separate buffer to hold received packets.

Also consider the impact of the UDP packet sizes. While UDP packets can be up to 64 kilobytes (KB), the underlying IP transport can only transport packets up to the Maximum Transmission Unit (MTU) of the link. When a packet bigger than an MTU comes along, IP fragmentation occurs—the IP protocol divides the sent packet into fragments smaller than MTU, and reassembles them at the destination, but only if all fragments were received.

Not relying on IP fragmentation, but instead limiting your packets to MTU, increases the reliability of the transmission and reduces packet loss. Ethernet MTU is usually 1500 bytes, but some connections (such as dial-up and some tunneling protocols) have an MTU of 1280 bytes. Practically speaking, it is hard to discover the minimum MTU of a link, so choosing a packet size of less than 1280 (or 1500) will increase reliability.

Another concern is that sending UDP packets too fast will result in some packets being silently discarded. Earlier I pointed out that SendTo calls do not block. This can surprise developers who are accustomed to the TCP behavior of blocking the Send call until the data can be sent.

What constitutes too fast? A few packets a second are not an issue; hundreds or thousands may be an issue, depending on the computer and network.

All those factors show that even on a local network with perfect connectivity, you cannot rely on all sent UDP packets being received. And for applications using the Internet, packet loss can be a major issue, reaching more than 5 percent in some areas, with worse packet loss experienced on specific links or at specific times.

Most UDP-based applications need to implement some kind of reliability mechanism. Reliability is often seen in the form of guarantees. The three guarantees you may be familiar with in TCP are: non-duplication (each packet is received once at most); transmission (each packet is received at least once or an error is reported to the sender); and packet order (if packet A was sent before packet B on the same TCP connection, it will be read before packet B). Figure 4 shows an example of the problems caused by dropped and out-of-order packets.

Figure 4 Dropped and Out-of-Order Packets
Figure 4 Dropped and Out-of-Order Packets

With UDP, each of those guarantees, if desired, needs to be implemented by the application. If a specific guarantee is not needed or can be relaxed, you have more freedom to write a more efficient protocol.

In my version of the protocol, I deal with dropped packets by resending them—but I send only the dropped packets to conserve bandwidth. To do that, the application needs to detect which packets were dropped and which arrived successfully.

The common method (which is also used by TCP) is for the receiver to send an acknowledgment (ACK) for packets received. The protocol can save bandwidth if the receiver sends a single ACK for several received packets. To facilitate this, each sent packet will normally have a sequence number or a unique identifier.

Different packet-loss handling protocols select different ACK methods and timeouts for handling a packet that did not receive an ACK. They also differ in their handling of out-of-order packets.

To help test the UDP samples presented in this article, I wrote a utility called UDPForward that can simulate packet drops and delays. UDPForward is available in the code download for this article. If you try the samples in this article with simulated packet loss, you can see how packet loss and delay affects the receiver of messages, and how the reliability protocol I implemented resolves this issue. You can also use UDPForward to test your own UDP applications and get a feel for their behavior in the presence of packet loss.

Sample Protocol Implementation

To demonstrate how an application can use UDP to tailor a specific protocol for specific needs, I will implement a sample protocol for transmitting the current location of a PlayerInfo object. Using UDP, I will choose suitable ACK behaviors as well as control and communication parameters such as buffer sizes and timeouts.

The receiver of a sequence of PlayerInfo packets normally cares only about the last known or current state. Using this property, which is applicable to other applications such as streaming audio and video, I can better handle a dropped packet. If a newer update to the same information becomes available, I can avoid the ACK and the delay until the packet is resent. This approach results in better bandwidth utilization and lower delay.

Why would you ever resend a packet if you can constantly receive updated information? For the PlayerInfo application, I'm willing to accept a momentary stale display, but not a permanent one. What if the packet notifying the movement of player 7 was lost and therefore player 7 was not moved afterwards? Without an ACK, I will never know. In addition, the sender can determine when the receiver is no longer receiving any packets. And since the sender may send a packet twice, the receiver implements a mechanism to drop duplicate and stale copies of the packet received.

The important point is that this is an application-specific decision. In some applications, including the PlayerInfoBroadcaster example shown later and some voice streaming implementations, you can choose a different tradeoff.

The protocol I chose to implement optimizes sending PlayerInfo data by only requiring one ACK per PlayerInfo send if no newer PlayerInfo data was received. The sender will send one packet per update of PlayerInfo data, together with a constantly-increasing sequence number. The sender maintains a list of "waiting for ACK" data, but the sender would only process an ACK for the last update sent (so old ACKs are discarded). The sender also occasionally checks the waiting for ACK list and resends the PlayerInfo data for any item that has been waiting too long.

The receiver needs to perform two duties: duplicate detection and ACK sending. This implementation sends ACKs immediately for new data as well as for duplicate data, but duplicates are detected (since I maintain the last sequence received for this PlayerID) and will be dropped before they are passed to the application.

The application also detects a broken connection if too many attempts to resend data fail. In the real world, the user may be informed or some error recovery could be attempted.

I first define PacketInfo:

public struct PacketInfo
{
    public int sequenceNumber;
    public long sentTicks;
    public int retryCount;
}
The sender initiates the socket and listens for ACKs (see Figure 5). The sender maintains a list of ACKs that are needed and will handle incoming ACKs by retrieving the PlayerID and sequenceNumber from the ACK packet, then removing an entry from the ACK-needed list if the sequence number matches (see Figure 6).

Figure 6 Maintaining the List of ACKs (Close Figure 6)
static void OnReceiveAck(IAsyncResult result)
{
    UDP_protocol uf = (UDP_protocol)result.AsyncState;
    EndPoint remoteEndPoint = new IPEndPoint(0, 0);
    int bytesRead = uf.senderSocket.EndReceiveFrom(
        result, ref remoteEndPoint);
    uf.ProcessIncomingAck(uf.bufferAck, bytesRead);
    uf.ListenForAcks();
}

void ProcessIncomingAck(byte[] packetData, int size)
{
    ...
    // remove from 'needing ACK' list if ACK for newest info
    if (sentPacketInfo[playerID].sequenceNumber == sequenceNumber)
    {
        Console.WriteLine("Received current ACK on {0} {1}",
            playerID, sequenceNumber);
        ProcessPacketAck(playerID, sequenceNumber);
    }
    ...
}
Figure 5 Listening for ACKs (Close Figure 5)
void OpenSenderSocket(IPAddress ip, int port)
{
    senderEndPoint = new IPEndPoint(ip, port);
    senderSocket = new Socket(AddressFamily.InterNetwork,
        SocketType.Dgram, ProtocolType.Udp);
    EndPoint bindEndPoint = new IPEndPoint(IPAddress.Any, 0);
    senderSocket.Bind(bindEndPoint); // any free port
    ListenForAcks();

    // start 'ack waiting' thread
    ThreadPool.QueueUserWorkItem(CheckPendingAcks, 0);
}

public void ListenForAcks()
{
    EndPoint endPoint = new IPEndPoint(0, 0);
    senderSocket.BeginReceiveFrom (bufferAck, 0, SizeAck,
        SocketFlags.None, ref endPoint,
        onReceiveAck, (object)this);
}

public void ListenForAcks()
{
    EndPoint endPoint = new IPEndPoint(0, 0);
    senderSocket.BeginReceiveFrom (bufferAck, 0, SizeAck,
        SocketFlags.None, ref endPoint,
        onReceiveAck, (object)this);
}

When sending data, the sequenceNumber and PlayerID are written to the packet, the packet is sent, and the list of ACKs that are needed is updated, including current time, which will be used to decide if a resend is needed:

void SendPlayerInfoData(byte playerID, byte locationX, byte locationY)
{
    sequenceNumber++;
    ...
    SendPlayerInfo(info, sequence, false);
}

void SendPlayerInfo(PlayerInfo info, int sequenceNumber, bool retry)
{
    ...
    UpdatePlayerInfo(info, sequenceNumber, retry);
    senderSocket.SendTo(packetData, SizeData, SocketFlags.None,
        senderEndPoint);
}
Occasionally, the sender will resend packets that were not ACKed. If too many resends are attempted, then the sender will decide that the connection is broken (see Figure 7).

Figure 7 Resending Packets (Close Figure 7)
public void CheckPendingAcks(object o)
{
    Console.WriteLine("Checking for missing ACKs");
    byte currentPosition = 0;
    while(!closingSender)
    {
        Thread.Sleep(10);
        ResendNextPacket(ref currentPosition, 10,
            DateTime.Now.Ticks - TimeSpan.TicksPerSecond / 10);
    }
}

void ResendPlayerInfo(byte playerID)
{
    SendPlayerInfo(sentPlayerInfo[playerID],
        sentPacketInfo[playerID].sequenceNumber, true);
    sentPacketInfo[playerID].retryCount++;
    sentPacketInfo[playerID].sentTicks = DateTime.Now.Ticks;

    Console.WriteLine("Resending packet {0} {1} {2}", playerID,
        sentPacketInfo[playerID].sequenceNumber,
        sentPacketInfo[playerID].retryCount);
}

void ResendNextPacket(ref byte currentPosition,
    byte maxPacketsToSend, long olderThan)
{
    ...
    if ((sentPacketInfo[newPosition].sequenceNumber = 0;
        sentPacketInfo[newPosition].sentTicks < olderThan)
    {
        if (sentPacketInfo[newPosition].retryCount > 4)
        {
            Console.WriteLine("Too many retries, should fail connection");
        }
        else
        {
            ResendPlayerInfo(newPosition);
            packetsSent++;
        }
    }
}

The receiver will just listen for packets and check for duplicate or old data (see Figure 8). An ACK here is sent immediately. More elaborate implementations could reduce the number of ACKs by sending one ACK for several packets.

Figure 8 Listening for Packets (Close Figure 8)
void OpenReceiverSocket(int port)
{
    EndPoint receiverEndPoint = new IPEndPoint(IPAddress.Any, port);
    receiverSocket = new Socket(AddressFamily.InterNetwork,
        SocketType.Dgram, ProtocolType.Udp);
    receiverSocket.Bind(receiverEndPoint);
    ListenForData();
}

public void ListenForData()
{
    EndPoint endPoint = new IPEndPoint(0, 0);
    receiverSocket.BeginReceiveFrom (bufferData, 0, SizeData,
        SocketFlags.None, ref endPoint, onReceiveData, (object)this);
}

static void OnReceiveData(IAsyncResult result)
{
    UDP_protocol uf = (UDP_protocol)result.AsyncState;
    EndPoint remoteEndPoint = new IPEndPoint(0, 0);
    int bytesRead = uf.receiverSocket.EndReceiveFrom(result,
        ref uf.receiverAckEndPoint);
    uf.ProcessIncomingPlayerInfo(uf.bufferData, bytesRead);
    uf.ListenForData();
}

void SendAck(byte playerID, int sequenceNumber)
{
    ... // create packet with playerID and sequenceNumber
    receiverSocket.SendTo(packetData, SizeAck, SocketFlags.None,
        receiverAckEndPoint);
}

void ProcessIncomingPlayerInfo(byte[] packetData, int size)
{
    ... // create packet with playerID and sequenceNumber
    if (sequenceNumber >= recvSequenceNumber[info.playerID])
        SendAck(info.playerID, sequenceNumber);

    if (sequenceNumber > recvSequenceNumber[info.playerID])
    {
        ... // update internal data and process packet
        Console.WriteLine("Received update: {0} ({1},{2})",
            info.playerID, info.locationX, info.locationY);
    }
    // else older packet, don't process or send ACK
}

Broadcast and Multicast

A crucial feature supported by UDP but not TCP is sending a single message to multiple destinations. UDP supports both broadcast and multicast communication. Broadcasting a packet makes it available within a subnet mask. Multicasting is subscription-based in the sense that listeners have to join a multicast group to receive messages sent to that group.

A multicast group uses an IP address in the range of 224.0.0.0 through 239.255.255.255. To listen on a multicast group, you need to indicate that you want multicast messages from a specific multicast group by setting the add membership SocketOption after calling Bind, but before calling BeginReceiveFrom:

IPAddress multicastGroup = IPAddress.Parse("239.255.255.19");
socket.Bind(...);
socket.SetSocketOption(SocketOptionLevel.IP,
    SocketOptionName.AddMembership,
    new MulticastOption(multicastGroup));
socket.BeginReceiveFrom(...);
You do not need to set SocketOption in order to send to a multicast group, as shown here:
const int ProtocolPort = 3001;
Socket sendSocket = new Socket(AddressFamily.InterNetwork,
    SocketType.Dgram, ProtocolType.Udp);
EndPoint sendEndPoint = new IPEndPoint(multicastGroup, ProtocolPort);
sendSocket.SendTo(buffer, bufferUsed, SocketFlags.None, sendEndPoint);

If your application does not know the IP address or machine name of, say, a game server, that server can listen on a well-known multicast address, wait for a specific request, and respond directly to the application that made the inquiry.

In the sample I built that follows, the inquiring application will also use a dynamic port to receive the responses from zero, one, or more game servers. It also shows serialization of more complex data types. First I'll define the classes representing FindRequest and FindResult, as shown in the following:

class FindRequest {
    public int serviceID;
    public int responsePort;
    public int SerializeToPacket(byte[] packet) {...}
    public FindRequest(byte[] packet) {...}
}

class FindResult {
    public int serviceID;
    public int SerializeToPacket(byte[] packet) {...}
    public FindResult(byte[] packet) {...}
}
To advertise itself, a server would open a socket, set multicast options on it, and process any requests, as shown in Figure 9.

Figure 9 Starting a Server Session (Close Figure 9)
public AsyncCallback onReceiveRequest =
    new AsyncCallback(OnReceiveRequest);
IPAddress multicastGroup = IPAddress.Parse("239.255.255.19");

public void FindMe(int blockFor, int serviceID)
{
    responseSocket = new Socket(AddressFamily.InterNetwork,
        SocketType.Dgram, ProtocolType.Udp);
    EndPoint responseEndPoint = new IPEndPoint(IPAddress.Any,
        multicastPort);
    responseSocket.Bind(responseEndPoint);
    responseSocket.SetSocketOption(SocketOptionLevel.IP,
        SocketOptionName.AddMembership,
        new MulticastOption(multicastGroup));
    ListenForRequests();
}

public void ListenForRequests()
{
    EndPoint endPoint = new IPEndPoint(0, 0);
    responseSocket.BeginReceiveFrom(findRequestBuffer, 0, 12,
        SocketFlags.None, ref endPoint,
        onReceiveRequest, (object)this);
}

When a request arrives, the server identifies the sender of the request either by looking at the remoteEndPoint obtained or by including this data in the packet. The server then sends a response announcing that it is present and active (see Figure 10). The client looking for a server would send a request, then wait a given amount of time for a response as shown in Figure 11.

Figure 11 Client-Side Multicast Request (Close Figure 11)
public void Finder(int waitFor, int serviceID)
{
    // start listening for responses before sending the request
    responseSocket = new Socket(AddressFamily.InterNetwork,
        SocketType.Dgram, ProtocolType.Udp);
    EndPoint responseEndPoint = new IPEndPoint(IPAddress.Any, 0);
    responseSocket.Bind(responseEndPoint);
    responsePort = ((IPEndPoint)(responseSocket.LocalEndPoint)).Port;

    ListenForResponses();

    // prepare request
    FindRequest fr = new FindRequest();
    fr.serviceID = serviceID; fr.responsePort = responsePort;
    int requestLength = fr.SerializeToPacket(findRequestBuffer);

    //send request
    Socket requestSocket = new Socket(AddressFamily.InterNetwork,
        SocketType.Dgram, ProtocolType.Udp);
    EndPoint requestEndPointDestination =
        new IPEndPoint(multicastGroup, multicastPort);
    requestSocket.SendTo(findRequestBuffer, requestLength,
        SocketFlags.None, requestEndPointDestination);

    requestSocket.Close();

    //wait for responses
    Thread.Sleep(waitFor);
}

public void ListenForResponses()
{
    EndPoint endPoint = new IPEndPoint(0, 0);
    responseSocket.BeginReceiveFrom(findResultBuffer, 0, 8,
        SocketFlags.None, ref endPoint,
        onReceiveResponse, (object)this);
}

static void OnReceiveResponse(IAsyncResult result)
{
    UDP_finder uf = (UDP_finder)result.AsyncState;
    EndPoint remoteEndPoint = new IPEndPoint(0, 0);
    int bytesRead = uf.responseSocket.EndReceiveFrom(result,
        ref remoteEndPoint);
    FindResult response = new FindResult(uf.findResultBuffer);

    uf.ListenForResponses();

    Console.WriteLine(„Found service {0}", response.serviceID);
}
Figure 10 Responding to Requests (Close Figure 10)
static void OnReceiveRequest(IAsyncResult result)
{
    UDP_finder uf = (UDP_finder)result.AsyncState;
    EndPoint remoteEndPoint = new IPEndPoint(0, 0);
    int bytesRead = uf.responseSocket.EndReceiveFrom(result,
        ref remoteEndPoint);
    FindRequest request = new FindRequest(uf.findRequestBuffer);

    // prepare result
    FindResult fr = new FindResult();
    fr.serviceID = uf.currentServiceID;
    int requestLength = fr.SerializeToPacket(uf.findResultBuffer);

    //send result
    Socket requestSocket = new Socket(AddressFamily.InterNetwork,
        SocketType.Dgram, ProtocolType.Udp);
    EndPoint requestEndPointDestination =
        new IPEndPoint(((IPEndPoint)remoteEndPoint).Address,
        request.responsePort);
    requestSocket.SendTo(uf.findResultBuffer, requestLength,
        SocketFlags.None, requestEndPointDestination);

    requestSocket.Close();

    uf.ListenForRequests();
}

Another purpose is reducing traffic and server load by sending a message to multiple senders, since one packet sent can be read by many machines on the local network, when multiple machines are interested in the same data. Copying the same file to many machines can take advantage of multicast and save bandwidth and reduce CPU usage for the sending server.

Security

When building a UDP (or any) server, security is an important consideration. The first concern is a denial of service attack, where an attacker sends many requests and spends too much of the server's resources, especially CPU time. It is common to implement throttling so that, if messages are received too quickly, some of them will be rejected or ignored.

A more difficult problem to solve is authenticating incoming messages. Since an attacker can possibly send messages as well as observe sent messages, you can use standard cryptographic methods of verifying an identity, including signing and encrypting packets. Signing a message can prevent tampering and may contain some form of authorization. Encryption can prevent third parties from reading the information.

However, UDP messages pose the additional problem of controlling the response address. You can find out the source IP address by examining the EndPoint class passed to EndReceiveFrom, but this source address is not verified by the UDP stack. It is just the claimed source address—an attacker can craft UDP packets that fake this. An attacker can also copy a valid request spotted on the network, change the source address, and send the modified copy. Often, the application will either validate the claimed source address against the message authentication or ignore the claimed source address altogether and pass the reply information in the packet, where tampering is prevented by message signing.

Sometimes you may choose to encrypt a message rather than just sign it. Encryption can use symmetric algorithms such as Data Encryption Standard (DES) or asymmetric algorithms (using public/private key cryptography). Note that for messages of several kilobytes or larger, compressing the message before encrypting it can not only reduce bandwidth, but also reduce encryption time. That is because encryption is a CPU-intensive operation and the time it takes is a function of the number of bytes (or blocks of bytes) to encrypt. If the compression reduced the data size by 75 percent, the encryption stage will be reduced similarly.

Windows Communication Foundation

Windows Communication Foundation (WCF), formerly codenamed "Indigo," provides a new way to exchange messages between distributed applications. WCF can be used to expose Web services, a standardized way to exchange messages and expose services using SOAP and XML, and allow an easier interoperability experience. In the past, Web services were usually exposed over HTTP, but specifications for SOAP over UDP are available at soap-over-udp.pdf.

WCF exposes multiple network protocols (most notably TCP) and is extensible to other network protocols by writing a WCF transport channel. WCF supports UDP through its extensibility model, and the code for a basic UDP channel will be provided as a code sample in the WinFX® SDK.

WCF sends messages as XML representations of data passed between applications. It also provides a programming model that makes exchanging messages between applications easy. In addition to one-way messages, which are the equivalent of sending a UDP packet in the code samples, support is provided for duplex (two-way) and request-reply message exchange.

Although UDP does not provide reliability features out of the box, WCF supports a standard-based reliability protocol through WS-ReliableMessaging. The WS-ReliableMessaging protocol can work with any WCF transport channel and defines a standard method to send and request ACKs, resend a dropped message, and ensure message delivery. WS-ReliableMessaging exposes three different guarantees: AtMostOnce, AtLeastOnce, and InOrder. Using this protocol layer implementation can save you from hand-coding your own reliability layer.

The code to send a message using UDP and WS-ReliableMessaging isn't very complex. I first define an address, binding, and contract. The contract outlines the data that I want passed in the message. I will define a one-way contract, similar to the one implemented previously in this article:

[ServiceContract ]
public interface IUpdatePlayerInfoContract {
    [OperationContract(IsOneWay = true)]
    void Update(byte playerID, byte locationX, byte locationY);
}
You could also choose to implement a request-reply contract by making the Update method return a value and removing the one-way reference.

The binding is the link between the transfer protocol and the data. I have to specify that I want to use the UDP transport, and that I want to use a ReliableMessaging channel.

The WCF-based implementation of UDP is a one-way channel, but I need both a listener and a sender on each side—after all, the receiver (server) needs a return address to be able to send back ACKs. Support for a return address feature in an inherently one-way transport channel is provided by the CompositeDuplex channel, as shown in this code snippet:

BindingElementCollection bindingElements =
    new BindingElementCollection();
ReliableSessionBindingElement session =
    new ReliableSessionBindingElement();
session.Ordered = true;
bindingElements.Add(session);
bindingElements.Add(new CompositeDuplexBindingElement());
bindingElements.Add(new UdpTransportBindingElement());
Binding playerInfoBinding = new CustomBinding(bindingElements);
Had I opted for no reliability, the BindingElementCollection would be comprised of only the UdpTransportBindingElement, but the rest of the code would stay the same.

The address is simply the address the UDP server will listen on. In the case of the UDP transport, this address includes only a machine name and port:

Uri playerInfoAddress = new Uri( "soap.udp://localhost:16000/");

The code on the server should handle a change in the player information, like so:

class playerInfoService : IUpdatePlayerInfoContract
{
    public void Update(byte playerID, byte locationX, byte locationY)
    {
        Console.WriteLine("Player {0}: {1},{2}", playerID,
            locationX, locationY);
    }
}

The server code will also include something like this to set up the endpoint and start listening:

using(ServiceHost service = new ServiceHost(typeof(playerInfoService)))
{
    service.AddServiceEndpoint(typeof(IUpdatePlayerInfoContract),
        playerInfoBinding, playerInfoAddress);
    service.Open();
    ... // more service code here
}
The client would need to define a proxy—a class that will be used to call the server method:
public class PlayerInfoProxy : ClientBase<IUpdatePlayerInfoContract>,
    IUpdatePlayerInfoContract
{
    public PlayerInfoProxy(Binding binding, EndpointAddress address) :
        base(binding, address) {}

    public void Update(byte playerID, byte locationX, byte locationY)
    {
        base.InnerProxy.Update(playerID, locationX, locationY);
    }
}
And with an instance of a proxy, the client can send a message to update the PlayerInfo:
using(PlayerInfoProxy playerInfoProxy = new PlayerInfoProxy(
    playerInfoBinding, new EndpointAddress(playerInfoAddress)))
{
    for (byte i = 0; i < 10; ++i)
    {
        playerInfoProxy.Update(1, i, i);
        Console.WriteLine("Sent");
    }
}

You can also control the ReliableMessage parameters. For example, if message ordering is not important, a more efficient exchange would take place if you set the session's Ordered property to false:

session.Ordered = false;
You can easily configure the channel selection using a configuration file and avoid the code setting bindingElements, allowing an administrator to change the transport selection, the port, address, or even protocol, and replace UDP with TCP without changing the program source.

You can configure security on the exchange in a similar manner; WCF supports authentication either using cryptographic signatures (x.509 certificates) or Windows Authentication. It also allows for validating, signing, and encrypting messages.

If the provided reliability mechanism does not fit your application, you can use WCF extensibility to implement your own reliability protocol on top of UDP by implementing a custom reliability channel. Similarly, you can support other transport protocol and security mechanisms.

Conclusion

In this article, I've discussed both the benefits and challenges of implementing UDP communication in your own apps. You've seen two ways to use UDP in your .NET-based applications. Using the Socket class will be useful if you want to provide a full implementation of the application protocol and do the bookkeeping yourself. Meanwhile, using the Windows Communication Foundation classes will enable you to rely on a standardized implementation, reducing the code you need to write, optimize, and test. Either way, with UDP in your toolbox, you have more options available when designing your next networked application.

Note that the code samples shown here use IPv4, which is the version of IP most commonly used. Converting the sample code to use IPv6 is usually simple, and requires you to specify AddressFamily.InterNetworkV6 instead of AddressFamily.InterNetwork, IPAddress.IPv6Any instead of IPAddress.IPAny, and SocketOptionLevel.IPv6 instead of SocketOptionLevel.IP. You must also replace IPv4 addresses with IPv6 addresses. For simplicity, I skipped most error checking (such as bounds checking) in the code samples. For safety and stability, don't forget these details in your own code.

Yaniv Pessach is a Software Design Engineer at Microsoft working on Windows Communication Foundation Core Messaging. He contributed to the WS-Discovery and SOAP-over-UDP specifications, and is co-author of the upcoming book An Introduction to Windows Communication Foundation (Addison-Wesley, 2006). Reach Yaniv at www.yanivpessach.com.


Wandering Code: Write Mobile Agents In .NET To Roam And Interact On Your Network

Wandering Code

Write Mobile Agents In .NET To Roam And Interact On Your Network

Matt Neely
This article discusses:
  • Introduction to mobile agents
  • Creating a simple mobile agent system
  • Mobile agent development patterns
  • Security implications and solutions
This article uses the following technologies:
.NET Framework
Code download available at: MobileAgents.exe (145KB)
Contents
Introduction to Mobile Agents
A Simple Mobile Agent System
Assembly Resolution
The Traveling Pattern
The Task Pattern
The Interactive Pattern
Advanced Topics
Security and Mobile Agents
Conclusion

R
ecently I had the opportunity to return to school to complete a graduate degree. This experience made me realize two important things: there are some cool ideas in academia that seem to never find the light of day in the professional setting, and the academic world at large is not yet very familiar with the Microsoft® .NET Framework. Thus was born my goal of introducing lesser-known ideas to the professional masses while introducing the .NET Framework to the academic world. One particular concept has become the pivot point of this goal: mobile agents, a distributed computing paradigm.

The term agent originates in artificial intelligence and describes a logical entity that has some level of autonomy within its environment or host. A mobile agent has the added capability to move between hosts. In a computing context, a mobile agent is a combined unit of data and code that can move between different execution environments. This mobility gives mobile agents several potential advantages: reduced network traffic, decentralization, increased robustness and fault-tolerance, and an easy deployment scenario.

In this introduction to mobile agents, I'll create an example mobile agent system and develop several mobile agent applications, highlighting some of the problems that I encountered along the way as well as several ideas for overcoming them. Lastly, I'll touch on some advanced concepts that need to be addressed when creating a serious mobile agent system.

Introduction to Mobile Agents
Figure 1 Traveling Agent Hops Between Machines
Figure 1 Traveling Agent Hops Between Machines

There are three main design patterns for mobile agents which can be combined for any implementation. These patterns are named after the agent's primary purpose: traveling, task, and interaction. A traveling agent is mainly concerned with location. A task agent is used for processing (doing work or tasks). An interaction agent collaborates or interacts with other agents

Figure 2 Task Agent Allocates Tasks
Figure 2 Task Agent Allocates Tasks

An example of a traveling agent app could perform operations control. An agent is sent out with a list of machines on the local network it should traverse to inventory hardware and software (see Figure 1). A task agent application (see Figure 2) could be one that performs massive scientific calculations. An agent can determine what hosts are available on the network and subdivide its tasks accordingly, which essentially creates a load-balancing, parallel-processing application. To understand an interaction agent, think of a marketplace. In this environment, a buying agent can interact with various selling agents to determine the best price for some item (see Figure 3). The buying agent can then purchase from the chosen selling agent.

Figure 3 Interaction Agent
Figure 3 Interaction Agent

So what does .NET have to offer mobile agents? First, having an object-oriented framework makes construction of a mobile agent application much more intuitive. Next, there are built-in services that facilitate the componentization and mobility of code, namely object remoting and serialization. (Sending an object over the wire is a snap with .NET. Name that object Agent and you're close to having a mobile agent application.) In addition, threading and synchronization are crucial, as each agent is supposedly autonomous. And finally, the security features of the .NET Framework will be a great boon to the mobile agent environment. As I'll discuss later, downloading and running unknown code is far from a secure thing to do.

A Simple Mobile Agent System

Let me first say that my particular design of a mobile agent system is not a standard, absolute, or ideal version. It is merely one implementation of many. That disclaimer aside, I began my mobile agent application as a solution in Visual Studio® with three projects: a server process console application, a client process console application, and a common class library to be used by the first two projects. In the class library, I start out with two classes: Agent and AgentHost.

The Agent class is essentially going to be the interface that all agents must adhere to so that the host knows how to run them. Because I want to include some actual common functionality to all agent subclasses, I'm making the Agent class an abstract class and not merely an interface. The Agent class represents a mobile agent and so must have at least two areas of functionality: executing code and moving. As such, I gave the Agent class two brilliantly named methods: Run and Move. The Run method is marked abstract and therefore must be overridden in agent subclasses. Run will be invoked by an AgentHost once an Agent arrives there. The second method, Move, contains the internal communication code for this system, which I'll discuss later.

The AgentHost class (shown in Figure 4) needs only one method to start with to host an Agent class. I've called it simply "HostAgent." It has a single parameter, agent, of type Agent. When this method is invoked, it will call the Run method on the Agent class.

Figure 4 AgentHost Class (Close Figure 4)
public class AgentHost : MarshalByRefObject
{
    public static void Initialize(int port, string objectUri)
    {
        //Create a new TcpChannel on the given port and register it
        // with the runtime
        TcpServerChannel c =
            new TcpServerChannel("MyAgentHostChannel", port);
        ChannelServices.RegisterChannel(c);

        //Register the agent host type with the remoting runtime
        RemotingConfiguration.RegisterWellKnownServiceType(
            typeof(AgentHost), objectUri, WellKnownObjectMode.SingleCall);
    }

    public void HostAgent(Agent agent)
    {
        agent.Run();
    }
}

The communication mechanism that I'm using is .NET remoting. However, it is important to keep in mind that a mobile agent system can be created on top of any communication layer. Now that the Windows Communication Foundation (formerly code-named "Indigo") is available in beta, you might consider using that instead.

In order to implement the remoting capability, I added a method to the AgentHost class called Initialize, which will properly set up a TcpChannel on the given port with the given uniform resource identifier (URI). It will then register the AgentHost type with the remoting runtime so that the host will be externally available.

The client-side code for communication (which resides within the Move method of the Agent class) is even easier, thanks to the Activator type. Activator has a GetObject method overload that takes a URL. This method will return a proxy to an AgentHost object on the server. The Agent uses the proxy to call the HostAgent method, giving itself as the Agent parameter (see Figure 5).

Figure 5 Agent Class (Close Figure 5)
[Serializable]
public abstract class Agent
{
    public event System.EventHandler AgentMoved;

    public Agent() { }

    public abstract void Run();

    public void Move(string urlOfHostToMoveTo)
    {
        AgentHost h = (AgentHost)Activator.GetObject(
            typeof(AgentHost), urlOfHostToMoveTo);
        h.HostAgent(this);
    }
}

You can find the complete code for the server process that will host the AgentHost class in Figure 6. Also, the following code snippet shows the relevant client process code, which references a yet-to-be-discussed agent subclass and sends it to a host at a given remoting endpoint:

MyFirstAgent agent = new MyFirstAgent();
agent.Move("tcp://localhost:10000/MyAgentSample");

Figure 6 Agent Server Process (Close Figure 6)
class AgentServer
{
    static void Main(string[] args)
    {
        int port = 10000;
        if (args.Length > 0) port = int.Parse(args[0]);

        MobileAgents.AgentHost.Initialize(port, "MyAgentSample");
        Console.Out.WriteLine("Press enter to stop...");
        Console.In.ReadLine();
    }
}

Now, the only thing that is missing is an actual Agent implementation. The MyFirstAgent class (see Figure 7) gives a very simple implementation of a mobile agent. On creation, it merely records the name of the process that it was created in. When it runs, it gets the name of the process it now runs in. It outputs the two of them to the console in a formatted string. Note that any Agent subclass should also have the Serializable attribute in order for the runtime to properly serialize it.

Figure 7 MyFirstAgent Class (Close Figure 7)
[Serializable]
class MyFirstAgent : MobileAgents.Agent
{
    string _startingProcess = string.Empty;

    public MyFirstAgent()
    {
        using(Process p = Process.GetCurrentProcess())
            _startingProcess = p.ProcessName;
    }

    public override void Run()
    {
        using(Process p = Process.GetCurrentProcess())
            Console.WriteLine(
                 "I started in '{0}' but now am in '{1}'!",
                _startingProcess, p.ProcessName);
    }
}

In order to run this mobile agent system, you need to start a host with the server process executable. This will establish the system's communication to the outside world so that it can begin to accept mobile agents for hosting. Once the host is ready, the client process is run to create a MyFirstAgent instance and then move it to the host.

Assembly Resolution

If you try to run this seemingly complete example, you get an assembly load exception. The problem is that even though the AgentHost only needs the Agent base class for compilation (and that is in the same assembly), at run time it will also need to access the assembly of the derived base types. Without this information, types can not be correctly laid out in memory and any inheritance functionality would break.

So the problem becomes, "how do I get the client-side, custom-defined agent assembly's bits to the host in order for correct .NET assembly resolution?" This problem can be broken into two parts, assembly transfer and assembly management. Assembly transfer deals with how you get the assemblies that define Agent subclasses into the realm of the hosting AppDomain. The solution depends on the scope of your mobile agent system and what you can assume. For example, your system may be architected such that it knows of a file share, Web, or FTP site from which to download the assemblies. Precluding the existence of a commonly known area for assembly download, the client will have to hand over the bits directly to the server. This is the approach I took in this example as it minimizes any other dependency or deployment headache. However, it also presents some security concerns, to be discussed later on in this article.

To facilitate the transfer of assemblies on the host side, I added two methods to the AgentHost class: IsAssemblyInstalled and UploadAssembly. The first method takes an assembly's full name ("mscorlib, Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089") and returns a Boolean indicating whether the assembly is installed on the system. The second method will actually save the assembly's bits in a unique location for easy loading. These two methods are closely linked to assembly management.

On the client side of the communication channel, I altered the Agent's Move method to gather information about the agent assembly's references (see Figure 8). If the agent assembly is not installed on the host (determined by a call to the host's IsAssemblyInstalled method) then the code will recursively walk the agent assembly's dependency graph and track which assemblies need to be uploaded to the host. For each assembly found, a call to IsAssemblyInstalled is made. The assembly file will be uploaded (via the host's UploadAssembly method) if IsAssemblyInstalled returns false. This process is simple with the combined power of the System.Reflection and System.IO namespaces. (In a robust system, care should be taken to not upload different versions of core .NET runtime files. For example, if the Agent subclass depends on the Visual Studio 2005 runtime, you should probably not try to upload and install Visual Studio 2005 assemblies via this mechanism.)

Figure 8 Updated Agent with Reflection (Close Figure 8)
[Serializable]
public abstract class Agent
{
    public event System.EventHandler AgentMoved;

    public Agent() { }
    public abstract void Run();
    public void Move(string urlOfHostToMoveTo)
    {
        AgentHost h = (AgentHost)Activator.GetObject (
            typeof(AgentHost), urlOfHostToMoveTo);
        TransferAgentAssembliesToHost(h);
        h.HostAgent(this); // move to the host
    }

    private void TransferAgentAssembliesToHost(AgentHost h)
    {
        //Get the full name of the assembly this agent is defined in.
        AssemblyName thisAssemblyName = GetType().Assembly.GetName();

        //If this hasn't been installed yet, walk the
        //assembly dependency graph to see what assemblies are needed
        if (!h.IsAssemblyInstalled(thisAssemblyName.FullName))
        {
            Dictionary<string, AssemblyName> assembliesFound =
                new Dictionary<string, AssemblyName>();
            FindReferencedAssemblies(assembliesFound,
                GetType().Assembly.GetName());
            foreach (AssemblyName an in assembliesFound.Values)
            {
                if (!h.IsAssemblyInstalled(an.FullName))
                    h.UploadAssembly(an.FullName,
                        GetRawAssembly(Assembly.Load(an)));
            }
        }
    }

    private void FindReferencedAssemblies(
        Dictionary<string, AssemblyName> assembliesFound,
        AssemblyName assemblyToSearch)
    {
        //Check to see that we haven't encountered this assembly yet.
        if (!assembliesFound.ContainsKey(assemblyToSearch.FullName))
        {
            assembliesFound.Add(assemblyToSearch.FullName,
                assemblyToSearch);
            AssemblyName[] references = Assembly.Load(
                assemblyToSearch).GetReferencedAssemblies();
            foreach (AssemblyName reference in references)
                FindReferencedAssemblies(assembliesFound, reference);
        }
    }

    ...
}

Another thing to note is that this mechanism does not allow for in-memory assemblies or unreferenced assemblies loaded via reflection. Therefore, avoid such programming constructs in an agent implementation. In fact, I would recommend denying agents the ability to use most reflection APIs for security and robustness reasons. I'll discuss security later in this article.

As mentioned before, assembly management is the one sub-problem of assembly resolution. Now that the AgentHost has been given the bits of the assemblies, what does it do with them? Again, this depends on your implementation. If your system requires that you only accept strong-named assemblies, then you can save the bits to disk and install them into the Global Assembly Cache (GAC). However, there may be circumstances where you do not want to do this. For example, one such circumstance is if you need to use weakly named assemblies or else just want greater control and flexibility. You might also want to totally separate the downloaded assemblies from the rest of the system. Putting an assembly in the GAC makes it globally accessible. This has considerable robustness and security implications.

Another management possibility is to keep the assemblies totally in memory. This can be a benefit to startup performance and security (nothing is directly stored or accessed to disk) but once the Application Domain is unloaded, all assemblies go with it.

If you create your own assembly management infrastructure you will need to find a place to store assemblies. You will also need to be able to resolve failed assembly loads to your assembly store. To find a place to store the assemblies, I chose a directory location that would be unique to my application. The base directory is returned by a call to Environment.SpecialFolders with the Environment.SpecialFolders.CommonApplicationData enum value as the parameter. Subdirectories are then created with the company ("MSDN" in this example), the application name, and the application version. This should ensure that I'll have a unique segment of the file system to call my own.

Now that I have a unique spot for storing assemblies, there's another problem I need to address—the Windows® operating system and the .NET Framework identify uniqueness differently. You can only have one file of any given name in a directory. However, assembly uniqueness has four parts: name, version, culture, and public key. This means that you'll have to store your assemblies in a way that ensures this uniqueness. Since you're essentially creating a local assembly cache, it makes sense to look at what the GAC does. The GAC creates a file structure based around the four uniquely identifying parts of the assembly. I merely copied this structure, as shown in Figure 9. Now each assembly should be located in a unique place that is easily discovered.

Figure 9 File Hierarchy for an Assembly
Figure 9 File Hierarchy for an Assembly

There is still one last caveat, however. If you do allow weakly named assemblies, it is still possible for there to be identification problems. This is because weakly named assemblies do not have a public key and therefore only have three attributes (name, version, culture) to uniquely identify them. Given that the assemblies that the host will be receiving all contain Agent implementations, it is a good assumption that you'll see lots of assemblies named MyAgent.dll or the like, all with the neutral culture and version 1.0.0.0. This name collision (in addition to security issues, discussed later) is a good reason to not support weakly named assemblies in a robust system.

In order to facilitate this assembly management, I created a helper class appropriately called AssemblyManager (see Figure 10). In its static constructor, it hooks into the current AppDomain's AssemblyResolve event. This event will be fired any time the .NET runtime is unable to load an assembly via established methods. See How the Runtime Locates Assemblies for information on how the common language runtime (CLR) resolves assemblies. The ResolveEventArgs class is the second parameter to this method (the first being the AppDomain object that was unable to resolve the assembly). This class has a Name property that gives the full name of the assembly the runtime couldn't find. Using the AssemblyName(string) constructor, I can obtain the four parts I mentioned earlier—name, version, culture, and public key token. Knowing these allows me to construct a path to the given assembly and then load it.

Figure 10 AssemblyManager (Close Figure 10)
public class AssemblyManager
{
    private static string _assemblyStoreBaseDir = string.Empty;

    static AssemblyManager()
    {
        //Create root directory for agent assemblies
        _assemblyStoreBaseDir = Path.Combine(Environment.GetFolderPath(
            Environment.SpecialFolder.CommonApplicationData),
            @"MSDN\MobileAgentsSample\v1.0");
        if (!Directory.Exists(_assemblyStoreBaseDir))
            Directory.CreateDirectory(_assemblyStoreBaseDir);

        //Hook AppDomain for assembly resolution failures
        AppDomain.CurrentDomain.AssemblyResolve +=
            new ResolveEventHandler(CurrentDomain_AssemblyResolve);
    }

    private static Assembly CurrentDomain_AssemblyResolve(
        object sender, ResolveEventArgs args)
    {
        string fileName, dirPath;
        GetDirectoryAndFileForAssembly(args.Name,
            out dirPath, out fileName);
        string fullPath = Path.Combine(dirPath, fileName);
        if (File.Exists(fullPath)) return Assembly.LoadFrom(fullPath);
        return null;
    }

    private static void GetDirectoryAndFileForAssembly(
        string assemblyFullName, out string directoryName,
        out string fileName)
    {
        fileName = string.Empty;
        directoryName = string.Empty;

        //Parse the assembly name
        string name, version, culture, publicKeyToken;
        ParseAssemblyFullName(assemblyFullName, out name, out version,
            out culture, out publicKeyToken);

        fileName = name + ".dll";
        string relPath = string.Format(@"{0}\{1}\{2}\{3}", name,
            culture, version, publicKeyToken);
        directoryName = Path.Combine(_assemblyStoreBaseDir, relPath);
    }

    private static void ParseAssemblyFullName(string assemblyFullName,
        out string name, out string version, out string culture,
        out string publicKeyToken)
    {
        name = version = culture = publicKeyToken = string.Empty;
        AssemblyName an = new AssemblyName(assemblyFullName);
        name = an.Name;
        version = an.Version.ToString();
        culture = an.CultureInfo.NativeName;
        publicKeyToken = ConvertBytesToHexString(an.GetPublicKeyToken());
    }

    private static string ConvertBytesToHexString(byte[] bits)
    {
        StringBuilder sb = new StringBuilder(bits.Length*2);
        foreach (byte b in bits) sb.AppendFormat("{0:X2}", b);
        return sb.ToString();
    }

    internal static void SaveAssemblyBits(string assemblyFullName,
        byte[] assemblyBits)
    {
        //Get the directory this assembly should go in.
        string fileName, dirPath;
        GetDirectoryAndFileForAssembly(assemblyFullName, out dirPath,
            out fileName);

        if (!Directory.Exists(dirPath))
            Directory.CreateDirectory(dirPath);

        //Create the assembly file if it doesn't already exist.
        string filePath = Path.Combine(dirPath, fileName);
        if (!File.Exists(filePath))
        {
            using(FileStream fs = File.OpenWrite(filePath))
                fs.Write(assemblyBits, 0, assemblyBits.Length);
        }
    }

    internal static bool IsAssemblyInstalled(string assemblyFullName)
    {
        try
        {
            AssemblyName an = new AssemblyName(assemblyFullName);
            Assembly asm = Assembly.Load(an);
            if (asm != null) return true;
        }
        catch (FileNotFoundException) { }
        return false;
    }
}

The AssemblyManager class also has two internal methods that are called by the AgentHost class to facilitate assembly transfer: IsAssemblyInstalled and SaveAssemblyBits. The IsAssemblyInstalled method will simply attempt to load the assembly given the full name passed in as the single parameter. If a failure occurs, the assembly must not be installed yet. The second method will save the assembly's bits to the appropriate location by again parsing the full name of the assembly.

The Traveling Pattern

To exemplify a traveling agent, I wrote a simple scenario where an agent would have an itinerary and would go to each host listed in it. Once at each host, it will merely output a message on the host process' console saying that it has arrived (in lieu of actual work) and then move to the next host. The implementation is straight- forward. I create a class MyTravelingAgent (see Figure 11) derived from Agent. I give it a private System.Collections.Queue<string> class, _destinations, for its itinerary. I then add another method, AddDestination, which will add a URL to the queue. The implementation of the Run override method merely outputs to the console, sleeps a bit, dequeues the next URL from the queue, and then moves there.

Figure 11 MyTravelingAgent (Close Figure 11)
[Serializable]
class MyTravelingAgent : MobileAgents.Agent
{
    Queue<string> _destinations = new Queue<string>();
    public void AddDestination(string url) { _destinations.Enqueue(url); }

    protected override void Run()
    {
        Console.WriteLine("I'm here now: {0}", DateTime.Now);
        System.Threading.Thread.Sleep(5000);
        if (_destinations.Count > 0)
        {
            string nextDestination = _destinations.Dequeue();
            this.Move(nextDestination);
        }
    }
}

When an agent moves to a new location, it has a synchronous connection to the remote host and won't relinquish it until the Agent is moved and finishes executing. In a traveling agent pattern, part of the execution involves moving to another host to execute there. This operation also performs synchronously. Thus, the originating call to Agent.Move waits for the traveling agent to move to its final destination and then finishes executing. Needless to say, this is not a good design. One could easily imagine a traveling agent that has no final destination but has a continual route that it travels performing useful work.

Although there are ways to address this via Remoting, I want a protocol-agnostic solution. The System.Threading namespace offers an easy workaround in the form of ThreadPool. I'll use the ThreadPool.QueueUserWorkItem static method to forward the incoming Agent to another thread which will then execute it. This action allows the caller to return immediately:

public void HostAgent(Agent agent)
{
    ThreadPool.QueueUserWorkItem(this.RunAgent, agent);
}

private void RunAgent(object context)
{
    ((Agent)context).Run();
}

The Task Pattern

The task pattern-based agent application will use a type of task pattern known as the parent-child pattern. In this scenario, one agent is the parent or controller agent and it spawns or communicates with child agents that do its bidding. My implementation of this pattern merely creates a parent agent and tells it on which hosts to perform work. When run, the parent agent will create a child agent for each host to visit and then send the child agent there. Each child agent will then perform some valuable work, like enforcing strong access control lists (ACLs) on all file shares or clearing the Recycle Bin. My child agent will merely list the system's logical drives (with a call to Directory.GetLogicalDrives) and again output the results to the host's console (see Figure 12).

Figure 12 Creating Parent and Child Agents (Close Figure 12)
[Serializable]
class MyParentTaskAgent : MobileAgents.Agent
{
    List<string> _hostsToCheck = new List<string>();

    public void AddHost(string url) { _hostsToCheck.Add(url); }

    protected override void Run()
    {
        foreach (string hostUrl in _hostsToCheck)
        {
            AgentProxy proxy = Agent.CreateAgent(
                typeof(MyChildTaskAgent));
            proxy.Move(hostUrl);
        }
    }
}

[Serializable]
class MyChildTaskAgent : MobileAgents.Agent
{
    protected override void Run()
    {
        foreach (string drive in Directory.GetLogicalDrives())
        {
            Console.WriteLine("Found drive: '{0}'", drive);
        }
    }
}

The Interactive Pattern

The interactive agent application consists of selling and buying agents. Sellers will set up shop on a host. The buying agent will then visit the shops that it knows about, looking for the best deal for some particular item. For this application I created two Agent subclasses: MySellingAgent and MyBuyingAgent.

MySellingAgent is not too complex. When a selling agent gets to a host to set up shop, it will register itself with a "registry," so to speak. This registry is really a static dictionary that will store the agent's ID (a simple integer in this scenario) mapping to the selling agent instance itself. MySellingAgent has a static dictionary, _sellers, a static constructor that initializes the dictionary, and two public static methods for buyers to find seller instances: GetSellersOnHost and GetSellerById. The first will return an array of MySellingAgent objects, the second will return a single instance of a MySellingAgent object.Generic Interaction Mechanisms

I'm sure that many of you see another shortcoming of the interactive buyer-seller scenario. That is, the buyer needs compile-time knowledge of the seller. Unless you code all agents involved in an interaction, you can't control this. If you can't control it, a generic mechanism needs to exist to allow for discovery and use of other agents' services. The AgentProxy class I described gets a step closer as you can invoke methods without compile-time knowledge. However, there is no discovery mechanism to see what services or functionality an agent provides.

Although it is fun to reinvent the wheel, you can take advantage of some protocols that currently exist for this functionality, for example, Universal Description, Discovery, and Integration (UDDI), Web Services Description Language (WSDL), and SOAP (see www.w3c.org for descriptions of these protocols). I can envision a world where each host acts as a UDDI server giving agents the ability to find other agents to use their services and communicate. Once an agent is found, WSDL can be used to dynamically discover the communication protocols and SOAP can be used for the actual communication. This essentially makes these mobile agents mobile Web services. That is a powerful, albeit complex, paradigm.

On the instance level, the selling agent has an ID (again, merely an integer) and a dictionary of items for sale. The dictionary maps the item's name to an internal class ItemForSale, which holds the item's name and price:

[Serializable]
public class ItemForSale
{
    public ItemForSale(string name, int price)
    {
        this.ItemName = name;
        this.ItemPrice = price;
    }

    public string ItemName = string.Empty;
    public int ItemPrice = 0;
}

The selling agent's instance constructor merely copies the parameters to member variables. Three public methods exist for interaction with a buyer: IsItemForSale, GetItemPrice, and BuyItem. In the overridden Run method, the seller registers itself on the host by adding itself to the static _sellers dictionary. It then waits to be called by a buying agent.

MyBuyingAgent, shown in Figure 13, is a bit more complex as it needs to maintain more state. The buying agent is given the name of an item to buy, a value for the maximum amount it can spend for that item, and an array of host URLs that the agent needs to visit in order to find the item at the lowest price.

Figure 13 MyBuyingAgent (Close Figure 13)
[Serializable]
class MyBuyingAgent : MobileAgents.Agent
{
    ...

    public MyBuyingAgent(
        string itemToBuy, int maximumPriceForItem, string[] urlsToVisit)
    {
        this._itemToBuy = itemToBuy;
        this._maxPriceForItem = maximumPriceForItem;
        foreach (string url in urlsToVisit) _sitesToVisit.Enqueue(url);
    }

    protected override void Run()
    {
        Console.WriteLine("Buying agent is here.");
        System.Threading.Thread.Sleep(2000);

        if (this._currentHost == this._winnerHost) BuyFromWinner();
        else
        {
            FindItem();
            if (_sitesToVisit.Count == 0)
                this.GoToMarketplace(this._winnerHost);
            else
            {
                string nextHost = this._sitesToVisit.Dequeue();
                this.GoToMarketplace(nextHost);
            }
        }
    }

    private void BuyFromWinner()
    {
        MySellingAgent seller =
            MySellingAgent.GetSellerById(this._winnerId);
        if (seller != null)
        {
            seller.BuyItem(this._itemToBuy);
            Console.WriteLine(
                "We bought '{0}' from seller {1} at {2} for ${3}!",
                _itemToBuy, _winnerId, _winnerHost, _winnerPrice);
        }
    }

    private void FindItem()
    {
        MySellingAgent[] sellers = MySellingAgent.GetSellersOnHost();
        foreach (MySellingAgent seller in sellers)
        {
            if (seller.IsItemForSale(this._itemToBuy))
            {
                int sellersPrice = seller.GetItemPrice(this._itemToBuy);
                if (sellersPrice <= this._maxPriceForItem)
                {
                    if (_winnerId == -1 || sellersPrice < _winnerPrice)
                    {
                        this._winnerHost = this._currentHost;
                        this._winnerId = seller.Id;
                        this._winnerPrice = sellersPrice;
                    }
                }
            }
        }
    }

    public void GoToMarketplace(string hostUrl)
    {
        this._currentHost = hostUrl;
        this.Move(this._currentHost);
    }
}

As the buying agent travels, it will need to enumerate each seller on each host, determine if that seller has the item desired, and then find the item's price. Since the agent wants to find the best price, it will have to track which seller it is going to buy from. I call this the winning seller. The agent will have to store the location of the winning seller, the ID of the winning seller, and the price the winning seller is offering for the desired item.

Finally, the buying agent will need one more piece of information—its current location. Since there is currently no facility in the AgentHost to give this information, I'll have to cheat. Instead of directly calling the base class's Move method directly, I created a wrapper function called GoToMarketplace. This will set a member variable to the URL of the host that it is about to move to and then it will call Move:

public void GoToMarketplace(string hostUrl)
{
    this._currentHost = hostUrl;
    this.Move(this._currentHost);
}
When the agent appears at the destination host, it will have its current host member set to where it is located.

The logic in the overridden Run method is as follows. If the current host is the same as the winning host, you have already determined that you are going to buy from a seller here. If so, get that seller by calling the static MySellingAgent.GetSellerById method. The object instance returned can now be invoked, calling its BuyItem method. If the current host is not the winning host, the agent will try looking for it on the local host. After this I check for any more host URLs to move to. If there are none, then the agent will move to the winning host to purchase the item. If there are more host URLs, the agent simply moves to the next host and continues.

Advanced Topics

Developers creating a mobile agent system need to manage local agent object references. As I'm sure some of you have noticed, merely calling the Move method doesn't do anything to the local object instance. It still exists. It can be invoked and run without the knowledge or consent of its clone that you just sent across the network. Just keeping a reference to the object will keep it in memory and prevent it from being garbage collected. In effect, the supposedly autonomous agent is at the mercy of anyone who has a reference to it.

Most agent implementations (especially in managed environments like that provided by the .NET Framework) use an agent proxy and never give out direct references to the agents themselves. A possible way to create such a proxy system is to implement an agent factory pattern. Instead of returning an agent, it will return a proxy to the agent. By using a basic eventing mechanism, all proxies to an agent can be notified when that agent moves. Once notified, they can ensure that no one can use the agent from that point on. Furthermore, the class implementing the proxy can use a WeakReference object to allow for greater agent object independence. (See WeakReference Class for a description of the WeakReference class.) Figure 14 shows the simple AgentProxy implementation based on the code download available from the MSDN®Magazine Web site. The creation and use of an agent would then look like this:

AgentProxy proxy = Agent.CreateAgent(typeof(MyTravelingAgent));
proxy.InvokeMethod("AddDestination",
    "tcp://localhost:10000/MyAgentSample");
proxy.InvokeMethod("AddDestination",
    "tcp://localhost:10002/MyAgentSample");
proxy.Move("tcp://localhost:10001/MyAgentSample");

Figure 14 AgentProxy Class (Close Figure 14)
[Serializable]
public class AgentProxy
{
    private WeakReference _agent = null;
    private bool _agentLeft = false;

    internal AgentProxy(Agent agentToProxy)
    {
        _agent = new WeakReference(agentToProxy);
        agentToProxy.AgentMoved += agentToProxy_AgentMoved;
    }

    public void Move(string hostUrl)
    {
        if (_agent.IsAlive && !_agentLeft)
        {
            Agent agent = (Agent)_agent.Target;
            agent.Move(hostUrl);
        }
        else throw new Exception("The agent no longer exists here.");
    }

    public object InvokeMethod(string methodName,
        params object[] methodArguments)
    {
        if (_agent.IsAlive && !_agentLeft)
        {
            Agent a = (Agent)_agent.Target;
            Type agentType = a.GetType();
            object retVal = agentType.InvokeMember(methodName,
                BindingFlags.Public | BindingFlags.InvokeMethod |
                BindingFlags.Instance, null, a, methodArguments);
            return retVal;
        }
        else throw new Exception("The agent no longer exists here.");
    }

    private void agentToProxy_AgentMoved(object sender, EventArgs e)
    {
        _agentLeft = true;
        _agent.Target = null;
    }
}

For bonus points, some implementations of agent systems also use proxies as a location-agnostic way of interfacing with agents. With this design, if some object has a proxy to agent A on machine X and A moves to machine Y, the user of the proxy doesn't care. It can call to agent A's proxy as before and the proxy takes care of all the details, making sure that A (now on Y) gets the message.

Unfortunately, this object reference problem is not merely due to external handles to the agent. One of the shortcomings of building an agent system on the .NET Framework is the inability to stop thread execution, store stack frames and registers, and then truly resume execution on the destination host. Because of this, there still exists the potential for an agent to misuse itself. If an agent makes a call to this.Move but then continues processing, it is violating the intention for one agent to exist in only one place at one time. Therefore, agent implementers need to think of a call to this.Move as if it is the final word in processing (exempting object cleanup, of course).

Agents should also have some internal mechanism to make sure that during race conditions they are not moved twice by accident. This goes mostly for the host side. By giving each agent instance a unique identifier, a host can add rules to make sure agents are not moved to it multiple times.

Another topic for advanced systems is the support given by the host. As seen in the buying agent example, there was no means for an agent to know where it was unless I engineered an ugly solution. Shouldn't the host itself provide this information? What services should the host give executing agents? Maybe the host will allocate file or database space. Perhaps the host will provide persistence services for long-running agents. The more services a host provides, the more interaction between agent and host, the more complexities arise. A production system will need more careful design than the samples given here.

The code samples available in the download provide a simple context class, HostContext, that each agent can utilize by simply accessing the static property HostContext.Current. The HostContext in the code exposes only the current host location, but other services can be added here to give agents a more functional playground in which to execute. (For details on using the AgentProxy class to invoke methods without compile-time knowledge, see the sidebar "Generic Interaction Mechanisms.")

Security and Mobile Agents

When I describe the mobile agent paradigm to people, their first reaction is usually that it sounds like a virus. Surely, allowing the download, installation, and execution of unknown code has significant security implications. A thorough understanding of security issues and a comprehensive threat model should be a minimum requirement in any production system.

I'll briefly discuss some different threats unique to a mobile agent system and how .NET facilities can help mitigate the risks. First though, I must make the disclaimer that the threats described here may or may not apply (nor are the threats I describe the only ones that exist), based on the particular function of the mobile agent system designed. For example, the threats for a publicly available interactive mobile agent system greatly differ from an internal, task-based mobile agent system. Don't forget that threat model. To determine the threats inherent to a mobile agent system, you need to look at the protagonists. As such, the system needs to be secure from malicious clients, agents, and hosts. (The downloadable code implements several security enhancements.)

The first possible threat is a malicious client attempting to harm an established host. Here I am defining a client as any program that communicates with a host via its outward-facing interface. A lot of the mitigation for this threat depends on your communication protocol. Will you allow anonymous calls? Will you authenticate? How? Securing the communication protocol is beyond the scope of this article, but it is very important and should be thoroughly analyzed during the design phase. I recommend a protocol that can both authenticate and encrypt the requests. Doing so will hinder many types of attacks on the host. This is where a framework like that provided by Windows Communication Foundation comes in extremely handy.

As for the host itself, a good rule of thumb in any security design is to reduce the attack surface, so be very careful of what functionality you choose to expose to the outside world (see Attack Surface: Mitigate Security Risks by Minimizing the Code You Expose to Untrusted Users for more information about attack surface reduction). The .NET Framework 2.0 has a new facility called Transparency, which can be used in external-facing code to always run with the caller's permissions (see Are You in the Know? Find Out What's New with Code Access Security in the .NET Framework 2.0).

The AgentHost class presented here has three externally visible methods: IsAssemblyInstalled, UploadAssembly, and HostAgent. Although it might seem innocuous, IsAssemblyInstalled could pose a security risk if it gives information away to a potential hacker. For example, if there is a known security issue in a particular version of a core .NET assembly, malicious code merely has to query the host to see if that assembly (and thus, vulnerability) is available for attack. Furthermore, the current implementation attempts to see if an assembly exists by loading it. If an assembly that poses a security risk is loaded, it is loaded into the host's application domain. So much for isolation. The code download contains a slightly different solution where the host creates a separate AppDomain for the sole purpose of loading assemblies during the call to IsAssemblyInstalled.

UploadAssembly needs caution because it is how a malicious user would upload malicious code onto the host machine. In the mobile agent system this article presents, the assembly bits are saved to disk at a known location and then loaded later if assembly resolution fails. Because the assemblies are not loaded directly into memory, you can delay the problem of uploading malicious agents. That is, you can delay this problem only until the call to HostAgent. However, once an assembly is uploaded and saved to storage, it is possible to inspect it for security purposes. Here you can enforce such rules as requiring a strong name. Strong-named assemblies help reduce risk in several ways, such as proving that the assembly hasn't be tampered with post build.

Of course, you should only trust the builder within certain limitations—you shouldn't trust an assembly just because it has a strong name. Finally, remember that assemblies that are loaded from a local hard disk are full-trust assemblies by default. I'll address that in a moment.

The HostAgent method is where execution of the uploaded agent code will begin, which brings me to a discussion of agent-host threats. When an agent is to be hosted for the first time, the CLR looks for the agent's assembly. Because I store the assemblies in an obscure location, default assembly resolution will fail. It is then up to the AppDomain.AssemblyResolve event handler to load the appropriate assembly. It would be a very good idea to place those assemblies in a sandboxed environment before you load and use them.

In the code download, I have bolstered the security of the system by creating separate application domains to separate agents from the host domain and other agents. I chose to allow the agent to specify the name of the application it is a part of. Each application will have its own app domain. This architecture is more suitable for interactive agent scenarios where agents from different sources can mingle more freely. You can segregate agents however you want. Your scenario may prompt you to launch an application domain for each agent type or even each agent instance that is hosted. The built-in separation given with the AppDomain construct allows you to unload these assemblies if you either determine they are a threat or if activity wanes and you need to reclaim some memory.

For creating an AppDomain, the .NET Framework 2.0 has some new APIs that allow you to specify the code access security (CAS) policy that you desire. New AppDomain.CreateDomain overloads allow you to pass in the default Evidence, PermissionSet, and the names of the full-trust assemblies for that new application domain. If you choose to go this route, I recommend you read Shawn Farkas's blog at blogs.msdn.com/shawnfa on creating custom AppDomainManagers and HostSecurityManager subclasses as well as his recent MSDN Magazine article concerning hosting add-ins at Do You Trust It? Discover Techniques for Safely Hosting Untrusted Add-Ins with the .NET Framework 2.0.

In the download code, I took another approach. My approach alters the CAS policy for the local machine. Using the SecurityManager class, I added a new CodeGroup with a restricted permission set (derived from the Internet-named permission set, then altered). The membership condition for this CodeGroup is a URL condition that points to the base file location where the assemblies were saved. This is set up and checked every time an agent AppDomain is created. And the best part is that the settings are now machine-wide. If any other process loads these assemblies they will have the same restrictive permission set. (See Code Access Security for more information on CAS in the CLR.)

Finally, I'll touch on the threat of a malicious host towards your agent. If you send data in the form of an agent to a host, that host has full control and use of the agent. So I'd recommend not storing things like credit card or social security numbers in them. Since you can't control what a malicious host might do to your agent, the best course of action would be to prevent your agent from going to such a host. Again, your communication protocol could take this on with mutual authentication. Remember, security is based on trust. If you don't trust a host, stay away from it.

Conclusion

There are many ways to engineer a networked application. Mobile agents have their uses and their pros and cons. The autonomous and mobile nature of mobile agents can lead to reduced network traffic, decentralization, increased robustness and fault-tolerance, and easy deployment. Now that you know a bit of what mobile agents can do, let's look at these advantages.

A mobile agent system has the capability of reducing network traffic by taking the processing local to the resource being used. If you are doing a text search of files across the network, you can dramatically improve performance by doing the search locally and then returning only the results.

A mobile agent system can be decentralized for increased robustness. Since the mobile agent contains code, you can implant into it any fault-tolerance and error mechanisms you want. Think of searching text files across the network again. If the machine that originally dispatched the mobile agents to the remote hosts crashes, processing can continue. The mobile agents can even camp out on remote hosts until they can determine that it is safe to return. It is also possible to survive a denial-of-service attack this way.

Deployment is made easier since mobile agents essentially deploy themselves. The only prerequisite is that an agent host is installed and configured on that machine. Once that condition is met, you're open for business.

Unfortunately, mobile agents do have a downside. They present a paradigm that is different from other development paradigms and can be more difficult to comprehend. Also, security is a much bigger issue with the download and execution of code. Search for more information about mobile agents on MSN® and look at the various implementations that currently exist. Then look at how many even attempt to tackle the issue of security—most don't. Downloading and executing unknown code is not a good way to control security.

I believe that mobile agents are a very useful paradigm that should receive more attention than they have been getting in the realm of academia. The features that are provided by the .NET Framework should allow mobile agents to proliferate to the full extent of their capabilities.

Matt Neely is a Software Design Engineer at Microsoft where he spends most of his time looking busy when the manager walks by. He enjoys playing the guitar and spending time with his wife, four children, three cats, and a puppy.


WSE Security: Protect Your Web Services Through The Extensible Policy Framework In WSE 3.0

WSE Security

Protect Your Web Services Through The Extensible Policy Framework In WSE 3.0

Tomasz Janczuk
This article discusses:
  • Policy framework architecture in WSE 3.0
  • Using the policy framework to secure Web services
  • Wire format of secure SOAP messages
  • Writing custom extensions to the policy framework
This article uses the following technologies:
Web Services Enhancements 3.0
Code download available at: WSE.exe (119KB)
Contents
WSE Policy Framework
The Policy Object Model
Applying Policy
Security Assertions
Username/Password Authentication
Certificate Authentication
Kerberos Authentication
Adding Replay Detection
Conclusion

W
eb Services Enhancements (WSE) 3.0 is an add-on to Visual Studio® 2005 and the Microsoft® .NET Framework 2.0. It provides advanced Web services capabilities, helping to keep pace with the evolving Web services protocol specifications.

The WSE policy framework provides a mechanism to describe the constraints and requirements that need to be enforced for a Web service. In this article I will describe how the policy framework works in WSE. I will discuss security scenarios supported by WSE 3.0, particularly the wire formats of secure SOAP messages generated and consumed by WSE. I will then explore the mechanisms that can be used to extend the framework, allowing you to apply and enforce custom constraints and requirements. My example code will use this capability to add a replay detection feature to protect recipients of SOAP messages.

WSE Policy Framework

The WSE policy framework describes constraints and requirements for communicating with an ASMX (ASP.NET Web services framework) or WSE Web service. Policy is a deployment-time concept. An instance of a policy is translated into run-time components that apply the requirements at the sender or enforce them at the receiver. This process of translation, referred to as policy compilation, takes place before any messages are exchanged between the client and the service.

The atomic run-time components are called SOAP filters and operate at the level of a single SOAP envelope. Their primary responsibility is to inspect or transform the SOAP envelope. For example, a SOAP filter on the sender may digitally sign the message and add a Security header to it, as defined in the WS-Security specification. Similarly, a SOAP filter on the receiver may extract the Security header from the SOAP envelope and require that the digital signature it contains be successfully verified before dispatching the message to the application code.

A single policy contains an ordered list of policy assertions. Each policy assertion defines a single requirement of a Web service. Two corresponding run-time components enforce these requirements. In Figure 1, the policy and policy assertion deployment-time concepts correspond to the pipeline and SOAP filter run-time concepts, respectively. For example, the ordering of requirements in a policy allows an assertion that describes authentication requirements to take effect prior to an assertion that describes authorization requirements—this way the identity of the caller, which is determined in the authentication process, can be used during authorization enforcement. The order of policy assertions within a policy controls the order in which related requirements will be applied or enforced at run time. Figure 1 shows how policy compilation preserves the ordering of requirements expressed as policy assertions in the order of SOAP filters in the pipeline.

Figure 1 Policy Implemented Through Assertion of SOAP Filters
Figure 1 Policy Implemented Through Assertion of SOAP Filters

During the policy compilation process, each policy assertion generates up to four SOAP filters. Each SOAP filter executes at a different stage of the request/response message exchange. Two distinct SOAP filters process the input and output messages on the client, and two filters process the input and output messages on the service. SOAP filters generated by policy assertions follow the order of assertions in the policy. This collection of input and output filters is represented by a pipeline.

Even though a single policy assertion may create up to four SOAP filters, not all requirements expressed as policy assertions need all four hook points in the request/response message pattern. For example, a replay detection requirement would only need two SOAP filters for incoming messages on the client and service. No special processing of outgoing messages is necessary in this case.

The Policy Object Model

A look at the object model behind the policy framework helps to illustrate these concepts. The PolicyAssertion abstract base class declares four methods for creating SoapFilters. These methods execute to process a request or response on the client or service. At run time, two of these methods will be called (which two will depend on whether a policy assertion is applied to the client or the service) to contribute SoapFilters to the pipeline. Any of these methods may return null, which indicates the assertion does not require any processing at that point in the message exchange.

public abstract class PolicyAssertion{
    public abstract SoapFilter CreateClientInputFilter(
        FilterCreationContext context);
    public abstract SoapFilter CreateClientOutputFilter(
        FilterCreationContext context);
    public abstract SoapFilter CreateServiceInputFilter(
        FilterCreationContext context);
    public abstract SoapFilter CreateServiceOutputFilter(
        FilterCreationContext context);
    ...
}

SoapFilter is a run-time class responsible for an atomic transformation of a SOAP message. Its contract is quite straightforward: the only method that must be implemented accepts a SoapEnvelope to be inspected or transformed and returns a SoapFilterResult. The SoapFilterResult indicates whether the pipeline should either continue processing or terminate. This termination feature is only useful in advanced scenarios involving handling of infrastructure messages (for example, WS-Reliability protocol messages), that should not be dispatched directly to the Web service class:

public abstract class SoapFilter
{
    public abstract SoapFilterResult ProcessMessage(
        SoapEnvelope envelope);
    ...
}

A PolicyAssertion is ordered into an instance of the Policy class to capture the desired set of requirements for a service or a proxy. The Policy can then act as a factory for creating client or service pipelines, as shown in the following:

public class Policy : IPipelineProvider
{
    public Collection<PolicyAssertion> Assertions { get; }
    public Policy(params PolicyAssertion[] assertions) { }
    public Pipeline CreateClientPipeline(
        PipelineCreationContext context) { }
    public Pipeline CreateServicePipeline(
        PipelineCreationContext context) { }
}
The Pipeline class represents an ordered collection of SoapFilter objects for processing incoming or outgoing messages. As such, Pipeline can be used to customize the run-time behavior of a service or service proxy.

Applying Policy

Policies in WSE can be defined imperatively in code or declaratively in an external XML file. Both approaches address specific scenarios. When defining policy imperatively in code, you take full control over defining the requirements that need to be satisfied in order to use the service. When the compiled application is distributed, the administrator cannot change these requirements.

Declaring policies in an XML file addresses the opposite scenario: the developer only provides the application logic in code and delegates the responsibility of specifying policy requirements to the administrator. This approach makes more sense when the policy requirements depend on the deployment environment and cannot be decided upon at development time. For example, the same Web service can be exposed for users of a Windows domain and the Internet. In the Windows domain, the security requirements could use Kerberos. Meanwhile, on the Internet, username/password and X.509 certificates can be used.

WSE defines a PolicyAttribute, which can be added to a class representing the ASMX Web service or a service proxy to apply a specific policy:

[WebService]
[Policy("HelloWorldPolicy")]
public class HelloWorld : WebService
{
    [WebMethod]
    public string Hello(string request)
    {
        return String.Format("Hello, {0}", request);
    }
}
In this example, a policy named HelloWorldPolicy is applied to the HelloWorld Web service. When the service class is instantiated for the first time in an application domain, the WSE runtime will resolve this policy name to a Policy instance declared in an XML file referenced in the configuration file. Afterwards, the Policy instance will be compiled into a pipeline with SoapFilters for incoming and outgoing messages. All SOAP requests and responses directed to this ASMX Web service will be processed through this pipeline. PolicyAttribute is not supported at the Web method scope.

The name of the XML file containing policy declarations is provided in the configuration file as shown in the following:

<configuration>
    <microsoft.web.services3>
        <policy fileName="policies.config"/>
    </microsoft.web.services3>
</configuration>
The schema of the XML file containing the policies allows you to declare any number of named policies. This enables different services that are running in the same application domain (and therefore sharing a single configuration file) to use different policies. For example, this policies.config file contains definitions of two policies—HelloWorldPolicy and Policy17, as shown here:
<policies xmlns="http://schemas.microsoft.com/wse/2005/06/policy">
    <policy name="HelloWorldPolicy">
        <usernameForCertificateSecurity/>
    </policy>
    <policy name="Policy17">
        <kerberosSecurity/>
    </policy>
</policies>
Both policies have only one policy assertion each—one is using a usernameForCertificateSecurity assertion and the other a kerberosSecurity assertion. The assertions are built into WSE 3.0 and offer additional configuration options.

PolicyAttribute can also be used to apply policy to a service proxy on the client side. This can be done by adding the attribute on the class representing the ASMX proxy. However, the ASMX proxy class is typically autogenerated using the Add Web Reference feature of the Visual Studio environment or the wsdl.exe tool. PolicyAttribute would therefore have to be added to this class each time it is regenerated, which is not convenient. Thus, the recommended way to assign policy to a client proxy is to call the SetPolicy method on it as follows:

HelloWorld serviceProxy = new HelloWorld();
serviceProxy.SetPolicy("HelloWorldPolicy");

SetPolicy is provided by the WebServicesClientProtocol class, which the ASMX proxy class (HelloWorld in this example) must derive from. WSE integrates closely with the Add Web Reference feature of the Visual Studio environment to ensure all newly generated ASMX proxy classes derive from WebServicesClientProtocol for WSE-enabled projects.

Referencing policy by name in your code is the best approach when you want to leave the decision about the specifics of the policy to the administrator. However, if you want to control the policy entirely in code, you can achieve this in two ways: you can indicate a common language runtime (CLR) type of the policy class using PolicyAttribute, or you can provide an instance of a Policy object using the SetPolicy method.

In Figure 2, the MyPolicy class derives from a Policy base class and populates the Assertions collection with an instance of KerberosAssertion in the constructor, thereby creating a fully specified policy in code. Next, PolicyAttribute is applied to the ASMX service class to indicate the CLR type of a class complying with the Policy contract. This class will be instantiated by the WSE runtime by the default constructor when the service instance is created. The resulting Policy instance is compiled into a pipeline. Figure 3 summarizes recommended and available methods of setting policy on the ASMX and SoapClient/SoapService services and proxies.

Figure 3 Methods for Setting Policy (Close Figure 3)

Service/Proxy | PolicyAttribute with Policy Name | PolicyAttribute with Policy CLR Type | SetPolicy Method with Policy Name | SetPolicy Method with Policy Instance
ASMX Service |  |  |  |
ASMX Proxy |  |  |  |
SoapSender |  |  |  |
SoapReceiver |  |  |  |
Recommended [image] Available [image] Not Applicable
Figure 2 Specifying a Policy Type (Close Figure 2)
class MyPolicy : Policy
{
    public MyPolicy() : base()
    {
        this.Assertions.Add(new KerberosAssertion());
    }
}

[WebService]
[Policy(typeof(MyPolicy))]
public class HelloWorld : WebService
{
    [WebMethod]
    public string Hello(string request) {
    {
        return String.Format("Hello, {0}", request);
    }
}

Security Assertions

The policy framework in WSE 3.0 provides six ready-to-use assertions that address common scenarios, helping to secure Web services. In general, these assertions use WS-Security and WS-SecureConversation mechanisms to provide message integrity and confidentiality as well as authentication. Figure 4 summarizes the features offered by the six assertions.

Figure 4 Security Assertions in WSE 3.0 (Close Figure 4)

Policy Assertion | Integrity | Confidentiality | Secure Conversation | Client Authentication | Service Authentication
UsernameForCertificateAssertion |  |  |  | Username/password | X.509
MutualCertificate10Assertion |  |  |  | X.509 | X.509
MutualCertificate11Assertion |  |  |  | X.509 | X.509
AnonymousForCertificateAssertion |  |  |  |  | X.509
UsernameOverTransportAssertion |  |  |  | Username/password |
KerberosAssertion |  |  |  | Kerberos |

Using any of the six security assertions offered by WSE facilitates future interoperability with Windows Communication Foundation. Currently, messages secured using any of the WSE 3.0 assertions are wire-level compatible with the Windows Communication Foundation Beta 2 release. If you are going to write a WSE client application that needs to call a service written on another platform, or your WSE Web service will be consumed by clients created on other platforms, it is important to understand the features of the WS-* specifications that these assertions use, as well as the actual layout of elements within the Security SOAP header.

Except for UsernameOverTransportAssertion, all security assertions support integrity, confidentiality, and secure conversation. Integrity features allow the recipient of a message to check whether the message was modified in transit. This is achieved by digitally signing the message and the recipient validating this signature. The WS-Security standard requires that the digital signature follow the XML Digital Signature standard. This standard supports signing of selected parts of an XML document (a SOAP envelope in this case). Security assertions in WSE allow you to decide which parts of the SOAP envelope need to be signed. By default, all WS-Addressing SOAP headers, the SOAP body containing the application payload, and the timestamp within the Security header are signed by the sender and required to be signed by the recipient.

Confidentiality features allow the sender of the message to ensure that the content of the message will only be available to the intended recipient. This is achieved by encrypting the message. WS-Security requires that encryption of the message follow the XML Encryption standard, which supports encrypting only selected elements of an XML document. Standard assertions in WSE only support encryption of the body of the SOAP envelope, which is the most common scenario. This is also the default behavior if you decide not to leave the body in plain text.

All assertions in WSE except UsernameOverTransportAssertion allow you to customize the parts of the message that are signed and to decide whether the body of the message should be encrypted. The APIs for all assertions are the same because these features are handled by a common base SecurityPolicyAssertion class.

These protection requirements will be uniformly applied to all Web methods of a service after the policy containing this assertion is applied to the service class. More advanced scenarios may require that protection requirements differ between operations exposed by a service. The two most common scenarios are related to performance and interoperability. WSE allows you to differentiate the protection requirements for individual operations handled by a single policy. For this purpose, operations of a service are uniquely identified using the value of the Action header of the request message as defined in the WS-Addressing specification. (The WSE SDK contains a sample that demonstrates how to use the first child element of the SOAP body as a key on which to pivot your protection requirements.) For example, if you want to encrypt the body of the SOAP message for all operations except one identified by the uri:hello action, you could write code like this:

KerberosAssertion assertion = new KerberosAssertion();
assertion.Protection.DefaultOperation.Request.EncryptBody = true;
assertion.Protection.DefaultOperation.Response.EncryptBody = true;
OperationProtectionRequirements operationRequirements =
    new OperationProtectionRequirements();
operationRequirements.Request.EncryptBody = false;
operationRequirements.Response.EncryptBody = false;
assertion.Protection.Operations["uri:hello"] = operationRequirements;

Whenever you add a custom SOAP header to the message, you may want to define protection requirements for it with a policy. WSE security assertions support specifying integrity requirements for custom SOAP headers. If your header has namespace uri:example and local name myHeader, and is only present on requests, you can require it to be signed with the following code:

assertion.Protection.DefaultOperation.Request.CustomSignedHeaders.Add(
    new XmlQualifiedName("myHeader", "uri:example"));
You can control all of these integrity and confidentiality requirements by using the external XML policy file. For example, the policy file shown in Figure 5 defines a policy that uniformly signs and encrypts all operations except for those with a request action equal to uri:hello.

Figure 5 Signing Operations that Don't Request uri:hello (Close Figure 5)
<policies>
  <policy name="HelloWorldPolicy">
    <kerberosSecurity>
      <protection>
        <request encryptBody="True" />
        <response encryptBody="True" />
        <fault encryptBody="False" />
      </protection>
      <protection requestAction="uri:hello">
        <request signatureOptions="IncludeAddressing, IncludeSoapBody"
                 encryptBody="False">
          <signedHeader name="myHeader" namespace="uri:example"/>
        </request>
        <response signatureOptions="IncludeAddressing, IncludeSoapBody"
                  encryptBody="False" />
        <fault signatureOptions="IncludeNone " encryptBody="False" />
      </protection>
    </kerberosSecurity>
  </policy>
</policies>

All security assertions in WSE except for UsernameOverTransportAssertion provide support for establishing secure conversations following the WS-SecureConversation 1.1 specification. The idea behind a secure conversation is that a session is established between the client and the service. The handshake protocol defined by WS-SecureConversation is secured using credentials specific to the selected security assertion. For example, in the case of UsernameForCertificateAssertion, the client authenticates to the service during the handshake using username and password credentials, and the service authenticates to the client using its X.509 certificate. At the end of the handshake, the service issues a SecurityContextToken to the client. This token contains a symmetric key that is later used by both the client and service when exchanging subsequent messages.

The use of a symmetric cryptographic key over multiple message exchanges is, in most cases, more efficient than using the credentials that were used to establish the secure conversation. If you plan to use a single client proxy instance for multiple calls to the service and performance is important, then enabling secure conversation in the policy will be beneficial.

Establishing a secure conversation is handled entirely by the pipeline generated by the security assertions. The application code does not have to deal with the handshake messages defined by WS-SecureConversation at any point. Therefore, enabling secure conversation in a security assertion is as easy as simply turning it on:

KerberosAssertion assertion = new KerberosAssertion();
assertion.EstablishSecurityContext = true;
The security context token established between the client and the service is associated with the service proxy instance and is used for all messages sent on that proxy. However, the token may expire during the lifetime of the proxy. In that case, the proxy will, by default, throw an exception when you try to send new requests. (If it didn't, the service would likely reject the message anyway.)

There is another feature of security assertions in WSE that allows secure conversations to be reestablished once they expire, which provides a seamless experience for the user of the proxy. However, the new secure conversation bears no cryptographic relationship to the original one. (In the process of reestablishing the secure conversation, the security context token is not renewed; a new one is obtained.) Since the application utilizing the proxy may depend on the session semantics of the secure conversation, the reestablishment behavior is turned off by default. To enable it, all you need to do is set one more property on the assertion instance:

assertion.RenewExpiredSecurityContext = true;
The settings controlling the secure conversation behavior are fully accessible from the XML policy file as well:
<policies>
  <policy name="HelloWorldPolicy">
    <kerberosSecurity establishSecurityContext="True"
                      renewExpiredSecurityContext="True" />
  </policy>
</policies>

Username/Password Authentication

WSE provides two assertions that support client authentication using a username/password credential: UsernameForCertificateAssertion and UsernameOverTransportAssertion. UsernameForCertificateAssertion offers both client and server authentication using WS-Security protocols. UsernameOverTransportAssertion only provides client authentication using WS-Security, and relies on a secure transport (such as HTTPS) to provide message protection and server authentication.

When using UsernameForCertificateAssertion, the service must be configured with an X.509 certificate and its associated private key. The client needs to be configured with its own username/password credential as well as with the X.509 certificate of the service (public key only). It is assumed that the client knows the X.509 certificate of the service out of band. Note that WSE does not support obtaining a service's certificate at run time using protocols like Transport Layer Security (TLS) and secure sockets layer (SSL).

Configuration of the client or server credentials can be done declaratively in the XML policy file:

<usernameForCertificateSecurity>
  <clientToken>
    <username username="tomasz" password="zsamot"/>
  </clientToken>
  <serviceToken>
    <x509 storeLocation="LocalMachine" storeName="My"
          findValue="CN=example.com"
          findType="FindBySubjectDistinguishedName" />
  </serviceToken>
</usernameForCertificateSecurity>
The service's certificate is referenced from the certificate store, and the username/password credential is directly embedded in the XML policy file. Please note the client's username/password credential is only required on the client side. Even though it is possible, you should avoid storing passwords in an XML file. I recommend that you don't include the clientToken element in the policy configuration file at all, but instead use the imperative method of providing the username/password credential in the client code by calling the SetClientCredential method on a proxy:
string username = ..., password = ...;
HelloWorld helloWorld = new HelloWorld();
helloWorld.SetClientCredential(new UsernameToken(username, password));
This allows you to query the user of your application for credentials at run time and pass them on to the proxy, without the need to store them in an external XML file. You can also provide X.509 credentials in code using a similar mechanism, which may come in handy if you want to use the same proxy class and policy to talk to services with different X.509 certificates:
X509Certificate2 serviceCertificate;
helloWorld.SetServiceCredential(
    new X509SecurityToken(serviceCertificate));

If you provide both client and service credentials in code, your policy can be reduced to the following:

<policies>
  <policy name="HelloWorldPolicy">
    <usernameForCertificateSecurity/>
  </policy>
</policies>
So how does the UsernameForCertificateAssertion secure the message exchange? Figure 6 shows a sample layout of the Security header for the request and response message.

Figure 6 Messages Secured UsernameForCertificateAssertion
Figure 6 Messages Secured UsernameForCertificateAssertion

To secure the request message, WSE creates a random symmetric key, wraps this key using the public key from the service's X.509 certificate, and uses this symmetric key to sign and encrypt the parts of the message that require integrity and confidentiality. The symmetric key is serialized into the Security header in the form of an xenc:EncryptedKey element. The reference list inside this element points to the xenc:EncryptedData element containing a username token representing the client's credentials. This way the client's password does not appear in the clear on the wire. The last element in the Security header of the request is dsig:Signature, which signs all message parts that require integrity, the timestamp, and the plain text wsse:UsernameToken element encrypted within xenc:EncryptedData.

Note that the service's X.509 certificate does not appear in the Security header of the request message. It is not required since both the client and the service have access to it. Instead, an external reference describing the properties of this certificate is used.

The response message uses the symmetric key passed by the client to the service in the request to sign and encrypt the message. The Security header of the response contains the timestamp, an xenc:ReferenceList pointing to the encrypted message parts, and a signature signing the timestamp and parts of the message that require integrity protection.

UsernameForCertificateAssertion verifies that a response is signed and encrypted with the same symmetric key that was used on the request. This proves that the response was sent by the owner of the X.509 certificate for which the symmetric key was encrypted in the request, since only the holder of the associated private key could decrypt the symmetric key. This mechanism allows the assertion to provide server authentication assurances on the client.

The wire format may be different depending on the integrity and confidentiality requirements as well as the assertion configuration. In particular, the layout will differ if you change the order in which the signature and encryption is applied to the message. By default, the message is first signed and then encrypted. Using the MessageProtectionOrder property, you can also configure the assertion to first encrypt and then sign the message, or sign the message, encrypt it, and then encrypt the message signature.

Another configurable aspect of the assertion is related to key derivation. By default, derived keys are not used, but you can change this using the RequireDerivedKeys property. When this feature is turned on, each time a message is signed or encrypted using a symmetric key, a derived key is first computed based on this root symmetric key, and the signature or encryption uses the newly derived key. In that case, you would see two additional elements in the Security header representing the derived key tokens used for signatures and encryption.

One last feature supported by security assertions in WSE is signature confirmation as defined in WS-Security 1.1. By default this behavior is turned off, but you can use the RequireSignatureonfirmation property on the assertion to require that the recipient of the request confirms all signatures from the request in the response. In this case, you would see additional wsse:SignatureConfirmation elements in the Security header of the response.

The second assertion in WSE that offers client authentication using username/password credentials is UsernameOverTransportAssertion. This assertion relies on a secure transport protocol like HTTPS to provide integrity, confidentiality, and server authentication assurances. WS-Security features are only used to carry the username/password credentials from the client to the service in the Security header of the SOAP message. WSE does not enforce that the transport is actually secure when this assertion is applied, so you must exercise extra caution when using it. If the transport is not secure, the password will appear in plain text on the wire.

Configuration of UsernameOverTransportAssertion is minimal. On the server, no credential configuration is necessary, as server authentication is not performed at the SOAP level. Your XML policy file can be as simple as the following:

<policies>
  <policy name="HelloWorldPolicy">
    <usernameForCertificateSecurity/>
  </policy>
</policies>
On the client side, you have the option of specifying the client's username/password credential in the XML policy file or in code. Again, for security reasons it is recommended you provide the credentials in code. Since server authentication is handled at the transport layer, there is no need to provide the service's X.509 certificate on the client side. Given all that, your client-side XML policy can look identical to the one on the server side.

Note that, when UsernameOverTransportAssertion is applied, the username and associated password are sent over the wire in plain text within the Security header of the request message. It is therefore crucial that confidentiality is provided for the entire message at the transport level.

Certificate Authentication

WSE provides three security assertions supporting authentication based on X.509 certificates. AnonymousForCertificateAssertion allows the client to remain anonymous while authenticating the service using its X.509 certificate. MutualCertificate10Assertion and MutualCertificate11Assertion both perform mutual authentication using the client and service's X.509 certificates. The difference between them is in the WS-Security specification features they use. MutualCertificate10Assertion does not require any WS-Security 1.1 features and is therefore compliant with a broader range of interoperability targets. MutualCertificate-11Assertion uses the WS-Security 1.1 feature of referencing the xenc:EncryptedKey element as a token to achieve better performance in securing request/response exchanges. However, this requires both communicating endpoints to support the newer WS-Security specification.

To use AnonymousForCertificateAssertion, both the client and service must be configured with the service's X.509 certificate. This certificate can be provided either through the XML policy file or through code. The mechanism is identical to what I already described for UsernameForCertificateAssertion. The wire format, shown in Figure 7, is even simpler.

Figure 7 Messages Secured with AnonymousForCertificateAssertion
Figure 7 Messages Secured with AnonymousForCertificateAssertion

Notice that the layout of the Security header of the request message is similar to that of UsernameForCertificateAssertion and the layout of the Security header of the response message is identical. What's missing in the request's Security header is an xenc:EncryptedData element containing the client's username/password credentials in the form of wsse:UsernameToken. This way the client remains anonymous, while the service is authenticated with its X.509 certificate, and both request and response can be signed and encrypted using the encrypted key token. Because the response message contains a reference to the encrypted key token from the request, AnonymousForCertificateAssertion therefore requires that both of the communicating parties support the WS-Security 1.1 specification.

MutualCertificate11Assertion builds on AnonymousForCertificateAssertion by adding client authentication using an X.509 certificate. The client's certificate can be provided in the XML policy file (see Figure 8).

Figure 8 Client Certificate Specified by Policy (Close Figure 8)
<mutualCertificate11Security>
  <clientToken>
    <x509 storeLocation="CurrentUser" storeName="My"
          findValue="CN=client.com"
          findType="FindBySubjectDistinguishedName" />
  </clientToken>
  <serviceToken>
    <x509 storeLocation="LocalMachine" storeName="TrustedPeople"
          findValue="CN=service.com"
          findType="FindBySubjectDistinguishedName" />
  </serviceToken>
</mutualCertificate11Security>

You can also omit the specification of the client's certificate in the XML file and provide it directly in code using the same SetClientCredential method I described earlier:

X509Certificate2 clientCertificate = ...;
HelloWorld helloWorld = new HelloWorld();
helloWorld.SetClientCredential(
    new X509SecurityToken(clientCertificate));
Figure 9 shows the effect of applying MutualCertificate11Assertion to the exchange. The response message is identical to that generated by AnonymousForCertificateAssertion, which requires support for WS-Security 1.1.

Figure 9 Messages Secured with MutualCertificate11Assertion
Figure 9 Messages Secured with MutualCertificate11Assertion

Compared with AnonymousForCertificateAssertion, there are two additional elements when using MutualCertificate11Assertion: a binary security token representing client's X.509 certificate, and a second signature that uses this token to sign the primary message signature (the signature that signs all the parts of the message that require integrity). The effect of this second signature is to operate as if the client's certificate was used to sign the same parts of the message the primary signature signs. This is possible because the serialized form of the primary signature follows the XML Digital Signature specification, which means it already contains digested values representing the message parts that require integrity protection. The second signature will therefore be called an endorsing signature. Computing the endorsing signature is typically less expensive than computing the primary signature simply because the XML representing the primary signature is often smaller than the XML of all the message parts it covers.

MutualCertificate10Assertion achieves the same authentication and message protection effect as MutualCertificate11Assertion. The difference is that it can do its job using only WS-Security 1.0 features. The benefit is a greater number of products this assertion will interoperate with. This comes at a cost of degraded performance: securing the full request/response exchange requires four public key cryptographic operations as opposed to the three required by MutualCertificate11Assertion.

MutualCertificate10Assertion has the same configuration requirements as MutualCertificate11Assertion. But notice that the wire format shown in Figure 10 is very different.

Figure 10 Messages Secured with MutualCertificate10Assertion
Figure 10 Messages Secured with MutualCertificate10Assertion

The request contains a binary security token with the client's X.509 certificate. This token is used directly to sign all the parts of the message that require integrity, including the timestamp. Encryption is done the same way as in AnonymousForCertificateAssertion and MutualCertificate11Assertion: using a symmetric key token encrypted for the service's X.509 certificate.

The response message follows a pattern similar to the request. The message is signed with the service's X.509 certificate and encrypted with a symmetric key token encrypted for the public key of the client's X.509 certificate. However, the response does not contain a binary security token with the service's X.509 certificate. This is because the client already has this certificate (it was used to help encrypt the request), so it is sufficient to provide an external reference to it.

Kerberos Authentication

When both the client and service are within the same Kerberos trust domain (for example, they are joined to the same Windows domain), Kerberos authentication can be used. It does not require as much deployment effort as any of the certificate-based authentication assertions supported by WSE, since certificates do not need to be distributed to all participants. In addition, Kerberos is based on symmetric cryptography, which is faster than public key cryptography. WSE provides KerberosAssertion to help you secure message exchanges with Kerberos tickets.

KerberosAssertion does not require any configuration on the service side. On the client, the Service Principal Name (SPN) of the targeted service needs to be specified. The SPN is used by the client to obtain a Kerberos ticket for the target service. You can specify the SPN in the XML policy file like this:

<policies>
  <policy name="HelloWorldPolicy">
    <kerberosSecurity>
      <token>
        <kerberos targetPrincipal="HOST/myServer"/>
      </token>
    </kerberosSecurity>
  </policy>
</policies>
Alternatively, you can provide the KerberosToken directly to your proxy in code using the SetClientCredential method:
HelloWorld helloWorld = new HelloWorld();
helloWorld.SetClientCredential(
    new KerberosToken("HOST/myServer"));

There is one limitation of using the SetClientCredential method with Kerberos tokens—the Windows operating system prevents a given instance of a Kerberos ticket from being used more than once in order to prevent replay attacks. This means before making each call, you have to set a new instance of a KerberosToken on the proxy using the SetClientCredential method as shown previously. Alternatively, if you specify your entire policy in code, you can set a Kerberos token provider on the KerberosAssertion directly. The token provider acts as a factory of KerberosTokens. Whenever the WSE runtime needs to secure a new request, it calls into this provider to ask for a fresh instance of a KerberosToken. This is how you could write code to set the Kerberos token provider:

KerberosAssertion assertion = new KerberosAssertion();
assertion.KerberosTokenProvider =
    new KerberosTokenProvider("HOST/myServer");
HelloWorld helloWorld = new HelloWorld();
helloWorld.SetPolicy(new Policy(assertion));
Figure 11 shows how a message exchange is secured with a KerberosAssertion.

Figure 11 Messages Secured with KerberosAssertion
Figure 11 Messages Secured with KerberosAssertion

Adding Replay Detection

Security assertions in WSE provide support for integrity, confidentiality, and client and service authentication. But these assertions do not mitigate a replay attack where an attacker intercepts an acceptable message sent by a valid client and then sends a copy of that message to the service at a later time. Such attacks are typically mitigated with replay detection, where the recipient can detect and reject duplicate messages. A replay detection feature can be added to WSE by building on the extensibility capabilities of the policy framework.

You first need to make some design decisions. How are you going to determine message uniqueness? Obviously you need to compare some parts of incoming messages. The parts of the SOAP envelope for which you have ensured uniqueness between messages should be signed by an authenticated sender of the message; otherwise, an attacker can circumvent your replay comparison check by modifying the messages, which would pass undetected on the receiver. Ideally, you would be able to compare all the parts that were signed by the sender. However, this is impractical because XML comparison is computationally expensive, and you would need to store substantial sections of incoming messages in order to prevent future replay attacks.

However, if you can assume the parts you care about are digitally signed with the XML Digital Signature required by the WS-Security specification, you have a very concise part of the SOAP envelope that carries enough entropy (or randomness) for replay detection purposes: the signature value itself.

Using the signature value has three advantages: it carries enough entropy to represent the parts of the message the recipient cares about, it's cheap and easy to compare, and it's compact to store. An ideal implementation of a replay detection mechanism would leverage signature value for that purpose. However, since I want to highlight the extensibility capabilities of the WSE policy framework, I will take an even simpler route by using MessageId, a WS-Addressing SOAP header that is recommended to carry a globally unique Uniform Resource Identifier (URI) identifying the message. Every self-respecting service will require this header to be signed, so it contributes to the entropy of the message signature value.

The replay detection mechanism needs to store the value of the MessageId header of past messages in order to compare them to new incoming messages. This process would obviously lead to ever increasing memory allocation. You could solve this problem by introducing control over the memory consumption and the assurances provided by the replay detection mechanism. Let's assume you want to limit the maximum number of past messages our replay detection mechanism will remember, therefore putting a cap on memory allocation. At the same time you can introduce control over the minimum quality of service, which in this case means the minimum time span during which replayed messages are guaranteed to be detected. Finally, you must consider the desired behavior of the system when a new message arrives at the time the memory consumption is at its limit. To live up to the quality of service assurances and the memory quota at the same time, you don't have many options other than rejecting this message. The core of the replay detection mechanism is implemented by a custom SoapFilter (see Figure 12).

Figure 12 Replay Detection Based on wsa:MessageId (Close Figure 12)
class ReplayDetectionSoapFilter : SoapFilter
{
    Dictionary<string, DateTime> replayCache =
        new Dictionary<string, DateTime>();
    int maximumCacheSize = 60 * 60 * 10; // 60 msg/sec for 10 minutes
    TimeSpan minimumReplayProtectionTime = new TimeSpan(0, 10, 0);

    public override SoapFilterResult ProcessMessage(SoapEnvelope envelope)
    {
        if (envelope == null) throw new ArgumentNullException("envelope");

        string uniqueMessageIdentifier =
            GetUniqueMessageIdentifier(envelope);
        lock (replayCache)
        {
            EnsureMessageIdentifierUniqueness(uniqueMessageIdentifier);
            AddMessageIdentifierToCache(uniqueMessageIdentifier);
        }
        return SoapFilterResult.Continue;
    }

    string GetUniqueMessageIdentifier(SoapEnvelope envelope)
    {
        return envelope.Context.Addressing.MessageID.Value.ToString();
    }

    void EnsureMessageIdentifierUniqueness(string identifier)
    {
        if (replayCache.ContainsKey(identifier))
        {
            replayCache[identifier] = DateTime.Now;
            throw new InvalidOperationException(
                «A duplicate message was received»);
        }
    }

    void AddMessageIdentifierToCache(string identifier)
    {
        if (replayCache.Count >= maximumCacheSize)
        {
            // purge the cache from entries we no longer need to keep
            DateTime retireTime = DateTime.Now.Subtract(
                minimumReplayProtectionTime);
            List<string> entriesToRetire = new List<string>();
            foreach (KeyValuePair<string, DateTime> entry in replayCache)
            {
                if (entry.Value < retireTime)
                    entriesToRetire.Add(entry.Key);
            }
            foreach (string key in entriesToRetire)
            {
                replayCache.Remove(key);
            }

            // if the cache size is still over quota, can't process
            if (this.replayCache.Count >= this.maximumCacheSize)
                throw new InvalidOperationException(
                    «Replay detection cache is full»);
        }

        replayCache[identifier] = DateTime.Now;
    }
}

The ReplayDetectionSoapFilter constructor assumes default values for the maximum cache size for MessageId header values as well as the minimum duration for which replays are guaranteed to be detected. The data structure to keep track of MessageId values is a dictionary that maps the MessageId value to the time the message with that value was last received.

The core of the implementation is in the ProcessMessage method. It first extracts the MessageId value from the incoming message. Then the EnsureMessageIdentifierUniqueness method is called to make sure another message with the same MessageId hasn't been registered within the minimum replay detection time span. If the new message happens to have a MessageId value that matches one already in the cache, the code assumes it is a replayed message and rejects it by throwing an appropriate exception from the EnsureMessageIdentifierUniqueness method. Otherwise, AddMessageIdentifierToCache is called to store the MessageId value along with the current time in the cache.

If the cache size has reached its quota, the application first tries to purge the cache by removing all entries not required to maintain quality of service as determined by the minimum replay detection time span. If the cache is still full after the attempted purging, the code will throw an exception because it cannot guarantee the quality of service and keep the memory consumption within the quota at the exact same time.

Pay close attention to the locking around the operations on the cache, which takes place in the ProcessMessage method. The ProcessMessage method on the SoapFilter must be designed to be thread safe because it may be called from multiple threads to process different messages.

The next step is to develop a policy assertion that will create the ReplayDetectionSoapFilter and allow this functionality to be added to a policy. Figure 13 shows the minimal implementation of ReplayDetectionPolicyAssertion. Note that ReplayDetectionSoapFilter is only returned to process incoming messages on the service. Theoretically, replay detection also makes sense when processing incoming messages on the client, but for the simplicity of this example, I have decided to provide this feature only on the service side.

Figure 13 Implementation of ReplayDetectionPolicyAssertion (Close Figure 13)
public class ReplayDetectionPolicyAssertion : PolicyAssertion
{
    public override SoapFilter
        CreateClientInputFilter(FilterCreationContext context)
    {
        return null;
    }

    public override SoapFilter
        CreateClientOutputFilter(FilterCreationContext context)
    {
        return null;
    }

    public override SoapFilter
        CreateServiceInputFilter(FilterCreationContext context)
    {
        return new ReplayDetectionSoapFilter();
    }

    public override SoapFilter
        CreateServiceOutputFilter(FilterCreationContext context)
    {
        return null;
    }
}

ReplayDetectionPolicyAssertion is now ready to be used imperatively. This is how you add this assertion to a policy and install it on a service:

class MyPolicy : Policy {
    public MyPolicy() : base() {
        this.Assertions.Add(new KerberosAssertion());
        this.Assertions.Add(new ReplayDetectionPolicyAsertion());
    }
}

[WebService]
[Policy(typeof(MyPolicy))]
public class HelloWorld : WebService {
    ...
}
Note how the ReplayDetectionPolicyAssertion is installed as a second assertion in MyPolicy, after KerberosAssertion. If you look back at Figure 1, it will become clear that such ordering will cause ReplayDetectionSoapFilter to execute before the filter created by KerberosAssertion when processing incoming messages on the service. In this case it makes perfect sense: if the message appears to be replayed, you want to reject it before investing time and memory in validating security.

To make the ReplayDetectionPolicyAssertion usable declaratively when specifying policy in the XML file, you need to add serialization capability. PolicyAssertion defines a three-method contract that each serializable policy assertion needs to follow. The three methods are ReadXml, WriteXml, and GetExtensions. Their implementation is shown in Figure 14.

Figure 14 ReplayDetectionPolicyAssertion (Close Figure 14)
public class ReplayDetectionPolicyAssertion : PolicyAssertion
{
    public override IEnumerable<KeyValuePair<string, Type>>
        GetExtensions()
    {
        return new KeyValuePair<string, Type>[] {
            new KeyValuePair<string, Type>(
                "replayDetection",
                typeof(ReplayDetectionPolicyAssertion)) };
    }

    public override void WriteXml(XmlWriter writer)
    {
        if (writer == null) throw new ArgumentNullException("writer");

        writer.WriteStartElement("replayDetection");
        writer.WriteEndElement();
    }

    public override void ReadXml(XmlReader reader,
        IDictionary<string, Type> extensions)
    {
        if (reader == null) throw new ArgumentNullException("reader");

        bool isEmpty = reader.IsEmptyElement;
        reader.ReadStartElement("replayDetection");
        if (!isEmpty) reader.ReadEndElement();
    }

    ...
}

The ReadXml and WriteXml methods are responsible for reading the policy assertion from and writing it to the XML policy file, respectively. The concept of an extension is used to provide a mapping between a local element name in the XML policy file with a CLR type representing this element. When a policy specified in an XML file contains a custom assertion (such as ReplayDetectionPolicyAssertion), this assertion type must be declared as an extension to the policy framework in a special extensions section at the beginning of the file, like this:

<policies>
  <extensions>
    <extension name="replayDetection"
       type="WSE3ReplayDetection.ReplayDetectionPolicyAssertion,
             WSE3ReplayDetection" />
  </extensions>
  <policy name="HelloWorldPolicy">
    <kerberosSecurity/>
    <replayDetection/>
  </policy>
</policies>

The extensions section may contain many extension elements, each of which defines and associates a local element name with a fully qualified CLR type name. When the policy parser comes across one of the names declared in the extensions section whenever an assertion is expected, an instance of the associated CLR type is created and the ReadXml method is called on it.

The extensions section can also be leveraged when your custom assertion is designed to expose an extensibility point. One deficiency of the ReplayDetectionPolicyAssertion described in this article is that the cache of MessageId values representing past messages is held in memory. When the application domain is recycled or a new pipeline instance is created in any other way, this state is lost. Holding the replay detection state in memory also does not allow you to detect replays across server farm nodes.

One way to solve this problem is to make the implementation of the replay detection cache an extensibility point. In scenarios that require persistency, an implementation can use a database instead of memory. Once you have a CLR type that defines a contract for such an extensibility point, you need to expose a property of this type on the ReplayDetectionPolicyAssertion as well as provide serialization and deserialization code for its parameters (such as a database connection string). The assertion deserialization code only requires that a class complies with the contract, but is not interested in its configuration details. The extensions mechanism can be used to register a mapping between a specific local element name and a CLR type implementing this contract, therefore providing a uniform mechanism for capturing extensibility points in the XML policy file. I have provided a full implementation of the extensible ReplayDetectionPolicyAssertion with the code download for this article.

Conclusion

As you've seen in this article, the policy framework in WSE 3.0 provides a general mechanism for transforming SOAP messages sent to and from Web services. Extensions to the policy framework implemented by WSE itself offer support for a broad range of security scenarios. Using the security policy assertion implementations included in WSE 3.0 can help you create apps that will interoperate with services created using Windows Communication Foundation. The policy framework is extensible and allows custom features to be implemented when processing SOAP messages while taking advantage of imperative and declarative policy specifications.

Tomasz Janczuk has worked on security for the Windows Communication Foundation since 2001. He also works on development for WSE 3.0. Tomasz's focus has been on SOAP-level security protocols and standards, most notably WS-SecurityPolicy 1.0 and 1.1.


Smart Clients: Build A Windows Forms Control To Consume And Render WSRP Portlets

Smart Clients

Build A Windows Forms Control To Consume And Render WSRP Portlets

Carl Nolan
This article discusses:
  • Understanding WSRP interactions
  • Building a WSRP viewer control
  • Security considerations for WSRP clients
This article uses the following technologies:
.NET Framework, Web Services, Windows Forms
Code download available at: WSRP.exe (317KB)
Contents
The WSRP Makeup
WSRP Viewer Architecture
Establishing Relationships and Capabilities
Obtaining and Rendering Markup
Processing Interactions
State Persistence and Destruction of Relationships
Viewer Security
Wrap-Up

T
oday's organizations need sites that deliver interactive functionality and data in a secure environment. Building them could translate into a lot of development time, if you were to start from scratch. The portlet—a form of distributed Web component—simplifies this task.

When working with portlets, well-defined interfaces are essential; they allow you to easily integrate portlets into Web applications. The Web Services for Remote Portlets (WSRP) standard defines such an interface, allowing the aggregation of remote, interactive Web service-based presentation components. WSRP provides the means for interactive content to be produced and then consumed by portal applications, with minimal programming effort. The portal then acts as an intermediary between the user and the aggregated services provided by the producer.

WSRP represents a noble goal, but there are often better alternatives. Web services that pass around data rather than UI constructs are more flexible, and Visual Studio®, in concert with the Microsoft® .NET Framework, makes quick work of consuming such services and supplying quality UI. That said, if you need to work with a service that is already providing interactive Web components as WSRP portlets, you can incorporate these portlets seamlessly into a smart client application. This is appealing because smart client applications use local resources, provide a rich client experience, and support intelligent install and update mechanisms. Web services offer powerful interoperability and application integration features. Combining the two makes it possible to develop integrated applications that can incorporate data from disconnected sources.

To really understand portlets it helps to take a look at a Windows® Forms control that implements the WSRP specifications—consuming third-party portlets exposed as WSRP compliant Web services. In this article, I will walk you through the implementation of just such a control and the classes used to implement the WSRP standard (the code for this control is available from the MSDN®Magazine Web site).

The WSRP Makeup

WSRP defines a set of APIs that allows applications to utilize remote interactive Web services in the form of portlets. These APIs are built on top of the existing SOAP, WSDL, and UDDI Web services technologies.

The WSRP specification describes the conversation between producers and consumers on behalf of the users. These producers are the presentation-oriented Web services that host portlets, rendering the markup and processing user interactions. The consumers are the applications that integrate these Web services, presenting markup to users and managing user interaction.

WSRP provides a set of standard operations that enable a consumer to request valid markup fragments (such as HTML) from a producer based on an existing state. Users can then interact with the markup, while the state is preserved across these calls. Figure 1 and Figure 2 show the key operations and types that will be consumed by my WSRP viewer control and service class. This is not a definitive set, but represents the key elements required.

Figure 2 WSRP Types (Close Figure 2)

Type | Description
ServiceDescription | The return type from a getServiceDescription call, containing the service metadata.
RegistrationContext | Type used to pass the registration state and handle to the WSRP producer. Also the return type from a WSRP register request.
PortletDescription | Type describing the portlets. Array of offered portlets contained within the ServiceDescription type.
RegistrationData | Type used to pass the fields needed by the WSRP producer to return the registration context.
UserContext | Type used to pass the user information to the WSRP producer.
SessionContext | Type used to pass the session information to the WSRP producer.
RuntimeContext | Type used to pass the runtime information to the WSRP producer.
PortletContext | Type used to pass the portlet information to the WSRP producer. Includes the portlet handle and interactive state.
MarkupParams | Type used to pass the set of fields needed by the WSRP producer to generate markup for the user.
MarkupResponse | Type containing the fields returned from a WSRP getMarkup request.
MarkupContext | Type containing the fields from a markup invocation, including the actual markup. Contained within the MarkupResponse type.
InteractionParams | Type used to pass the set of fields needed by the WSRP producer when performing a user interaction.
BlockingInteractionResponse | Type containing the fields returned from a WSRP performBlockingInteraction request.
UpdateResponse | Type containing the fields from an interaction invocation. Contained within the BlockingInteractionReponse type.
UploadContext | Type containing the fields specific to uploading data to the portlet.
Figure 1 WSRP Operations (Close Figure 1)

Operation | Description
getServiceDescription | Acquires the WSRP producer's metadata.
Register | Establishes a relationship with a WSRP producer.
modifyRegistration | Modifies a relationship with a WSRP producer.
initCookie | Allows the consumer to assist the WSRP producer in creating cookies.
clonePortlet | Allows the consumer to request the creation of a new portlet from an existing one.
getMarkup | Requests the markup for rendering the current state of a portlet.
performBlockingInteraction | Allows for the potential modification of the state of a portlet. Initiated through an end user interaction.
Deregister | Destroys a relationship with a WSRP producer.
releaseSessions | Informs the WSRP producer that a set of sessions are no longer required.
destroyPortlets | Allows the destruction of portlets no longer required by the consumer.

When the WSRP viewer control is displayed, the user can select from a list of available portlets through the toolbar. This interaction creates relationships with the WSRP producer and determines its capabilities. Next, the markup, which contains action links that the user can interact with, is acquired and rendered. When the user interacts with the action links, a blocking interaction takes place. This modifies the state of the portlet. Then new markup is acquired and rendered. Finally, once the user interaction is complete, all previous relationships are terminated. The interaction flow of my smart client WSRP viewer is illustrated in Figure 3.

Figure 3 Interaction Flow in a WSRP Viewer (Close Figure 3)

WSRP Viewer Architecture

Before delving into WSRP, I'll outline the principles that have guided my implementation. I set out to create a Windows Forms control that would encapsulate the interaction with a WSRP producer. This would include all the required visual elements that allow the user to interact with the WSRP portlets. As part of this effort, I've created a reusable WSRP class library, which is consumed by the Windows Forms control to encapsulate all schematics of calling WSRP-compliant Web services.

The WSRP Viewer will demonstrate the loading and displaying of the HTML markup by using the appropriate Web browser control and HTML DOM library. WSRP calls will be transported over HTTP and will support HTML markup only.

I will also demonstrate how to handle the scope of the WSRP sessions through session and registration calls. This includes defining user-defined registration properties. Finally, I will demonstrate how the WSRP Viewer can provide offline storage of the registration and portlet state for later reuse.

Figure 4 illustrates a high-level design model that encompasses the main components of the WSRP Viewer architecture. It includes the Windows Forms viewer control, the WSRP service class, and the supporting class libraries. Other components are present, but as helper and event delegate classes.

Figure 4 WSRP Viewer Design Model
Figure 4 WSRP Viewer Design Model

WsrpViewerControl is the Windows Forms control. This component's primary functions are to present HTML (rendering markup and capturing HTML events for processing), handle user interaction, and initiate WSRP state changes. The graphical elements handle all the required user interactions: connecting to a predefined WSRP producer, portlet selection, and transitions between WSRP modes and views. Figure 5 shows the viewer control when connected to a document management portlet.

Figure 5 Rendering a Document Management Portlet
Figure 5 Rendering a Document Management Portlet

The WsrpService class is the wrapper for the WSRP Web service requests, and represents an implementation of the WSRP specification. The PersistenceState type, through XML serialization and deserialization, will persist state and configuration data between invocations of the service class. This type aggregates registration and portlet state information and is persisted through the use of the PersistenceStateStorage class.

The WsrpServiceUrl class is a simple wrapper for the four required WSRP URLs: service description, markup operations, registration, and portlet management. Each instance of the WsrpService class is tied to a specific URL endpoint and is therefore defined by this property during an initialization call. This component also supports a type converter derived from ExpandableObjectConverter to allow for integration into the Visual Studio properties window.

The code base for the WSRP Viewer is too large to cover in a single article. Therefore, I will break down the WSRP processing into its essential elements, outlining how the WsrpService class handles these interactions and how the WsrpViewerControl interacts with the service. Later, I will discuss security and exceptions.

Establishing Relationships and Capabilities

Before you can interact with a WSRP producer, you need to establish a relationship with the producer and determine its capabilities. This is achieved through the WSRP getServiceDescription and register operations, which are wrapped within the service class InitializeService method. During this initialization, the operating mode of the WSRP interactions is defined (through an enumeration called WsrpOperatingMode) as one of the values shown in Figure 6.

Figure 6 WsrpOperatingMode Values (Close Figure 6)

Value | Description
ModifyGlobalState | Support global changes and no portlet management.
UsePortletManagement | Support user state persistence and portlet cloning through the portlet management interfaces, if supported.
Stateless | Support user state persistence but no portlet management cloning; the WSRP producer can, however, create a portlet clone for the user during an interaction. This is the default state.

The operating mode parameter is important because it determines the scope of portlet state changes. Internally, two properties are defined: SupportGlobalChanges and SupportPortletManagement.

The SupportGlobalChanges flag determines whether changes are made to the global portlet state or individual user clones. By default all portlet state changes are per user and do not affect the global state. Setting this to true, however, will allow modifications to the global portlet state.

The SupportPortletManagement flag determines the portlet cloning method used. If the portlet has been enabled for user-specific state (the WSRP hasUserSpecificState indicator set to true), then a WSRP clonePortlet request will be made for the user. Otherwise, the WSRP cloneBeforeWrite indicator will be used on the WSRP performBlockingInteraction requests (the default mode).

The SupportPersistence property also affects the portlet state operation. This flag, which defaults to true, determines whether the previous registration and portlet state information is initialized from a previous interaction. This defines a value for the internal RegistrationPstate and PortletPstate array properties.

The main purpose of the InitializeService method is to acquire registration data and determine the producer's capabilities. This includes determining the list of portlets with which the user can interact. Therefore, once called, the URL defining the interaction becomes immutable. The core code for this is in Figure 7.

Figure 7 Registration Data and Capabilities (Close Figure 7)
// make the initial service description call with blank registration
_serviceDescription = wsrpGetServiceDescription();

// is required for the WSRP producer? make call again
_usesRegistration = _serviceDescription.requiresRegistration;
if (AcquireRegistration())
    _serviceDescription = wsrpGetServiceDescription();

// iterate through the portlets; persist the portlet title and state
_portletDescriptions = _serviceDescription.offeredPortlets;
if (_portletDescriptions.Length > 0)
{
    _portletTitles = new string[_portletDescriptions.Length];
    _portletPstates = new PortletPstate[_portletDescriptions.Length];
    for (int idx = 0; idx <_portletDescriptions.Length; idx++)
    {
        _portletTitles[idx] = _portletDescriptions[idx].title.value;
        _portletPstates[idx] =
            new PortletPstate(_portletDescriptions[idx].portletHandle);
    }
}
// now look to see if cookie assistance is required
if (_serviceDescription.requiresInitCookie != CookieProtocol.none)
{
    // set up the collection for the cookies for markup calls
    _cookieCollection = new Hashtable();
    _usesCookies = true;
}
else
{
    // ensure markup has a cookie collection although not persisted
    _wsrpMarkup.CookieContainer = new CookieContainer();
    _cookieCollection = null;
    _usesCookies = false;
}

When a user registers with a WSRP producer, a registration context is returned. This registration context must be used on subsequent markup and interaction calls. If a WSRP producer requires registration, the service description request must be made again with the new registration context. This actual registration process is wrapped in the AcquireRegistration method. The core code is shown here:

RegistrationContext registrationContext = wsrpRegister();
if (registrationContext != null)
{
    _registrationPstate.RegistrationHandle =
        registrationContext.registrationHandle;
    _registrationPstate.RegistrationState =
        registrationContext.registrationState;
}

All the portlet information is persisted in an array of PortletDescription types. While processing the markup and interactions, this information (along with the user portlet state information) is used to determine the exact markup that is rendered.

Since the producer can ask for assistance with cookies, the final process for the service description is to determine the cookie requirements. At this point, the initialization process merely defines a holder for the cookie containers. The container contents are then defined and used during WSRP getMarkup and performBlockingInteraction requests. The service maintains a table of CookieContainers based on the GroupId of the portlet or the calling user. The final determination of which container is to be used is based on the producer's requirements and the portlet specifications. When a new cookie is required, the WSRP initCookie request is made and the cookie is added to the collection (see Figure 8).

Figure 8 Initializing Cookie Groups (Close Figure 8)
string cookieGroup;
if (_serviceDescription.requiresInitCookie == CookieProtocol.perUser ||
        _portletDescriptions[_portletIndex].groupID == null)
    cookieGroup = _callingUser;
else cookieGroup = _portletDescriptions[_portletIndex].groupID;

// see if the cookie container has been defined
if (_cookieCollection.ContainsKey(cookieGroup))
{
    // return the previously defined cookie container
    cookie = (CookieContainer)_cookieCollection[cookieGroup];
}
else
{
    // cookie not in the collection, so create and add into
    // the cookie group
    cookie = wsrpInitCookie();
    _cookieCollection.Add(cookieGroup, cookie);
}

The WSRP producer relationship is not automatically established when the viewer control is initially displayed. Since the service URL can be defined either programmatically or within the designer, this relationship is not established until requested either programmatically or through a user interaction.

Figure 9 WSRP Connection Settings
Figure 9 WSRP Connection Settings

To assist the initialization process, the viewer control supports a connect toolbar and context menu item. Both of these display the WSRP Producer Connect dialog box shown in Figure 9. This allows the user to select a portlet and initial state. Upon rendering the dialog, an instance of the WsrpService class is constructed and the InitializeService method is executed, making the WSRP getServiceDescription request. The control properties are used to determine the operating mode and whether the previous state information requires rehydration.

Although a toolbar and context menu is integral, portlet initialization can also be managed programmatically, as shown here:

wsrpViewerControl.WsrpInitializeProducer();
wsrpViewerControl.WsrpPortletHandle =
    "781F3EE5-22DF-4ef9-9664-F5FC759065DB";

Code initialization offers the additional advantage that the initial portlet can be selected through the PortletHandle property. This property is the handle assigned to the portlet by the WSRP producer. To find its value, the appropriate portlet can be selected manually and the handle property inspected and persisted offline. Using this technique, you can predetermine the portlet that is displayed to the user.

Obtaining and Rendering Markup

Once a relationship to the WSRP service has been established, you can obtain the required portlet markup. However, you need to determine whether a portlet requires cloning prior to making markup calls. This determination is based on whether the portlet supports clones and the operating mode, after which a WSRP clonePortlet request is made:

if (_portletClones && _supportManagement && !_supportGlobal)
{
    PortletContext portletContext = wsrpClonePortlet();
    _portletPstates[_portletIndex].CloneHandle =
        portletContext.portletHandle;
    _portletPstates[_portletIndex].PortletState =
        portletContext.portletState;
}

Upon making the WSRP clonePortlet request, the returned clone handle is persisted and used for all subsequent calls. To support the persistence of clone handles, the PortletPstate type defines two portlet handles: one for the original and one for the clone. PortletPstate also supports a CurrentHandle method to determine the appropriate handle for interactions.

A WSRP getMarkup request is used to obtain the HTML markup and corresponding markup title, passing in the requested portlet handle and state information. The core code for making the request is shown in Figure 10.

Figure 10 Retrieving HTML Markup (Close Figure 10)
getMarkup getMarkupParam = new getMarkup();
getMarkupParam.registrationContext = GetRegistrationContext();
getMarkupParam.markupParams = GetMarkupParams();
getMarkupParam.portletContext = GetPortletContext();
getMarkupParam.runtimeContext = GetRuntimeContext();
getMarkupParam.userContext = GetUserContext();

if (_usesCookies) _wsrpMarkup.CookieContainer = GetCookieContainer();

// make the WSRP get markup call
MarkupResponse markupResponse = _wsrpMarkup.getMarkup(getMarkupParam);

// after successful call ensure parameters are persisted
if (_usesCookies) SetCookieContainer(_wsrpMarkup.CookieContainer);

_sessionContext = markupResponse.sessionContext;

// Extract the markup and perform post processing
MarkupContext markupContext = markupResponse.markupContext;
ExtractMarkup(markupContext, out html, out title);
if (markupContext.requiresUrlRewriting)
    html = ProcessWsrpRewrite(html);

When defining the parameters for the WSRP getMarkup request, there are several WSRP types that must be defined: RegistrationContext, PortletContext, RuntimeContext, UserContext, MarkupParams, and CookieContainer. The MarkupParams type defines the parameters, determining the markup that will be returned. MarkupParams includes fields for locale information, MIME types, allowed character sets, WSRP mode, window state, and the navigational state. The WSRP mode and window state are those selected by the user through the viewer control—although, they can be changed when performing an interaction.

Initially, the navigational state is undefined, but is then modified during interactions. A new value is derived from an action URL or as a return value from a blocking interaction request. Between instances of the service class, the navigational state (along with other state information) is persisted, allowing the initial markup request to return markup for a previous state condition.

Exception handling is an important process during markup acquisition. When a WSRP getMarkup request is made, there are a number of exceptions that warrant further processing. These include invalid cookies, session information, portlet clone handle, and registration information. Through analysis of the request exception, you can potentially handle each of these failures and reprocess the WSRP getMarkup request (see Figure 11).

Figure 11 Analyze and Reprocess Requests (Close Figure 11)
if (ex.WsrpFaultCode == WSRP_FAULTCODE_INVALIDCOOKIE)
{
    // clear out the cookie and try the call again
    reprocess = ReleaseCookie();
}
else if (ex.WsrpFaultCode == WSRP_FAULTCODE_INVALIDSESSION)
{
    // release the session and try again
    reprocess = ReleaseSession();
}
else if (ex.WsrpFaultCode == WSRP_FAULTCODE_INVALIDHANDLE)
{
    // see if using clone and if so ensure blanked out and try again
    if (!_portletPstates[_portletIndex].IsCloneEmpty)
    {
        _portletPstates[_portletIndex].CloneHandle = null;
        AcquireClonePortlet();
        reprocess = true;
    }
}
else if (ex.WsrpFaultCode == WSRP_FAULTCODE_INVALIDREGISTRATION)
{
    // blank out any registration data to make the call again
    if (!_registrationPstate.IsEmpty && !_usesRegistration)
    {
        // have data and should not so reprocess
        _registrationPstate = new RegistrationPstate();
        reprocess = true;
    }
    else
    {
        // try and acquire new registration data if needed
        reprocess = AcquireRegistration();
    }
}

Finally, your WSRP consumer must process the markup URLs. URL rewriting within a Windows Forms control is simplified since there is no intermediate consumer agent. The service class, within the ProcessWsrpRewrite method, takes the simple approach of replacing the following string values with an empty value:

wsrp_rewrite?
/wsrp_rewrite
wsrp-urlType=resource&wsrp-url=
wsrp-urlType=resource&wsrp-requiresRewrite=true&wsrp-url=

This simple approach works for a smart client application since the viewer control captures all events and a URL is not required. However, there are some caveats to consider. This approach will not work for a portal consumer application, and some complex routines are needed to complete this process.

When returning the HTML markup and corresponding title, the viewer control needs to render the information. The title is a viewer control property that is displayed within the toolbar. The HTML markup is displayed as the innerHTML of the Microsoft Web Browser control.

At this point, I should take a moment to discuss the HTML rendering process. To have access to the HTML DOM (the mshtml type library), the Web browser must first load an appropriate document. To achieve this, the viewer control navigates to the "about:blank" document and, in effect, loads the HTML DOM. From this point onwards all markup is rendered by modifying the document body innerHTML property.

You direct the Web browser to a URL through its Navigate method. Navigation completion is indicated by the document complete event. To obtain the HTML document, the Web browser Document property is cast to an mshtml.HTMLDocument interface object. This object can be queried for its properties and collections as well as to access the HTML body (supporting the mshtml.HtmlBody interface) so as to manipulate the HTML DOM:

document = (mshtml.HTMLDocument)this.webBrowser.Document;
body = (mshtml.HTMLBody)document.body;

Don't assume this is a straightforward process, however. There are some complexities associated with URL fixing due to the use of "about:blank" navigation. This involves some COM interop. Similar processing is required to suppress the default context menu of the Web browser and instead display one more suitable to WSRP processing.

The BeforeNavigate2 event, exposed by the Web browser control, handles markup interactions. It allows you to capture, cancel, and internally process any navigation event. This is necessary within a WSRP viewer control because any interactions require processing through a WSRP call, with the parameters determined from the navigation URL. The DocumentComplete event captures the initial "about:blank" navigation to perform post-navigation processing.

Note that when processing the HTML markup, there may be tags present that the HTML specification specifically prohibits from occurring outside the head element of the document. The actual markup fragment returned from the producer can contain any number of elements, but it cannot use body, frame, frameset, head, html, or title tags. Other tags will require additional processing, including the script, style, and link tags. If these tags are not managed separately, their values will be lost when translated into the innerHTML property.

Script tags and style tags are handled in a similar fashion. The corresponding DOM element is created using the markup values. These Elements are then appended into the body of the document. Some special processing is required for link tags since the contents of a linked stylesheet can contain URLs that require rewriting. Therefore, any stylesheets referenced within a link tag are retrieved, processed, and cached in memory for inclusion into the rendered markup. In a standard portal consuming app, these resources—including referenced images—are cached on the consumer Web site and the URL is rewritten to point to the consuming portal.

This processing of special tags is encapsulated with the SetTextMetadata method, shown in Figure 12. Regular expressions are used extensively here to parse the returned markup for the required tags. A helper method, GetResourceString, collects the linked stylesheet resources and caches them. This optimizes performance because the stylesheets may be required for rendering markup in a portlet.

Figure 12 HTML Tag Processing (Close Figure 12)
private void SetTextMetadata(string html)
{
    // define source variables
    string scriptSource = null, scriptTag = null,  styleSource = null;
    string styleTag = null, linkTag = null;

    // extract any script and style tags for later processing
    if (html != string.Empty)
    {
        // see if have a link tag for processing
        Regex linkExpression = new Regex(LINK_PARSE_EXPRESSION,
            RegexOptions.IgnoreCase | RegexOptions.Multiline |
            RegexOptions.Singleline | RegexOptions.ExplicitCapture);
        Match linkMatch = linkExpression.Match(html);
        // see if a match was found and if so process
        if (linkMatch.Success)
        {
            linkTag = linkMatch.Result(LINK_TAG_PARSE_MATCH);
        }
        // see if a style tag has been included in the return
        if (Regex.IsMatch(html, STYLE_PARSE_PRE_EXPRESSION,
            RegexOptions.IgnoreCase | RegexOptions.Multiline |
            RegexOptions.Singleline))
        {
            Regex styleExpression = new Regex(STYLE_PARSE_EXPRESSION,
                RegexOptions.IgnoreCase | RegexOptions.Multiline |
                RegexOptions.Singleline | RegexOptions.ExplicitCapture);
            Match styleMatch = styleExpression.Match(html);
            // see if a match was found and extract the script
            if (styleMatch.Success)
            {
                styleTag = styleMatch.Result(STYLE_TAG_PARSE_MATCH);
                styleSource = styleMatch.Result(STYLE_SOURCE_PARSE_MATCH);
            }
        }
        // see if a script tag has been included in the return
        if (Regex.IsMatch(html, SCRIPT_PARSE_PRE_EXPRESSION,
                RegexOptions.IgnoreCase | RegexOptions.Multiline |
                RegexOptions.Singleline))
        {
            // define a regular expression for the Html Body parsing
            Regex scriptExpression = new Regex(
                SCRIPT_PARSE_EXPRESSION,
                RegexOptions.IgnoreCase | RegexOptions.Multiline |
                RegexOptions.Singleline | RegexOptions.ExplicitCapture);
            Match scriptMatch = scriptExpression.Match(html);
            // see if a match was found and extract the script
            if (scriptMatch.Success)
            {
                scriptTag = scriptMatch.Result(SCRIPT_TAG_PARSE_MATCH);
                scriptSource = scriptMatch.Result(
                    SCRIPT_SOURCE_PARSE_MATCH);
            }
        }
    }
    // process link depending on if they are null or not
    if (linkTag != null)
    {
        // see if have a processable link tag
        bool linkFound = false;
        // extract the rel and type properties of the link tag
        Match relMatch = Regex.Match(linkTag, REL_PARSE_EXPRESSION,
            RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture);
        Match typeMatch = Regex.Match(linkTag, TYPE_PARSE_EXPRESSION,
            RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture);
        Match hrefMatch = Regex.Match(linkTag, HREF_PARSE_EXPRESSION,
            RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture);
        string relValue = (relMatch.Success) ?
            relMatch.Result(REL_VALUE_PARSE_MATCH) : string.Empty;
        string typeValue = (typeMatch.Success) ?
            typeMatch.Result(TYPE_VALUE_PARSE_MATCH) : string.Empty;
        string hrefValue = (hrefMatch.Success) ?
            hrefMatch.Result(HREF_VALUE_PARSE_MATCH) : string.Empty;
        if (IsStringEqual(relValue, REL_VALUE_STYLESHEET) &&
            IsStringEqual(typeValue, TYPE_VALUE_TEXT_CSS) &&
            hrefValue != string.Empty)
        {
            // populate the string value from the text
            string source = _wsrpService.GetResourceString(hrefValue);
            if (source != null)
            {
                // have a new style resource include document
                linkFound = true;
                if (styleSource == null)
                {
                    // have no previous style so create
                    styleSource = source;
                    styleTag = STYLE_TAG_VALUE;
                }
                else
                {
                    // append to previous style sheet
                    styleSource = source + styleSource;
                }
            }
        }
        if (!linkFound)
        {
            // have link but not found so include in document
            mshtmlDomNode bodyNode = (mshtmlDomNode)body;
            mshtmlDomNode linkNode =
                (mshtmlDomNode)document.createElement(linkTag);
            linkNode = bodyNode.appendChild(linkNode);
            linkNode = linkNode.swapNode(bodyNode.firstChild);
        }
    }

    // process style depending on if they are null or not
    if (styleSource != null)
    {
        mshtmlDomNode bodyNode = (mshtmlDomNode)body;
        mshtmlDomNode styleNode =
            (mshtmlDomNode)document.createElement(styleTag);
        styleNode = bodyNode.appendChild(styleNode);
        styleNode = styleNode.swapNode(bodyNode.firstChild);
        mshtmlStyleElement style = (mshtmlStyleElement)styleNode;
        style.styleSheet.cssText = styleSource;
    }

    // process script depending on if they are null or not
    if (scriptSource != null)
    {
        mshtmlDomNode bodyNode = (mshtmlDomNode)body;
        mshtmlDomNode scriptNode =
            (mshtmlDomNode)document.createElement(scriptTag);
        scriptNode = bodyNode.appendChild(scriptNode);
        mshtmlScriptElement script = (mshtmlScriptElement)scriptNode;
        script.text = scriptSource;
    }
}

After the initial markup is rendered, users can modify the WSRP mode, the window state, and the selected portlet. When a state change occurs, the viewer modifies the state within the service class and performs a WSRP getMarkup request.

The portlet connect dialog box lets the user switch portlets. If the connect dialog box is presented when a relationship has previously been established, it will display the previously acquired portlet information. If a new portlet is selected, the viewer control notifies the service class via the SelectPortlet method and performs a WSRP getMarkup request.

Processing Interactions

After markup has been rendered, the next step is to process interactions. Rendered markup contains action links structured as form post operations with an action attribute and anchor tags with a href attribute:

<FORM action=wsrp-urlType=blockingAction&amp;wsrp-interactionState=add
      method=post>
<A href="http://wsrp-urlType=render&amp;wsrp-navigationalState=add">
      Add Quote</A>
<A href="http://wsrp-urlType=render&amp;wsrp-navigationalState=title">
      Change Title</A>
Here, the referenced URLs are parsable WSRP expressions that appear in the form of parameters collections. The protocol, host name, port, and path have been excluded during URL rewriting because the viewer control is acting as the portal consumer.

During an interactive operation, the viewer control obtains the URL, processes the optional form data and collects file names to be uploaded. Once this data has been captured, the service class can assume control to make the WSRP message requests and return the new markup.

The service class ProcessInteraction method processes the WSRP interactions for the viewer control. The first step is to parse the requested URL, extracting the WSRP parameters. These parameter names, defined in the specification and configured using URL templates, determine the required navigational state, interaction state, WSRP mode, window state, and URL type. The URL type, in turn, determines the process that will take place. In this case, you're interested in render, resource, and blockingAction types.

The render type results in fetching new markup based on the new state, as defined by the other properties. The resource type provides a URL to which navigation is required. The blockingAction type results in the execution of a WSRP performBlockingInteraction request and the subsequent fetching of new markup. This is managed within the PerformBlockingInteraction method, which wraps the wsrpPerformBlockingInteraction method (see Figure 13). When executed, either an update response or redirection URL is returned.

Figure 13 PerformBlockingInteraction (Close Figure 13)
// obtain the interaction response
BlockingInteractionResponse interactionResponse =
    wsrpPerformBlockingInteraction(interactionState,
        formCollection, filenames);
// persist any user specific information for later calls
UpdateResponse updateResponse = interactionResponse.updateResponse;
// see if one has an update response of redirectUrl
if (updateResponse != null)
{
    // persist any session context including a null value
    _sessionContext = updateResponse.sessionContext;
    // persist any new portlet information
    if (updateResponse.portletContext != null)
    {
        _portletPstates[_portletIndex].CloneHandle =
            updateResponse.portletContext.portletHandle;
        _portletPstates[_portletIndex].PortletState =
            updateResponse.portletContext.portletState;
    }
    // persist any new navigational and WSRP state
    _portletPstates[_portletIndex].NavigationalState =
        updateResponse.navigationalState;
    if (updateResponse.newMode != null &&
        updateResponse.newMode != string.Empty)
    {
        _portletPstates[_portletIndex].WsrpMode =
            WsrpConversion.GetMode(updateResponse.newMode);
    }
    if (updateResponse.newWindowState != null &&
        updateResponse.newWindowState != string.Empty)
    {
        _portletPstates[_portletIndex].WindowState =
            WsrpConversion.GetWindowState(updateResponse.newWindowState);
    }
    // obtain the markup and title response
    MarkupContext markupContext = updateResponse.markupContext;
    ExtractMarkup(markupContext, out html, out title);
}
    // will return with unchanged information
else RedirectUrl = interactionResponse.redirectURL;

The update response provides all the information relating to a state change in the portlet interactions. The portlet context, if returned, may contain a clone handle that requires persistence. This process is identical to the operation performed when a WSRP clonePortlet request is made.

Finally, any markup returned in the update response is processed. If the markup is not returned in this fashion, and no redirection URL is given, you must make a WSRP getMarkup request.

The parameters for a WSRP performBlockingInteraction request are the same as for a WSRP getMarkup request, but with the addition of the InteractionParams type. This contains the properties interactionState, portletStateChange, formParameters, and uploadContexts.

The main aspects of the interaction parameters are the form parameters and the upload context array. The form data is merely a collection of the WSRP NamedString types. The definition of the uploadContext property, consisting of a mimeType value and uploadData byte array, is a little more complex.

The specification for uploading binary data for WSRP is a bit unclear. The technique that has been adopted by most vendors is to pass the MIME contents, derived directly from a browser post operation, into the uploadData byte array with the MIME type defined as multipart/form-data. The MIME contents include all the attributes and form data, in addition to the binary data. The service class library provides a WsrpMimeBuilder helper class to support generating such a MIME stream. This helper class also encapsulates the urlmon.dll function FindMimeFromData to locate the content type of the data stream.

Once a blocking interaction has been performed, the markup or redirection URL is returned to the viewer control for processing.

As previously mentioned, the viewer control is responsible for obtaining the interaction URL, processing form data, and collecting file names to be uploaded. This is achieved by capturing and canceling the Web browser BeforeNavigate event and then executing the service class ProcessInteraction method. This returns new markup or a redirection URL.

The URL is easily obtained from the event arguments, but the post data and file names require more work. You can parse the post data, but since this may not contain a complete set of data when FileUpload elements are present, you can instead parse the HTML DOM. The parsing is outlined in Figure 14.

Figure 14 Parsing Markup (Close Figure 14)
private void WsrpProcessInputs(
    out NameValueCollection eventCollection,
    out NameValueCollection formCollection,
    out NameValueCollection filenames)
{
    eventCollection = new NameValueCollection();
    formCollection = new NameValueCollection();
    filenames = new NameValueCollection();

    // see if active control is a submit button; ensure it's not cancel
    if (IsStringEqual(document.activeElement.tagName, HTML_INPUT_TAG)) {
        mshtmlInputElement activeElement =
            (mshtmlInputElement)document.activeElement;
        if (IsStringEqual(activeElement.type, FORM_SUBMIT_TYPE))
        {
            formCollection.Add(activeElement.name, activeElement.value);
            eventCollection.Add(activeElement.name, activeElement.value);
            if (IsStringEqual(activeElement.value, FORM_CANCEL_TEXT))
                return;
        }
    }

    // locate all the input elements on the form
    mshtmlElementCollection inputs =
        body.getElementsByTagName(HTML_INPUT_TAG);
    foreach (mshtmlElement element in inputs) {
        mshtmlInputElement inputElement = (mshtmlInputElement)element;
        string elementType = inputElement.type, filename = null;
        string elementName = inputElement.name, elementValue = null;
        bool eventArg = false;
        if (IsStringEqual(elementType, FORM_CHECKBOX_TYPE) ||
            IsStringEqual(elementType, FORM_RADIO_TYPE))
        {   // see if input was a check box or radio button
            if (inputElement.@checked) elementValue = bool.TrueString;
            eventArg = true;
        }
        else if (IsStringEqual(elementType, FORM_TEXT_TYPE))
        {   // was text, excluding password and hidden values
            elementValue = inputElement.value;
            eventArg = true;
        }
        else if (IsStringEqual(elementType, FORM_PASSWORD_TYPE) ||
                 IsStringEqual(elementType, FORM_HIDDEN_TYPE))
        {   // was text, including password and hidden values
            elementValue = inputElement.value;
        }
        else if (IsStringEqual(elementType, FORM_FILE_TYPE))
        {   // was a file name
            filename = inputElement.value;
            eventArg = true;
        }

        if (elementValue != null) { // if have value, add to collection
            formCollection.Add(elementName, elementValue);
            if (eventArg) eventCollection.Add(elementName, elementValue);
        }
        if (filename != null) { // if have file, add it to the collection
            filenames.Add(elementName, filename);
            if (eventArg) eventCollection.Add(elementName, filename);
        }
    }
}

The parsing process iterates through all the DOM elements, looking for the various input elements. The main complication is deciding which button was pressed during posting. This is done by looking at the active element and deciding if it was a cancel button. If so, the HTML DOM parsing is halted and only the cancel element is included within the form collection.

In addition to capturing the form elements for the WSRP interaction, the parsing process defines an additional form data collection that excludes hidden and password text. Whenever a user performs an interaction, the viewer control will raise an event containing this data collection along with the interaction URL. Other events are raised by the viewer control for user interactions such as portlet changes and WSRP mode changes.

State Persistence and Destruction of Relationships

WSRP producers can maintain state in the form of consumer registrations, portlets, and sessions. A consumer can create a registration context by calling the register service operation. In the scope of a consumer registration, portlets can be created by making a WSRP clonePortlet request. In regards to a portlet, the producer may create sessions that expire after a period of inactivity.

It's essential that a consumer application manage the destruction of state information. When a consumer no longer needs a portlet, it should execute a WSRP destroyPortlets request, passing the portlet handle, so the producer can release associated resources and session information. When a consumer no longer needs the producer's services, it should execute the WSRP deregister request, passing the consumer registration handle.

The service class needs to persist state information between sessions. It uses the PersistenceStateStorage helper class to do this, serializing and deserializing a PersistenceState type that aggregates a RegistrationPstate and an array of PortletPstate types. The PortletPstate type contains the portlet and clone handles, portlet state, navigational state, WSRP mode, and window state. During a normal interactive session, this information changes and all these changes need to be captured.

The storage mechanism is an isolated user storage file for the app domain. The file name is derived from the producer URL. Of course, there are times when isolated storage will not be the preferred mechanism, for example when an application needs to manage state centrally. To support this, the state information is exposed through the RegistrationPstate and PortletPstate properties and, thus, isolated storage persistence can be disabled.

An important aspect of the service class is the implementation of the IDisposable interface. The Dispose method ensures that any state information is persisted to isolated storage. The Dispose method of the viewer control correctly disposes of the service class so that, when the parent container is disposed, all state information is correctly persisted for the next session.

Since the state information is persisted in a file whose name is based upon the producer's URL, multiple service instances for a single producer will share the same persistence information. This, however, may not always be appropriate. Therefore, the service constructor supports a property used to differentiate service class instances. This value determines the isolated storage file name.

Once the producer state information is persisted, the consumer is responsible for ensuring that the resources are destroyed when no longer needed. For this, the service class exposes several methods, including ReleaseCookie, ReleaseSession, ReleasePortletClones, and ReleaseRegistration. The ReleaseCookie method merely initializes the cookie container associated with the portlet. ReleaseSession performs a WSRP releaseSession request, destroying the session information associated with the portlet interaction, while keeping any portlet clone and registration information intact.

The ReleasePortletClones method performs a WSRP destroyPortlets request for each clone created during interaction. In releasing the portlet clones, any interaction state, including edits made through the portlets, will be lost. This effectively initializes the state of the interaction. The heart of this process is shown in Figure 15.

Figure 15 Destroying Portlets (Close Figure 15)
// create an array of the clones to be destroyed
ArrayList handleList = new ArrayList();
for (int index = 0; index < _portletPstates.Length; index++)
{
    if (!_portletPstates[index].IsCloneEmpty)
    {
        // add the clone to the list and clear out value
        handleList.Add(_portletPstates[index].CloneHandle);
        _portletPstates[index].CloneHandle = null;
    }
}
// if have items then call the destroy routine
if (handleList.Count > 0)
{
    string[] handles = new string[handleList.Count];
    handleList.CopyTo(handles);
    DestroyPortletsResponse destroyPortletsResponse =
        wsrpDestroyPortlets(handles);
}

To standardize the destruction of relationships when a service class is no longer needed, two termination methods are exposed: TerminateInteraction and TerminateRelationship. TerminateInteraction releases session information before executing the Dispose method. TerminateRelationship releases all persisted information, ending the consumer relationship with the producer, as shown here:

ReleasePortletClones();
ReleaseSession();
ReleaseRegistration();
_stateStorage.ClearPersistenceState();
_registrationPstate = new RegistrationPstate();
_portletPstates = null;
Dispose();

The viewer control toolbar provides a disconnect action that lets the user control the destruction of WSRP persisted information. The user can choose to preserve or destroy the interaction information. If the data is to be preserved, the viewer control merely ends the WSRP interaction by calling the service class TerminateInteraction method. If the user disconnects from the WSRP producer, all persisted information is destroyed by calling TerminateRelationship.

Viewer Security

Security is important, especially when external data producers are involved. WSRP does not handle authentication and authorization; it relies on the underlying protocols and Web services standards. Thus, the WSRP service class relies on HTTP and WS-Security.

WSRP allows you to indicate the authentication to employ, but not how it is implemented. The options are wsrp:none, wsrp:password, or wsrp:certificate, and the value is passed in RuntimeContext. Consequently, the service class provides a means for specifying the authentication type and credentials. The authentication type is specified during the service initialization and can be set through AuthenticationMode to None, UseDomain, UseCertificate, or UsePassword. Domain forces the default login credentials to be used, and Password requires user credentials. To use these credentials, a class derived from ICredentials is defined to be used as the security credentials for the Web service proxy class:

ICredentials credentials;
switch (_authenticationMode)
{
    case WsrpAuthenticationMode.UsePassword:
        credentials = new NetworkCredential(_userName, _userPassword);
        break;
    case WsrpAuthenticationMode.UseDomain:
        credentials = CredentialCache.DefaultCredentials;
        break;
    default:
        credentials = null;
        break;
}

To support the use of alternate credentials, a SetCredentials method is available. In this instance, the viewer control parent application has the responsibility of collecting and possibly saving these values in a secure manner.

The preferred alternative is to implement the WS-Security specification. This allows the use of a policy file for determining the security requirements, but it requires that the WSRP-compliant Web service supports this standard. If necessary, you can implement this feature by modifying the WSRP proxy class.

Note that certain resources, such as stylesheets, are obtained through a Web request so the security mechanism for these requests follows the same pattern as those for the Web service requests.

As the service description for any WSRP producer is standardized, you can generate the proxy class by pointing the .NET Framework WSDL tool to any sample WSDL such as www.pavonis.co.uk/devnet/wsrp/wsrpservice.wsdl.

Wrap-Up

Microsoft is a member of OASIS WSRP. Since the release of the 1.0 specification, Microsoft announced products that support the WSRP standard including two XML-based toolkits designed for SharePoint®. (See WSRP Web Part Toolkit for SharePoint Products and Technologies.) For the .NET implementation of a WSRP producer, I used NetUnity. The company provides access to a WSRP producer containing a number of sample portlets.

Carl Nolan is an independent technical architect and principle of Pavonis Consulting Services. He specializes in .NET Framework and SQL Server solutions. Previously, Carl was employed by Microsoft Consulting Services. Reach Carl at msdn@pavonis.co.uk.


Paste As Visual Basic: A Visual Studio Add-In That Converts C# Code To Visual Basic

Paste As Visual Basic

A Visual Studio Add-In That Converts C# Code To Visual Basic

Scott Swigart
This article discusses:
  • Using code conversion Web services
  • Unit testing code
  • Creating a Visual Studio add-in
  • Adding menu items to Visual Studio
This article uses the following technologies:
Visual Studio 2005, Visual Basic .NET
Code download available at: PasteAs.exe (875KB)
Contents
Creating the Add-In
Testing Times
Adding an Add-In
Form of Conversion
Conclusion

T
he "Paste as..." functionality in applications like Microsoft® Word has become indispensable for me. I often copy something from the Web and want to paste it into a document without all of the HTML formatting. Paste as is the tool for the job.

While perusing the Web one day, looking for a code example, it occurred to me that Paste as would also be beneficial for Visual Studio®. However, instead of simply converting the text formatting, it could convert the language of the code example. It's common to be coding in C# and find a Visual Basic® example that does exactly what you want, or vice versa. Using such code requires copying the sample to the clipboard, going to one of the many great code conversion Web sites, pasting in the code, converting it, copying the new code, and finally pasting the converted code into Visual Studio.

When I find a C# code example on a Web site, I'd really like to eliminate the intermediate conversion steps and just Paste as Visual Basic right into the code editor. All the necessary building blocks are in place to make this work; I just need some code to glue it all together.

Creating the Add-In

There are a number of Web-based code converters available, so I wanted to architect the Paste as Visual Basic add-in so that the user can choose whichever converter he wants to use. When you're dealing with Web solutions, there's no guarantee that a particular Web site will remain operational indefinitely, or that it won't change its implementation in a way that breaks your code. Hooking up to more than one conversion service makes it more likely that there will always be a service available to perform the conversion. Also, different code converters use different algorithms, and one may work better or worse for a particular piece of code. Finally, I wanted it to be easy for the Visual Studio add-in to support additional converters in the future.

To abstract the converter implementation from the rest of the code, I crafted the IConvertCode interface. This describes the conversion operation in general, but will leave the specific implementation to other classes. The IConvertCode interface is pretty simple:

Public Interface IConvertCode
    Function Convert(ByVal csCode As String) As String
    ReadOnly Property ConverterName() As String
End Interface
The Convert method takes C# code as a string argument and returns Visual Basic code as a string. The ConverterName property returns the name of the converter so that the user can choose the converter that they would like to use.

The first implementation of this interface uses Kamal Patel's C# to Visual Basic code converter Web service, currently available at ConvertCSharp2VB. Once a reference is added to this service, the code to call it is very straightforward, as shown in the code in Figure 1.

Figure 1 Calling a Conversion Web Service (Close Figure 1)
Public Class WSConvert
    Implements IConvertCode

    Public Function Convert(ByVal csCode As String) As String _
            Implements IConvertCode.Convert

        Dim codeConverter As _
            New CSToVBWebService.ConvertCSharp2VBService()
        Return codeConverter.Execute(csCode)

    End Function

    Public ReadOnly Property ConverterName() As String _
            Implements IConvertCode.ConverterName
        Get
            Return "Kamal Patel's Converter"
        End Get
    End Property
End Class

The Web service exposes an Execute method that takes C# code and returns Visual Basic code. Since the Web service just happens to use the same signature as IConvertCode.Convert, the arguments to IConvertCode.Convert can just be passed through to the Web service, and the results can be returned.

Another great tool is Carlos Aguilar Mares's AJAX-powered code converter, available at CodeTranslator. This converts between C# and Visual Basic as you type. Behind the scenes, the code is posted to carlosag.net/Tools/CodeTranslator/translate.ashx as a set of form fields. Another implementation of IConvertCode uses this facility to perform translation, as shown in Figure 2.

Figure 2 Using a Form-Based Converter (Close Figure 2)
Public Class CAConvert
    Implements IConvertCode

    Public Function Convert(ByVal csCode As String) As String _
            Implements IConvertCode.Convert

        Dim formFields As New NameValueCollection
        formFields.Add("code", csCode)
        formFields.Add("Language", "C#")
        formFields.Add("DestinationLanguage", "VB")

        Dim client As New WebClient
        Return Encoding.ASCII.GetString( _
            client.UploadValues( _
                "http://www.carlosag.net/Tools/" & _
                "CodeTranslator/translate.ashx", _
                "POST", formFields _
            ) _
        )

    End Function

    Public ReadOnly Property ConverterName() As String _
            Implements IConvertCode.ConverterName
        Get
            Return "Carlos Aguilar's Converter"
        End Get
    End Property
End Class

This implementation posts the C# code as form fields to a Web address. The Visual Basic code is returned as the body of the HTTP response. To send the C# code, a NameValueCollection is used. The C# is added as the code item, the source language is specified as the Language item, and the Visual Basic result is specified as the DestinationLanguage item. Once the collection is populated, it is then sent to the Web address using the WebClient.UploadValues method. This method posts the form fields and gets the results back as an array of bytes. Encoding.ASCII.GetString is used to convert the array of bytes back to a string. This string contains the Visual Basic translation.

You can see that although the internal implementations are quite different, both converters satisfy the IConvertCode interface in that they provide methods that take C# code as a string and return Visual Basic code as a string.

Testing Times

I've placed the implementations of code conversion into a class library project. In the past, I probably would have created a simple Windows Form application to test my two converters to make sure they're functioning properly. However, Visual Studio 2005 Team Edition for Software Developers provides built-in unit testing capabilities. All I have to do is right-click inside of a method and I can quickly generate a test for that method.

The first time you create a unit test, a new project is added to your solution and the test code is placed in that project. The unit test for one of the conversion methods is shown in Figure 3.

Figure 3 Unit Test for Form-Based Conversion (Close Figure 3)
<TestMethod()> _
Public Sub CAConvertTest()
    Dim target As IConvertCode = New CAConvert()

    Dim csCode As String = My.Computer.FileSystem.ReadAllText( _
        DataDir & "\CSCode.txt")

    Dim expected As String = My.Computer.FileSystem.ReadAllText( _
        DataDir & "\CAVBCode.txt")
    Dim actual As String

    actual = target.Convert(csCode)
    Assert.AreEqual(expected, actual, _
        "PasteAsVB.CAConvert.Convert did not return the expected value.")
End Sub

The test loads the C# source code from a text file. The code is converted to Visual Basic and compared with another file that contains the expected results. If the results match, then the test passes. The Test Manager window lets me quickly run my unit tests. If I add more converters in the future, or make changes to my existing logic, these tests should give me a quick way to make sure that everything is working properly.

Adding an Add-In

Now that I know the converters are working as expected, it's time to create an add-in for Visual Studio 2005. You create an add-in by adding a new add-in project to your solution. Add-ins are available under Extensibility projects, as shown in Figure 4.

Figure 4 Creating an Add-In Project
Figure 4 Creating an Add-In Project

The wizard walks you through a number of questions to assist in building the skeleton for the add-in. You're prompted to specify a name, whether you want the add-in to show up in the toolbar, whether the add-in should load when Visual Studio starts, and whether it's always modeless so that it's safe to use from command-line builds.

Unlike in Visual Studio .NET 2003, the add-in project for Visual Studio 2005 contains the boilerplate code needed to correctly connect your add-in into the development environment. The add-in class implements IExtensibility2, which is the interface that Visual Studio will use to communicate with the add-in. Some key methods, like OnConnection and QueryStatus, are also written for you. The Exec method—the method that's called when the user invokes your add-in—is stubbed out, too.

When you set the add-in as the start-up project and press F5, an amazing thing happens: a new instance of Visual Studio launches with the add-in installed. You can use this new instance of Visual Studio to test the add-in. You can even set breakpoints and otherwise debug your add-in just like any other piece of code. Launching the add-in in a test instance of Visual Studio is a new and very welcome feature in Visual Studio 2005.

There are a couple changes that I want to make to the boilerplate at this point in the process. First, the add-in shows up under the Tools menu by default. I really want it to show up under the Edit menu, right after the Paste menu item. The code in Figure 5 shows the modified OnConnection method.

Figure 5 Modified OnConnection Method (Close Figure 5)
Public Sub OnConnection(ByVal application As Object, _
        ByVal connectMode As ext_ConnectMode, _
        ByVal addInInst As Object, _
        ByRef custom As Array) Implements IDTExtensibility2.OnConnection

    _applicationObject = CType(application, DTE2)
    _addInInstance = CType(addInInst, AddIn)
    If connectMode = ext_ConnectMode.ext_cm_UISetup Then

        Dim commands As Commands2 = CType( _
            _applicationObject.Commands, Commands2)
        Dim editMenuName As String

        Try
            Dim resourceManager As System.Resources.ResourceManager = _
                New System.Resources.ResourceManager( _
                    "PasteAsVBAddIn.CommandBar", _
                    System.Reflection.Assembly.GetExecutingAssembly())

            Dim cultureInfo As System.Globalization.CultureInfo = _
                New System.Globalization.CultureInfo( _
                    _applicationObject.LocaleID)
            editMenuName = resourceManager.GetString( _
                String.Concat(cultureInfo.TwoLetterISOLanguageName, _
                    "Edit"))

        Catch e As Exception
            editMenuName = "Edit"
        End Try

        Dim commandBars As CommandBars = _
            CType(_applicationObject.CommandBars, CommandBars)
        Dim menuBarCommandBar As CommandBar = commandBars.Item("MenuBar")

        Dim editControl As CommandBarControl = _
            menuBarCommandBar.Controls.Item(editMenuName)
        Dim editPopup As CommandBarPopup = CType(editControl, CommandBarPopup)

        Try
            Dim command As Command = commands.AddNamedCommand2( _
                _addInInstance, "PasteAsVBAddIn", _
                "Paste as Visual Basic", _
                "Convert the clipboard content from C# " & _
                    "to Visual Basic and paste", _
                True, 59, Nothing, _
                CType(vsCommandStatus.vsCommandStatusSupported,Integer)+ _
                CType(vsCommandStatus.vsCommandStatusEnabled, Integer), _
                vsCommandStyle.vsCommandStylePictAndText, _
                vsCommandControlType.vsCommandControlTypeButton)

            command.AddControl(editPopup.CommandBar, 12)
        Catch argumentException As System.ArgumentException
        End Try

    End If
End Sub

There are a few important modifications made to the boilerplate for this method. First, the project includes a resource file called CommandBar.resx, which contains the text for the top-level Visual Studio menu items in many languages. The following line of code looks up a menu item, prefixed with your two-letter language name:

editMenuName = resourceManager.
    GetString( _
    String.Concat(cultureInfo.TwoLetterISOLanguageName, "Edit"))
If you're running this on a computer set for U.S. English, it looks up a resource string named enEdit in CommandBar.resx. If the computer was configured for German, it would look up deEdit. This will return culture-specific text for the menu item, which is needed to locate that menu object programmatically. Once the top-level menu is located, you can add submenus to it. The following lines of code locate the actual menu object:
Dim editControl As CommandBarControl = _
    menuBarCommandBar.Controls.Item(editMenuName)
Dim editPopup As CommandBarPopup = CType(editControl, CommandBarPopup)

Once the menu bar is located, the new Paste as Visual Basic menu item can be added to it. This is done by creating a command and then adding the command to the menu:

Dim command As Command = commands.AddNamedCommand2( _
    _addInInstance, "PasteAsVBAddIn", "Paste as Visual Basic", _
    "Convert the clipboard content from C# to Visual Basic and paste", _
    True, 59, Nothing, _
    CType(vsCommandStatus.vsCommandStatusSupported, Integer) + _
    CType(vsCommandStatus.vsCommandStatusEnabled, Integer), _
    vsCommandStyle.vsCommandStylePictAndText, _
    vsCommandControlType.vsCommandControlTypeButton)

command.AddControl(editPopup.CommandBar, 12)
The command is added to the menu at position 12, which places it right after the Paste menu item (see Figure 6).

Figure 6 Add-In on Edit Menu
Figure 6 Add-In on Edit Menu

It's also important that the add-in only be enabled when it could conceivably be used. You would only want to Paste as Visual Basic if you were actually editing a Visual Basic code file. Enabling and disabling the add-in is handled by the QueryStatus method. The code shown in Figure 7 will cause the add-in to display only when there's an active document that ends with the extension .vb, and there's text on the clipboard.

Figure 7 Control Whether Add-In is Enabled (Close Figure 7)
Public Sub QueryStatus( _
    ByVal commandName As String, _
    ByVal neededText As vsCommandStatusTextWanted, _
    ByRef status As vsCommandStatus, _
    ByRef commandText As Object) Implements IDTCommandTarget.QueryStatus

    If neededText = _
        vsCommandStatusTextWanted.vsCommandStatusTextWantedNone Then

        If commandName = "PasteAsVBAddIn.Connect.PasteAsVBAddIn" Then
            If _applicationObject.ActiveDocument IsNot Nothing _
            AndAlso _
                LCase(_applicationObject.ActiveDocument.FullName) _
                    .EndsWith(".vb") _
            AndAlso My.Computer.Clipboard.GetText <> String.Empty Then
                status = CType(vsCommandStatus.vsCommandStatusEnabled + _
                    vsCommandStatus.vsCommandStatusSupported, _
                    vsCommandStatus)
            Else
                status = CType(vsCommandStatus.vsCommandStatusSupported, _
                    vsCommandStatus)
            End If
        Else
            status = vsCommandStatus.vsCommandStatusUnsupported
        End If
    End If
End Sub

At this point in the project, the add-in is properly wired into Visual Studio and nearly ready to be used to convert the code. The only remaining task is to connect the Exec method of the add-in to the code that will actually perform the conversion and paste the results into the editor.

Form of Conversion

In order to provide a rich experience for the user, I've created a form that allows the user to select from various conversion options and preview the conversion results. Figure 8 shows the conversion form that's displayed when the user selects the Paste as Visual Basic menu command.

Figure 8 Conversion User Interface
Figure 8 Conversion User Interface

The C# source area shows the current contents of the clipboard. A listbox allows the user to choose the converter that they want to use. Clicking the Preview button will cause the output of the conversion to be shown at the bottom of this form, to insure that the conversion will be acceptable before pasting it into your code file. Clicking Convert will perform the conversion and paste the results directly into the open code file. If the Format Code After Convert checkbox is checked, then the code will be reformatted to insure proper indenting after the converted code is pasted in.

The code for the form is pretty simple and uses the conversion classes that were shown at the beginning of this article:

Private converters() As IConvertCode = { _
    New PasteAsVB.CAConvert(), _
    New PasteAsVB.WSConvert() _
    }

Private Sub ConverterForm_Load(ByVal sender As System.Object, _
        ByVal e As System.EventArgs) Handles MyBase.Load
    CsSourceTextBox.Text = My.Computer.Clipboard.GetText
    ConverterListBox.ValueMember = "ConverterName"
    ConverterListBox.DisplayMember = "ConverterName"
    ConverterListBox.DataSource = converters
End Sub
When the form loads, an array of converters is instantiated. These converters all implement the IConvertCode interface. In FormLoad, the listbox is bound to this array, and the ConvertName property is set to be displayed. This causes the name of each converter to appear in the listbox. If you wanted to add more converters, you would just create additional classes that implement IConvertCode, and then add them to the converters array. This could also be modified to pull the list of converters to be used from a configuration file.

When Preview is clicked, the selected converter is used to perform the conversion and display the results in the preview textbox. When Convert is clicked, the converted code is assigned to a property of the form, and the form closes. The code for the preview and convert functionality is shown in Figure 9.

Figure 9 Preview and Convert Functionality (Close Figure 9)
Private Sub Preview_Click(ByVal sender As System.Object, _
    ByVal e As System.EventArgs) Handles Preview.Click

    Dim oldCursor As Cursor = Cursor
    Cursor = Windows.Forms.Cursors.WaitCursor

    Dim convertedCode As String = GetConvertedCode(CsSourceTextBox.Text)
    VbResultsTextBox.Text = convertedCode
    Cursor = oldCursor
End Sub

Private Sub ConvertButton_Click(ByVal sender As System.Object, _
    ByVal e As System.EventArgs) Handles ConvertButton.Click

    Dim oldCursor As Cursor = Cursor
    Cursor = Windows.Forms.Cursors.WaitCursor

    VBCode = GetConvertedCode(CsSourceTextBox.Text)
    Cursor = oldCursor
    Me.DialogResult = Windows.Forms.DialogResult.OK
End Sub

Private Function GetConvertedCode(ByVal csCode As String) As String
    Dim converter As IConvertCode = ConverterListBox.SelectedItem
    Dim convertedCode As String = converter.Convert(csCode)
    Return convertedCode
End Function

The conversion form is now complete and simply needs to be invoked from the Exec method of the add-in as shown in Figure 10. When the add-in is invoked, the conversion form is displayed. If the Convert button is clicked, the conversion is performed, and the results are stored in the VBCode property of the form. The form then returns with a result of DialogResult.OK. The add-in checks the result and then inserts the converted code into the code editor. If the format check-box is checked, the add-in also executes the Edit.FormatDocument command to reformat the current code file.

Figure 10 Performing a Conversion (Close Figure 10)
Public Sub Exec(ByVal commandName As String, _
    ByVal executeOption As vsCommandExecOption, _
    ByRef varIn As Object, ByRef varOut As Object, _
    ByRef handled As Boolean) Implements IDTCommandTarget.Exec

    handled = False
    If executeOption=vsCommandExecOption.vsCommandExecOptionDoDefault Then
        If commandName = "PasteAsVBAddIn.Connect.PasteAsVBAddIn" Then
            Dim f As New ConverterForm()
            If f.ShowDialog() = System.Windows.Forms.DialogResult.OK Then
                _applicationObject.ActiveDocument _
                    .Selection.Insert(f.VBCode)
                If f.Reformat Then
                   _applicationObject.ExecuteCommand( _
                        "Edit.FormatDocument")
                End If
            End If
            handled = True
            Exit Sub
        End If
    End If
End Sub

Conclusion

Visual Studio 2005 simplifies the task of creating add-ins. First, it generates boilerplate code that will hook your add-in into the menu of Visual Studio. Second, when you press F5, a test instance of Visual Studio will be launched that contains your add-in. This makes debugging your add-in very simple.

The add-in shown here performs a very useful task—it allows you to paste C# code into your project as Visual Basic code. The add-in includes the ability to use two existing Web-based converters, but it's constructed in such a way that additional converters would be easy to add.

If you just want to install this add-in and go, the code download accompanying this article (on the MSDN Magazine Web site) includes an installer project that will let you install Paste as Visual Basic into any Visual Studio 2005 development environment. I hope that you see that creating your own add-ins doesn't need to become a daunting task. If you find yourself saying, "I wish I had an add-in that accomplished...," you just might be able to build that add-in yourself more easily than you may think.

Scott Swigart spends his time consulting, writing, and speaking about converging and emerging technologies. Scott is the author of several books on .NET, a certified Microsoft trainer (MCT) and developer (MCSD), and a Microsoft MVP. You can contact Scott at scott@swigartconsulting.com.


Editor's Note: Code Name Confusion

Editor's Note
Code Name Confusion

We'd like to have a word about product code names this month.

You've probably worked on a software project whose beta versions existed under a code name. These names can be meaningful (like "Longhorn," which is a bar between Whistler and Blackcomb), quirkily descriptive (like "Aurum," the SQL Server 2000 data-mining code name and also the Latin word for gold), or just plain goofy (we'll withhold examples of this one).

If you are old enough to remember "Get Smart," you might recall the episode when the Chief is put under oath and reveals his name is Thaddeus, much to Max's amusement. Even though "Thaddeus" is his given name, everyone still thinks of him as "Chief."

Many times a beta code name will stay in the public consciousness for some time. Even now that the next version of Windows has received its real name, Windows Vista, many of us are still working on the mental reassociation. But Windows Vista is a nice name. It lends itself to images of wide-open spaces.

Then there are other technologies that have a pretty snappy code name, but when the final product name is announced, it is more descriptive than fun. We're talking about you, Windows Communication Foundation. WCF is going to be around for many years. It's a powerful, complex technology. And as hard as we try, we still think "Indigo" when we hear it mentioned.

Sometimes we become accustomed to thinking of a program as "Chief." When we discover it's actually "Thaddeus," it takes a while to adapt to the real name. Windows Communication Foundation is still in the Chief/Thaddeus crossover phase.

We do wish, though, that new software names would be chosen to be conducive to eye-catching headlines. Have you ever tried to fit "Windows Communication Foundation" on a cover line? There's simply not enough room. By contrast, a name like "Indigo" would've left us with all kinds of options. And don't get us started on "Windows Presentation Foundation!"

From Chief to Thaddeus
Longhorn | Now known as Windows Vista, this is the next client version of Windows. The server version is currently still known as its code name "Longhorn Server."
Yukon | SQL Server 2005.
Whidbey | Visual Studio 2005. Also the .NET Framework 2.0.
Blackcomb | Looking further down the road, "Blackcomb" is the code name for the release of Windows following Vista.
Orcas | The code name for the version of Visual Studio after 2005. We'll be discussing Orcas in more depth once you're more comfortable with Visual Studio 2005.
WinFX | The next generation of managed APIs provided by Microsoft. WinFX encompasses a lot of technologies; more info can be found at msdn.microsoft.com/winfx.
Avalon | Now known as Windows Presentation Foundation, this is the new unified presentation subsystem for Windows. WPF is part of WinFX.
Indigo | The messaging subsystem part of WinFX, now known as Windows Communication Foundation. WCF will help you create secure, reliable Web services, and like WPF, it is part of WinFX.

Here's a handy list of some of the recent code names Microsoft has used. Sometimes it can seem overwhelming, but when you think back to the early 1990s and the types of programs that were being created for Windows 3.0, and then you look at the amazing software being created today, all the churn starts to make more sense. You can easily create code now that most people would only have dreamed of just a few years ago. It takes a lot of code names to develop all that technology.

—J.T.

Thanks to the following Microsoft technical experts for their help with this issue: Jonathan Caves, Shawn Farkas, Mike Fitzmaurice, Mark Fussell, Forest Gouin, Mike Harder, Ashok Kamath, Ilya Mironov, Kieran Mockford, John Nisi, Jay Roxe, Andres Sanabria, Stefan Schackow, Jason Shirk, and Steve Swartz.

IntelliSense, Microsoft, MSDN, MSN, SharePoint, Visual Basic, Visual C++, Visual Studio, Windows, WinFX, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation. Other trademarks or tradenames mentioned herein are the property of their respective owners.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


Toolbox: User-Friendly UIs, Online Data Entry Controls, and More

Toolbox
User-Friendly UIs, Online Data Entry Controls, and More
Scott Mitchell

Contents
Build Office-Like UIs
Add Rich Data Entry Controls to Your Site
Generate PDF Documents
The Bookshelf

Build Office-Like UIs

Microsoft® Office is one of the most well-known and most-used consumer products. For many people, their only experience with a computer is when using Office, be it checking e-mail, creating presentations, or writing a document. Due to Office's ubiquity, many programs strive to offer a similar end-user experience by mimicking its user interface. While the Microsoft .NET Framework ships with common controls for building Windows® Forms applications, these controls are much simpler than their Office equivalent, lacking in features, aesthetics, and pizzazz.

To overcome the limitations of the default Windows Forms controls, many developers turn to third-party controls. DotNetBar Suite by DevComponents offers developers a rich suite of controls designed to provide an Office-like user experience. DotNetBar Suite encompasses a total of 14 eye-pleasing controls that make it easy to add menus, balloons, dockable windows, panels, side bars, tabs, and other common UI elements to your .NET Windows Forms-based applications.

Being an ASP.NET developer by trade and lacking any sort of artistic skill, my strength lies in creating really unattractive applications when it comes to Windows Forms development. Regardless, with DotNetBar I was able to easily and quickly create a very appealing and user-friendly UI whose menu and toolbar looked as slick as Office. The ease of design is due in large part to the DotNetBarManager control, which, once added to a form, allows you to create and customize the UI elements through the DotNetBar Designer. The designer window provides a Layout Preview pane, making it easy to see immediately how additions and changes to the controls on the page and their properties affect the results seen by the end user.

The most recent addition to the Suite is the Ribbon control, a new user interface concept introduced with the upcoming Office 12. The Ribbon control groups common tasks together, where each task can have a unique set of controls and user interfaces. With DotNetBar Suite's Ribbon control, creating professional-looking Ribbons are a breeze. Simply add on a RibbonControl item to start and then drag in a RibbonBar control for each set of common tasks to appear in the Ribbon. Buttons, comboboxes, textboxes, or other UI items can be added to the RibbonBar controls.

Building attractive, Office-like user interfaces offers more than just eye candy—it can also make your programs more intuitive and usable. End users who are familiar with the Office UI will likely find an app that uses DotNetBar Suite more familiar than one that relies on the Windows Forms controls.

Price: $189 for a single developer license.

www.devcomponents.com/dotnetbar

Add Rich Data Entry Controls to Your Site

My first real-world experience with Web development was back in 1998 when I was tasked with moving a company's internal business processes from a paper-based system to the company intranet. The main bulk of this job required taking the data currently captured by the paper forms and providing Web pages to collect the same information.

By moving these processes to the Web, the company was able to assess the data more quickly and view aggregate information more easily. When building such an online data entry system, ease of data entry and input validation are primary concerns. Users will be remiss to move to an online-based system if it's more burdensome to enter the data through a browser than it is through pen and paper. While with paper-based forms, it's acceptable if the user forgets to include the year when writing in a date, or spells out a number by writing in "two" instead of "2", a computer is not as forgiving.

Creating a data-entry application can be done entirely with the builting ASP.NET controls. The TextBox, CheckBox, and DropDownList controls can be used to collect data, and the ASP.NET validation controls can be used to help ensure that the data entered is valid. Speaking from personal experience, while such user interfaces do meet the minimum requirements, they are often less than ideal. The ASP.NET built-in validation controls only allow for trivial validation scenarios. The built-in Web controls are often too generic for more specialized data entry forms.

Before trying to kludge ASP.NET data entry and validation controls together, give the PeterBlum.com Professional Validation And More toolkit a try. Validation And More (VAM) offers over 40 ASP.NET controls that make creating efficient, usable data entry forms with rich validation support a breeze. Improving over the default TextBox control, the VAM provides the IntegerTextBox, CurrencyTextBox, FilteredTextBox, and others, which emit JavaScript to prevent users from entering invalid characters, thus helping to reduce typos and errors. In addition, VAM offers a client-side toolkit that includes controls to help inject JavaScript in a page to make it more interactive.

Need to have controls become enabled or disabled based on the state of another control? Want to display a running total that is updated as the user enters values into a series of textboxes? All of these tasks, and more, can be done with the client-side toolkit of VAM.

Where Professional Validation And More really shines is in its validation capabilities. While ASP.NET provides 5 validation controls, VAM includes 25. Furthermore, the VAM controls can be turned on or off based on an assortment of conditions. That textbox may not be a required field unless a particular checkbox is checked—the VAM controls offer this level of flexibility. There's also ASP.NET 2.0 support, including callback support for the CustomValidator control, localization, and themes and skins support. After having used Professional Validation And More in several projects, I can't imagine going back to using vanilla textbox controls and ASP.NET-provided validation functionality.

Price: $200 for the Web server license.

www.peterblum.com/VAM

Generate PDF Documents

Over the years I've had the opportunity to use a few different PDF-generating components, the most recent one being TallPDF.NET by TallComponent. TallPDF provides two mechanisms through which you can create PDFs: programmatically, via the TallPDF object model or declaratively, through XML.

In the programmatic approach, you use classes like Document, Section, Paragraph, Image, and so on to lay out the content of the PDF. While this approach allows for the greatest degree of flexibility, I've found that creating a detailed PDF document in this manner can quickly lead to code bloat. For example, adding some formatted text involves creating a TextParagraph object, adding a Fragment object to the TextParagraph, specifying formatting settings, and then adding the TextParagraph to the appropriate Document's Section. However, if you need a highly customizable page layout based on dynamic criteria, this is a small price to pay.

TallPDF can also have its structure specified declaratively, through XML files. The XML content defines the various elements in the PDF document, such as sections, paragraphs, fragments, and so on, along with each element's style settings. The XML content can even include elements whose value is populated at run time.

For example, within a <paragraph> element you could add a dynamic <fragment> element like so: <fragment fontsize="14" font="verdana" type="MyObject.PropertyName" />. This fragment would, when rendered, insert the value of the specified object's specified property.

TallPDF is designed simply to create PDFs. That means that with TallPDF you'll need to write the code or define the XML that specifies the document's complete structure. If you want to be able to take an existing PDF document and dynamically insert content into placeholders, you'll have to upgrade to TallComponents's PDFKit.NET. In addition to TallPDF.NET and PDFKit.NET, TallComponents also offers PDFTextBox .NET, a Windows Forms component for authoring PDF documents.

Price: $499 for the standard server license.

www.tallpdf.net

The Bookshelf

Have you ever taken your commonly used code and wrapped it up inside of a class, making it easier to reuse, test, and extend? Or maybe you were tasked with creating reusable code libraries for your development team at work? Do you work for a company that sells a product aimed at the developer market, such as a Windows Forms control library, or, like TallComponents, a toolkit for creating PDFs? If you fall into any of these categories, then you know how easy it is with the .NET Framework to create, deploy, and extend reusable code libraries.

Framework Design Guidelines

While the mechanics of building a reusable .NET library are trivial, it is anything but to design an easy-to-use library that will gracefully handle future changes and additions. The naming conventions, types, members, exceptions, and design patterns that make up the library greatly affect its overall success. Code libraries that aren't intuitive, that are inconsistent, or that deviate from standard conventions are apt to frustrate its users. When designing a code library, then, why not turn to Microsoft for advice? After all, Microsoft has released the largest reusable code library for .NET—the .NET Framework Base Class Library! Framework Design Guidelines: Conventions, Idioms, and Patterns for Reusable .NET Libraries (Addison-Wesley, 2005), by Krzysztof Cwalina and Brad Adams, shares the framework design lessons learned by Microsoft since .NET development began in the late 90s.

The book begins with an interesting discussion of what qualities make up a well-designed framework. Among other things, the authors note that a well-designed framework is both expensive to design and full of trade-offs. In other words, if you want to create a great reusable code library it's going to take time to properly design it, and even then it won't be perfect.

Next, the book covers the fundamentals of framework design, hammering home the importance of making libraries that are scenario-driven and offer a low barrier to entry. Design for specific-use cases and make sure that your library makes it trivial to accomplish easy tasks. The book also examines guidelines for naming, types, members, exceptions, and so on, along with a look at designing for extensibility.

What I liked best about this book was the style in which it was written. Rather than being a stodgy tome on framework design, it is very readable thanks to the many "discussions" interjected throughout the book by various Microsoft architects. These discussions, which appear in offset boxes after particular guidelines, give the book a very conversational feel, almost as if you're sitting in on a meeting at the Microsoft campus.

For example, in the "Type Design Guidelines" chapter, after the guideline that reads, "Do favor defining classes over interfaces," Krzystof Cwalina adds that in talking with developers on the team, many have regretted shipping some API as an interface, while no one has regretted shipping a class. Jeffrey Richter then interjects, recommending that framework developers use both an interface and then an abstract base class that implements the interface, as this gives the end-user developer the luxury of choosing whether to define their own type based on your interface (for convenience) or abstract base class (for flexibility).

While this book is best suited for those creating sufficiently large frameworks, anyone who creates reusable code libraries of any size would benefit from the lessons offered.

Price: $39.99, 346 pages.

www.awprofessional.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

All prices were confirmed at press time and are subject to change.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at mitchell@4guysfromrolla.com or via his blog, at ScottOnWriting.NET.


Basic Instincts: Introducing ASP.NET Web Part Connections

Basic Instincts
Introducing ASP.NET Web Part Connections
Ted Pattison

Code download available at: BasicInstincts0602.exe (763KB)
Contents
Creating Web Parts for ASP.NET 2.0 Apps
Designing Connectable Web Parts
Timing a Web Part Connection
Defining Static Web Part Connections
Named Connection Points
Establishing Connections Dynamically

When you begin to work with the Microsoft® .NET Framework 2.0 and ASP.NET, you discover that the new Web Parts infrastructure adds some very powerful functionality to the underlying platform. In the September 2005 issue of MSDN®Magazine, Fritz Onion and I have an article on programming Web Parts titled "ASP.NET 2.0: Personalize Your Portal with User Controls and Custom Web Parts". In this month's column, I am going to build on the information we presented in that article by discussing how Web Part connections works.

For this column, I assume that you already understand the basics of Web Parts such as how to work with the WebPartManager control, Web Part zones, editors, catalogs, and persistence properties. If you don't, I encourage you to read the aforementioned article before continuing with this one.

Creating Web Parts for ASP.NET 2.0 Apps

There are two ways in which you can create a Web Part. The first involves creating a custom Web Part class that inherits from the WebPart class defined in the System.Web.UI.WebControls.WebParts namespace. When using this approach, it often makes sense to package custom Web Part classes in an assembly DLL to provide more control over reuse, versioning, and Visual Studio® 2005 integration. If you are familiar with building custom controls with previous versions of ASP.NET, many of the same techniques apply to building custom Web Parts into a DLL assembly.

A second approach for creating ASP.NET 2.0 Web Parts involves the use of User Controls. While this approach doesn't yield the same levels of reuse and version control, it does allow you to create the user interface aspects of a Web Part using the Visual Studio forms designer. If you like to create applications by dragging and dropping controls for user input, validation, and data binding onto a design surface, this approach is for you. Of course, it's also a good approach to take if you've already spent time creating a User Control that you'd like to use as a Web Part.

When you create a User Control that's specifically designed to be a Web Part, it's recommended that you implement the IWebPart interface. This allows the code behind your Web Part to programmatically assign several of its own internal Web Part properties such as its Title and TitleIconUrl.

The code sample that accompanies this month's column uses a custom base class named WebPartBase that inherits from UserControl and implements IWebPart. The definition for this base class is deployed in a source file named WebPartBase.vb in the App_Code directory. Whenever you create a new Web Part using a User Control, all you have to do is change the base class in the codebehind file to take advantage of this technique:

Partial Class WebParts_Customers
        Inherits WebPartBase

  Sub New()
    Title = "NorthWind Customer List"
    TitleIconImageUrl = "~\img\Customers.gif"
  End Sub

End Class

Designing Connectable Web Parts

Using Web Part connections, you can make it easier for your users to visualize the relationships that exist between items of data. For example, Web Part connections can model a master-detail scenario where one Web Part displaying a customer list is connected to another Web Part that displays the details of the currently selected customer. Figure 1 shows an example of what the user interface might look like with such a design.

Figure 1 Visualizing Relationships with Web Part Connections (Close Figure 1)

Web Part connections can also be used to model one-to-many relationships. As an example, one Web Part that displays a customer list can be connected to another Web Part that displays all the orders for the currently selected customer.

Another scenario that's commonly modeled using Web Part connections is query-by-form. In such a scenario, one Web Part provides a user interface that allows the user to choose search or filter criteria for a query on data such as a database table. This Web Part is then connected to another Web Part that displays the results of the query. The Web Part connection is used to pass the filter criteria from one Web Part to the other before the query is run.

Web Part connections are based on the notion of providers and consumers. A provider Web Part supplies information to one or more consumer Web Parts through a programmatic interface. The information that is exchanged between a provider and a consumer could be a simple data item such as a number or a string or it could be something more exotic such as a reference to a complex array or to a collection of custom objects.

If you have programmed Web Parts for Windows® SharePoint® Services 2.0 (WSS), you might already be familiar with its model for connecting Web Parts. In WSS, Web Parts can only be connected using a set of predefined interface pairs. Examples of these interface pairs are ICellProvider and ICellConsumer as well as IRowProvider and IRowConsumer.

The Web Part connections model in ASP.NET 2.0 is easier and more flexible than the older model in WSS because you can use your own custom interfaces. That means you are not required to use an interface definition created by someone at Microsoft. Furthermore, you are not required to work in terms of interface pairs which must be implemented by both the provider and the consumer. With ASP.NET 2.0, only the provider is required to implement an interface.

To see how it all works, let's start by creating a connection between two Web Parts. For the examples I am presenting in this month's column, I decided to use the Northwind database because it has a Customers table and an Orders table. This allows me to show you how to design Web Parts for both master-detail and one-to-many relationships. Just note that if you are using SQL Server™ 2005, the sample Northwind database is not installed during product installation. To install it you must download and run the script available on the Microsoft Web site (see Microsoft SQL Server Home).

Now, imagine you want to establish a Web Part connection between a Web Part showing a customer list and a consumer Web Part showing the details of the currently selected customer such as the one shown in Figure 1. The Web Part showing the customer list will play the role of the provider while the Web Part showing the details of the currently selected customer will be the consumer. In this situation, you want the provider to supply the CustomerID field for the currently selected customer to the consumer.

First, create a simple interface named ICustomerIDProvider:

Public Interface ICustomerIDProvider
    ReadOnly Property CustomerID() As String
End Interface
In the code sample that accompanies this month's column, I have created the provider Web Part using a User Control with a SqlDataSource control and a GridView control to display customers from Northwind. The Web Part source files are Customers.ascx and Customers.ascx.vb, shown in Figure 2.

Figure 2 Customers.ascx.vb (Close Figure 2)
Imports System
Imports System.Web
Imports System.Web.UI
Imports System.Web.UI.WebControls
Imports System.Web.UI.WebControls.WebParts

Partial Class WebParts_Customers
        Inherits WebPartBase
        Implements ICustomerIDProvider

    Sub New()
        Title = "NorthWind Customer List"
        TitleIconImageUrl = "~\img\Customers.gif"
    End Sub

    <ConnectionProvider("Customer ID Provider")> _
    Public Function GetCustomerProvider() As ICustomerIDProvider
        Return Me
    End Function

    Public ReadOnly Property CustomerID() As String _
            Implements ICustomerIDProvider.CustomerID
        Get
            If gridCustomers.SelectedDataKey Is Nothing Then
                 Return String.Empty
            Else
                 Return Me.gridCustomers.SelectedDataKey.Value
            End If
        End Get
    End Property
End Class

As you can see, WebParts_Customers acts as a provider and implements the interface used for the Web Part connection. In this case, WebParts_Customers implements the ICustomerIDProvider interface. While the most common pattern is that the provider WebPart implements the connection interface itself, it is not required to do so. The only actual requirement is that the ConnectionProvider method return an instance of the specified interface. So, as an alternative, the provider Web Part could instead return a helper object that implements the connection interface. This is typically necessary if a provider Web Part has more than one connection point with the same interface type.

The WebParts_Customers class implements the CustomerID property by returning the value from the GridView control's SelectedDataKey property. The GridView control has been set up to display records from the Northwind Customers table and it also recognizes the CustomerID field as the SelectedDataKey value.

You should notice that the WebParts_Customers class has a method named GetCustomerProvider with a return type defined in terms of the ICustomerIDProvider interface. In this case, since the Web Part itself implements the desired interface, the GetCustomerProvider can simply return the Me reference to the current instance of the class. Also note that this method has been defined using the ConnectionProvider attribute:

<ConnectionProvider("Customer ID Provider")>

The WebPartManager is responsible for connecting Web Parts at run time. When the WebPartManager sees that a Web Part contains a method defined with ConnectionProvider attribute, it knows the Web Part exposes a connection point and can, therefore, act as a provider and be connected to a consumer. When it's time to connect two Web Parts together, the WebPartManager will call the GetCustomerProvider method to obtain a strongly typed reference to the provider Web Part.

It's possible to define whether a provider Web Part accepts multiple connections to consumers. In some case it is fine for a provider to have connections to multiple consumers at the same time. In other situations, you might want to limit a provider so it can have at most a single connection to one consumer Web Part. By default, providers allow multiple connections, and consumers do not. To change this, when you apply the ConnectionProvider attribute, you can make use of the named parameter AllowsMultipleConnections, as shown here:

<ConnectionProvider("Customer ID Provider", _
    AllowsMultipleConnections:=False)>

Now that you have seen how to create a method that exposes a connection point in the provider Web Part, let's see how this is complemented within a consumer Web Part. A consumer WebPart exposes a connection point by supplying a method defined with the ConnectionConsumer attribute. The consumer's connection point method is different from the provider's connection method because it does not define a return value. Instead, it takes a single parameter defined using the connection's interface type:

<ConnectionConsumer("Customer ID Consumer")> _
Sub RegisterCustomerProvider(ByVal provider As ICustomerIDProvider)
    ... ' implementation
End Sub

Keep in mind that the names of the provider's connection point method and the consumer's connection point method are not important. The only thing that matters is that each method be defined using the ConnectionProvider attribute and the ConnectionConsumer attribute, respectively.

Now let's see how the connection is established by the WebPartManager at run time. The WebPartManager calls the provider's connection point method to acquire the reference to the provider object. Next, the WebPartManager calls the consumer's connection point method to pass it a strongly typed reference back to the provider.

Once the WebPartManager has done its work, the consumer Web Part has an active connection back to the provider Web Part. At this point, the consumer can interact directly with the provider by using this reference to access the methods and properties defined in the interface. However, the ASP.NET team recommends that consumer Web Parts should not use methods or properties on the provider interface until the PreRender phase. Specifically, they should not use methods or properties on the provider interface in the <ConnectionConsumer()> method itself. The reason is that connections may have dependencies on each other. You may have a ProviderWebPart, connected to a ProviderConsumerWebPart, connected to a ConsumerWebPart. The ConsumerWebPart cannot query the provider interface until both connections have been established, and the order in which the connections are established is up to the Framework.

A complete code listing for the consumer Web Part in CustomerDetails.ascx.vb is shown in Figure 3. You can see that the consumer Web Part is responsible for holding on to a reference so it can track its connection to the provider. This consumer Web Part contains a private field named provider that is defined in terms of the ICustomerIDProvider interface.

Figure 3 Consumer Web Part (Close Figure 3)
Partial Class WebParts_CustomerDetails
        Inherits WebPartBase

    Sub New()
        Me.Title = "Customer Details"
        Me.TitleIconImageUrl = "~\img\Customer.gif"
    End Sub

    Private provider As ICustomerIDProvider

    <ConnectionConsumer("Customer ID Consumer")> _
    Sub RegisterCustomerProvider(ByVal provider As ICustomerIDProvider)
        Me.provider = provider
    End Sub

    Protected Sub SqlDataSource1_Selecting(ByVal sender As Object, _
            ByVal e As SqlDataSourceSelectingEventArgs) _
            Handles SqlDataSource1.Selecting
        SqlDataSource1.FilterExpression = _
            "CustomerID='" & provider.CustomerID & "'"
    End Sub

End Class

When the WebPartManager calls RegisterCustomerProvider, the consumer takes the incoming reference parameter and assigns it to the provider field. Once the provider field has been assigned the reference, the consumer Web Part can then directly interact with the provider Web Part. When you design the interface for a connection, you should add whatever methods and properties will provide the interaction you require.

There are certain situations in which a consumer Web Part may be designed to work regardless of whether there is an active connection to a provider. In such a design, you will want to add contingency code into the consumer Web Part that works correctly in cases when the provider field has a value of Nothing:

If provider IsNot Nothing Then
    ... ' interact with provider
Else
    ... ' contingency code goes here if required
End If

Timing a Web Part Connection

When you begin to design Web Parts that support connections, your understanding of the timing involved is critical. Figure 4 shows a trace of a page running a provider Web Part and a consumer Web Part during an HTTP GET. While there are more page-level events that fire during an HTTP POST, the timing for when the connection is established remains the same.

Figure 4 Page Trace (Close Figure 4)

The trace information shown in Figure 4 illustrates when each of the connection methods is fired with respect to the standard ASP.NET 2.0 page-level events. You should be able to see from this trace information that the WebPartManager connects Web Parts together during the page-level LoadComplete command.

It's important to remember that the Web Part connection has not been established when page-level events PreInit, Init, PreLoad, and Load are executing. That means you should never attempt to access the provider inside a consumer Web Part's handler method that is bound to one of these events. The code inside a consumer Web Part must wait until after the LoadComplete event has executed before trying to access the provider Web Part.

In this example, the consumer Web Part handles the Selecting event of the SqlDataSource control, which fires during the page-level PreRender event. At this point it is safe to access the provider and retrieve the customer ID.

Defining Static Web Part Connections

Now that you have seen how to create Web Parts that support connections and you understand the timing involved, it's time to explore how to actually connect them together. As you will see, you can add tags directly to a Web Part page definition to establish a static Web Part connection. Web Part connections can also be established dynamically at run time through either code or user interaction. I am going to begin by showing how to create a static connection between two Web Parts because it is the most straightforward approach.

To create a static Web Part connection between two Web Parts on a page, you add a StaticConnections element inside the WebPartManager tag:

<asp:WebPartManager ID="WebPartManager1" runat="server">
    <StaticConnections>
        <asp:WebPartConnection ID="c1"
            ProviderID="Customers1"
            ConsumerID="CustomerDetails1" />
    </StaticConnections>
</asp:WebPartManager>

For this code to work properly, the provider Web Part named Customers1 and the consumer Web Part named CustomerDetails1 must also be statically defined and properly named within Web Part Zones on the same page.

It's important to remember that there can only be one WebPartManager control per page. However, in many application designs involving Web Parts, you'll find that it's convenient to add the WebPartManager to a User Control or a Master Page so that it can be reused across many pages.

When a Web Part page is based on a Master Page or is using a User Control that contains a WebPartManager control, you cannot add a second instance of the WebPartManager control to define a StaticConnections tag. For these situations, the ASP.NET 2.0 Web Part control set provides the ProxyWebPartManager control. Here's an example of how to use it:

<asp:ProxyWebPartManager ID="ProxyWebPartManager1" runat="server">
    <StaticConnections>
        <asp:WebPartConnection ID="c1"
            ProviderID="Customers1"
            ConsumerID="CustomerDetails1" />
    </StaticConnections>
</asp:ProxyWebPartManager>

The value of the ProxyWebPartManager control is that it allows you to add static connections at the page level in situations where the page cannot contain a WebPartManager tag. In the sample page default.aspx that accompanies this column, the ProxyWebPartManager must be used to establish a static Web Part connection because the WebPartManager has been encapsulated within a User Control named WebPartManagerPanel.ascx.

Named Connection Points

In the example I have built up so far, the connection between the provider Web Part and the consumer Web Part have been based on default connection points. However, it is possible for a Web Part connection method to provide a named connection point. To add a named connection point to the provider, you simply add a second string parameter to the ConnectionProvider attribute:

<ConnectionProvider("Customer ID Provider", "CustomerIDProvider")> _
Public Function GetCustomerProvider() As ICustomerIDProvider
    Return Me
End Function

One motivation for named connection points is that a provider or a consumer can have more than one connection point and, therefore, must be able to differentiate between them. To add a named connection point to the consumer Web Part, you can add a second string parameter to the ConnectionConsumer attribute:

<ConnectionConsumer("Customer ID" & "Consumer", "CustomerIDConsumer")> _
Sub RegisterCustomerProvider(ByVal provider As ICustomerIDProvider)
    Me.provider = provider
End Sub

When you begin to use named connection points, you must supply two extra attribute values when you define a StaticConnections tag at the page level for the ProviderConnectionPointID and the ConsumerConnectionPointID.

Establishing Connections Dynamically

There are times when you will want to connect Web Parts together where you will not be able to rely on static Web Part connections. For example, this will be the case if you want to connect Web Parts that have been dynamically created either through custom code or by users who have added Web Parts to a page using a Web Part catalog.

In cases where static Web Part connections cannot be used, you must connect Web Parts using a dynamic technique. This can be done either through custom code or by using the ConnectionsZone control that is included with the ASP.NET 2.0 Web Part control set.

Let's start by examining custom code that creates two Web Parts and connects them together dynamically, as you can see in Figure 5. This code creates Web Part instances from user controls for the provider and the consumer and adds them to existing Web Part zones on the hosting Web Part page. It then establishes a connection between them. Note that the Web Parts and connections will be saved as part of a user's personalization information, so they should only be added to the WebPartManager once. Additional APIs are available if you don't want the Web Parts and connections saved in this fashion.

Figure 5 Web Part Connections Established at Run Time (Close Figure 5)
'*** get WebPartManager object
Dim wpMgr As WebPartManager = _
    WebPartManager.GetCurrentWebPartManager(Me.Page)

'*** create and add provider Web Part
Dim uc1 As UserControl = Me.Page.LoadControl("~\WebParts\Customers.ascx")
uc1.ID = "wp1"
Dim wp1 As GenericWebPart = wpMgr.CreateWebPart(uc1)
wp1 = wpMgr.AddWebPart(wp1, LeftWebPartZone, 0)

'*** create and add consumer Web Part
Dim uc2 As UserControl = _
    Me.Page.LoadControl("~\WebParts\CustomerDetails.ascx")
uc2.ID = "wp2"
Dim wp2 As GenericWebPart = wpMgr.CreateWebPart(uc2)
wp2 = wpMgr.AddWebPart(wp2, RightWebPartZone, 0)

'*** get desired connection points for provider and consumer
Dim cp1 As ProviderConnectionPoint = _
  wpMgr.GetProviderConnectionPoints(wp1)("CustomerIDProvider")
Dim cp2 As ConsumerConnectionPoint = _
  wpMgr.GetConsumerConnectionPoints(wp2)("CustomerIDConsumer")

'*** dynamically establish Web Part connection
wpMgr.ConnectWebParts(wp1, cp1, wp2, cp2)
Figure 6 Connect Display Mode
Figure 6 Connect Display Mode

You can see that the technique shown in Figure 5 requires the use of a ProviderConnectionPoint object and a ConsumerConnectionPoint. These objects can be retrieved by calling methods supplied by the WebPartManager and passing the string identifiers for those named connection points.

The other technique you and your users can employ to establish a dynamic Web Part connection involves the ConnectionsZone control. To use this technique effectively, you should create a Web Part page with a right-hand task pane that contains the ConnectionsZone control. Once the user puts the page into connect view display mode, a Connect command will be added to the Web Part menu of each Web Part that exposes connection points, as shown in Figure 6.

When the user selects the Connect command, the Web Part page then displays the ConnectionsZone control and allows the user to see all the compatible connection points from all the connectable Web Parts on that page (see Figure 7). Using this technique, you can connect a consumer to a provider. Likewise, you can also connect a provider to the consumer.

Figure 7 ConnectionsZone Control (Close Figure 7)

Send your questions and comments for Ted to instinct@microsoft.com.

Ted Pattison is an author and trainer who delivers hands-on training through his company, the Ted Pattison Group. Ted is currently researching and writing a new book focusing on Windows SharePoint Services "V3" and Office 12 servers technology.


CLR Inside Out: Improving Application Startup Time

CLR Inside Out
Improving Application Startup Time
Claudio Caldato

Contents
Planning for Performance
Load Fewer Modules at Startup
Avoid Unnecessary Initialization
Place Strong-Named Assemblies in the GAC
Use NGEN
Avoid Rebasing
Application Configuration
The Impact of AppDomains

Visual Studio is a wonderful development environment, whose IntelliSense®, integrated debugging, online help, and code snippets help boost your performance as a developer. But just because you're writing code fast doesn't mean you're writing fast code.

Over the few past months, the CLR performance team met with several customers to investigate performance issues in some of their applications. One recurring problem was client application startup time. So in this column, I'll present lessons we learned analyzing these applications.

Planning for Performance

Your success in reaching your performance goals depends on the process you will be using. A good process can help you achieve the level of performance you need. These four simple rules will help:

Think in Terms of Scenarios Scenarios can help you focus on what is really important. For instance, if you are designing a component that will be used at startup, it is likely that the component will be called only once (when the app starts). From a performance point of view you want to minimize the use of external resources, such as network or disk, because they are likely to be a bottleneck. If you don't take into account that the component will be used at startup, you could spend time optimizing code paths without seeing any significant improvement. The reason is that most of the startup time will be spent loading DLLs or reading configuration files.

For startup scenarios you should analyze how many modules are loaded and how your app is going to access configuration data (files on disk, the registry, and so on). Refactoring your code by removing some dependencies or by delay-loading modules (which I'll cover later) could result in big performance improvements.

For code that is called repeatedly (such as a hash or parse function), speed is key. To optimize, you need to focus on the algorithms and minimize the cost per instruction. Data locality is also important. For example, if the algorithm touches large regions of memory, it is likely that L2 cache misses will prevent your algorithm from running at the fastest speed. Two metrics that you can use in this scenario are CPU cost per iteration and allocations per iteration. Ideally you want them both to be low. These examples should illustrate that performance is very context-dependent, and playing out scenarios can help you to tease out important variables.

Next time, before you start writing code, spend some time thinking about the scenarios in which the code will run, and identify which are the metrics and what are the factors that will impact performance. If you apply these simple recommendations, your code will perform well by design.

Set Goals It's a trivial concept, but sometimes people forget that, in order to decide if an application is fast or slow, you need to have goals to measure against. All performance goals you define (for instance, that the main window of your application should be fully painted within three seconds of application launch) should be based on what you think is the customer expectation. Sometimes it is not easy to think in terms of hard numbers early in the product development cycle (when you are supposed to set your performance goals), but it is better to set a goal and revise it later than not to have a goal at all.

Make Performance Tuning Iterative The process should consist of measuring, investigating, refining/correcting. From the beginning to the end of the product cycle, you need to measure your app's performance in a reliable, stable environment. You should avoid variability that's due to external factors (for instance, you should disable anti-virus or any automatic update such as SMS, so they don't interfere with performance test execution). Once you have measured your application's performance, you need to identify the changes that will result in the biggest improvements. Then change the code and start the cycle again.

Know Your Platform Well Before you start writing code, you should know the cost of each feature you will use. You need to know, for instance, that reflection is generally expensive so you'll need to be careful using it. (This doesn't mean that reflection should be avoided, just that it has specific performance requirements.)

Now let's move past the planning stage and tackle some coding problems. Startup time can be a problem for client applications with complex UI and connections to multiple data sources. End users expect the main window to appear as soon as they double-click on the app's icon, so startup time has a big impact on how customers view your application. Knowing the two types of startup scenarios you will be dealing with, cold and warm startup, will help you focus your efforts.

An example of cold startup is when your application starts for the first time after a reboot. Another could be if you start an application, close it, and then launch it again after a long period of time. Cold startup is dominated by hard faults. When an application starts up, if the pages required (code, static data, registry, and so forth) are not present in the OS memory manager's standby list, disk access is needed to bring those pages into memory. These page requests or page faults are known as hard faults.

In the warm startup scenario (for instance, you have already run a managed application once), it is likely that most of the pages for the main common language runtime (CLR) components are already loaded in memory from where the OS can reuse them, saving expensive disk access time. This is why a managed application is much faster to start up the second time you run it. These soft faults dominate warm startup.

Now that you know what cold and warm startup are, let's see how you can improve them. The following sections address concrete measures you can take.

Load Fewer Modules at Startup

Cold startup can benefit from loading fewer modules (as you can imagine, the result is less disk access). Even warm startup benefits from loading fewer modules because the associated CPU overhead is avoided. To figure out which modules your application loads, you can use VAdump. VAdump is included in the Platform SDK. If you type vadump –sop <process#> you will see something like Figure 1.

Figure 1 VAdump on HelloWorld.exe (Close Figure 1)
Category                        Total        Private Shareable    Shared
                           Pages    KBs         KBs       KBs       KBs
      Page table pages        33       132       132         0         0
      Other system            12        48        48         0         0
      Code/StaticData       1739      6956       552       500      5904
      Heap                   113       452       452         0         0
      Stack                    9        36        36         0         0
      Teb                      4        16        16         0         0
      Mapped data             98       392         0        28       364
      Other data             105       420       416         4         0

      Total modules         1739      6956       552       500      5904
      Total dynamic data     329      1316       920        32       364
      Total system            45       180       180         0         0
Grand Total working set     2113      8452      1652       532      6268

Module Working Set Contributions in pages
    Total   Private Shareable    Shared Module
        4         2         2         0 HelloWorld.exe
       55         3         0        52 ntdll.dll
       21         3         0        18 mscoree.dll
       16         1         0        15 ADVAPI32.dll
       51         3         1        47 KERNEL32.dll
        9         1         0         8 RPCRT4.dll
       29         2         0        27 SHLWAPI.dll
       20         2         0        18 GDI32.dll
       36         2         0        34 USER32.dll
       39         4         0        35 msvcrt.dll
      278        17         3       258 mscorwks.dll
       54         8         0        46 MSVCR80.dll
      270        14        17       239 mscorlib.ni.dll
       30         4         0        26 ole32.dll
       21         2         0        19 uxtheme.dll
       53         3         0        50 MSCTF.dll
       36         8         0        28 shell32.dll
       39         2         0        37 comctl32.dll
       14         3         0        11 comctl32.dll
       59         1         0        58 mscorjit.dll
       70         7         2        61 System.ni.dll
       34         4         1        29 System.Drawing.ni.dll
      245        17        12       216 System.Windows.Forms.ni.dll
       98        12        86         0 System.Configuration.ni.dll
       52         4         1        47 System.Xml.ni.dll
       94         7         0        87 gdiplus.dll
       12         2         0        10 OLEAUT32.DLL

There are some CLR modules that must be loaded every time but you may have some flexibility with others. In the previous example, if you are not using XML and you see System.Xml.ni.dll in the list of modules loaded, it means there is a module in your application that references System.Xml.dll. You can check the code and verify that the reference is actually necessary. If you remove the unnecessary reference, you can improve the startup profile and working set of your app.

To reduce the number of loaded modules you can also avoid code paths that require additional modules. Very often, minor code refactoring can help avoid loading additional DLLs. For example consider the following code:

void Start() {
    try {
        LaunchApplication();
    } catch (Exception e) {
        TypeInAnotherAssembly.DisplayMessageBox("error");
    }
}

When the CLR just-in-time (JIT) compiles the Start method it needs to load all assemblies referenced within that method. This means that all assemblies referenced in the exception handler will be loaded, even though they might not be needed most of the time the application is executed. In such cases, the code in the exception handler can be moved to a separate method, say ProcessException(Exception). Assuming ProcessException is large enough that it will not get inlined (otherwise the Microsoft® intermediate language, or MSIL code, would be very similar to the code generated for the previous example), the type from the other assembly is not referenced in the Start method, so the CLR JIT will not need to load it (if it is small enough to be inlined, you can use a MethodImplAttribute on the method to prevent inlining). It will be loaded only when the code throws the exception and ProcessException needs to be compiled. Of course, this is only a simple example; other times you might need to make more significant changes in order to achieve the same result.

Another way to reduce the number of modules loaded is by merging multiple modules into one. Clearly this applies only if you have control over them. In terms of the CPU, assembly loads have fusion binding and CLR assembly-loading overhead in addition to the LoadLibrary call, so fewer modules mean less CPU time. In terms of memory usage, fewer assemblies also mean that the CLR will have less state to maintain.

Avoid Unnecessary Initialization

It may seem obvious, but avoiding unnecessary initialization can also improve startup time, and it's easy to get this wrong. In the Microsoft .NET Framework, any initialization that needs to happen for a class is performed in the class constructor. If that code references other classes, it can cause a cascading effect where a large number of class constructors are executed. Figure 2 shows a simple example.

Figure 2 Cascading Class Constructors (Close Figure 2)
class DataType
{
    //Create facet checkers
    static FacetsChecker stringFacetsChecker =
        new StringFacetsChecker();
    static FacetsChecker miscFacetsChecker = new MiscFacetsChecker();
    ...
}

class StringFacetsChecker : FacetsChecker
{
    static Regex languagePattern = new Regex(
        "^([a-zA-Z]{1,8})(-[a-zA-Z0-9]{1,8})*$", RegexOptions.None);
    ...
}

The DatatType class initializes two fields, among others. This triggers the class constructors of the referenced classes. One instance is of the StringFacetChecker class, which creates an instance of the Regex class in its constructor. Even if the Regex instance is rarely used, you'll still have to pay the cost of its initialization. By doing the Regex initialization on demand rather than as part of the class constructor, you can reduce the performance cost of the DataType class for most of the applications that will use it.

Place Strong-Named Assemblies in the GAC

If an assembly is not installed in the Global Assembly Cache (GAC), you will pay the cost of hash verification of strong-named assemblies along with native code generation (NGEN) image validation if a native image for that assembly is available in the machine. In other words, if an assembly is strong named, the CLR will ensure the integrity of the assembly binary by verifying that the cryptographic hash of the assembly matches the one in the assembly manifest. But if the assembly is in the GAC, this verification can be skipped because the verification is performed as part of installation into the GAC and any update requires administrative permissions. So the CLR is basically assured that changes have not occurred.

The hash verification process is expensive because it involves touching every page in the assembly, which can be bad for cold startup. Also, the hash computation is CPU-intensive and thus impacts warm startup, too. The extent of the impact depends on the size of the assembly being verified.

If an assembly has been precompiled using NGEN but it is not installed in the GAC, then during binding, fusion needs to verify that the native image and the MSIL assembly are the same version (to avoid cases where a newer version of the assembly is deployed on the machine but a newer version of the native image is not generated). In order to accomplish that, the CLR needs to access pages in the MSIL assembly, which can hurt cold startup time.

As an aside, if you are going to deploy assemblies marked with the AllowPartiallyTrustedCallersAttribute to the GAC, make sure you have carefully reviewed them to ensure they don't make you vulnerable to any security exploits. Assemblies installed in the GAC can be called by any managed code application, including potential dangerous code downloaded from unsafe sites.

Use NGEN

The JIT compiler compiles methods as they are required during execution. This runtime compilation has several effects on performance. First, JIT compilation consumes CPU cycles. Second, compiled code lives in dynamically allocated heaps which are private to each process. This could have a big impact in scenarios like Terminal Server, where the app scalability can benefit from sharing pages among user sessions. Finally, lots of metadata pages are touched within referenced assemblies, an operation that otherwise might not be required.

CPU consumption during JIT compilation can become a bottleneck in warm startups, and the additional disk accesses incurred can also have a significant impact on cold startup scenarios.

The NGEN tool (installed with the .NET Framework) is used to precompile all methods in an assembly and installs a native image file on the machine so it can be used instead of JIT-compiling the code.

Using NGEN can improve startup because no CPU resources are consumed by the JIT compiler, fewer pages are touched (since the CLR does not need to look up metadata in referenced assemblies), and page sharing across processes is increased because code and data lie in NGEN image pages. (A large number of NGEN image pages are read-only and thus can be shared among processes).

Note the subtle tradeoff here. Using NGEN means trading CPU consumption for more disk access, since the native image generated by NGEN is likely to be larger than the MSIL image. You might be wondering if this could hurt cold startup as it results in increased disk activity. Interestingly, the CLR Performance team has observed that if JIT compilation is completely eliminated, cold startup time typically improves. This is because CLR loads far fewer pages from referenced assemblies as mentioned before, and it does not load mscorjit.dll and the MSIL assembly files.

Anyway, you should always measure cold startup time to determine the impact of NGEN on your scenario and then decide what is the best choice for you. You can get more information about NGEN at Native Image Generator and in the MSDN®Magazine article "NGEN Revs Up Your Performance With Powerful New Features" by Reid Wilkes.

Avoid Rebasing

If you use NGEN, you need to be aware that rebasing could occur when the native images are loaded in memory. If a DLL does not get to load at its preferred base address (because that address range is already allocated to another module or allocation), the OS loader will load it wherever it sees fit. This can be a very expensive operation because the loader has to update all addresses referring to locations within the DLL based on the new address where the DLL is loaded. From a performance point of view, this is bad because the OS loader has to read every page that contains an address, and once a page is written to, it becomes private to that process: the page now needs to be backed by the page file. In addition, cold startup time is impacted because there is a CPU cost associated with updating the DLL addresses, and there is more disk access because more pages must be touched. If you build more than one DLL as part of your application, rebasing will definitively occur when the application is loaded, since the default base address assigned to every DLL is always the same (0x400000).

The quickest way to approximate if one or more modules have been rebased is to use VAdump (vadump –sop <process ID>) and check if there are modules where all the pages are private. If so the module might have been rebased to a different address and thus its pages cannot be shared. You can also use tlist.exe (the Platform SDK command-line equivalent of the task manager) and cross-check if modules are loaded at their preferred address. tlist <process id> will list all modules loaded by the process with the address where they are loaded. You can find out which is the preferred base address by looking at the MSIL image using the Ildasm tool (installed with Visual Studio® 2005 under the SDK directory). If you double-click on the assembly's manifest you will see, among other information, output like that shown in Figure 3.

Figure 3 Preferred Base Address (Close Figure 3)
.module System.dll
// MVID: {62EF48F3-0E13-4BDC-A6F6-FCD87801E067}
.custom instance void [mscorlib]System.Security.UnverifiableCodeAttribute::.ctor() =
    ( 01 00 00 00 )
.imagebase 0x7a440000
.file alignment 0x00001000
.stackreserve 0x00100000
.subsystem 0x0003       // WINDOWS_CUI
.corflags 0x00000009    //  ILONLY
// Image base: 0x037B0000

The NGEN tool uses the imagebase property in the assembly's manifest to set the base address for the native image. You can also use Link –dump –headers <native image file> to find out the preferred base address. Native image files can be found under %SystemRoom%\assembly\NativeImages_<.NET Framework version>.

If you detect that one or more modules are rebased, you can fix the problem by recompiling the code, specifying a different base address with the /baseaddress option. In Visual Studio, you can set the base address option from the advanced tab in the project properties by clicking on the Advanced button.

Also, you can use the Rebase tool that ships with the platform SDK, which does not require a recompilation. Native images are usually larger than the corresponding MSIL files, so make sure you take that into account when you set the preferred base addresses to avoid conflict due to the fact that native images are bigger. The Rebase tool does not work for strongly signed assemblies because it would invalidate the signature, and the assembly would not be considered valid.

Application Configuration

Application configuration also affects startup performance. The .NET Framework provides support for retrieving application configuration settings stored in XML format. While this is a convenient feature, you need to be aware of its performance cost. If your application has simple configuration requirements and has strict startup time goals, registry entries or a simple INI file might be a better alternative.

The table in Figure 4 compares two scenarios: using an XML-based config file and a simple text file to read some configuration settings for a simple client application.

Figure 4 Configuration Approaches (Close Figure 4)

 | Total Working Set (KB) | Number of DLLs
Text File | 4120 | 18
Config File | 6880 | 24

As you can see, using config files has an impact on both working set and the number of DLLs that are loaded at startup time. There are, of course, scenarios where XML config files make sense, for instance to save complex settings (debugging/tracing options, configuration for different libraries used in your application, and so on), but they are an unnecessary cost if you have very simple config requirements. For instance, using an XML file to save only the location of the app's main window is not a wise decision.

The Impact of AppDomains

Often, especially for security reasons, you cannot avoid using multiple AppDomains. However, doing so can limit performance at startup. You can reduce the impact of multiple App Domains if you take into account the following recommendations.

Load Assemblies as Domain Neutral If an assembly is loaded as domain neutral, it means its code can be reused in another AppDomain. If the assembly is loaded in more than one AppDomain as domain bound (which is the default) each AppDomain gets its own copy of the code. This has several bad performance characteristics. First there's CPU cost. If there is a native image for the assembly, only the first AppDomain can use the native image. All other AppDomains will have to JIT-compile the code which can result in a significant CPU cost.

Next, the JIT-compiled code resides in private memory, so it cannot be shared with other processes or AppDomains. If the assembly did have an NGEN image, then the first AppDomain uses the image. All the other AppDomains have to JIT-compile the code, which means that the MSIL DLL for that assembly is also loaded. This is the worst possible scenario from a cold startup perspective because disk access for that assembly would double.

Loading the assembly as domain neutral ensures that the native image, if one exists, gets used in all AppDomains created in the application. If a native image does not exist there is still a benefit in loading assemblies as domain neutral because code gets compiled just once and then shared by all AppDomains in the application.

Enforce Efficient Cross-AppDomain Communication Needless to say, fewer cross-AppDomain calls are better from a performance standpoint. Calls with no arguments or with simple primitive type arguments offer the best performance.

For methods with reference type arguments, the more complex the object graph the poorer the performance is. In the .NET Framework 2.0, most cross-AppDomain calls have been optimized to perform much better than they did in the.NET Framework 1.1. But there are still cases where method calls may not be able to take advantage of the superior performance. Some common examples are calls on interfaces from domain-bound assemblies, calls with "ref" or "out" arguments, calls to AppDomains with shadow-copying of assemblies turned on, and partial trust AppDomains.

Use NeutralResourcesLanguageAttribute When asked for a resource, the ResourceManager first checks for the existence of satellite assemblies for the current UI culture, then for the parent of the current UI culture, and finally for the neutral culture. If the current UI culture is the neutral culture too, then the CLR can avoid two satellite assembly lookups by directly accessing the neutral culture resources. The NeutralResourcesLanguageAttribute allows a developer to tell the ResourceManager what the neutral culture is. If the ResourceManager finds that the current UI culture is the same as the neutral culture for that assembly, it will access the neutral culture resources directly. This avoids unsuccessful assembly lookups, which tend to be expensive in terms of CPU usage.

Use Serialization Wisely Using serialization (or deserialization) at startup can have a significant negative impact on performance. These are inherently expensive operations in terms of CPU and memory allocation. Moreover, a lot of code is loaded that would probably not have been needed otherwise.

If you must use serialization, it is better if you use the BinaryFormatter class instead of the XmlSerializer class. BinaryFormatter is implemented in the Base Class Library (BCL), or mscorlib.dll. XmlSerializer is implemented in System.Xml.dll, which might represent an additional DLL to load in some scenarios. BinaryFormatter also tends to be faster than XmlSerializer. But as I said, these are general guidelines that you need to test for your own scenario. The recommended approach is to measure speed and memory consumption of the two serialization approaches and then decide which one performs better in your scenario.

If you must use the XmlSerializer, you can achieve better performance if you pre-generate the serialization assembly, an option new to the .NET Framework 2.0. XmlSerializer works by generating an assembly to perform the serialization. This assembly is either generated on the fly, or can be pre-generated using the sgen tool (sgen ships with the .NET Framework 2.0 SDK). In addition to the actual serialization work, the on-the-fly generation involves the use of the CodeDOM to generate C# code. This means invoking the C# compiler to compile the code into MSIL, using reflection to load that assembly, and finally JIT-compiling code within that assembly. I don't need to remind you how expensive these operations can be.

A must read is "Improving .NET Application Performance and Scalability". In addition, you can learn more about garbage collection and performance in general at Rico Mariani's blog and Maoni Stephens' blog.

Send your questions and comments to clrinout@microsoft.com.

Claudio Caldato is a Program Manager for Performance and Garbage Collector in the Common Language Runtime team. A special thank you to Ashok Kamath for his performance work with startup scenarios and his precious notes I used for this column.


Cutting Edge: Build Providers for Windows Forms

Cutting Edge
Build Providers for Windows Forms
Dino Esposito

Code download available at: CuttingEdge0602.exe (185KB)
Contents
ASP.NET Build Providers
Build Providers in Practice
Build Providers in Detail
Comparing Build Engines
What is MSBuild?
Custom Tasks with MSBuild
Using Custom Files in Visual Studio 2005
Custom Tools in Visual Studio
Registry Configuration
Putting It All Together

In ASP.NET and Windows Forms projects, some file types are treated differently than others. For example, ASPX and ASCX files are dynamically parsed and compiled to an assembly on the fly. The contents of an XML Schema Definition (XSD) file are used to create a new, strongly typed DataSet-based object at design time. When you add a reference to an external Web service in Visual Studio®, you download the Web Services Description Language (WSDL) file and Visual Studio creates a proxy class to let you place calls to it through a special API. These special features apply to any kind of .NET application—ASP.NET, Windows® Forms, and even console applications.

In ASP.NET 2.0, the parse-and-compile model is taken to the next level. Each and every file in a project can be associated with a build provider (a component that parses the contents of the file and produces a compilable unit of code) and parsed at build time, and executable code can be generated on the fly. This build provider model allows you to extend the build engine of ASP.NET with custom code generators, and enables you to use custom syntax (typically, but not necessarily, XML-based syntax) to describe certain application-specific contents that will be translated into C# or Visual Basic® and then compiled. Build providers are a great option if you want to avoid writing boilerplate C# or Visual Basic code.

Unfortunately, Windows Forms projects don't support build providers just yet. In this column, after a brief overview of the build provider model in ASP.NET, I'll explore a couple of ways to extend Windows Forms projects in Visual Studio 2005 using a feature that simulates the build providers offered by ASP.NET 2.0.

ASP.NET Build Providers

The compilation model has undergone a significant refactoring, and in ASP.NET 2.0 it can now parse and dynamically compile a number of new file types in addition to pages, user controls, handlers, and Web services. Newly supported file types include XSD for strongly typed DataSets, WSDL for describing external Web services, resource files, themes, and master pages. Furthermore, the model is extensible and you can add your own file types to a list of those recognized and automatically handled by the host environment—be it Visual Studio 2005 or the Web server. The new compilation model eliminates the need for an explicit compilation step handled by Visual Studio when deployed files undergo changes.

Build providers serve two main purposes. They let you deploy application files that are always compiled on demand and automatically recompiled whenever necessary (for example, after a file has been changed). Of course, if you don't like leaving source files lying around the Web server, you can simply choose to precompile the site for deployment and install only binaries and fictitious endpoints for publicly accessible resources.

The second benefit is that you can use build providers as user-defined code generators. Assume, for example, that you place an XML file with a given extension in your project and bind it to a custom provider. At build time, that provider will parse the source file and generate a helper class that will be automatically compiled to an assembly. Plus, this helper class is a legitimate part of the project and enjoys full IntelliSense® support in Visual Studio 2005.

Build Providers in Practice

Now, let's look at the sample XML file in Figure 1. It describes mappings between database tables and classes. After parsing this code, a build provider can generate a C# or Visual Basic file that contains a class named Employee with as many public properties as there are columns in the Select command. The class will belong to the specified namespace and, optionally, will be marked as partial.

Figure 1 Mapping Database Tables with Classes (Close Figure 1)
<mappings namespace="MyApp.Entities">
    <mapping connectionString=
        "SERVER=.;DATABASE=northwind;Integrated Security=SSPI"
        tableName="Employees" className="Employee"
        selectCommand="SELECT firstname AS FirstName,
            lastname AS LastName, title As Title FROM employees"
        allowPartialClass="false" allowCollectionClass="false"
        collectionClassName="EmployeeCollection"
            allowGatewayClass="false"
        gatewayClassName="Customers">
    </mapping>
    <mapping connectionString=
        "SERVER.;DATABASE=northwind;Integrated Security=SSPI"
        tableName="Customers" className="Customer"
        selectCommand="SELECT companyname AS CompanyName,
            contactname AS Contact, country As Country FROM customers"
        allowPartialClass="true" allowCollectionClass="true"
        collectionClassName="CustomerCollection"
        allowGatewayClass="true" gatewayClassName="Customers">
    </mapping>
</mappings>

A provider working on the file in Figure 1 could also generate an EmployeeCollection class and even the skeleton of a gateway class if you wanted to build your data access layer in accordance with the principles of the Table Data Gateway design pattern.

Figure 2 Dynamically Created Class
Figure 2 Dynamically Created Class

You add the XML file in Figure 1 to the project and have ASP.NET generate the underlying set of classes dynamically. You'll never see the source code of these files, in much the same way you are never exposed to the real source code used to serve a request for an ASP.NET page. You will see the programming interface of the classes created from the XML description through IntelliSense. Figure 2 shows the Customer class created from the code of Figure 1 in action in a Visual Basic project in Visual Studio 2005.

Build Providers in Detail

It's important to know that build providers are an integral part of the build engine of ASP.NET. Build providers are not simply a way to extend the build engine with custom components—they are the only way to build code for ASP.NET applications. There's a build provider behind ASPX and ASCX files. And build providers are used to serve C#, Visual Basic, and many other types of files that may appear in ASP.NET applications. All these build providers are non-public, sealed classes defined in the System.Web.Compilation namespace inside the System.Web assembly. And they all derive from one common root: the BuildProvider class.

To create your own build provider, therefore, you must define a class that inherits from BuildProvider, compile it to an assembly, and override the <compilation> section of the Web.config file to add a new entry. You can register a build provider at various levels—machine, site, or application. Here's what a sample build provider might look like when registered:

<compilation>
    <buildProviders>
        <add extension=".map" type="MsdnMag.MapBuildProvider" />
        ...
    </buildProviders>
</compilation>

The structure of a build provider looks like the following:

Public Class MyBuildProvider
        Inherits BuildProvider

    Public Sub New ()
    End Sub

    Public Overrides Sub GenerateCode(ByVal ab As AssemblyBuilder)
        Dim fileName As String = MyBase.VirtualPath
        Dim code As CodeCompileUnit = BuildCodeTree(fileName)
        ab.AddCodeCompileUnit(this, code)
    End Sub

    ...

End Class

The GenerateCode method is virtual but has an empty body in the base class. At the very minimum, you need to override it to build a provider. The VirtualPath property on the base class provides the virtual name of the source file. The BuildCodeTree is a provider-specific function that parses the source file and returns a CodeDOM tree representing the class to create. The CodeDOM tree is then passed as an argument to the AddCodeCompileUnit method of the AssemblyBuilder object. The assembly builder object is a helper class passed to GenerateCode that hides all the details involved in generating the source code for the CodeDOM in the right language and compiling the assembly in a valid location. CodeDOM is the Microsoft® .NET Framework API that you use to represent any piece of code as an object graph, similar in form to an Abstract Syntax Tree for those of you familiar with compiler construction. The graph, in turn, can be passed to a language-specific code provider object and translated into compilable code.

Once you have built your own code generator to return a CodeDOM hierarchy, you're pretty much finished if you're working with ASP.NET. Note that you are in no way bound to CodeDOM to generate code on the fly for an ASP.NET build provider. By overriding a different set of methods, you can also pass the code to compile through a TextWriter object. In this case, you just output code as plain strings.

If you're interested in a step-by-step guide on how to create build providers, I recommend reading Fritz Onion's blog posting located at Jaw-dropping experience with custom build providers. The post was a great source of inspiration for me, and I imagine it will be for you, too.

Comparing Build Engines

As of the .NET Framework 2.0, build providers are an exclusive feature of ASP.NET. Period. But it would be amazing to have compilers that accept external plug-ins to preprocess file types and perform some conversion into the proper language. In this scenario, you declaratively set a list of associations between file types, and build plug-ins to pass this information to the compiler. When the C# compiler meets, say, a .map file on its command line, it knows that it has to invoke a code provider component to get the real C# source to work on. (There was a plan for something like this in the early days of the .NET Framework 2.0. Unfortunately, this model is not here yet.)

ASP.NET build providers extend the tailor-made compilation engine of ASP.NET applications. You can't really get anything like that in Windows Forms until the compilers are refactored or a brand new compilation model for Windows applications is invented and implemented. That said, you can quite easily simulate build providers for Windows Forms with a little help from the Visual Studio environment. After all, both Visual Studio .NET 2003 and Visual Studio 2005 let you add an XSD file to a project and have the IDE generate a corresponding C# or Visual Basic .NET reference file. While this is radically different than using ASP.NET 2.0 build providers, the final effect is very similar. You still have an external and reusable component to generate some good code for you. The key differences are in the timing and the actor.

For Windows Forms applications, you take advantage of the characteristics of the Visual Studio build machinery. The file is created locally and then added to the project. It is then passed on to the language compiler as part of the project. In ASP.NET 2.0, this same sequence of steps is abstracted by the compiler infrastructure. To an external observer, in ASP.NET you really compile, say, .map files to assemblies. In Windows Forms, you rely on Visual Studio to generate C# or Visual Basic files with respect to a .map input file. This code generation can happen both at design and build time.

In both Visual Studio .NET 2003 and Visual Studio 2005, content files such as the .map file shown in the code in Figure 1 can be preprocessed using a special component known as a custom tool. In Visual Studio 2005, you have another option: using an MSBuild custom task in the project file.

What is MSBuild?

MSBuild is the new build system integrated in Visual Studio 2005 and also the first release of a new build system designed for the Microsoft platform. Built entirely in managed code, MSBuild processes a new XML-based project file format and allows developers to describe what items need to be built and how to build them. MSBuild doesn't require Visual Studio 2005, so you can build applications on machines even if those machines don't have Visual Studio 2005 installed. The project file format supports a syntax that enables developers to define reusable build rules in separate files. These files, known as targets, can be imported in different projects.

An in-depth look at MSBuild is far beyond the scope of this column, but is worth tackling in the future. For a primer on MSBuild, take a look at the wiki located at Channel9 Wiki: HomePage. A variety of blogs discuss useful tips for MSBuild development. My favorite is the development team's blog located at MSBuild Team Blog.

For this discussion, the key point I want to express about MSBuild is that when you create a Visual Studio 2005 project, the source code of the .csproj or .vbproj file is nothing more than an MSBuild script. This means that if you want Visual Studio 2005 to treat certain content files in a particular way, you must use the proper MSBuild syntax.

Custom Tasks with MSBuild

An MSBuild project file contains references to things like targets, property groups, item groups, and tasks. A target is any final result you expect from the build process. For example, if you're going to compile a few C# files, an assembly is the expected target. Each project file can have one or more targets. Targets refer to a collection of tasks, each with its own set of properties to carry out a particular operation. A group of items represents a collection of related input files to be processed by the system. Finally, a property group defines project-wide properties that can be used to configure tasks and targets.

Figure 3 shows the MSBuild file that compiles a couple of Visual Basic files and generates an assembly named MyApp.exe. The contents of a Visual Studio 2005 .vbproj file are more complex and articulated, but follow the same principles and schema.

Figure 3 Sample MSBuild Project File (Close Figure 3)
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
    <PropertyGroup>
        <AssemblyName>MyApp</AssemblyName>
    </PropertyGroup>

    <ItemGroup>
        <Compile Include="test1.vb" />
        <Compile Include="test2.vb" />
    </ItemGroup>

    <Target Name="Build" Inputs="@(Compile)"
            Outputs="$(OutputPath)$(AssemblyName).exe">
       <Vbc Sources="@(Compile)"
            OutputAssembly="$(OutputPath)$(AssemblyName).exe" />
    </Target>
</Project>

The visible part of the MSBuild engine is a console application (msbuild.exe) that is a thin wrapper around the Build API. This API is implemented in the Microsoft.Build.* assemblies in the installation path of the .NET Framework 2.0.

A task represents an action you want to accomplish at some point during the build process. Predefined Visual Studio 2005 targets for C# and Visual Basic already define a number of tasks and properties that you can use in your own scripts. An MSBuild task is a class that inherits from the Task class defined in the Microsoft.Build.Utilities namespace.

Figure 4 illustrates the source code of a sample task. The task, named MakeDirectories, creates a set of directories in the current folder. The names of the directories to be created are specified through a public property. The following code snippet shows one possible use of the MakeDirectories task:

<ItemGroup>
    <PublishDir Include="c:\temp\publish" />
</ItemGroup>
<Target Name="AfterBuild">
    <MakeDirectories Condition="!Exists('@(PublishDir)')"
        Directories="@(PublishDir)" />
    <Copy SourceFiles="$(TargetPath)"
        DestinationFolder="@(PublishDir)" />
</Target>

Figure 4 MakeDirectories Sample MSBuild Task (Close Figure 4)
Imports System.IO
Imports Microsoft.Build.Utilities

Public Class MakeDirectories
        Inherits Task

    Private _directories() As String

    Public Property Directories() As String()
        Get
            Return _directories
        End Get
        Set(ByVal value As String())
            _directories = value
        End Set
    End Property

    Public Overrides Function Execute() As Boolean
        For Each directory As String In _directories
            Directory.CreateDirectory(directory)
        Next
        Return True
    End Function

End Class

To preprocess a custom file to a C# or Visual Basic class, you need a custom task that takes input and output file paths and converts the custom file to a class. Figure 5 shows the source code of the custom task named MapGeneratorTask. The task has a required property, FileName, which indicates the name of the source file to be preprocessed and another attribute flagged with the <Output()> attribute. The output property indicates an element that represents the task's contribution to the output of the parent target.

Figure 5 MapGeneratorTask (Close Figure 5)
Imports System
Imports System.CodeDom
Imports System.CodeDom.Compiler
Imports System.Text
Imports System.IO
Imports Microsoft.Build.Framework
Imports Microsoft.Build.Utilities

Namespace MsdnMag
Public Class MapGeneratorTask
        Inherits Task
    Private _language As String = "CS"
    Private _fileName As String
    Private _outFileName As String

    Public Property Language() As String
        Get
            Return _language
        End Get
        Set(ByVal value As String)
            _language = value
        End Set
    End Property

    <Required()> _
    Public Property FileName() As String
        Get
            Return _fileName
        End Get
        Set(ByVal value As String)
            _fileName = value
        End Set
    End Property

    <Output()> _
    Public Property OutputFileName() As String
        Get
            Return _outFileName
        End Get
        Set(ByVal value As String)
            _outFileName = value
        End Set
    End Property

    Public Overrides Function Execute() As Boolean
        Log.LogMessage("Transforming into a " & Language & _
            " file: " & FileName)

        ' Read the file contents
        Dim reader As StreamReader = New StreamReader(FileName)
        Dim inputFileContent As String = reader.ReadToEnd()
        reader.Close()

        ' Build a CodeDOM tree of the file to return
        Dim unit As CodeCompileUnit = _
            CodeDomHelpers.BuildCodeTreeFromMapFile(inputFileContent)

        ' Output to the current language
        Dim writer As StreamWriter = New StreamWriter(OutputFileName)
        Dim provider As CodeDomProvider = _
            CodeDomProvider.CreateProvider(_language)
        provider.GenerateCodeFromCompileUnit(unit, writer, Nothing)
        writer.Close()

        ' Success
        Return True
    End Function
End Class
End Namespace

The task takes, as input, a custom file and transforms it into a Visual Basic or C# class. This will be added to the output of the target and can be used as input to another MSBuild target. In Figure 5, the CodeDomHelpers static class contains the logic necessary to create a CodeDOM tree out of the input file. There are still a few points to clarify, the first of which is to associate the task with .map files. In addition, you'll need to know how to register the custom task with MSBuild and how to pass the newly created files to the core build engine for actual compilation.

Figure 6 shows a sample MSBuild project file that preprocesses all .map files in the current directory using the sample MapGeneratorTask and then passes the results to the Visual Basic compiler. The <UsingTask> node declares the new task and indicates full class name and assembly. The project file contains two targets: Build and Mapping. The Build target compiles the Visual Basic files listed through a <Compile> node and all the input it receives from the Mapping target. The Inputs attribute of the target lists the input sources. The DependsOnTarget attribute indicates that the Mapping target must be built before proceeding with the core compilation. All input that the main target receives is passed to the Vbc task (the Csc task for C# sources) for actual file compilation.

Figure 6 Using a Custom Task for .map Files (Close Figure 6)
<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <UsingTask TaskName="MsdnMag.MapGeneratorTask"
        AssemblyFile="c:\MyMsBuild\Bin\MyBuildTasks.dll" />

    <PropertyGroup>
        <AssemblyName>MyAsm</AssemblyName>
    </PropertyGroup>

    <ItemGroup>
        <Compile Include="test.vb" />
    </ItemGroup>

    <ItemGroup>
        <Content Include="*.map" />
    </ItemGroup>

    <Target Name="Build"
        Inputs="@(Mapping);@(Compile)"
        Outputs="$(OutputPath)$(AssemblyName).exe"
        DependsOnTargets="Mapping">
        <Vbc Sources="@(Compile);@(Mapping)"
            OutputAssembly="$(OutputPath)$(AssemblyName).exe" />
    </Target>

    <Target Name="Mapping">
        <MapGeneratorTask Language="VB"
            FileName="%(Content.FullPath)"
            OutputFileName="$(OutputPath)%(Content.FullPath).vb">
            <Output TaskParameter="OutputFileName" ItemName="Mapping"/>
        </MapGeneratorTask>
    </Target>

</Project>

The following code snippet shows a realistic usage of the MapGeneratorTask. The FileName property owes its value to the FullPath metadata property of the Content node where .map files are bound. The OutputFileName property is created in the project's output path and has the same name as the input file plus the language-specific extension:

<Target Name="Mapping">
    <MapGeneratorTask Language="VB" FileName="%(Content.FullPath)"
            OutputFileName="$(OutputPath)%(Content.FullPath).vb">
        <Output TaskParameter="OutputFileName" ItemName="Mapping"/>
    </MapGeneratorTask>
</Target>

The Output node indicates which of the task's properties are used to form the target's output and the name used to reference it in other targets. Figure 7 shows the msbuild.exe application at work on the project file you saw in Figure 6. In the end, the final executable has been built from one source class and a couple of .map files. The build task embedded in the MSBuild project file actually works on three source files—the original file and two additional files that were created from .map files. Note that by writing a slightly more complex MSBuild script, you can make these source classes invisible or temporary by generating them in a separate folder or removing them through a post-build target with a RemoveFile task.

Figure 7 Msbuild.exe in Action on .map Files (Close Figure 7)

Using Custom Files in Visual Studio 2005

For the example I just discussed, I utilized a custom project file. Another option, however, is to extend the project file that Visual Studio 2005 creates for a Visual Basic or C# project, enabling it to support .map files via a custom task. This is a two-step procedure.

First, register the new task with the <UsingTask> element. Next, create a <Target> section to manage .map files; the Mapping target shown earlier is just fine. You don't need to explicitly modify the project file to add <Content> sections for .map files. It's sufficient to add .map files to the Visual Studio 2005 project and select a Content build action for them. This will automatically create the following markup underneath the ItemGroup section with files to process:

<Content Include="yourfile.map"/>

One more thing. You still need to inform the principal target about the new dependency. Given the default structure of the .csproj and .vbproj files, here's what you have to enter:

<PropertyGroup>
    <CoreBuildDependsOn>
        $(CoreBuildDependsOn);
        [YourTarget]
    </CoreBuildDependsOn>
</PropertyGroup>

Of course, the YourTarget placeholder will be replaced by the actual name of the target. Note that when you edit project files outside Visual Studio 2005, you will see a security dialog warning that informs you the project has been customized. Simply choose to load the project normally and go.

Custom Tools in Visual Studio

The second possible route for processing custom .map files in Windows Forms is to use custom tools. Custom Tool support is a little known feature of Visual Studio .NET (available since version 2002) and basically refers to the IDE's ability to use external components to preprocess non-class files. Figure 8 shows where you can set the Custom Tool of any item added to the project in the Windows Forms designer of Visual Studio 2005. Built-in custom tools exist for a variety of content files including XSD, RESX, and WSDL. And these represent the chief technology that enables the creation of reference files for certain well-known resources. Basically, whenever you get a typed DataSet in response to the addition of an XSD file, there's a custom tool working behind the scenes.

Figure 8 Setting the Custom Tool
Figure 8 Setting the Custom Tool

In the October 2002 issue of MSDN®Magazine, Jon Flanders and Chris Sells wrote the article "Top Ten Cool Features of Visual Studio .NET Help You Go From Geek to Guru". In the article, custom tools were number two on the list. Custom tools work nearly the same way in Visual Studio 2005.

A Visual Studio custom tool is a COM object that is expected to implement a few interfaces, including IVsSingleFileGenerator and IObjectWithSite. If you want to write custom tools with managed code, you need to employ the BaseCodeGeneratorWithSite class. The class is defined in the newest Microsoft.VisualStudio.Design system assembly, but it is marked as internal and cannot be used to derive from. So what do you do? Well, the development team provided equivalent code in an assembly that I've included in this month's sample code. Say, for example, that a custom tool is a class that inherits from the BaseCodeGeneratorWithSite class. The code in Figure 9 shows a Visual Studio custom tool that uses the same CodeDOM generator employed in the MSBuild task to create a class file based on a .map file.

Figure 9 Visual Studio Custom Tool for .map Files (Close Figure 9)
Imports System
Imports System.Collections.Generic
Imports System.Runtime.InteropServices
Imports Microsoft.CustomTool
Imports System.CodeDom
Imports System.CodeDom.Compiler
Imports System.IO

Namespace MsdnMag

<ComVisible(True)> _
<Guid("DB336537-7F85-441b-909B-97595D6E9590")> _
<CLSCompliant(False)> _
Public Class TableMappingGenerator
    Inherits BaseCodeGeneratorWithSite

    Public Overrides Function GetDefaultExtension() As String
        Return ".map" + MyBase.GetDefaultExtension()
    End Function

    Protected Overrides Function GenerateCode( _
            ByVal inputFileName As String, _
            ByVal inputFileContent As String) As Byte()

        ' Build a CodeDOM tree of the file to return
        Dim unit As CodeCompileUnit =
            CodeDomHelpers.BuildCodeTreeFromMapFile(inputFileContent)

        ' Output to the current language
        Dim writer As New StringWriter
        CodeProvider.GenerateCodeFromCompileUnit(unit, writer, Nothing)
        Dim code As String = writer.ToString()
        writer.Close()

        Return System.Text.Encoding.ASCII.GetBytes(code)
    End Function
End Class

End Namespace

Registry Configuration

You don't have to be a rocket scientist to understand how the code of a custom tool works. It consists of overriding the GenerateCode method to return an array of bytes and optionally the GetDefaultExtension method to specify the extension of the generated file. Note that, unlike MSBuild tasks, the Visual Studio Custom Tool feature is specifically designed to transform one file into another with different content. More importantly, you don't have to do anything to force the compilation of these dynamically created reference files. Visual Studio automatically tweaks the project file to request compilation.

The trickiest part of the story is how you make a custom tool visible to the Visual Studio 2005 IDE. First and foremost, you need to register the custom tool class as a COM object. You do this with a call to regasm.exe.

regasm /codebase YourCustomTool.dll

If you don't install the assembly that you are registering into the global assembly cache, you'll need to use the /codebase switch. Next, you need to list your custom tool with the built-in custom tools in the registry. The entry to look for is: HKLM\SOFTWARE\Microsoft\VisualStudio\8.0\Generators. You'll find three subkeys, each with a GUID. These GUIDs correspond to installed languages (in other words, you could opt to register a custom tool for one language but not another). For brevity, I'm not showing the entire GUID here but will limit it to the first three characters. The GUID that starts with "164" is for VB; "e6f" is for J#; "fae" is for C#. You choose your language (or languages) and create a key with the name of the custom tool. This name is important as it represents the string to type in the dialog box shown in Figure 8. The key requires two entries—CLSID and GeneratesDesignTimeSource. The former is a string that points to the GUID of the custom tool class—the value of the [Guid] attribute in Figure 9. The latter is a DWORD entry and must be set to 1.

Putting It All Together

You're now all set and can start working with a sample Windows Forms project with a form and the nwind.map file shown in Figure 1. Make sure the .map file has a build action of Content and remember to set Custom Tool to the name of the generator—for example, TableMappingGenerator. If you double-click the .map file and click Run Custom Tool, you should see a child node appear beneath the .map file. If not, make sure you have the Show All Files feature turned on (see Figure 2).

Figure 10 offers a preview of the code you get from either the custom tool or the custom build action. Incidentally, it is the same code you can obtain in ASP.NET 2.0 using a build provider. Actually, all that you need is a made-to-measure CodeDOM library to describe the code you want to obtain with respect to a given input file. Once you have such a library up and running, you can create a variety of wrappers for it—build providers, custom tools, and MSBuild tasks. And note that the .map file can generate a partial class. When this happens, you can extend the code of the dynamically generated class through other partial classes throughout the project.

Figure 10 Class Behind the .map File (Close Figure 10)
'-------------------------------------------------------------------------
' <auto-generated>
'  This code was generated by a tool.
'  Runtime Version:2.0.50215.44
'
'  Changes to this file may cause incorrect behavior and will be lost if
'  the code is regenerated.
' </auto-generated>
'-------------------------------------------------------------------------

Option Strict Off
Option Explicit On

Imports System.Collections.Generic

Namespace MsdnMag

    Partial Public Class Customer

        Private _companyname As String
        Private _contact As String
        Private _country As String

        Public Overridable Property CompanyName() As String
            Get
                Return Me._companyname
            End Get
            Set
                Me._companyname = value
            End Set
        End Property
        Public Overridable Property Contact() As String
            Get
                Return Me._contact
            End Get
            Set
                Me._contact = value
            End Set
        End Property

        Public Overridable Property Country() As String
            Get
                Return Me._country
            End Get
            Set
                Me._country = value
            End Set
        End Property
    End Class

    Public Class CustomerCollection
        Inherits List(Of Customer)
    End Class

    Partial Public Class Customers
        Public Shared ReadOnly Property ConnectionString() As String
            Get
             Return "SERVER=.;DATABASE=northwind;Integrated Security=SSPI"
            End Get
        End Property
    End Class
End Namespace

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Reach him at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Wicked Code: The SQL Site Map Provider You've Been Waiting For

Wicked Code
The SQL Site Map Provider You've Been Waiting For
Jeff Prosise

Code download available at: WickedCode0602.exe (117KB)
Contents
The New and Improved SQL Site Map Provider
Using SqlSiteMapProvider
Creating the Site Map Database

Now that ASP.NET 2.0 is a shipping product, it seems appropriate to revisit an issue that tops the new features wish lists of many developers: a SQL Server™ site map provider.

As you probably know, ASP.NET 2.0 vastly simplifies the process of building data-driven site navigation interfaces. You build a site map, slap a SiteMapDataSource control onto the page, and bind a Menu or TreeView control to the SiteMapDataSource. The SiteMapDataSource uses the default site map provider (typically XmlSiteMapProvider) to read the site map, and then passes site map nodes to the Menu or TreeView, which renders the nodes into HTML. For good measure, you can add a SiteMapPath control to the page, too. SiteMapPath displays the familiar breadcrumb element showing the path to the current page. Figure 1 shows the components of the site navigation subsystem and illustrates how they fit together.

Figure 1 Navigation System
Figure 1 Navigation System

The one drawback to site navigation is that XmlSiteMapProvider is the one and only site map provider included in the box with ASP.NET 2.0, which means that site maps must be stored in XML files. Even before ASP.NET 2.0 shipped, developers were clamoring for a means to store site maps in databases.

The June 2005 installment of Wicked Code presented one solution in the form of a custom site map provider named SqlSiteMapProvider. Unlike XmlSiteMapProvider, SqlSiteMapProvider reads site maps from SQL Server databases. And it takes advantage of the ASP.NET 2.0 provider architecture to integrate seamlessly with the site navigation subsystem. Just create the site map database and register SqlSiteMapProvider as the default provider, and then, like magic, everything else just works.

So, why do I want to revisit an issue that has been addressed before? Three reasons. First, after spending most of the summer digging deep into the provider architecture, I realized that my original SqlSiteMapProvider implementation needed a few improvements in order to be more consistent with the built-in XmlSiteMapProvider. Second, ASP.NET 2.0 saw some significant changes between Beta 2 and RTM, and I wanted to update SqlSiteMapProvider for the shipping platform. Finally, and most importantly, I wanted to add a feature that several readers e-mailed me about: automatic reloading of the site map following a change to the site map database. Without this feature, it seems that many readers consider a site map stored in SQL Server to be about as useful as an airplane without wings—something I learned about the hard way this summer when my favorite radio-control airplane clipped a fence post. But that's a story for another day.

The New and Improved SQL Site Map Provider

The upshot of this new and improved version of SqlSiteMapProvider is that it meets all my objectives and more. Its architecture is consistent with that of the built-in providers; it compiles and runs on the retail release of ASP.NET 2.0; and it uses the ASP.NET 2.0 SqlCacheDependency class to monitor the site map database and refresh the site map if changes occur. XmlSiteMapProvider has a similar feature that reloads the site map if the underlying XML site map file changes.

Figure 2 lists the source code for the new SqlSiteMapProvider. The Initialize method, which is present in all providers, is a special one that ASP.NET calls after loading the provider. ASP.NET passes Initialize a NameValueCollection named config that contains all the configuration attributes (and their values) found in the configuration element that registered the provider. The Initialize method's job is to apply configuration settings and do anything else required to initialize the provider. SqlSiteMapProvider's Initialize method performs the following tasks:

  1. It demands SqlClientPermission to make sure it has permission to access databases. Without that permission, SqlSiteMapProvider is powerless to operate.
  2. It calls the base class's Initialize method, which, among other things, processes the securityTrimmingEnabled configuration attribute, if present.
  3. It processes the connectionStringName and sqlCacheDependency configuration attributes, if present.
  4. It throws an exception if the element that registers the provider contains unrecognized configuration attributes.

Figure 2 SqlSiteMapProvider.cs (Close Figure 2)
[SqlClientPermission(SecurityAction.Demand, Unrestricted=true)]
public class SqlSiteMapProvider : StaticSiteMapProvider
{
    ... // static error messages omitted

    const string _cacheDependencyName = "__SiteMapCacheDependency";

    private string _connect;
    private string _database, _table;
    private bool _2005dependency = false;
    private int _indexID, _indexTitle, _indexUrl, _indexDesc,
        indexRoles, _indexParent;
    private Dictionary<int, SiteMapNode> _nodes =
        new Dictionary<int, SiteMapNode>(16);
    private SiteMapNode _root;
    private readonly object _lock = new object();

    public override void Initialize (
        string name, NameValueCollection config)
    {
        // Verify parameters
        if (config == null) throw new ArgumentNullException("config");
        if (String.IsNullOrEmpty(name)) name = "SqlSiteMapProvider";

        // Add a default "description" attribute to config if the
        // attribute doesn't exist or is empty
        if (string.IsNullOrEmpty(config["description"]))
        {
            config.Remove("description");
            config.Add("description", "SQL site map provider");
        }

        // Call the base class's Initialize method
        base.Initialize(name, config);

        // Initialize _connect
        string connect = config["connectionStringName"];
        if (String.IsNullOrEmpty(connect))
            throw new ProviderException(_errmsg5);
        config.Remove("connectionStringName");

        if (WebConfigurationManager.ConnectionStrings[connect] == null)
            throw new ProviderException(_errmsg6);
        _connect = WebConfigurationManager.ConnectionStrings[
            connect].ConnectionString;
        if (String.IsNullOrEmpty(_connect))
            throw new ProviderException(_errmsg7);

        // Initialize SQL cache dependency info
        string dependency = config["sqlCacheDependency"];

        if (!String.IsNullOrEmpty(dependency))
        {
            if (String.Equals(dependency, "CommandNotification",
                StringComparison.InvariantCultureIgnoreCase))
            {
                SqlDependency.Start(_connect);
                _2005dependency = true;
            }
            else
            {
                // If not "CommandNotification", then extract
                // database and table names
                string[] info = dependency.Split(new char[] { ':' });
                if (info.Length != 2)
                    throw new ProviderException(_errmsg8);
                _database = info[0];
                _table = info[1];
            }

            config.Remove("sqlCacheDependency");
        }

        // Throw an exception if unrecognized attributes remain
        if (config.Count > 0)
        {
            string attr = config.GetKey(0);
            if (!String.IsNullOrEmpty(attr))
                throw new ProviderException(
                    "Unrecognized attribute: " + attr);
        }
    }

    public override SiteMapNode BuildSiteMap()
    {
        lock (_lock)
        {
            // Return immediately if this method has been called before
            if (_root != null) return _root;

            // Query the database for site map nodes
            using(SqlConnection connection = new SqlConnection(_connect))
            {
                SqlCommand command = new SqlCommand(
                    "proc_GetSiteMap", connection);
                command.CommandType = CommandType.StoredProcedure;

                // Create a SQL cache dependency if requested
                SqlCacheDependency dependency = null;
                if (_2005dependency)
                    dependency = new SqlCacheDependency(command);
                else if (!String.IsNullOrEmpty(_database) &&
                         !String.IsNullOrEmpty(_table))
                    dependency = new SqlCacheDependency(_database,
                        _table);

                connection.Open();
                SqlDataReader reader = command.ExecuteReader();
                _indexID = reader.GetOrdinal("ID");
                _indexUrl = reader.GetOrdinal("Url");
                _indexTitle = reader.GetOrdinal("Title");
                _indexDesc = reader.GetOrdinal("Description");
                _indexRoles = reader.GetOrdinal("Roles");
                _indexParent = reader.GetOrdinal("Parent");

                if (reader.Read())
                {
                    // Create the root SiteMapNode and add it to site map
                    _root = CreateSiteMapNodeFromDataReader(reader);
                    AddNode(_root, null);

                    // Build a tree of SiteMapNodes under the root node
                    while (reader.Read())
                    {
                        // Create another site map node and add it
                        AddNode(CreateSiteMapNodeFromDataReader(reader),
                            GetParentNodeFromDataReader(reader));
                    }

                    // Use the SQL cache dependency
                    if (dependency != null)
                    {
                        HttpRuntime.Cache.Insert(_cacheDependencyName,
                            new object(), dependency,
                            Cache.NoAbsoluteExpiration,
                            Cache.NoSlidingExpiration,
                            CacheItemPriority.NotRemovable,
                            new CacheItemRemovedCallback(
                              OnSiteMapChanged));
                    }
                }
            }

            // Return the root SiteMapNode
            return _root;
        }
    }

    protected override SiteMapNode GetRootNodeCore ()
    {
        return BuildSiteMap();
    }

    ... // Helper methods CreateSiteMapNodeFromDataReader and
        // GetParentNodeFromDataReader
}

The sqlCacheDependency attribute is the one that allows you to take advantage of SqlSiteMapProvider's ability to refresh the site map if the underlying database changes. Setting sqlCacheDependency to "SiteMapDatabase:SiteMap" instructs the provider to refresh the site map if a table named SiteMap in a SQL Server 7.0 or a SQL Server 2000 database changes. ("SiteMapDatabase" indirectly specifies the database name by referring to an entry in the <databases> section of the <sqlCacheDependency> configuration section.) If the site map lives in a SQL Server 2005 database, you set sqlCacheDependency equal to "CommandNotification" instead. That's the high-level view; the details will come shortly.

The heart of SqlSiteMapProvider is its BuildSiteMap method. This method is called by ASP.NET sometime after the provider is loaded to build the site map, which is simply a collection of SiteMapNodes linked together to form a tree. Each SiteMapNode represents one node in the site map and is distinguished by the following properties: Title, which specifies the text that a navigation control displays for the node; Url, which specifies the URL the user is sent to when the node is clicked; Description, which specifies the descriptive text that's displayed if the cursor hovers over the node; and Roles, which specifies the role or roles that are permitted to view the node if security trimming is enabled ("*" if anyone can view it). Multiple roles can be specified using commas or semicolons as separators.

SqlSiteMapProvider's implementation of BuildSiteMap queries the site map database. Then it iterates over the records one by one, transforming them into SiteMapNodes. At the end, it hands the site map over to ASP.NET by returning a reference to the root site map node. And because all provider code outside the Initialize method must be thread-safe, SqlSiteMapProvider wraps everything in BuildSiteMap in a lock statement in order to serialize concurrent thread accesses.

In addition to querying the database and building the site map, BuildSiteMap also creates the basic infrastructure that enables SqlSiteMapProvider to refresh the site map if the site map database changes. If the configuration element that registered the provider contains a sqlCacheDependency="CommandNotification" attribute, BuildSiteMap creates a SQL Server 2005-compatible SqlCacheDependency object that wraps the SqlCommand used to query the site map database:

// In Initialize
SqlDependency.Start(_connect);

// In BuildSiteMap
dependency = new SqlCacheDependency(command);

If, on the other hand, the configuration element contains the kind of sqlCacheDependency configuration string used in SQL Server 7.0 or SQL Server 2000, (for example, "SiteMapDatabase:SiteMap"), BuildSiteMap creates a SqlCacheDependency object that wraps the supplied database name and table name:

// Initialize
_database = info[0];
_table = info[1];

// BuildSiteMap
dependency = new SqlCacheDependency(_database, _table);

Regardless of which type of SqlCacheDependency it created, BuildSiteMap later inserts a trivial object into the ASP.NET application cache and creates a dependency between that object and the database by including the SqlCacheDependency in the call to Cache.Insert:

if (dependency != null)
{
    HttpRuntime.Cache.Insert(_cacheDependencyName,
        new object(), dependency,
        Cache.NoAbsoluteExpiration,
        Cache.NoSlidingExpiration,
        CacheItemPriority.NotRemovable,
        new CacheItemRemovedCallback(OnSiteMapChanged));
}

The final parameter to Cache.Insert instructs ASP.NET to call the provider's OnSiteMapChanged method if the SqlCacheDependency triggers a cache removal—that is, if the site map database changes. OnSiteMapChanged clears out the old site map and calls BuildSiteMap to build a new one.

It may seem odd that SqlSiteMapProvider uses the ASP.NET application cache when there's really nothing for it to cache (after all, the object it inserts into the cache is simply a marker that contains no meaningful data), but doing so enables SqlSiteMapProvider to capitalize on a key feature of ASP.NET 2.0.

ADO.NET 2.0 has a SqlDependency class that enables application code to query SQL Server 2005 databases and receive callbacks if the underlying data changes, but it has no comparable feature for SQL Server 7.0 or SQL Server 2000. The ASP.NET 2.0 SqlCacheDependency class, by contrast, works with SQL Server 7.0, SQL Server 2000, and SQL Server 2005. Placing a marker object accompanied by a SqlCacheDependency in the cache and registering for cache removal callbacks is a convenient way to take advantage of the extra smarts built into SqlCacheDependency. If the underlying database changes, the marker object is removed from the cache and the callback method is called and can take whatever action it deems appropriate—in this case, refreshing the site map (see Figure 2).

You can deploy SqlSiteMapProvider by copying SqlSiteMapProvider.cs to your Web site's App_Code folder. (In ASP.NET 2.0, source code files in that directory are automatically compiled.) Once the provider is deployed, you need to register it and make it the default site map provider. If you want to use its SQL cache dependency feature, you have to configure that, too. How you do it depends on whether the site map is stored in a SQL Server 7.0 or SQL Server 2000 database or a SQL Server 2005 database.

Using SqlSiteMapProvider

The web.config file in Figure 3 shows how to configure SqlSiteMapProvider to use SQL cache dependencies if the site map is stored in a table named SiteMap in a SQL Server 7.0 or SQL Server 2000 database. The <connectionStrings> section defines a connection string named SiteMapConnectionString that identifies the database. The provider uses this to query the database and monitor the site map portion of it for changes. Obviously, you'll need to replace the ellipsis ("...") with a real connection string.

Figure 3 Cache Dependencies (SQL Server 7.0 and 2000) (Close Figure 3)
<configuration>
  <connectionStrings>
    <add name="SiteMapConnectionString"
      connectionString="..."
      providerName="System.Data.SqlClient" />
  </connectionStrings>
  <system.web>
    <siteMap enabled="true" defaultProvider="AspNetSqlSiteMapProvider">
      <providers>
        <add name="AspNetSqlSiteMapProvider"
          type="SqlSiteMapProvider"
          securityTrimmingEnabled="true"
          connectionStringName="SiteMapConnectionString"
          sqlCacheDependency="SiteMapDatabase:SiteMap" />
      </providers>
    </siteMap>
    <caching>
      <sqlCacheDependency enabled="true" pollTime="5000">
        <databases>
          <add name="SiteMapDatabase"
            connectionStringName="SiteMapConnectionString" />
        </databases>
      </sqlCacheDependency>
    </caching>
  </system.web>
</configuration>

The <siteMap> section registers SqlSiteMapProvider and makes it the default site map provider. It also includes a sqlCacheDependency attribute that identifies the database and table where site map information is stored. The presence of this attribute tells SqlSiteMapProvider to create a SQLCacheDependency to monitor the site map database for changes; if you want to use SqlSiteMapProvider without cache dependencies, simply omit the sqlCacheDependency attribute.

The <sqlCacheDependency> section enables SQL cache dependencies in ASP.NET and supplies needed configuration information, including the polling interval which specifies how often ASP.NET checks the database for changes (in this example, every five seconds). The database name in this section maps to the database name specified in the provider's sqlCacheDependency attribute; the connection string name maps to the connection string in the <connectionStrings> section.

In order for these configuration settings to work, you must first prepare the site map database and the table containing site map data to support SQL cache dependencies. Two minutes with the aspnet_regsql.exe utility that comes with ASP.NET 2.0 is all it takes. If the database is a local one named SiteMapDatabase, first run the following command to prepare the database:

aspnet_regsql –S localhost –E –d SiteMapDatabase -ed

Then, if site map data is stored in the database's SiteMap table, run this command to prepare the table:

aspnet_regsql –S localhost –E –d SiteMapDatabase –t SiteMap -et

The first command adds a change notification table to the database, as well as stored procedures for accessing the table. The second adds an insert/update/delete trigger to the SiteMap table. When fired, the trigger inserts an entry into the change notification table, indicating that the contents of the SiteMap table have changed. ASP.NET 2.0 polls the change notification table at preprogrammed intervals to detect changes to the SiteMap table and to any other tables being monitored with SQL cache dependencies.

If you use SqlSiteMapProvider with a SQL Server 2005 database, you don't have to prepare the database to use SQL cache dependencies. You don't even have to identify the database and table containing the site map data; you just enable SQL cache dependencies with a <sqlCacheDependency> element and set SqlSiteMapProvider's sqlCacheDependency attribute to "CommandNotification," as shown in Figure 4. (You also need to run the ASP.NET worker process with dbo privileges for SQL Server 2005 cache dependencies to work automatically.)

Figure 4 Cache Dependencies (SQL Server 2005) (Close Figure 4)
<configuration>
  <connectionStrings>
    <add name="SiteMapConnectionString"
      connectionString="..."
      providerName="System.Data.SqlClient" />
  </connectionStrings>
  <system.web>
    <siteMap enabled="true" defaultProvider="AspNetSqlSiteMapProvider">
      <providers>
        <add name="AspNetSqlSiteMapProvider"
          type="SqlSiteMapProvider"
          securityTrimmingEnabled="true"
          connectionStringName="SiteMapConnectionString"
          sqlCacheDependency="CommandNotification" />
      </providers>
    </siteMap>
    <caching>
      <sqlCacheDependency enabled="true" />
    </caching>
  </system.web>
</configuration>

Configured thusly, SqlSiteMapProvider takes advantage of the SQL Server 2005 support built into ASP.NET by wrapping a SqlCacheDependency object around the SqlCommand object used to query the database for site map data. SqlCacheDependency, in turn, uses SQL Server 2005 query notifications to receive asynchronous callbacks indicating that data returned by the query has changed. No polling occurs and no special tables, stored procedures, or triggers are required. If you're looking for an excuse to upgrade your current database to SQL Server 2005, this feature alone is worth the price of admission for data-driven ASP.NET applications.

Creating the Site Map Database

A site map table created for SqlSiteMapProvider must conform to a predefined schema that lends itself to the representation of hierarchical data in a relational database. The SQL script in Figure 5 creates one such table named SiteMap and seeds it with sample site map nodes.

Figure 5 SQL Script for Creating a SiteMap Table (Close Figure 5)
-- Create the site map node table

CREATE TABLE [dbo].[SiteMap] (
    [ID]          [int] NOT NULL,
    [Title]       [varchar] (32),
    [Description] [varchar] (512),
    [Url]         [varchar] (512),
    [Roles]       [varchar] (512),
    [Parent]      [int]
) ON [PRIMARY]
GO

ALTER TABLE [dbo].[SiteMap] ADD
    CONSTRAINT [PK_SiteMap] PRIMARY KEY CLUSTERED
    (
        [ID]
    )  ON [PRIMARY]
GO

-- Add site map nodes

INSERT INTO SiteMap (ID, Title, Description, Url, Roles, Parent)
VALUES (1, 'Home', NULL, '~/Default.aspx', NULL, NULL)

INSERT INTO SiteMap (ID, Title, Description, Url, Roles, Parent)
VALUES (10, 'News', NULL, NULL, '*', 1)

INSERT INTO SiteMap (ID, Title, Description, Url, Roles, Parent)
VALUES (11, 'Local', 'News from greater Seattle', '~/Summary.aspx?CategoryID=0', NULL, 10)

INSERT INTO SiteMap (ID, Title, Description, Url, Roles, Parent)
VALUES (12, 'World', 'News from around the world', '~/Summary.aspx?CategoryID=2', NULL, 10)

INSERT INTO SiteMap (ID, Title, Description, Url, Roles, Parent)
VALUES (20, 'Sports', NULL, NULL, '*', 1)

INSERT INTO SiteMap (ID, Title, Description, Url, Roles, Parent)
VALUES (21, 'Baseball', 'What''s happening in baseball', '~/Summary.aspx?CategoryID=3', NULL, 20)

...

-- Create the stored proc used to query site map nodes

CREATE PROCEDURE proc_GetSiteMap AS
    SELECT [ID], [Title], [Description], [Url], [Roles], [Parent]
    FROM [SiteMap] ORDER BY [ID]
GO

Each record added to the table represents one site map node, and each has fields that map to SiteMapNode properties of the same name as well as fields that denote relationships between nodes. Each node must have a unique ID, which is stored in the ID field. To parent one node to another, you set the child node's Parent field equal to the ID of the parent. All nodes except the root node must have a parent. In addition, because of the way BuildSiteMap is implemented, a node can only be parented to a node with a lesser ID. (Note the ORDER BY clause in the database query.) For example, a node with an ID of 100 can be the child of a node with an ID of 99, but it can't be the child of a node with an ID of 101.

By default, SqlSiteMapProvider assumes that the table where site map data is stored is named SiteMap. SqlSiteMapProvider uses the stored procedure named proc_GetSiteMap to query the database for site map nodes, and this targets the SiteMap table. If you want to change the name of the site map table, simply change the table name in the database and in the stored procedure.

SqlSiteMapProvider demonstrates how compelling new features can be added to ASP.NET 2.0 through custom providers. To learn more about the ASP.NET 2.0 provider model and how to write custom providers of your own, check out ASP.NET 2.0 Provider Model: Introduction to the Provider Model.

Send your questions and comments for Jeff to wicked@microsoft.com.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET (Microsoft Press, 2002). He's also a cofounder of Wintellect, a software consulting and education firm.


Pure C++: Live Source Browsing in Visual C++

Pure C++
Live Source Browsing in Visual C++
Boris Jabes and Ameya Limaye

Contents
What is Source Browsing?
How Does Live Source Browsing Work?
Limitations of Live Source Browsing
Browsing Enhancements
Wrap-Up

If you're wondering where Stan Lippman is, we're happy to report that he has graciously granted us the use of his column this month to talk about some of the recent work the Visual C++ team has been doing to improve developer productivity. Unlike most of Stan's columns, this one will not focus on C++ itself, but will discuss the tool we use to develop programs—Visual Studio® 2005.

While Visual Studio 2005 supports many languages, we're going to discuss the C++ features. From the simplest syntax highlighting, all the way to powerful integrated debugging, IDEs like Visual Studio 2005 can be a major source of productivity. Visual C++® has supported source browsing as a feature for many years now, but we have given the 2005 version a major overhaul and have renamed it live source browsing to more accurately describe its features.

What is Source Browsing?

As developers, we spend most of our time in front of the machine writing code, but there are a number of peripheral activities we perform daily. These include reviewing, compiling, and debugging code. Sometimes we need to understand a foreign code base such as an API, and at other times we need to navigate through a large one. Source browsing is the feature that enables this.

Source browsing goes beyond showing the code; it focuses on examining the relationships between code elements, such as what member functions are in a class and which classes derive from another. Browsing abstracts the physical structure of the code; in other words, it exposes this information for a whole project (for those unfamiliar with Visual C++, projects represent a set of files, which are ultimately built into an executable or library).

During the build process in previous versions of Visual Studio, the IDE was able to create a browse information file called Browser Source Cache (BSC) that contains the data necessary to enable browsing features. This process involves the generation of an additional file during the compilation of each source file, adding an additional overhead of 10-15 percent to the build time. This additional compilation step leads to another significant deficiency in BSC-based browsing: in order to reflect any code modifications, we must rebuild the browse information file. Thus, the code needs to be in a fully compilable state before we can generate a BSC file for it.

When making significant modifications to code or starting a new project from scratch, the program as a whole cannot be compiled. In these situations, BSC-based browsing is not possible. In Visual C++ 2005, we have created a new architecture in order to solve these problems, ensuring that the browsing features are always accessible and reflect the current state of the code (even unsaved changes). This is the "live" part of live source browsing.

How Does Live Source Browsing Work?

Live source browsing is built on top of the Visual C++ IntelliSense® engine. The IntelliSense engine maintains a store that contains the data required for the editor to provide intelligent auto-completion and tooltips. We call this store the No Compile Browse (NCB) file. To populate it, we created an auxiliary parser, which specializes in parsing fuzzy C++ code. This parser makes accommodations for improper or incomplete statements (such as missing brackets), unlike a full-fledged compiler. As users work in the editor, the IDE monitors files for changes. When changes occur, the parser reparses the affected files in the background and updates the NCB file if necessary. Another difference when using NCB-based browsing is the treatment of C++ preprocessor macros. In the BSC model, the compilation process evaluates the preprocessor macros and their definitions become unrelated to their use. On the other hand, the NCB file keeps the macros in its store just like it keeps other symbols. Consider this simple piece of code:

#define FUNC func
void func() { }
void main() {
   FUNC();
}

Here, BSC browsing identifies FUNC() as a call to the function func(); however, it does not track macros and cannot relate the reference to the macro FUNC to its definition. On the other hand, NCB browsing is unable to determine that the reference to the macro FUNC is also a call to the function func(); it will simply relate the call to the macro FUNC. Figure 1 illustrates this example using three callers graphs (given a function or method, a hierarchy of functions that call it or that it calls is generated). In Figure 1, the callers graphs from top to bottom are NCB-based browsing for the function func, BSC-based browsing for the function func, and NCB-based browsing for the macro FUNC (note that BSC browsing cannot be done for FUNC).

Figure 1 Callers Graphs
Figure 1 Callers Graphs

Limitations of Live Source Browsing

There are a few limitations in this new architecture. They are the result of the workings of our fuzzy parser and the way we store data in the NCB file. The most significant limitation is the one mentioned earlier, namely that when a macro expands to a function call, our engine cannot equate the underlying function call with its definition. Another type of call that goes undetected is a call through a function pointer or through a delegate (including events). Finally, implicit calls such as constructors or operators also go undetected. We are committed to removing all these limitations, with time being our only constraint. Note, however, that some of these limitations exist in BSC browsing, too.

Browsing Enhancements

Visual C++ 2005 provides myriad features to aid developers with browsing-related tasks, some brand new. What follows is a look at the most important features and enhancements.

Figure 2 Inheritance Tree
Figure 2 Inheritance Tree
Class View Class View has been a staple of Visual C++ for many years. It's a window (accessible with Ctrl+Shift+C) that displays all elements, such as classes and methods, for a project. Double-clicking on an element causes the editor to jump to its definition in code. As we mentioned before, browsing code means examining relationships; to achieve this effect, Class View organizes types and namespaces into a tree structure with child nodes. For example, a class node might be a child of a namespace node.

We display members in a second pane whose contents depend on the currently selected element in the upper pane. Of course, all these details sound very familiar if you have used the IDE, but we have made some noteworthy additions in 2005. The first and foremost is the inheritance graph (see Figure 2). When looking at a type, you can now expand its node to see all classes that derive (subclass) from this type as well as all classes that this type derives from. The second enhancement is the addition of a filtering box at the top of the class view window (see Figure 3). When the user types into this box, Class View automatically filters the full list of symbols by pattern matching the string entered.

Figure 3 Filtering Class View
Figure 3 Filtering Class View

Call Browser The Call Browser is a very powerful feature brand new to Visual Studio 2005. As we mentioned earlier, given a function or method, Class Browser generates a hierarchy of functions called by or calling the given function, which are named Callers Graph and Call Graph, respectively. These graphs can be expanded such that if method A calls B, and B calls C, the callers are properly reflected in the graph (see Figure 4). Although this sounds simple, exposing these relationships is tremendously powerful when trying to decipher the logical flow of code without running it. For example, often you have pinpointed the location of a bug but you're lacking a call stack. The call browser allows you to visualize all possible paths to this function, as well as the paths out of it. For each level of the tree, the graph also lists each site in the code where the calls are made. In Visual C++ 2005, you can generate these graphs by right-clicking on a function anywhere in code (or in Class View) and selecting the graph you want under the Call Browser menu.

Figure 4 Superfunc Callers Graph
Figure 4 Superfunc Callers Graph

Object Browser The Object Browser (Ctrl+Alt+J) is an expanded version of the Class View window. Instead of limiting itself to a small space, it comes up like an editor document and uses the full space available. It has three panes: the tree of classes, the member fields and methods, and a short description of whatever element is selected. This information is gleaned from the declaration of the type as well as the comments that accompany it. In the case of managed code, the information is retrieved from the metadata that is compiled into the binary. Unlike the class view, the purpose of the object browser is not to jump around the code but more to explore an API or a framework. To further facilitate this, you can look at elements outside of your project in the object browser. You can add external libraries (managed, COM, or native) and peer into their exposed functionality.

Find Symbols and Find All References Sometimes you just want to find a symbol that you know is used somewhere in your code base. However, performing a classic text search causes all sorts of irrelevant strings to be returned, most notably information in comments. In order to handle this scenario more efficiently, we have a command called Find Symbol (accessible with Alt+F12) that searches through the list of symbols stored in the browsing database. The results are presented in a new window (aptly named Find Symbol Results) so that finding and navigating to specific instances of the symbols is fast and efficient. However, what if you have a hypothetical class, CFoo, defined with the same name in a number of different files and namespaces? In these cases, Find Symbol will not be able to differentiate between them. After all, you cannot tell them apart unless you specify this in a search string. In order to handle these types of situations, we have a command called Find All References (accessible with a right-click on a symbol) that performs a search that is more tightly scoped than the previous one. Indeed, it will only return instances of the exact symbol you select in the editor or class view.

Useful Shortcuts Live source browsing enables a number of very simple yet powerful shortcuts in the IDE. The two notable shortcuts, which are not new, are Go To Definition (F12) and Go To Declaration (Ctrl+Alt+F12), which are self-explanatory. These commands are also available in the editor context menu if you prefer using the mouse in these situations. Veterans of the IDE might be familiar with an unfortunate consequence of these shortcuts that jump the editor to other parts of your project: it's hard to get back to your previous position. We are happy to announce that the following commands have been added to Visual Studio 2005: Navigate Backward (Ctrl+-), Navigate Forward (Ctrl+Shift+-), and Pop Browse Context (Ctrl+Num*). The first two act more like the Internet Explorer Back and Forward shortcuts while the third moves backwards through the browsing history.

Wrap-Up

Live source browsing is a major improvement over the previous browsing model. By no longer requiring you to recompile your code, browsing can become part of the coding routine. The advantage is not limited to saving time by not recompiling. Browsing can now be used as a coding tool, not merely for examining large code bases. While we feel we have made major advancements in browsing with Visual Studio 2005, we will continue to improve these features in future releases until they cover all nuances and aspects of the complex C++ language. To learn more about the architecture of our IntelliSense engine, which encompasses aspects of browsing, see IntelliSense Support in Visual C++.

Send your questions and comments to purecpp@microsoft.com.

Boris Jabes is a program manager on the Visual C++ IDE team responsible for a number of features in the IDE. He holds degrees in Math from the University of Waterloo and Information Networking from Carnegie Mellon.

Ameya Limaye has been a developer on the Visual C++ Compiler team for four years and on the Visual C++ IDE team for the past two years, working mainly on the IntelliSense, Browsing and Editor feature areas. He holds a Masters Degree in Computer Science from Purdue University.


C++ at Work: Event Programming

C++ At Work
Event Programming
Paul DiLascia

Code download available at: CAtWork0602.exe (175KB)

Q The Microsoft® .NET Framework lets me define events for managed classes and handle them using delegates and operator+=. Is there some way to do the same thing in native C++? It seems like this would be useful.

Several Readers

A Indeed it would be! Visual C++® .NET has something called the Unified Event Model that lets you implement native events the same way you would for a managed class (using the __event keyword), but alas native events have some obscure technical problems which the Redmondtonians don't plan to fix, so they've asked me to officially discourage you from using them. Does that mean C++ programmers have to live without events? Of course not! There's more than one way to skin a cat. I'll show you how to implement your own nifty event system with very little fuss.

But before I do, let me write some words about events and event programming in general. It's an important topic. You can't program in the modern world without a solid understanding of events—what they are and when to use them.

Successful programming is all about managing complexity. A long time ago, when functions were called "subroutines" (I'm dating myself, I know), one of the main ways to manage complexity was top-down programming. You start with some high-level goal like "model the universe," then break it into smaller tasks like "model the galaxies" and "model the solar system" and so on until the tasks become small enough to implement in a single function. Top-down programming still works for procedural tasks, but it doesn't work well for systems that respond to real-world events that happen in a nondeterministic order. The classic example is a GUI where your program must respond when the user does something like press a key or move the mouse. Indeed, event programming was largely spurred by the advent of graphical user interfaces.

In the top-down model, high-level components at the top of the food chain bark orders to lower-level components by calling functions like DoThis and DoThat to perform various tasks. But sooner or later, the lower-level components need to talk back. In Windows®, you can call Rectangle or Ellipse to draw a rectangle or ellipse, but eventually Windows needs to call your app to paint its window. But your app doesn't even exist yet, it's still behind schedule! So how can Windows know which function to call? This is where events come in.

Figure 1 Top-Down vs. Bottom-Up
Figure 1 Top-Down vs. Bottom-Up

At the core of every Windows-based program—whether directly coded in C, or hidden behind layers of MFC or .NET Framework classes—is a window procedure that processes messages like WM_PAINT, WM_SETFOCUS, and WM_ACTIVATE. You (or MFC or .NET) implement the window procedure and pass it to Windows. When it's time to paint, change the focus, or activate your window, Windows calls your procedure with the appropriate message code. The messages are the events. Your window procedure is the event handler.

If procedural programming is top-down, event programming is bottom-up. In a typical software system, function calls flow downward from higher-level components to lower-level ones; whereas events percolate up in the opposite direction. Figure 1 illustrates this pattern. Of course, the real world in which we romp is not always so neatly hierarchical. Many software systems look more like Figure 2.

Figure 2 A Mixed Model
Figure 2 A Mixed Model

So what exactly is an event? Essentially, it's a callback. Instead of calling a function whose name is known at compile time, the component calls a function you provide at run time. In Windows, it's the window proc. In the .NET Framework, it's a delegate. Whatever the terminology, events provide a way for one software component to call another without knowing until runtime what function to call. The callback is called the event handler. Raising or firing an event means calling the handler. To get the ball rolling, the receiving component first gives the source component a pointer to its event handler, a process called registration.

Here are some common situations in which events are used.

To Notify Clients About Real-World Events The user pressed a key; the clock struck midnight; the fan failed and the CPU is on fire.

To Report Progress of Lengthy Operations When copying files or searching a huge database, a component might periodically raise an event to report how many files have been copied or how many records have been searched.

To Report When Something Significant or Potentially Interesting Has Happened If you use IWebBrowser2 to host Microsoft Internet Explorer in your app, the browser will notify you before and after navigating to a new page, or when it creates a new window—among other things.

To Invoke an Application-Supplied Algorithm The C runtime library function qsort sorts an array of objects, but you must supply the function that compares objects. Many STL containers let you play the same trick. Most programmers wouldn't call the qsort callback an event, but there's no reason you can't think of it that way. It's the "time to compare" event.

Some readers occasionally ask: What's the difference between an exception and an event? The main difference is that exceptions represent unexpected conditions that aren't supposed to occur. For example, your program runs out of memory or encounters divide-by-zero. Oops. These are aberrant situations you hope won't happen, and yet if they do your program has to cope. Events, on the other hand, are part of normal everyday operation and are fully expected. The user moves the mouse or presses a key. The browser just navigated to a new page. From a control-flow perspective, an event is a function call, whereas an exception is a long jump across the stack, with unwinding semantics to destroy lost objects.

A common misconception about events is that they're asynchronous. While events are often used to handle user input and other actions that happen asynchronously, the events themselves happen in synchronous fashion. Raising an event is the same thing as calling the event handler(s). In pseudocode, it looks something like the following snippet:

// raise Foo event
for (/* each registered object */) {
  obj->FooHandler(/* args */);
}

Control passes immediately to the event handler and doesn't return until it's done executing. Some systems provide a way to raise events asynchronously; for example, Windows lets you use PostMessage instead of SendMessage. Control returns immediately from PostMessage and the message gets processed later. But .NET Framework events and the events I'll discuss here are handled immediately when you raise them. Of course, you can always raise an event from code that runs in a separate thread, or you can use asynchronous delegate invocation to execute each event handler in the thread pool, in which case the event happens asynchronously relative to the main thread.

The way Windows does events, with its omniversal window proc and type-deadly WPARAM/LPARAM parameters, is downright primitive by modern programming standards. Yet every Windows program uses this mechanism, even today. Some programmers even create invisible windows just to pass events. The window proc is not a true event mechanism because Windows allows only one window proc per window, though multiple procs can be linked if each proc calls the one before it. This process is known as subclassing. In a true event system, more than one recipient can register for the same events in a non-hierarchical way.

In the .NET Framework, events are full-fledged citizens. Any object can define events, and multiple objects can listen to them. In .NET, events work using delegates, which is Framework terminology for a callback. Most important, delegates are type-safe. No more void*'s or WPARAM/LPARAM.

To define an event using the Managed Extensions, you use the __event keyword. For example, the Button class in Windows::Forms has a Click event:

// in Button class
public:
  __event EventHandler* Click;
EventHandler is a delegate for a function that takes an Object (the sender) and EventArgs:
public __delegate void EventHandler(
   Object* sender,
   EventArgs* e
);

To receive events, you implement a handler member function with the right signature and create a delegate to wrap it, then invoke the event's operator+= to register your handler/delegate. For the Click event, it looks like this:

// event handler
void CMyForm::OnAbort(Object* sender, EventArgs *e)
{
  ...
}
// register my handler
m_abortButton->Click += new EventHandler(this, OnAbort);

Note that the handler function must have the signature defined by the delegate. All of this is Managed Extensions 101. But you didn't ask about managed events, you asked about native events—how can you implement events in native C++? C++ has no built-in event mechanism, so what can you do? You could use a typedef to define a callback and make clients supply it, something similar to qsort—but that's so old-hat. Not to mention cumbersome if you are working with several events. And it's especially ugly if you want to use member functions as the event handlers, as opposed to static extern functions.

A better way is to create an interface that defines the events. That's how COM does it. But you don't need all the overhead of COM code in C++; you can use a simple class. I wrote a class called CPrimeCalculator that shows how; it finds prime numbers. Figure 3 shows the code. CPrimeCalculator::FindPrimes(n) finds the first n prime numbers. While it's working, CPrimeCalculator can raise two kinds of events: a Progress event and a Done event. The events are defined in an interface IPrimeEvents. IPrimeEvents isn't an interface in the .NET or COM sense; it's a plain old C++ abstract base class that defines the signature (parameters and return type) for each event handler. Clients that handle CPrimeCalculator events must implement IPrimeEvents, then call CPrimeCalculator::Register to register their interface. CPrimeCalculator adds the object/interface to its internal list. As it tests every integer for primeness, CPrimeCalculator periodically reports how many primes it has found so far:

// in CPrimeCalculator::FindPrimes
for (UINT p=2; p<max; p++) {
   // figure out if p is prime
   if (/* every now and then */)
      NotifyProgress(GetNumberOfPrimes());
   ...
}
NotifyDone();

Figure 3 CPrimeCalculator (Close Figure 3)

Prime.h
#pragma once

#include <vector>
#include <list>
using namespace std;

/////////////////
// Interface for prime events. Not an interface in the .NET or COM sense,
// just an ordinary old abstract base class. Clients that handle prime
// events must derive from this and implement the virtual functions.
//
class IPrimeEvents {
protected:
   IPrimeEvents() { }
   virtual ~IPrimeEvents() { };
public:
   virtual void OnProgress(UINT nPrimes) = 0;
   virtual void OnDone() = 0;
};

// List (vector) of prime numbers.
typedef vector<UINT> PRIMELIST;

//////////////////////////////////////////////////////////////////
// Class to calculate 1st n prime numbers.
//
class CPrimeCalculator {
protected:
   list<IPrimeEvents*> m_clients; // list of client objects to notify
   PRIMELIST m_primes;            // list (vector) of primes numbers found

   // Helpers
   BOOL IsPrime(UINT p);          // is prime?
   void AddPrime(UINT p)          // add prime to list
   {
      m_primes.push_back(p);
   }

   // Raise events
   void NotifyProgress(UINT nFound);
   void NotifyDone();

public:
   CPrimeCalculator() { }
   ~CPrimeCalculator() { }

   // Find 1st n prime numbers.
   UINT FindPrimes(UINT n);

   // Get number of primes found so far.
   UINT GetNumberOfPrimes() {
      return (UINT)m_primes.size();
   }

   // Get all the primes found.
   const PRIMELIST GetPrimes() {
      return m_primes;
   }

   // Register client to receive events
   void Register(IPrimeEvents* client)
   {
      m_clients.push_back(client);
   }

   // Unregister client from receiving events.
   void Unregister(IPrimeEvents* client)
   {
      m_clients.remove(client);
   }
};
Prime.cpp
#include "StdAfx.h"
#include "Prime.h"

const NOTIFYINCR = 100;                 // notify every 100 tries
const FIRSTPRIME = 2;                   // important!

//////////////////
// Find 1st n prime numbers.
//
UINT CPrimeCalculator::FindPrimes(UINT nFind)
{
   m_primes.reserve(nFind);
   for (UINT p=FIRSTPRIME; GetNumberOfPrimes()<nFind; p++) {
      if (IsPrime(p)) {                 // if prime:
         AddPrime(p);
      }
      if (p % NOTIFYINCR == 0) {
         // report progress by raising event
         NotifyProgress(GetNumberOfPrimes());
         Sleep(30); // simulate work; otherwise too fast!
      }
   }
   NotifyDone();
   return GetNumberOfPrimes();
}

//////////////////
// Helper fn to test if number p is prime:
// Try dividing by all primes found so far, up to SQRT(p).
//
BOOL CPrimeCalculator::IsPrime(UINT p)
{
   PRIMELIST::iterator it;
   for (it=m_primes.begin(); it!=m_primes.end(); it++) {
      UINT factor = *it;
      if (p%factor == 0) {              // if divisible:
         return FALSE;                  // ..it's not prime
      }
      if (factor*factor > p)            // if no more primes to try:
         return TRUE;                   // ..number must be prime
   }
   // Should only get here when p==2 (list is empty).
   ASSERT(p==2);
   return TRUE;
}

//////////////////
// Notify clients of progress so far.
//
void CPrimeCalculator::NotifyProgress(UINT nFound)
{
   list<IPrimeEvents*>::iterator it;
   for (it=m_clients.begin(); it!=m_clients.end(); it++) {
      (*it)->OnProgress(nFound);
   }
}

//////////////////
// Notify clients I'm done.
//
void CPrimeCalculator::NotifyDone()
{
   list<IPrimeEvents*>::iterator it;
   for (it=m_clients.begin(); it!=m_clients.end(); it++) {
      (*it)->OnDone();
   }
}

CPrimeCalculator calls internal helper functions NotifyProgress and NotifyDone to raise the events. These functions iterate the client list, calling the appropriate event handler for each client. In code, it looks like this:

void CPrimeCalculator::NotifyProgress(UINT nFound)
{
  list<IPrimeEvents*>::iterator it;
  for (it=m_clients.begin(); it!=m_clients.end(); it++) {
    (*it)->OnProgress(nFound);
  }
}

In case you're not up on your STL, recall that for iterators the dereference operator returns the current object, so the line inside the for loop in the preceding snippet is equivalent to:

IPrimeEvents* obj = *it;
obj->OnProgress(nFound);

There's a similar function NotifyDone that raises the Done event, which has no parameters. Figure 3 shows the code. You might think there's no need for a Done event since the client knows CPrimeCalculator is done when FindPrimes returns control. And you'd be right—except for one thing. There may be more than one client registered to receive events, and possibly not the same one that calls CPrimeCalculator::FindPrimes. Figure 4 shows my test program, PrimeCalc. PrimeCalc implements two different event handlers for prime events. The first handler is the main dialog itself, CMyDlg, which implements IPrimeEvents using multiple inheritance. The dialog handles OnProgress and OnDone by displaying the progress in a dialog window (see Figure 5) and beeping when done. The other event handler, CTracePrimeEvents, also implements IPrimeEvents. Its implementation displays information in the diagnostic (TRACE) stream. Figure 6 shows a sample run in my TraceWin applet. (See my March 2004 column, or download TraceWin from www.dilascia.com.) I wrote CTracePrimeEvents to show how more than one client can register for the same events.

Figure 4 PrimeCalc.cpp (Close Figure 4)
#include "stdafx.h"
#include "StatLink.h"
#include "resource.h"
#include "Prime.h"
#include "TraceWin.h"

const NFIND = 1000; // find first 1000 primes

//////////////////
// This class displays IPrimeEvents in the diagnostic (TRACE) stream. Used
// to show that multiple objects can register for the same event stream.
//
class CTracePrimeEvents : public IPrimeEvents
{
public:
   void OnProgress(UINT nPrimes)
   {
      UNREFERENCED_PARAMETER(nPrimes);
      TRACE(_T(" PrimeCalc: Found %d primes.\n"), nPrimes);
   }
   virtual void OnDone()
   {
      TRACE(_T(" PrimeCalc: Done\n"));
   }
};

//////////////////
// Main dialog: implements IPrimveEvents to handle events from
// CPrimeCalculator.
//
class CMyDlg : public CDialog, public IPrimeEvents {
protected:
   .
   .
   CEdit m_wndFeedback;

   // IPrimeEvents handlers
   void OnProgress(UINT nPrimes);
   void OnDone();

   // message handler
   void CMyDlg::OnCompute()
};

//////////////////
// User pressed "Compute" button.
//
void CMyDlg::OnCompute()
{
   CWaitCursor wc;                      // display wait cursor

   m_wndFeedback.SetWindowText(_T("Calculating..."));

   // create prime calculator and hook events
   CPrimeCalculator pc;       // Prime calculator
   CTracePrimeEvents tn;      // TRACE notifier
   pc.Register(this);         // register myself for events
   pc.Register(&tn);          // .. and trace notifier too
   pc.FindPrimes(NFIND);      // find prime numbers

   // Build message showing prime numbers.
   CString msg,temp;
   int any = 0;
   PRIMELIST primes = pc.GetPrimes();
   PRIMELIST::iterator it;
   for (it=primes.begin(); it!=primes.end(); it++) {
      UINT p = *it;
      if (any)
         msg += _T(", ");
      else
         any++;
      temp.Format(_T("%d"),p);
      msg += temp;
   }

   // show results in feedback window
   m_wndFeedback.SetWindowText(msg);
}

//////////////////
// Event handler: Show progress.
//
void CMyDlg::OnProgress(UINT nPrimes)
{
   CString s;
   s.Format(_T(«Found %d primes so far...»), nPrimes);
   m_wndFeedback.SetWindowText(s);
   m_wndFeedback.UpdateWindow();
}

//////////////////
// Event handler: Beep when done.
//
void CMyDlg::OnDone() {
   MessageBeep(0);
}
Figure 5 PrimeCalc in Action
Figure 5 PrimeCalc in Action

From the perspective of a programmer writing an application that uses CPrimeCalculator, handling events is simple and straightforward. Derive from IPrimeEvents, implement the handler functions, then call Register. For programmers writing classes that raise events, the process is a bit more tedious. First you have to define the event interface. That's not bad. But then you have to write Register and Unregister functions, not to mention a NotifyFoo function for each Foo event. Quite a bore if you have 15 events, especially since each NotifyFoo function has the same pattern:

void CMyClass::NotifyFoo(/* args */)
{
  list<IPrimeEvents*>::iterator it;
  for (it=m_clients.begin(); it!=m_clients.end(); it++) {
    (*it)->OnFoo(/* args */);
  }
}

Figure 6 PrimeCalc Output in TraceWin
Figure 6 PrimeCalc Output in TraceWin

NotifyFoo iterates the client list, calling the appropriate OnFoo handler for each registered client and passing whatever parameters are required. Isn't there some way to implement this generically using macros or templates or something, in order to encapsulate the drudgery and spare yourself the misery of writing such boring boilerplate code? In fact, there is. I'll show you how next month. Same bat time. Same bat channel. Until then—Happy programming!

Send your questions and comments for Paul to cppqa@microsoft.com.

Paul DiLascia is a freelance software consultant and Web/UI designer-at-large. He is the author of Windows++: Writing Reusable Windows Code in C++ (Addison-Wesley, 1992). In his spare time, Paul develops PixieLib, an MFC class library available from his Web site, www.dilascia.com.


{ End Bracket }: Building Voice User Interfaces

{ End Bracket }
Building Voice User Interfaces
Alex Acero

The GUI is the interface of choice for scenarios when the user has a large keyboard, mouse, and display. But most cell phones have just a keypad and a small display. Drivers want to get directions without taking their eyes off the road or their hands off the wheel. And we make many telephone calls where we start interacting with a machine before we reach a human. Speech recognition is a technology that can enable users to access information more efficiently in those scenarios.

GUIs are deterministic (the same action will always produce the same result), whereas voice user interfaces (VUIs) are not (the user can say the same word twice and the system will come up with different answers). Although speech recognition error rates decrease every year due to advances in the technology, they will never be zero. Humans carry on conversations when they don't understand everything but they handle ambiguity gracefully. Likewise, an effective VUI must handle speech recognition errors gracefully. We've been making progress in this area over the last few years.

Voice applications in telephony systems are typically represented as a state machine that takes the caller from one state to another. A banking application could play a WAV file to the user (InitialPrompt="Please say account balance, make a payment, or transaction history") and go to one of three different states depending on the caller's response. The speech recognition engine selects the choice of words Wi that maximizes the posterior probability:

A represents the acoustic waveform, is the so-called acoustic model, and is the language model. Acoustic models, estimated with a technique called Hidden Markov Models, are built-in. Language models are represented with a finite state grammar (FSG) and need to be provided by the developer. Unless prior information is available, developers would set all three choices to be equally likely, and write the grammar in the W3C format:

<one-of>
    <item weight="0.33">Account balance</item>
    <item weight="0.33">Make a payment</item>
    <item weight="0.33">Transaction History</item>
</one-of>

In a VUI, developers need to handle issues such as speech recognition errors, barge-in, and discoverability of options available to callers, which are not problems in a GUI. For example, if the engine's confidence score does not exceed a threshold, a voice application will request confirmation (ConfirmationPrompt="Do you want to hear your account balance?"). If the confidence score is even lower, often because the caller said something other than the three choices, the system reprompts the user (MumblePrompt="Just say one of the following choices:..."). If the caller does not respond, because she does not know what to say, the system tries again, perhaps with another prompt.

To simplify authoring, a graphical dialog designer (see Figure) allows developers to choose from a palette of high-level controls that encapsulate best practices of VUI development, such as handling mumbles and confirmations, without the need for custom code. Such controls include QA, Statement, Get & Confirm, Menu, Recorded Message, and Navigatable List. These have properties specifying all the necessary prompts and recognition branches.

Unfortunately, callers' answers are not always covered by simple grammars like in the previous example. Since manually building robust FSGs is difficult, we have developed technology that can automatically generate the grammar by adapting the system's probabilistic dictation grammar with example sentences provided by the developer (balance, account balance, and checking account balance, for instance, all map to the state <account_balance>). The compiled probabilistic FSG is complex but covers most possible sentences a caller might say. Since the system might recognize "I want to chat my balance" when the caller actually said "I want to check my balance," a statistical classifier is optionally built to determine the callers' intention. In this example, the word balance is strong evidence for it to conclude that the caller wants account balance.

Our goal is to create tools that simplify the development of voice applications much like Visual Basic® simplified the development of GUI applications. More information is available at Speech Technology.

Alex Acero is Research Area Manager at Microsoft Research. He directs speech recognition activities and has contributed to technology shipped in Microsoft speech product.