January 2009
First Look: WCF And WF Services In The .NET Framework 4.0 And “Dublin”
We explore some of the key new WCF and WF features in .NET Framework 4.0 as well as the new application server capabilities provided by the “Dublin” extensions. Aaron Skonnard
Geneva Framework: Building A Custom Security Token Service
A Security Token Service, or STS, acts as a security gateway to authenticate callers and issue security tokens carrying claims that describe the caller. See how you can build a custom STS with the “Geneva” Framework. Michele Leroux Bustamante
Silverlight: Build Line-Of-Business Enterprise Apps With Silverlight, Part 1
Take a walk through the creation of a call center client application to learn how to build real-world enterprise solutions using Silverlight. Hanu Kommalapati
Windows Mobile: Use GPS And Web Maps For Location-Aware Apps
We show you how to build a location-aware task list application for Windows Mobile devices that uses GPS to remind you of a task when and where it’s appropriate. Christopher Mitchell
VSTS 2010: Agile Planning Tools In Visual Studio Team System 2010
Visual Studio Team System (VSTS) 2010 introduces product backlog and iteration backlog workbooks and reports that will help agile teams plan and manage releases and iterations. Ajoy Krishnamoorthy
ASP.NET Workflow: Web Apps That Support Long-Running Operations
Integrating workflow into ASP.NET applications means communicating with activities via a workflow queue and hosting the runtime in the global application class. We’ll show you how. Michael Kennedy
Columns
Editor's Note: A Look Back As We Move Forward: Here’s a look at a year’s worth of coverage and changes at MSDN Magazine in response to reader requests.Howard Dierking | Toolbox: C# JavaScript, Eric Lippert’s Blog, And More: Writing your JavaScript code in C# and having it converted, Eric Lippert’s programming blog, and reading up on T-SQL 2008.Scott Mitchell | CLR Inside Out: Best Practices For Managed And Native Code Interoperability: There are many factors to consider when building your app with both managed and native code. Find out how to employ interop and how to choose the interop that’s right for you.Jesse Kaplan
Basic Instincts: Inspecting COM Objects With Reflection: Reflection is useful for debugging and logging and otherwise providing the type information you need. Here you’ll see how to use reflection on COM types.Lucian Wischik | Cutting Edge: Managing Dynamic Content Delivery In Silverlight, Part 1: This month Dino tackles the problem of large download size for Silverlight applications, explaining when to use streaming, when to divide the download, and other techniques for better performance over the wire.Dino Esposito | Test Run: Analyzing Project Exposure And Risk Using PERIL: Here James McCaffrey uses some statistical methods and a technique he calls PERIL to estimate risks to software projects involving timing, costs, and other potential resource shortfalls.Dr. James McCaffrey
Service Station: An Introduction To RESTful Services With WCF: We discuss some of the basic tenets of REST as well as present an implementation of a RESTful service using WCF.Jon Flanders | Security Briefs: Getting Started With The SDL Threat Modeling Tool: The Security Development Lifecycle (SDL) threat modeling tool helps you develop great threat models as a backbone of your security process. We'll show you how it works.Adam Shostack | Extreme ASP.NET: Routing with ASP.NET Web Forms: See how to get the benefits of rerouting while alleviating the risks with this in-depth look at URL rewriting in ASP.NET.Scott Allen
Foundations: Easily Apply Transactions To Services: Managing state and error recovery using transactions is the topic of this month’s installment of Foundations.Juval Lowy | Net Matters: Round-Robin Access To The ThreadPool: Stephen Toub shows you how to add round-robin scheduling support on top of the ThreadPool for more granular processing control.Stephen Toub | Usability in Practice: When Things Go Wrong: This month our usability experts explain what it takes to create informative, useful error messages.Dr. Charles Kreitzberg and Ambrose Little
{ End Bracket }: The Emergence Of Machine Translation: Vikram Dendi looks at how machine translation is poised to change the world and why it is so important to deliver information in multiple languages.Vikram Dendi


First Look
WCF And WF Services In The .NET Framework 4.0 And “Dublin”
Aaron Skonnard

This article is based on a prerelease version of the .NET Framework 4.0 and "Dublin." All information is subject to change.

This article discusses:
  • WF activity library and designer
  • WCF improvements in the .NET Framework 4.0
  • A guide to the “Dublin” extensions
  • Building and deploying services with “Dublin”
This article uses the following technologies:
.NET Framework 4.0, “Dublin”
Contents
Moving toward the .NET Framework 4.0
WF Base Activity Library
WF Activity Programming Model
More New WCF Features in the .NET Framework 4.0
The Need for "Dublin"
A Guide to "Dublin"
Building a WCF Workflow Service for "Dublin"
Deploying Applications with "Dublin"
Managing Running Applications
Updating Running Applications
Monitoring Running Applications
From "Dublin" to "Oslo"?
What about BizTalk Server?
At the Professional Developer's Conference (PDC) in October 2008, Microsoft released the details around numerous improvements that will ship in the Microsoft .NET Framework 4.0, specifically in the areas of Windows Communication Foundation (WCF) and Windows Workflow Foundation (WF). Microsoft also debuted some extensions to Windows Server—code-named "Dublin"—that provide an improved hosting and management experience for WCF and WF applications.
The integration of WF and WCF in the .NET Framework 4.0 will make it simpler to develop distributed service-oriented apps. You will be able to build stateful workflow services using a completely declarative model that offers more flexibility and business agility.
The new hosting and management extensions introduced by "Dublin" complement these framework advances. The combination of advances in the framework itself and in the operational tools that support the framework means that the application server capability in Windows Server will make a significant leap forward.
In this article, I will explore some of the key new WCF and WF features in the .NET Framework 4.0 as well as the new application server capabilities provided by the "Dublin" extensions.

Moving toward the .NET Framework 4.0
WCF and WF are complementary technologies. If you are unfamiliar with them, a simple way to think about the pair is: WCF on the outside, WF on the inside. WCF is used to expose the external service interface of an application, and WF is used to describe the internal flow, states, and transitions of an application.
The .NET Framework 3.5 introduced some compelling integration between the two, specifically in the form of the WF Send and Receive activities. With these activities, you can employ WF to simplify the process of coordinating multiple service interactions to fulfill complex, long-running workflows. You can also use these activities to extend the reach of your WF workflows by enabling them with WCF endpoints (see Figure 1). This essentially allows you to employ WF as the implementation for your WCF services, what I will call WCF workflow services throughout this article.
Figure 1 WCF Workflow Services
Although it's possible to implement WCF workflow services in the .NET Framework 3.5, it isn't easy. For starters, the integration layer between WCF and WF leaves something to be desired. In the .NET Framework 3.5, the WCF artifacts have to be defined and configured using the WCF programming and configuration model, while the workflow is defined using a different model. Ultimately, you end up with multiple artifacts that need to be deployed, configured, and managed separately.
Another reason for the difficulty is that the current base activity library concentrates on flow-control and logic activities and doesn't provide much in the way of work activities. So you must write a library of custom activities before you can implement real-world workflow services with WF. Given the complexity of that job, some developers have given up before they are able to experience the benefits of WF.
In addition to these issues, current WF also lacks tool support for eXtensible Application Markup Language (XAML)-only workflows, which are also referred to as declarative workflows because they're completely described in an XML file without any codebehind files. The XAML-only approach creates some compelling workflow hosting and deployment possibilities. The powerful thing about a declarative workflow is that it's just data that can be stored anywhere and executed within a WF runtime environment that knows about the activities being used.
A declarative workflow can be deployed to a runtime in the cloud or to a workstation under your desk (assuming the activities have been deployed to the runtime host). Declarative workflows are also easier to version, and they can be used in partial-trust scenarios (think "cloud"). The XAML-only model is also easier to build tools around since the tools are just working with XML files.
In general, the XAML-only model has always been the ultimate vision for WF as a technology, which is evident in the early writings by its architects. However, the current WF tool support doesn't fully realize that vision. While it's possible to build XAML-only workflows with the .NET Framework 3.5, you have to work around the current Visual Studio templates and give up important features, such as debugging.
The main goal for WCF and WF in the .NET Framework 4.0 is to simplify the developer experience around declarative workflows and services, fully embracing a XAML-only model. In addition, Microsoft wanted to take things one step further by making it possible to define declarative workflow services. That is, WCF services that are completely defined in terms of XAML, including the service contract definitions, endpoint configurations, and the actual service implementation (in the form of a XAML-based workflow).
To accomplish this, Microsoft has made numerous improvements in the .NET Framework 4.0, including an extended base class activity library, a simpler programming model for custom activities, a new flowchart workflow type, and numerous WCF-specific improvements.

WF Base Activity Library
The .NET Framework 4.0 comes with an enhanced base activity library containing several new activities (see Figure 2). Microsoft also plans to begin making additional WF activities available through CodePlex between major .NET Framework releases. You'll also begin to see more work activities (such as the PowerShellCommand activity) appear in future releases (or on CodePlex), thereby reducing the need for custom activity development. And because Microsoft is using CodePlex, you have a good opportunity to make your opinion heard on what additional activities you'd like to see.
The .NET Framework 4.0 introduces a few core activities that provide more flow-control options, including the FlowChart, ForEach, DoWhile, and Break activities. The new FlowChart activity is one of the most interesting additions because it offers a nice middle ground between the Sequential and StateMachine flow-control models. FlowChart lets you use a step-by-step approach, enhanced with simple decisions and switches, but it also allows you to return to previous activities in the workflow. Flowcharts in general seem more intuitive to many users. Figure 3 shows what the FlowChart designer looks like in the new workflow designer in Visual Studio 2010 (see "The New Workflow Designer" sidebar for more).
Figure 3 The New FlowChart Activity Designer
The .NET Framework 4.0 also introduces some new runtime activities for invoking CLR methods (MethodInvoke), for assigning values to workflow variables (Assign), and for explicitly persisting a running workflow instance (Persist).
Finally, the .NET Framework 4.0 comes with a new set of WCF-based activities that simplify the process of exposing workflows as services, or consuming services from within your workflows. The .NET Framework 3.5 provided two activities—Send and Receive—for sending and receiving messages through WCF. In version 4.0, you'll find SendMessage and ReceiveMessage activities—meant for sending and receiving one-way messages (similar to Send and Receive in version 3.5)—as well as a higher-level abstraction for request/response operations through the ClientOperation and ServiceOperation activities. A workflow that wants to expose a service operation should use the ServiceOperation activity. Whereas a workflow that wants to consume an external service should use the ClientOperation activity.
In addition to these core WCF activities, the .NET Framework 4.0 also provides support for correlation across different one-way operations to ensure that a particular message makes it back to the correct workflow instance. It comes with an activity for defining a new correlation scope (CorrelationScope) and for initializing the correlation values (InitializeCorrelation) before sending an outbound message through WCF.
The New Workflow Designer
A new workflow designer added to Visual Studio 2010 provides a compelling graphical user experience for many of the key WCF and WF features discussed in this article. It provides features such as improved workflow navigation (drilling into composite activities) with bread-crumb trails for going back in scope, in-place activity editing (reducing the need for the Properties window), zoom capabilities, and overview navigation. In addition, the new designer provides an improved model for customization and re-hosting. Visual Studio 2010 will also offer a suite of new or improved project templates that make it easy to get started quickly with flowcharts and XAML-only workflows, and it will provide full support for XAML-based debugging when working with declarative workflows and services.

WF Activity Programming Model
Even with these improvements, you may still need to write custom activities sometimes. So to make it easier, Microsoft redesigned the base classes for custom activities. The new base class for custom activities is called WorkflowElement, and there's another class that derives from it called Activity. The Activity class makes it easy to create new custom activities from existing activities without writing much, if any, code.
Figure 4, for example, shows how to define a new custom activity called CopyFile by deriving from Activity and overriding CreateBody. The implementation creates a customized PowerShellCommand instance configured to use the built-in copy-item command. You can also easily build custom activities like this one using the new workflow designer in Visual Studio 2010, through the graphical activity designer, if you'd rather avoid this type of code altogether.
Figure 4 A Custom CopyFile Activity
class CopyFile : Activity
{
    public InArgument<string> Source { get; set; }
    public InArgument<string> Destination { get; set; }

    protected override WorkflowElement CreateBody()
    {
        return new PowerShellCommand
        {
            CommandText = "copy-item",
            Parameters =
            {
                { "path", new InArgument<string>(Source) },
                { "destination", new InArgument<string>(Destination) } ,
                { "recurse",  new InArgument<bool>(false) }
            },
        };
    }
}
When you need to define a custom activity from scratch (something not based on existing activities), you must derive from WorkflowElement and override the Execute method. This approach requires a little more code, and it's similar to how it works today when deriving from Activity. However, Microsoft has further simplified things around writing these custom activities.
One of the things that makes writing a custom activity difficult is managing data flow into and out of the activity. For example, today it is not possible to define a set of typed arguments that get passed into and out of an activity. Normally, developers write custom classes that serialize and deserialize arguments from a workflow queue (for more, see Michael Kennedy's article, "Web Apps that Support Long-Running Operations.") Writing this plumbing code isn't difficult, but it is extra work that distracts from the central goal of modeling application flow. The .NET Framework 4.0 extends the activity programming model with three new data flow concepts that promise to simplify things significantly: arguments, variables, and expressions.
You use arguments to define the way data flows in and out of an activity. Each argument has a specified binding direction such as input, output, or input/output. The following example shows how to author a simple Audit activity with a single input argument called "AuditMessage":
public class Audit: WorkflowElement {
    public InArgument<string> AuditMessage{ get; set; }

    protected override void Execute(ActivityExecutionContext context) {
        WriteToEventLog(AuditMessage.Get(context));
    }
}
Variables provide a way for declaring named storage for data. You can define variables in code as well as in XAML-based workflows. Variables can also be defined at different scopes within a workflow (within nested workflow elements), and they are part of the workflow program definition at design time, while the values are stored in the workflow instance at run time.
The following example shows how to write a XAML workflow that defines a variable named message, assigns it a value (using the new Assign activity), and then passes the variable's value into the custom Audit activity defined previously:
<Sequence
    xmlns="http://schemas.microsoft.com/netfx/2009/xaml/workflowmodel"
    xmlns:p="http://schemas.microsoft.com/netfx/2008/xaml/schema"
    xmlns:s="clr-namespace:Samples;assembly=Samples"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
    <Sequence.Variables>
        <Variable Name="message" x:TypeArguments="p:String" />
    </Sequence.Variables>
    <Assign x:TypeArguments="p:String" To="[message]"
        Value="Audit message: something bad happened" />
    <s:Audit Text="[message]" />
</Sequence>
An expression is a construct that takes one or more input arguments, performs some operation or behavior on those input arguments, and then returns a value. Expressions are defined by deriving a class from ValueExpression, which itself derives from WorkflowElement, so expressions can be used anywhere an activity can be used. Expressions can also be used as a binding to another activity's argument. This example defines a simple Format expression:
public class Format : ValueExpression<string> {
    public InArgument<string> FormatString { get; set; }
    public InArgument<string> Arg { get; set; }

    protected override void Execute(ActivityExecutionContext context) {
        context.SetValue(Result,
            String.Format(FormatString.Get(context), Arg.Get(context)));
    }
}
You can incorporate this expression into your XAML workflows like any other activity. For example, the following workflow illustrates how to pass the result of the Format expression into the Audit activity to write the result to the event log (this assumes that Audit's content maps to the message argument):
<Sequence
    xmlns="http://schemas.microsoft.com/netfx/2009/xaml/workflowmodel"
    xmlns:p="http://schemas.microsoft.com/netfx/2008/xaml/schema"
    xmlns:s="clr-namespace:Samples;assembly=Samples"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml">
    <Sequence.Variables>
        <Variable Name="message" x:TypeArguments="p:String" />
    </Sequence.Variables>
    <s:Audit>
        <s:Format FormatString="Audit message: {0}" Arg="[message]"/>
    </s:Audit>
</Sequence>
Another thing to note is that there's nothing special about the root activity. Anything that derives from WorkflowElement can be used at the root or anywhere within the workflow. This lets you arbitrarily compose different workflow styles within one another (for example, a state machine within a flowchart within a sequence).
Most of these programming model changes help make declarative workflows first-class citizens because you no longer need imperative codebehind files for defining data-flow properties—it can all be done declaratively in XAML now. However, since these are pretty fundamental changes, they required some significant changes to the WF runtime that ultimately break compatibility with your .NET Framework 3.x activities (see the sidebar on migrating workflows for more information). Despite that, though, Microsoft believes that simplifying the WF programming model will greatly benefit WF developers over time.
Migrating Workflows to .NET 4.0
The new .NET Framework 4.0 activity programming model required some significant changes to the core WF runtime. As a result, custom activities designed for the .NET Framework 3.0 and 3.5 won't be able to run within a .NET Framework 4.0 workflow host without some special care.
To facilitate interoperability, the .NET Framework 4.0 comes with a special Interop activity that makes it easy to wrap a custom .NET 3.x activity within a .NET 4.0 host. This approach doesn't work for all .NET 3.x activities—it specifically does not work for root activities. Hence, when moving to Visual Studio 2010, you'll need to redesign your workflows using the new workflow designer (because it has also changed extensively), then you can wrap custom .NET 3.x activities using the new Interop activity within your new workflow definitions.

More New WCF Features in the .NET Framework 4.0
It's easy to see how you might implement a WCF service with a workflow, but to truly produce a declarative workflow service, you will also need a way to define service contracts and to configure endpoint definitions using the declarative XAML-only model. This is precisely what WCF in the .NET Framework 4.0 brings to the table. Suppose you have this WCF service contract definition:
[ServiceContract]
public interface ICalculator
{
    [OperationContract]
    int Add(int Op1, int Op2);
    [OperationContract]
    int Subtract(int Op1, int Op2);
};
In the .NET Framework 4.0, you can define the same contract declaratively using the following XAML definition:
<ServiceContract Name="ICalculator">
    <OperationContract Name="Add">
        <OperationArgument Name="Op1" Type="p:Int32" />
        <OperationArgument Name="Op2" Type="p:Int32" />
        <OperationArgument Direction="Out" Name="res1" Type="p:Int32" />
   </OperationContract>
   <OperationContract Name="Subtract">
        <OperationArgument Name="Op3" Type="p:Int32" />
        <OperationArgument Name="Op4" Type="p:Int32" />
        <OperationArgument Direction="Out" Name="res2" Type="p:Int32" />
   </OperationContract>
</ServiceContract>
Now that you have the service contract defined in XAML, the next step is to define how you want to project the contract onto the wire. The .NET Framework 4.0 introduces the concept of a contract projection to separate the logical contract definition from the representation of the messages that are sent and received. This allows you to define a single-service contract that can be projected differently to support different messaging styles.
For example, you can have one contract projection for SOAP-based messaging and another projection for REST/POX messaging, but both are based on the same logical service contract. Here's how you can define a SOAP contract projection for the service contract just defined:
<Service.KnownProjections>
    <SoapContractProjection Name="ICalculatorSoapProjection">
        <!-- service contract definition goes here -->
    </SoapContractProjection>
</Service.KnownProjections>
With this contract definition and projection in place, you are able to implement the service logic in XAML. The root element for the XAML service definition is Service. The Service element holds the contract and projection definitions in the Service.KnownProjections child element. The service implementation goes in the Service.Implementation element (which is a declarative workflow). And finally, the service's endpoint configuration goes in the Service.Endpoints element. Figure 5 shows the complete declarative service definition in XAML.
Figure 5 Declarative WCF Service
<Service xmlns="clr-namespace:System.ServiceModel;assembly=System.WorkflowServiceModel"
     xmlns:p="http://schemas.microsoft.com/netfx/2008/xaml/schema"
     xmlns:ss="clr-namespace:System.ServiceModel;assembly=System.ServiceModel"
     xmlns:sw="clr-namespace:System.WorkflowServiceModel;assembly=System.WorkfowServiceModel"
     xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
     xmlns:x2="http://schemas.microsoft.com/netfx/2008/xaml">
    <Service.KnownProjections>
        <SoapContractProjection x:Name="ICalculatorSoapProjection">
            <ServiceContract x:Name="ICalculatorContract"
                <OperationContract Name="Add" x:Name="Add">
                    <OperationArgument Name="Op1" Type="p:Int32" />
                    <OperationArgument Name="Op2" Type="p:Int32" />
                    <OperationArgument Direction="Out" Name="Result"
                     Type="p:Int32" />
                </OperationContract>
            </ServiceContract>
        </SoapContractProjection>
    </Service.KnownProjections>
    <Service.Implementation>
        <sw:WorkflowServiceImplementation>
              <!-- place service implementation here -->
        </sw:WorkflowServiceImplementation>
    </Service.Implementation>
    <Service.Endpoints>
        <Endpoint Uri="http://localhost:8080/calc"
         ContractProjection="{x2:Reference ICalculatorSoapProjection}">
            <Endpoint.Binding>
                <ss:BasicHttpBinding />
            </Endpoint.Binding>
        </Endpoint>
    </Service.Endpoints>
</Service>
One downside at this early stage is that there is no visual designer to help in building services declaratively. I hope that as the Community Technology Previews (CTPs) roll out, Microsoft will deliver a developer-friendly designer experience for working with the XAML definition.
In addition to the declarative service support and the WCF-based activities mentioned earlier, the .NET Framework 4.0 comes with several other lower-level WCF improvements. Some of these features make it possible to manage workflow services with the new "Dublin" extensions. These features include improved configuration, message correlation, durable duplex communication, durable timers, and Event Tracing for Windows (ETW) tracing integration. Other capabilities that add value when hosting your services in a managed environment include standard control endpoints and auto-start. You will hear more about these features in the months ahead.

The Need for "Dublin"
In the real world, another challenging aspect of working with WCF and WF is figuring out where to host your services and workflows in a server environment. For WCF, the answer is usually pretty easy—the most common choice is IIS and the Windows Process Activation Service (WAS) in Windows Server 2008.
Today, the combination of IIS and WAS provides several key features including process activation in response to incoming messages, process monitoring and health management, process recycling, CLR AppDomain integration, built-in security, and some basic management capabilities made available through IIS Manager and Windows PowerShell cmdlets. These combined features typically make IIS/WAS the right choice for hosting WCF services in a server environment because most folks don't want to build that kind of stuff themselves.
Although IIS/WAS provides a foundation for hosting WCF applications, it falls short in a few important areas. Its biggest shortcoming is in the area of service manageability. IIS/WAS doesn't really provide any WCF-specific service management capabilities such as service tracing, monitoring, and diagnosing running service instances. For example, an administrator cannot currently list all the WCF services that are configured in IIS—it's not possible to query the status of all the hosted services. IIS/WAS also doesn't provide built-in support for simplifying scale-out deployments or common WCF configuration tasks, which turns out to be a sore spot for WCF.
The hosting challenges are worse for WF applications in server environments because of the inherently stateful model required to support long-running workflows across server farms. Because of these complexities, Microsoft didn't ship a WF server host in the .NET Framework 3.0—developers had to write this themselves. It wasn't until the .NET Framework 3.5, with the introduction of the WorkflowServiceHost class, that it was possible to host WF workflows as WCF services out of the box, thereby making it possible to host your WF workflows within IIS/WAS as well.
WorkflowServiceHost was a good start, but it didn't come with any tool support for managing stateful workflows across an entire Web server farm, nor did it provide tools for monitoring and managing running workflow instances at run time.
In terms of service and workflow management features, most developers expect an experience similar to what's provided by BizTalk Server, yet simpler. Many organizations want the BizTalk management experience, but they don't need the integration capabilities or the reliability semantics (and corresponding performance hit) that's inherent in the BizTalk Server MessageBox. A lighter-weight model, specifically designed for WCF and WF applications, is more appropriate.

A Guide to "Dublin"
Microsoft has been working on a new set of extensions to Windows Server—code-named "Dublin"—that provides valuable hosting and management features for your WCF and WF applications. "Dublin" is essentially a set of service management extensions built on top of IIS/WAS that will ship as part of Windows Server. When using the "Dublin" extensions, you'll still be hosting your service and workflows in IIS/WAS, but your applications will enjoy additional WCF and WF-specific management capabilities and tools that don't currently exist in IIS/WAS.
The various "Dublin" extensions will ship in a future version of Windows Server as part of the Windows Server Application Server role. Hence, the set of features is often referred to as the Windows Application Server, although that's not the official name sanctioned by Microsoft. For the rest of this article, I'll refer to the new Windows Server Application Server extensions as simply "Dublin."
The key capabilities provided by the "Dublin" extensions include management support for reliable and durable services and long-running workflows. "Dublin" makes it possible to deploy an application to individual servers across a farm, and it provides the tools you need for specific service management tasks. I will be discussing these capabilities in more detail throughout the following sections, but first let's take a look at the overall architecture, which is illustrated in Figure 6.
Figure 6 The “Dublin” Architecture
As you can see, "Dublin" provides several runtime databases that lay a foundation for service persistence and monitoring. There is a layer of runtime components and services provided by the .NET Framework that build on top of these databases. "Dublin" further extends these runtimes to provide integrated hosting, persistence, monitoring, and messaging capabilities. This layer, in conjunction with the underlying runtime databases, constitutes what is referred to as "Dublin."
The top two layers in the architecture are what make "Dublin" usable by humans. There's a management API layer that makes the various capabilities scriptable via Windows PowerShell cmdlets. On top of that there is an IIS Manager experience that should feel familiar to today's IIS administrators because it actually builds on the Windows PowerShell cmdlets. So everything you can do in IIS Manager you can also do in Windows PowerShell.
Microsoft has added numerous UI extensions to IIS Manager for performing the various hosting and management tasks described in this section. There are extensions for deploying and configuring applications, managing applications, and monitoring applications. These extensions also provide a runtime dashboard of your system showing things such as running, suspended, and persisted workflow instances.

Building a WCF Workflow Service for "Dublin"
Visual Studio 2010 makes it really easy to build services that target the "Dublin" extensions through a new project template (see Figure 7). This project template gives you a web.config preconfigured with the "Dublin"-provided persistence provider and durable timer service.
Figure 7 Creating a New “Dublin” Service Library
Once you've generated the project, you can focus on implementing the WCF workflow service using the various techniques described earlier in this article. The new workflow designer experience makes it possible to implement your entire service using the graphical workflow development approach. Once you're finished building your complete workflow service, you're ready to deploy the service to a Windows Server enabled with the "Dublin" extensions.
You can deploy your services to IIS/WAS using whatever technique you normally use for Web applications (you can deploy it directly from Visual Studio if your environment allows that). You can also use Windows PowerShell cmdlets to automate these tasks. (These cmdlets allow you to build automated deployment scripts for your WCF and WF environments. If you want to try using any of these cmdlets, simply open Windows PowerShell on a Windows Application Server and type "help <command>" to learn how to get started with that particular command.) Once deployed, you can begin using IIS Manager to access the various "Dublin" extensions highlighted in Figure 8.
Figure 8 The “Dublin” Extensions in the IIS Manager

Deploying Applications with "Dublin"
Many environments do not allow developers to deploy services directly to their managed servers for security and isolation reasons. "Dublin" provides an alternate solution for deploying services through its application export/import features.
You can package up WCF and WF applications for deployment on other servers by selecting the Application Export feature in IIS Manager. This will bring up a dialog that asks you to select a specific application to export. Once you've specified a location, pressing Export will generate a file package with a .zip extension containing all the WCF and WF code as well as all the metadata and configuration settings for a single application. This package can then be moved and imported onto another server through the "Dublin" extensions. Now you can simply send the .zip file to the IT professional who manages your servers and let him take care of it.
The Windows PowerShell Export-Application cmdlet accomplishes the same thing. This Application Export capability is also very nice for system test and validation environments.
You can import a WCF and WF application by selecting the Application Import feature or by using the Import-Application cmdlet (you can also view the contents of a package by using the Get-PackageManifest cmdlet). The import feature will then prompt you to select the package (.zip file) that you want to import.
During the process, you can specify the application name, the application pool, and the physical path for the application, if desired. And since "Dublin" provides centralized persistence configuration, when you deploy your workflow services to another server, you don't need to worry about changing the persistence configuration at the application level. It will simply use the new persistence database associated with the new server. These features make the process of deploying WCF and WF applications in a server environment very approachable for the system administrator responsible for those tasks.
Once you've successfully deployed an application, you can begin configuring your services through the other extensions provided by "Dublin." For example, in some situations, the system administrator may need to manually reconfigure the runtime configuration for the persistence and tracking databases. With the "Dublin" extensions, this is quite simple. Simply select Services from the default view and then you'll see a list of all managed services, along with a right-hand pane that exposes various service configuration options (see Figure 9). These options allow you to easily change the persistence settings, tracking settings, and other WCF settings related to security and throttling. And you never have to touch a WCF configuration file.
Figure 9 Viewing and Configuring Services with the “Dublin” Extensions
There are multiple Windows PowerShell cmdlets for performing these tasks, including the Get-ServicePersistence, Set-ServicePersistence, Enable-ServiceTracking, and Get-TrackingParticipant. This means it is much easier to automate the setup and configuration of an application server environment.

Managing Running Applications
During the lifecycle of an application, developers and system administrators need the ability to monitor the application's health, to identify and view problematic workflow instances, and to terminate them when necessary. "Dublin" provides numerous extensions that target these common management needs.
If you select the Persisted Instances option in IIS Manager (see Figure 8), you'll be taken to a dashboard view that provides an overview of the running workflow instances that have been persisted and potentially suspended (see Figure 10). The Overview box displays the total number of applications and services deployed to the server, site, or application (depending on the selected scope).
Figure 10 Viewing Persisted Workflow Instances
The Persisted Instances box displays a summary of the running and persisted workflow instances, calling out those that may be blocked or suspended (meaning they ended in an error). Since the suspended instances are the ones that humans will typically have to deal with directly, they provide several other boxes that make it easy to isolate specific suspended instances based on virtual path, service name, or type of exception.
You can click on any of the links (shown in blue) to display the list of persisted instances and to view their details. At this point, you can suspend, terminate, or abort service instances manually by selecting from the actions shown in the right-hand pane (see Figure 11). Suspending a service instance stops the execution of the instance and prevents it from receiving new messages. Suspended instances can later be resumed, at which point they'll begin receiving messages again. Terminating a service instance stops the execution of the instance and removes it from the persistence store, which means it cannot be resumed. Finally, aborting a service instance clears the in-memory state pertaining to the given instance and reverts back to the last persistent point (which is stored in the persistence store).
Figure 11 Viewing Individual Persisted Instances
The Windows PowerShell Get-ServiceInstance and Stop-ServiceInstance cmdlets offer equivalent capabilities from the command-line; they provide numerous command-line options for identifying specific service instances.

Updating Running Applications
One of the particularly troublesome aspects of working with real systems is the need to periodically update them. For most complex distributed applications, this can be tricky to do correctly when the necessary changes require updates to the database, business logic, and service code simultaneously. Usually, it's not possible to apply significant updates atomically while the application is running.
One way to approach this problem is to take the application offline while you are performing the update. However, if you really think about it, there's not an easy way to take an IIS/WAS application offline while simultaneously performing an update to a Web site. This is another area where the "Dublin" extensions add value through a simple offline feature.
You can select an application in the IIS Manager and then select the Disable Protocols command in the right-hand pane shown in Figure 8 (it is important to note that this will likely be renamed in a future release). Doing so causes all instances of the application's services to end in a blocked state or to finish their execution naturally, without allowing any new incoming requests to be processed in the meantime.
At this point you have basically stopped the message flow for this application, meaning that clients will no longer be able to send messages to services without receiving errors (they won't be queued up like they otherwise would be in BizTalk Server). The way this works behind the scenes is quite simple: the "Dublin" extension simply removes all of the protocol handlers for this particular application and saves them so they can easily be restored once you are finished with the update.
Once the application is offline, you can perform whatever updates are required. When you are finished with the updates and are ready to bring the application back online again, you can then select the Restore Protocols command. The Windows PowerShell Disable-ApplicationMessageFlow and the Enable-ApplicationMessageFlow cmdlets can also be used to perform these same tasks from the command line.

Monitoring Running Applications
Businesses also need the ability to monitor running applications in order to see how the business is functioning and what changes might be necessary. The WCF and WF runtimes already come with a built-in tracking infrastructure that the "Dublin" extensions build on, making it easy to enable monitoring within your WCF and WF applications.
Figure 12 Configuring Basic Tracking
There are two main players in the .NET Framework 4.0 tracking architecture: tracking profiles and tracking participants. Developers define tracking profiles that tell the runtime what events to track, and then tracking participants can subscribe to those events.
"Dublin" comes with some built-in tracking profiles that make it easy to track to a common set of useful events. You can easily enable tracking for your application by navigating to a particular service in IIS Manager and selecting Tracking in the right-hand pane.
Then you'll see the dialog shown in Figure 12, which allows you to configure some basic tracking capabilities for your workflows and services. Once you've configured this, the appropriate updates will be made to your configuration and the WCF/WF tracking infrastructure will kick in.
If you would like, you can also view tracking data through the "Dublin" extensions. You can select View Tracking Data while inspecting persisted service instances (see Figure 11), which will run a SQL query against the monitoring store to produce a list of tracking events for you to view (see Figure 13). When you want to track custom events, you can define custom tracking profiles and configure them through the Tracking Profiles option on the main IIS Manager view.
Figure 13 Viewing Tracking Data
I don't have space to cover all of the "Dublin" features here—there are several other compelling features that merit further discussion (see the "Additional Dublin Features" sidebar for more)—but I hope you have a clearer understanding of the role of "Dublin" and some of the key features it will provide.

From "Dublin" to "Oslo"?
If you've heard about the platform code-named "Oslo," you're probably wondering at this point how "Dublin" relates to that initiative. First, "Oslo" is a new modeling platform being developed by Microsoft to simplify the way you design, build, and manage distributed applications. The modeling platform consists of three main components: the "Oslo" modeling language (also known as "M"), the "Oslo" repository, and the "Oslo" modeling tool (also known as "Quadrant"). "Oslo" is indeed a platform that other applications and technologies can build on to simplify the user experience through a model-driven approach.
Additional “Dublin” Features
"Dublin" comes with several other features I didn't have space to cover in detail in this article. But one worth calling out is support for scale-out deployments across server farms that integrate with existing load-balancing solutions such as those for managing persisted service instances across the farm through a centralized persistence database. The built-in error handling logic allows persisted instances to execute on any node in the farm and prevents race conditions when multiple nodes compete for the same instance.
The other important component is the Forwarding Service. This service makes it possible to intercept all incoming messages in order to perform central routing based on message content. In particular, this feature provides a nice foundation for building sophisticated service versioning solutions.
"Dublin" also comes with several key services for managing the lifecycle of service instances including the Durable Timer Server and the Instance Restart Service. "Dublin" also knows how to take advantage of the Instance Control Endpoint provided by the .NET Framework 4.0 and the new IIS/WAS auto-start feature, which enables you to start services when the machine starts up rather than waiting for the first message. More articles about these features will be appearing in the next several months.
"Dublin" will be one of the first technologies to leverage the "Oslo" modeling platform. You will be able to export "Oslo" apps from the repository and easily deploy them to "Dublin," where they can benefit from the various hosting and management features discussed in this article. Using models to describe and automate application deployments seems like a win for complex IT environments.
As both "Dublin" and "Oslo" continue to mature, it's likely that the integration between the two technologies will continue to grow. Microsoft has stated its intention that the two technologies will be very complementary to each other.

What about BizTalk Server?
Another common question is how "Dublin" relates to BizTalk Server. In a lot of ways, BizTalk Server inspired many of the features you see in "Dublin" today. Although both technologies provide similar management capabilities, there's a big difference between the two in terms of their focus. "Dublin" adds hosting and management extensions to Windows Server designed specifically for WCF and WF applications, whereas BizTalk Server is focused on application integration with non-Microsoft systems using a variety of different message formats, transports, and mapping techniques.
The primary focus of BizTalk Server has always been and will continue to be integration with non-Microsoft systems (line-of-business applications, legacy systems, RFID devices, and business-to-business protocols). BizTalk Server will remain focused on these core strengths in the years ahead. In general, you'll want to continue using BizTalk Server when you're focused primarily on these types of enterprise application integration (EAI) scenarios.
However, since many WCF and WF apps don't need these kinds of integration capabilities, BizTalk Server can often feel like overkill. This is precisely where "Dublin" fits into the picture—as a simpler alternative that provides similar management capabilities. In the end, "Dublin" will be more cost-effective for these scenarios than BizTalk Server since the "Dublin" extensions will ship as a core part of Windows Server and won't require you to buy integration adapters you don't need. It's likely that a future version of BizTalk Server will build upon the "Dublin" extensions in order to leverage the core management investments being made to Windows Server.
A special thanks to Eileen Rumwell, Mark Berman, Dino Chiesa, Mark Fussell, Ford McKinstry, Marjan Kalantar, Cliff Simpkins, Kent Brown, Kris Horrocks, and Kenny Wolf for their helpful assistance with this article.

Aaron Skonnard is a cofounder of Pluralsight, a premier Microsoft .NET training provider that offers both instructor-led and online training courses. Aaron is the author of numerous books, white papers, and articles as well as Pluralsight's training courses on REST, Windows Communication Foundation, and BizTalk Server. You can reach him at pluralsight.com/aaron.


Geneva Framework
Building A Custom Security Token Service
Michele Leroux Bustamante

This article is based on a prerelease version of the "Geneva" Framework. All information is subject to change.

This article discusses:
  • Implementing Security Token Services with Geneva Framework
  • Federated security
  • Claims transformation
This article uses the following technologies:
Windows Communication Foundation, ASP.NET, Geneva Framework
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
A Security Token Services Primer
Building a Custom Active STS
Extending SecurityTokenService
Hosting and Configuring the STS
Security Token Handlers
Building a Custom Passive STS
FederatedPassiveTokenService Control
SessionAuthenticationModule
User Authentication
Claims Transformation
Wrap-Up
The Microsoft claims-based access (CBA) platform strategy—code-named "Geneva"—includes the "Geneva" Framework, "Geneva" Server, and Windows CardSpace "Geneva." The Geneva Framework provides developers with tools to build claims-based applications and services that involve tokens issued by a Security Token Service (STS), as well as tools for building a custom STS and for building Windows CardSpace-enabled applications. Geneva Server is an enterprise STS, while the Geneva Framework makes it possible to build a custom STS for environments that don't require enterprise-level features. Windows CardSpace Geneva is the evolution of Windows CardSpace as the identity selector and identity provider on Windows client machines.
In my last article about the Geneva Framework I discussed a better way to build claims-based Windows Communication Foundation (WCF) services that rely on tokens issued by an STS. Here, I will build a custom STS with the Geneva Framework.
Before reading on, you should read the Geneva Framework white paper for developers written by Keith Brown and Sesha Mani and my last article, "Geneva Framework: A Better Approach for Building Claims-Based WCF Services."

A Security Token Services Primer
Whether the STS is based on Geneva Server or built with the Geneva Framework, its primary role is to act as a security gateway to authenticate callers and issue security tokens carrying claims that describe the caller. You'll recall from the aforementioned articles that there are several scenarios supported by STS authentication:
  • Decoupling apps and services from the authentication mechanism so that they can focus on authorizing relevant claims.
  • Supporting multiple credential types without complicating the implementation of applications and services.
  • Supporting federated scenarios where users are authenticated by their domain and granted access to resources in another domain—by establishing trust between each domain's STS.
  • Facilitating identity delegation scenarios where the authenticated user is granted access to downstream services.
  • Facilitating claims transformation so that relevant claims are available for authorization at applications and services.
Any of these scenarios can be based on passive federation (browser-based) or active federation (Windows client-based). Next, I will elaborate on these scenarios while describing how to build relevant logic into a custom STS built with the Geneva Framework.
Before diving into the implementation of an STS, let me first review some fundamentals. An STS used in active federation is an implementation of the WS-Federation Active Requestor Profile (see WS-Federation TC) and (primarily) the WS-Trust specification (see WS-Trust 1.3).
From a high level, WS-Trust describes a contract with four service operations: Issue, Validate, Renew, and Cancel. Respectively, these operations are called by clients to request a security token, to validate a security token, to renew an expired security token, or to cancel a security token that should no longer be used. Each operation is sent messages in the form of Request for Security Token (RST) and return messages in the form of RST Response (RSTR) according to the WS-Trust specification. For this article I am assuming that the issued token is a Security Assertion Markup Language (SAML) 1.1 or SAML 2.0 token.
Figure 1 illustrates the core contents of the RST and RSTR for active token issuance. The RST message includes necessary information to request a security token including the type of token to issue (SAML for this discussion), the claims requested by the relying party (RP) to be included in the issued token, the information about the RP (AppliesTo) including the URL and usually the certificate identifying the RP, and optionally (not shown) key material to be used for the proof-of-possession key (proof key) returned with the RSTR.
Figure 1 Token Issuance for an Active Federation Scenario
If token issuance is successful, the RSTR will include the issued SAML token and the proof key (assuming the STS decides which proof key to use and thus must return it in the RSTR). The SAML token will include relevant claims for the authenticated party, is signed by the STS to protect the token from being tampered with, contains the proof key encrypted for the RP, and is itself encrypted for the RP so that only the intended recipient can process the token.
The client uses the proof key to sign the message to the RP. The RP must be able to decrypt the proof key inside the SAML token or reject the message. If the proof key inside the token matches the signature on the message, this proves that the call to the RP was sent by the party that requested the token.
Passive federation scenarios are based on a WS-Federation Passive Requestor Profile, which involves browser-based communication. The underlying messaging is still based on WS-Trust; however, the RST is broken into query string parameters on the STS URL, and the RSTR is typically posted to the RP as a form parameter. The passive STS and RP intercept these parameters with federation Web handlers. The passive STS may process WS-Trust requests directly or pass them to an underlying WS-Trust implementation. Figure 2 illustrates how the RST and RSTR are handled in a passive federation scenario.
Figure 2 Token Issuance for a Passive Federation Scenario
One relevant difference between the active and passive federation scenarios is the type of SAML token issued. Active federation usually relies on a SAML token that uses a subject confirmation of type "holder-of-key," which means that there is a proof key used as I have described to prove that the client sending the token to authenticate is the subject that requested the token (also known as ActAs behavior). Passive federation scenarios usually involve a SAML token with a subject confirmation of type "bearer" (it is also known as a bearer token). This type of token does not include a proof key and is sometimes called a keyless token. It relies on the transport to securely procure it from the STS and transmit it to the RP.
These concepts—WS-Federation, WS-Trust, and SAML tokens—are important background information for the discussion that follows. First, I will describe how to build an active STS using the Geneva Framework. Following that, I'll discuss building a passive STS, and finally I'll present some extended scenarios for each.

Building a Custom Active STS
In a simple active federation scenario, such as that shown in Figure 3, the following participants are usually present:
  • An RP, which is the service called by a client.
  • A single STS, also implemented as a service, which supports the WS-Trust protocol. This STS authenticates callers and issues a security token with claims identifying the caller, also known as an Identity Provider or IP-STS.
  • A client, in this case a Windows-based application, that relies on a proxy to authenticate to the STS, to retrieve the resulting issued token, and to send messages to the RP supplying the issued token for authentication and authorization.
Figure 3 A Simple Federation Scenario with a Single RP and an Active IP-STS
Figure 4 illustrates the moving parts behind this implementation. Included are a custom SecurityTokenService implementation, the use of a ServiceHost extension (WSTrustServiceHost) to initialize the runtime for federation, configuration of one or more WS-Trust endpoints using a derivative of the WSTrustContract type, and other related configuration settings for the identity model runtime. In the sections that are to follow, I will review each one of these elements of a custom STS implementation based on the Geneva Framework.
Figure 4 Implementation Architecture for a Custom Active STS and an Active IP-STS

Extending SecurityTokenService
The Geneva Framework provides the core functionality to build a custom STS through the SecurityTokenService type from the Microsoft.IdentityModel.SecurityTokenService namespace. This abstract class handles the heavy lifting of processing RST and RSTR messages and generating security tokens. A custom STS type inherits this class and provides (at a minimum) the following functionality:
  • A constructor that accepts a custom SecurityTokenServiceConfiguration instance to configure some basic features of the STS (to be discussed later).
  • An override for GetScope to validate the target RP for the request and to supply an appropriate encrypting credential for that RP and a signing credential for the security token.
  • An override for GetOutputClaimsIdentity to supply claims for the resulting security token.
Figure 5 shows some code for a simple custom STS implementation with this functionality. Recall the flow of communication for an active STS from Figures 1 and 2. The STS implementation, IdentitySTS, validates the incoming RST when GetScope is called—by verifying that the AppliesTo element of the RST indeed points to a trusted URI. Presumably the STS manages a list of trusted RPs for which tokens can be issued, along with their certificates. GetScope sets the EncryptingCredentials property of the scope to the appropriate certificate if AppliesTo passes validation, in this case "RPKey". In addition, the SigningCredentials property is set to the appropriate certificate to be used for signing the issued token. This is usually the private key of the STS, in this case "IPKey".
Figure 5 A Simple Custom STS Implementation
public class IdentitySTS : SecurityTokenService
{
    public IdentitySTS(SecurityTokenServiceConfiguration config)
        : base( config )
    {
    }

    protected override IClaimsIdentity GetOutputClaimsIdentity(
        IClaimsPrincipal principal, RequestSecurityToken request,
        Scope scope)
    {
        IClaimsIdentity claimsIdentity = new ClaimsIdentity();

        claimsIdentity.Claims.Add(new Claim(ClaimTypes.Name,
            principal.Identity.Name));
        claimsIdentity.Claims.Add(new Claim(ClaimTypes.Role, "Users"));

        return claimsIdentity;
    }

    protected override Scope  GetScope(
        Microsoft.IdentityModel.Claims.IClaimsPrincipal principal,
        RequestSecurityToken request)
    {

        Scope scope = new Scope(request);
        scope.EncryptingCredentials = this.GetCredentialsForAppliesTo(
                                                     request.AppliesTo);
        scope.SigningCredentials = new
          X509SigningCredentials(CertificateUtil.GetCertificate(StoreName.My,
          StoreLocation.LocalMachine, "CN=IPKey"));
        return scope;
    }

    private X509EncryptingCredentials GetCredentialsForAppliesTo(Endpoint
        Address appliesTo)
    {
        if (appliesTo == null || appliesTo.Uri ==null ||
          string.IsNullOrEmpty(appliesTo.Uri.AbsolutePath))
        {
            throw new InvalidRequestException(
                "AppliesTo must be supplied in the RST.");
        }

        X509EncryptingCredentials creds = null;
        if (appliesTo.Uri.AbsoluteUri.StartsWith(
            "http://localhost:8000/RelyingPartyService"))
        {
            creds = new X509EncryptingCredentials(
                CertificateUtil.GetCertificate(StoreName.TrustedPeople,
                StoreLocation.LocalMachine,
                "CN=RPKey"));
        }
        else
            throw new InvalidRequestException(String.Format(
                "Invalid relying party address: {0}",
                appliesTo.Uri.AbsoluteUri));

        return creds;
    }
}
When GetOutputClaimsIdentity is called, a ClaimsPrincipal is passed by the runtime with the authenticated caller's identity. This identity is typically used to determine the appropriate claims to grant the caller. In Figure 5 the code generates a name claim and a hardcoded role claim for the caller, and it returns this in the form of a ClaimsIdentity. This ClaimsIdentity supplies claims to the runtime for the token to be issued.
This STS implementation could also be extended with the following functionality:
  • GetOutputClaimsIdentity could include code for looking up the user in a custom credential store and look up additional claims. For example, a list of roles, other relevant details about the user such as her e-mail address, or custom claims representing more granular application rights such as create, read, update, or delete.
  • GetScope could look up the AppliesTo URI in a custom database that lists all trusted RPs with their associated certificates.

Hosting and Configuring the STS
If you are familiar with WCF, you know that one or more endpoints must be configured in order for clients to send messages to a service. In the case of an STS, the service contract to be used for each endpoint must be based on WS-Trust protocol, which includes four operations: Issue, Validate, Renew, and Cancel. In fact, there are two versions of WS-Trust protocol that could be implemented by an STS:
  • WS-Trust 1.3: the latest version of the WS-Trust spec.
  • WS-Trust February 2005: the version of WS-Trust that many industry partners implemented while waiting on the standard to be ratified.
You can also provide asynchronous implementations for GetScope(), GetOutputClaimsIdentity()—among other methods—in your SecurityTokenService type. This improves scalability for I/O intensive operations such as accessing certificates or interacting with claims data. When you configure endpoints for your STS, you must select which contract to expose for the endpoint. The Microsoft.IdentityModel.Protocols namespace includes these two service contracts for STS endpoints: IWSTrust13SyncContract and IWSTrustFeb2005SyncContract. Figure 6 shows the configuration for an STS service with two endpoints, one for each contract. Note that there are also asynchronous versions of the contract, which would be used to implement an asynchronous proxy: IWSTrust13AsyncContract and IWSTrustFeb2005AsyncContract.
Figure 6 STS Service Config with Multiple WS-Trust Endpoints
<service name=
  "Microsoft.IdentityModel.Protocols.WSTrust.WSTrustServiceContract"
  behaviorConfiguration="stsBehavior">
  <endpoint address="WSTrustFeb05" binding="wsHttpBinding"
    contract="Microsoft.IdentityModel.Protocols.WSTrust.
    IWSTrustFeb2005SyncContract"/>
  <endpoint address="WSTrust13" binding="wsHttpBinding"
    contract="Microsoft.IdentityModel.Protocols.WSTrust.
    IWSTrust13SyncContract"/>
</service>
The STS should expose an endpoint based on the WS-Trust February 2005 for backward compatibility with preexisting clients. The service type that implements both contracts is the WSTrustServiceContract type found in the Microsoft.IdentityModel.Protocols.WSTrust namespace. This is the type that should be referenced in the <service> configuration section for the STS.
As illustrated in the diagram in Figure 4, the <service> configuration and its endpoints are used to initialize the host with the correct WSTrustServiceContract type. This type is also initialized with a reference to the custom SecurityTokenService implementation during host initialization. That's how the runtime directs messages to the custom STS.
In Figure 6, both STS endpoints rely on Windows credentials to authenticate callers (the default behavior of wsHttpBinding). The STS can expose multiple endpoints with alternate binding configurations to support different credential types. This also involves configuring an appropriate security token handler for each credential type. I'll discuss token handler configuration settings shortly.
The Geneva Framework supplies a custom ServiceHost type, WSTrustServiceHost, to be used for hosting STS instances. The following code illustrates how to construct the WSTrustServiceHost type in a self-hosting environment:
WSTrustServiceHost stsHost =
  new WSTrustServiceHost(new IdentitySTSConfiguration());
stsHost.Open();
WSTrustServiceHost relies on a custom SecurityTokenServiceConfiguration instance to initialize the runtime with WS-Trust endpoints, to enable the metadata exchange behavior for the STS, and to configure a metadata exchange endpoint.
When hosted in IIS, the WSTrustServiceHostFactory type is used to achieve the same results. In the .svc file, the @ServiceHost configuration specifies the factory type and the custom SecurityTokenServiceConfiguration type, as follows:
<%@ ServiceHost Factory="Microsoft.IdentityModel.Protocols.WSTrust.
  WSTrustServiceHostFactory"
  Service="STS.IdentitySTSConfiguration"  %>
The factory initializes the WSTrustServiceHost with the specified configuration upon activation.
A custom SecurityTokenServiceConfiguration type is required to initialize the WSTrustServiceHost for an STS. Figure 7 shows a custom implementation called IdentitySTSConfiguration.
Figure 7 A Custom SecurityTokenServiceConfiguration
public class IdentitySTSConfiguration: SecurityTokenServiceConfiguration
{

    public IdentitySTSConfiguration(): base("http://localhost:8010/sts")
    {

      this.TokenIssuerName = "http://localhost:8010/sts";

      this.SigningCredentials = new
        X509SigningCredentials(CertificateUtil.GetCertificate(StoreName.My,
        StoreLocation.LocalMachine, "CN=IPKey"));

      this.SecurityTokenService = typeof( IdentitySTS);

    }

}
This type must provide a URI for the STS, a signing credential, and a reference to the STS type with which the configuration is associated. The URL must be valid in the event that the STS issues managed cards—so that Windows CardSpace can import those cards. The base type requires a string value to be passed to the constructor for TokenIssuerName, but I recommend overriding that in code so that you can dynamically set the URI from configuration rather than hardcoding this value passed to the constructor.
The SecurityTokenServiceConfiguration type also exposes properties that can be used to set defaults for key size and token type, to disable access to metadata, to control token lifetime and clock skew, to set custom RST and RSTR serializers, to configure token handlers that authenticate callers, and to configure WS-Trust endpoints.
The following example shows how to disable metadata access and how to initialize WS-Trust endpoints (like those shown in Figure 6) programmatically rather than relying on <service> configuration settings:
IdentitySTSConfiguration config = new IdentitySTSConfiguration();
config.DisableWsdl = true;
config.TrustEndpoints.Add(new
  ServiceHostEndpointConfiguration("WSTrustFeb05", new WSHttpBinding(),
  typeof(IWSTrustFeb2005SyncContract)));
config.TrustEndpoints.Add(new ServiceHostEndpointConfiguration(
  "WSTrust13", new WSHttpBinding(),
  typeof(IWSTrust13SyncContract)));

WSTrustServiceHost stsHost = new WSTrustServiceHost(config);
The base SecurityTokenServiceConfiguration type also reads the <microsoft.identityModel> configuration section to initialize relevant STS configuration settings. Settings that can be declaratively configured for a custom STS include the maximum clock skew, security token handlers for authentication and issuance, a claims authentication manager, an issuer name registry, and token resolvers. Figure 8 shows a few useful settings configured for an STS.
Figure 8 <microsoft.identityModel> Config for an STS
<microsoft.identityModel>
  <maximumClockSkew value="00:05:00"/>
  <claimsAuthenticationManager type="STS.
    CustomClaimsAuthenticationManager, STS"/>
  <securityTokenHandlers>
    <remove type="Microsoft.IdentityModel.Tokens.
      WindowsUserNameSecurityTokenHandler,
      Microsoft.IdentityModel, Version=0.5.1.0, Culture=neutral,
      PublicKeyToken=31bf3856ad364e35" />
    <add type="Microsoft.IdentityModel.Tokens.
      MembershipUserNameSecurityTokenHandler,
      Microsoft.IdentityModel, Version=0.5.1.0, Culture=neutral,
      PublicKeyToken=31bf3856ad364e35">
        <usernameSecurityTokenHandlerRequirement
          membershipProvider="CustomProviders.CustomMembershipProvider,
          CustomProviders, Version=1.0.0.0, Culture=neutral,
           PublicKeyToken=c03h5a64f15d0b3f" />
    </add>
  </securityTokenHandlers>
</microsoft.identityModel>
A custom ClaimsAuthenticationManager type is invoked for non-Windows credentials giving the option to supply a custom ClaimsPrincipal type to the runtime prior to issuing claims in GetOutputClaimsIdentity. Custom security token handlers might be configured to supply settings that override the default behavior of a particular handler or to alter the choice of security token handler for a particular credential type.

Security Token Handlers
You probably expect a service behavior configuration to determine how authentication and authorization will take place for each STS endpoint. Recall from my last article that the Geneva Framework does things a little differently. In the case of authentication, the <securityTokenHandlers> collection specifies the SecurityTokenHandler types available to authenticate incoming requests.
This collection can only contain one entry for each SecurityTokenHandler type. For example, only one KerberosSecurityTokenHandler, UserNameSecurityTokenHandler, X509SecurityTokenHandler, Saml11SecurityTokenHandler, or Saml2SecurityTokenHandler can be registered to process requests for their respective credential type. If the STS endpoint expects a UserName credential, the default UserNameSecurityTokenHandler registered is the WindowsUserNameSecurityTokenHandler. Figure 8 illustrates the removal of WindowsUserNameSecurityTokenHandler and the addition of MembershipUserNameSecurityTokenHandler as its replacement—including related config settings for the membership provider.
You can also create custom SecurityTokenHandler types. Remember, as long as they derive from the appropriate base class matching the token category (such as UserNameSecurityTokenHandler), you can replace the default handler with the new custom handler. Figure 9 illustrates a custom UserNameSecurityTokenHandler implementation named CustomUserNameSecurityTokenHandler.
Figure 9 A Custom UserNameSecurityTokenHandler
public class CustomUserNameSecurityTokenHandler:
  UserNameSecurityTokenHandler
{
  public override ClaimsIdentityCollection ValidateToken(SecurityToken token)
  {
    UserNameSecurityToken userNameToken = token as UserNameSecurityToken;
    AuthenticateUser(userNameToken.UserName, userNameToken.Password);

    return new ClaimsIdentityCollection(new IClaimsIdentity[] {
      new ClaimsIdentity(
        new Claim(System.IdentityModel.Claims.ClaimTypes.Name,
         userNameToken.UserName), "CustomUserNameSecurityTokenHandler")});
  }

  public override bool CanValidateToken
  {
    get { return true; }
  }
}
At a minimum, ValidateToken and CanValidateToken must be overridden in the custom SecurityTokenHandler implementation. Inside ValidateToken you are responsible for authenticating against the appropriate credential store. The results of that authentication should be a set of claims that can be returned to the runtime to be attached to the ClaimsPrincipal for the request thread.
Figure 10 A Simple Federation Scenario with a Single RP and a Passive IP-STS
It is also very important to override CanValidateToken and return True. Without this override, the token handler will not be registered in the collection and it will never be invoked.

Building a Custom Passive STS
In a simple passive federation scenario, the same participants are present, as shown in Figure 3, for an active federation scenario—except that the client is a browser, the RP is a Web application, and the IP-STS is also fronted by a Web application to handle HTTP-based communication. Figure 10 illustrates the participants and flow of communication for passive federation.
The moving parts behind this scenario are similar to those shown in Figure 4, as far as the core STS is concerned, but there are some obvious differences in how the passive STS handles authentication and in how the underlying STS functionality is invoked. The diagram in Figure 10 illustrates these differences from a high level. The passive STS is implemented as a Web site that requires SSL encryption to secure the token issuance process. The default page (Default.aspx) hosts a control that facilitates communication with the underlying custom STS, which is configured just like an active STS.
The STS site must authenticate the caller prior to token issuance, and this is where classic ASP.NET configuration comes into play for authentication and authorization. In Figure 11, the STS application is configured for Forms authentication, so requests are redirected to a login page (Login.aspx) if they have not yet been authenticated by the FormsAuthenticationModule.
A passive STS can share the same core STS implementation as an active STS—with one minor change. In the GetScope override (shown in Figure 5) a passive STS must set the ReplyToAddress property so that the STS can redirect after issuing the token. Usually this is set to the default page for the RP, based on the AppliesTo address supplied with the RST:
Scope scope = new Scope(request);
scope.ReplyToAddress = scope.AppliesToAddress + "/default.aspx";
// other scope settings
Figure 11 Implementation Architecture for a Passive STS Using Forms Authentication
The Geneva Framework configuration for a passive STS is no different from an active STS. A SecurityTokenServiceConfiguration type is used to initialize the STS (shown in Figure 7), and any relevant settings in the <microsoft.identityModel> configuration section are also considered.

FederatedPassiveTokenService Control
The Geneva Framework provides a control that implements the required functionality of a passive STS. That is, it processes sign-in and sign-out HTTP requests, converting each request into an RST and then invoking the underlying STS implementation. The control also processes RSTR responses and handles redirection to the RP, and writing the session cookie for authenticated callers.
Place this control on the default page for the passive STS site and set its Service property to the custom SecurityTokenServiceConfiguration implementation as follows:
<idfx:FederatedPassiveTokenService ID="FederatedPassiveTokenService1"
  runat="server" Service="STS.IdentityProviderSTSConfiguration, STS">
</idfx:FederatedPassiveTokenService>
The control requires that the user is authenticated and checks this in its PreRender event. It is expected that the STS site is configured appropriately to ensure that users are redirected elsewhere to authenticate prior to reaching this default page.
As long as authenticated users are always directed to this default page, no other configuration is required to process requests. The control also provides the Error, PreSignInRequested, PostSignInRequested, PreSignOutRequested, and PostSignOutRequested events to handle exceptions, and to hook sign-in and sign-out requests.

SessionAuthenticationModule
As an alternative to the FederatedPassiveTokenService control, you can programmatically enable passive STS functionality. First enable federated authentication in the <microsoft.identityModel> configuration section:
<microsoft.identityModel>
  <federatedAuthentication enabled="true"/>
</microsoft.identityModel>
Next, enable the federation module for a passive STS, the SessionAuthenticationModule from the Microsoft.IdentityModel.Web namespace:
<modules>
  <add name="SessionAuthentication"
    type="Microsoft.IdentityModel.Web.SessionAuthenticationModule,
    Microsoft.IdentityModel, Version=0.5.1.0,
    Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
</modules>
This produces the same result as the FederatedPassiveTokenService control for requests sent to any page in the STS Web site. The module redirects unauthenticated callers to the login page. Upon successful login, callers are redirected to the STS page originally requested. This programmatic approach gives developers additional control beyond that provided by the FederatedPassiveTokenService control. For example, the module exposes the following events to interact with initialization, security token management, sign in, and sign out: ConfigurationLoading, ConfigurationLoaded, SecurityTokenReceived, SecurityTokenValidated, SessionSecurityTokenCreated, SessionSecurityTokenReceived, SignedIn, SigningOut, SignedOut, SignInError, and SignOutError.

User Authentication
The STS site is responsible for authenticating users according to supported credential types. While an active STS implemented with WCF can easily configure multiple endpoints to support different authentication mechanisms, a passive STS can only support one authentication mechanism by nature of the ASP.NET Web site configuration. Thus, a different passive STS site must be provided for each supported authentication mechanism. These sites can all share the same core STS implementation, of course.
Passive STS authentication is based on ASP.NET configuration techniques. Typical choices are Windows authentication, Forms authentication, and Windows CardSpace authentication. In the case of Windows authentication, the following configuration is used:
<authentication mode="Windows"/>
<authorization>
  <deny users="?"/>
</authorization>
The user will be authenticated through interactive dialogs prior to reaching the default page of the STS, so a custom login page is not required in this case.
Figure 11 illustrates an example whereby Forms authentication is used. The FormsAuthenticationModule redirects unauthenticated calls to the login page where the user can supply credentials. Once authenticated, the login page will redirect to the default STS page where the federation control will proceed to process the original request. The ASP.NET configuration for this would be as follows:
<authentication mode="Forms"/>
<authorization>
  <deny users="?"/>
</authorization>
In the case of Windows CardSpace authentication, the STS site can be configured for Forms authentication, but the login page will include an InformationCard control for Windows CardSpace authentication.

Claims Transformation
Claims transformation is essential to federated security—and this can happen at a variety of points in the flow of federation. In a simple federation scenario where the IP-STS and RP belong to the same domain, the IP-STS is responsible for transforming claims from the initial set of identity claims that the user presents during authentication, to those that the RP can rely on to authorize calls. Users can present any of the supported credential types to authenticate to the IP-STS, each evaluating to a set of claims representative of the credential.
The IP-STS transforms these claims into a normalized set of application claims that the RP relies on to authorize calls—which can be roles or more granular claims such as create, read, update, or delete rights. The diagram in Figure 12 illustrates such a scenario where the Admin user logs in and is granted a Role claim and several Action claims including Create, Read, Update, and Delete.
Figure 12 Claims Transformation at the IP-STS
This type of claims transformation is useful in that authentication to an STS results in a token carrying all of the claims granted to the user. There are cases where an alternate approach to claims transformation would be necessary; for example, to reduce the claims issued to those relevant only for the current call context; to protect the privacy of claims; or to facilitate federation across domains.
It isn't always desirable or appropriate to grant an authenticated caller a long list of claims related to all features exposed by the RP. Not only could the list be very long, but it is also possible that the claims to be granted depend on the call context and thus should not be issued without that context. For example, a user may only be granted the Delete claim if she is interacting with customer orders, but if she is interacting directly with customer records, she may not be granted that right.
In cases such as this one, it can be useful for the RP to request only a few claims from the IP-STS in order to identify the caller, and then request a new token with a specific set of additional claims only for the context of the call. For example, if the user is invoking the DeleteCustomer operation on an RP service, prior to authorizing access to the operation the RP calls to the RP-STS passing in the token from the IP-STS, requesting the Delete claim in the context of the DeleteCustomer operation. If the claim is present, the call is authorized. The diagram in Figure 13 illustrates this example.
There are also cases where the claims issued by an STS should not be shared with the RP directly. For example, rather than issuing an Age claim so that the RP knows the user's age, the RP can request the IsOver13 claim to ensure that the caller is old enough to use RP functionality. Thus, the actual value for the Age claim never leaves the STS. Of course, this implies that the ST supplies claims that avoid sharing personal details and yet still include data useful to the RP.
Figure 13 Claims Transformation at the RP-STS
Claims transformations also take place in a federated scenario where users belonging to one domain are granted access to an RP in another domain. In this case there are two STS involved—the user domain's IP-STS and the RP-STS for the domain that owns the RP.
And also in this case, IP-STS will grant some agreed-upon claims that RP-STS can understand; however, those claims are not likely to be directly useful in the RP application. Instead, the RP-STS would be responsible for transforming another trusted set of claims into claims that are understood in the RPs domain.
Figure 14 illustrates this scenario. When Joe tries to access the RP without a token, he will log in at the IP-STS in his own domain (Domain B). The request will ask for claims that the RP understands, in this case RPClaim, so that the IP-STS knows to issue a token that the RP can use. When the RP-STS receives this token it transforms the claims into RP-specific claims. For this federated scenario to work, there must be trust between the RP-STS and the IP-STS, and they must agree on a set of claims that the IP-STS should issue for its users that are to be granted access to the RP.
Figure 14 Claims Transformation in a Federated Scenario

Wrap-Up
The Geneva Framework is a very welcome utility for those interested in building a custom STS and who do not need a fully featured STS platform such as Geneva Server. Building a custom STS is not a trivial task, even with the Geneva Framework, and it is always recommended that you use a fully featured STS if possible to reduce exposure.
Regardless of the platform, the flow of communication for active and passive STS implementations remains the same, as do the ideas behind claims transformation. Some additional concepts related to STS implementations include identity delegation and step-up authentication. You can access samples and documentation related to these and other concepts in the Geneva Framework SDK.

Michele Leroux Bustamante is Chief Architect of IDesign Inc., Microsoft Regional Director for San Diego, and a Microsoft MVP for Connected Systems. Her latest book is Learning WCF. Reach her at mlb@idesign.net or visit idesign.net. Michele blogs at dasblonde.net.


Silverlight
Build Line-Of-Business Enterprise Apps With Silverlight, Part 1
Hanu Kommalapati

This article discusses:
  • The Silverlight runtime environment
  • Silverlight asynchronous programming
  • Cross-domain policies
  • A sample enterprise application
This article uses the following technologies:
Silverlight 2
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Silverlight Fundamentals: the CoreCLR
Silverlight Runtime
The Application Scenario
Push Notifications with Socket Server
Asynchronous I/O Loops
Modal Dialog Boxes in Silverlight
Push Notification Implementation
Cross-Domain Access of TCP Services
Cross-Domain Policies with TCP Services
Wrapping Up
When I presented an executive briefing on the business implications of Silverlight to a large company in Houston recently, the response was lukewarm. The cool demos that showcased DeepZoom, Picture-In-Picture and HD quality video, and high-quality animations should have easily excited the group. When I polled the audience members about their limited interest, it became clear that while dazzling graphics were great, there was very little real-world guidance available for building enterprise-quality, data-centric, line-of-business (LOB) applications with Silverlight.
Today, enterprise-class applications require secure delivery of LOB information across network boundaries, often over the Internet, with role-based UI and data trimming applied for the business context. Running Silverlight on the client and the Microsoft .NET Framework 3.5 on the server supplies excellent capabilities for building such scalable and secure LOB applications. The lightweight Silverlight runtime running inside a sandbox provides framework libraries for integration with back-office data services. In order to build robust applications with Silverlight, architects and developers need to understand the Silverlight programming model and its framework features in the context of a real-world application.
My main purpose in this article is to take an LOB scenario and build an application from the ground up, illustrating various aspects of Silverlight development along the way. The solution I'll discuss is a call center application; its logical architecture is shown in Figure 1. In this installment, I will focus on screen pop notification, the asynchronous programming model, Silverlight dialog boxes, and cross-domain TCP policy server implementation. In Part 2, I will discuss application security, Web service integration, application partitioning, and a number of other aspects of the application.
Figure 1 Silverlight Call Center Logical Architecture

Silverlight Fundamentals: the CoreCLR
Before I begin, let's have a refresher of Silverlight fundamentals. First I'll take a look inside the Silverlight runtime so you can get a better understanding of what's possible with Silverlight. The CoreCLR is the virtual machine used by Silverlight. It's similar to the CLR that powers the .NET Framework 2.0 and beyond and contains similar type-loading and garbage collection (GC) systems.
The CoreCLR has a very simple code access security (CAS) model—simpler than in the desktop CLR because Silverlight only needs to enforce security policies at the application level. This is because, as a platform-independent Web client, it can't rely on any particular enterprise or machine policies being in effect, and it should not allow a user to change existing policies. There are a few exceptions, such as OpenFileDialog and IsolatedStorage (storage quota change), where Silverlight needs the user's explicit consent for breaking the sandbox default rule set. OpenFileDialog is used for accessing the file system, while IsolatedStorage is used to access the namesake-isolated storage and increase the storage quota.
For desktop applications, each executable loads exactly one copy of the CLR, and the OS process will only contain one application. Each application has a system domain, a shared domain, a default domain, and a multitude of explicitly created AppDomains (see "JIT and Run: Drill into .NET Framework Internals to See How the CLR Creates Runtime Objects"). A similar domain model is found in the CoreCLR. In the case of Silverlight, several applications, possibly from different domains, will be running in the same OS process.
In Internet Explorer 8.0, each tab runs in its own isolated process; thus all the Silverlight applications hosted inside the same tab will be running in the context of the same CoreCLR instance, as illustrated in Figure 2. As each application can be from a different domain of origin, for security reasons, each application will be loaded into its own AppDomain. There will be as many instances of CoreCLR as there are tabs currently hosting Silverlight applications.
Similar to the desktop CLR, each AppDomain will get its own pool of static variables. Each domain-specific pool will be initialized during the AppDomain bootstrapping process.
Figure 2 Each Silverlight App Will Run in Its Own AppDomain
Silverlight applications can't create their own custom application domains; that ability is reserved for internal use. For a more detailed discussion of the CoreCLR, refer to the following CLR Inside Out columns from the CLR team: "Program Silverlight with CoreCLR" and "Security in Silverlight 2."

Silverlight Runtime
Silverlight is designed for a broad range of applications that require varying degrees of framework and library support. A simple application may, for example, just play audio files of a few bytes to help with the pronunciation of words on a dictionary Web site or display a banner ad. On the other hand, enterprise class LOB applications require security, privacy of data, state management, integration with other applications and services, and instrumentation support, to name a few. At the same time, Silverlight needs to have a smaller runtime so that deployment over the Internet will not be a problem on slower connections.
These requirements seem to conflict, but the Silverlight team tackled this by partitioning the framework into the layered view shown in Figure 2. CoreCLR + Silverlight runtime is called the "plug-in," which all users will install before they can run applications. The plug-in is sufficient for most consumer-centric applications. If an application requires the use of an SDK library (WCF integration or DLR runtimes such as Iron Ruby) or a custom library, the application must package these components into the XAP package so that Silverlight will know how to resolve the necessary types at run time (see Cutting Edge in this issue for more on XAPs).
The Silverlight runtime is approximately 4MB in size and, in addition to CoreCLR libraries such as agcore.dll and coreclr.dll, contains the necessary libraries required by app developers. These include the following fundamental libraries: mscorlib.dll, System.dll, System.Net.dll, System.Xml.dll, and System.Runtime.Serialization.dll. The runtime that supports the browser plug-in typically is installed under the C:\Program Files\Microsoft Silverlight\2.0.30930.0\ directory. This is the directory that gets created when a computer downloads and installs Silverlight as a part of the Web browsing session.
Developers who build and test applications on the same machine will have two copies of the runtime: one copy installed by the plug-in and the other through the SDK install. The latter copy can be located in the directory C:\Program Files\Microsoft SDKs\Silverlight\v2.0\Reference Assemblies. This copy will be used by Visual Studio templates as a part of the compile time references list.
The sandbox prevents the Silverlight application from interacting with most of the local resources, which is true for any typical Web application. By default, a Silverlight application can't access a file system (other than isolated storage), make socket connections, interact with devices attached to the computer, or install software components. This certainly imposes some constraints on the types of applications one could construct on the Silverlight platform. However, Silverlight has all the necessary ingredients for developing an enterprise-class, data-driven LOB application that needs to integrate with the back-end business processes and services.

The Application Scenario
The LOB application I'll be building here demonstrates a third-party call-control architecture where a centralized server taps into a private branch exchange (PBX) infrastructure to centrally control phones. Since my purpose is to focus on Silverlight as the UI surface, I will not be spending too much time on the telephony integration. Instead, I will use a simple call simulator to generate an incoming call event. The simulator will drop a packet of data that represents the call into a waiting queue of the Call Manager, which triggers the process that is central to this project.
My fictitious scenario requires the call center application to run inside a Web browser in a platform-independent manner while at the same time providing rich user interaction as a desktop application. Silverlight was the natural choice, as ActiveX is not very popular on non-Windows client environments.
Let's look at the app's architectural aspects. You'll be implementing push notifications, event integration, business service integration, caching, security, and integration with the cloud services.
Push Notifications These are required because the system needs to capture the incoming call event and transfer interactive voice response (IVR) data that was entered by the caller to do a "screen pop," or populating of the UI screen with the incoming call information. In addition, the user should be given an opportunity to accept or reject the call.
Event Streaming In a typical Web app, the Web server has all the knowledge of the business events as it performs the bulk of the business processes. In the case of a rich Internet application (RIA), however, the business process implementation will be shared by both the application running inside the Web browser and the server that implements business Web services. This means that the business events as well as technology events generated within the Silverlight application need to be sent to the server through a set of special Web services.
Examples of business events in this solution case are when the user (rep) rejects the call ("rep rejected the call") or accepts the call ("rep accepted the call"). Typical technology events are "Connection to Call Manager TCP server failed" and "Web service exception."
Business Service Integration The call center solution, just like any LOB application, needs to be integrated with data that may be stored in a relational database. I will use Web services as a vehicle for this integration.
Caching For a better user experience, I will cache the information locally in memory as well as on the disk. The cached information may include the XML files that indicate the rep's prompter scripts and other reference data that may not change often.
Security Application Security is a fundamental requirement of this kind of application. The security includes authentication, authorization, privacy of data in flight and at rest, and user profile-based data trimming.
Integration with the Cloud Services Integration with a cloud-based foundation service, such as storage service, requires special server-side infrastructures. This is so that the use of cloud services can be closely monitored and throttled for accountability and service levels.
I will cover integration with business services, application security, cross-domain policies for Web services, and application partitioning in Part 2 of this article.

Push Notifications with Socket Server
Screen pop is one of the fundamental requirements of a call center application for transferring call context from the telephony infrastructure to an agent's screen. The transferred call context may include any information spoken (for IVR systems) or keyed in by the phone-in customer.
The notification can be sent to the Silverlight application within the browser in one of two ways: through client polling or server push. Polling is fairly easy to implement, but it may not be the optimal choice for call center scenarios where state synchronization between the telephony events and the client application has to be precise. It is for that reason that I will use push notifications using Silverlight sockets.
One of the important capabilities of Silverlight is communication with TCP sockets. For security reasons, Silverlight only allows the connections to the server ports in the range of 4502 to 4532. This is one of the many security policies implemented in the sandbox. Another important sandbox policy is that Silverlight can't be a listener and, hence, cannot accept inbound socket connections. For these reasons, I am going to create a socket server listening on port 4530 and maintain a pool of connections with each connection representing an active call center rep.
The Silverlight socket runtime also enforces cross-domain opt-in policies on the server for all of the socket connections. When Silverlight application code attempts to open a connection to an IP endpoint on an allowed port number, opaque to the user code, the runtime will make a connection to an IP endpoint on the same IP address with the port number 943. This port number is hardwired to the Silverlight implementation and can't be configured by applications or changed by the application developer.
Figure 1 shows where the policy server fits into the architecture. When Socket.ConnectAsync is called, the message flow sequence is as shown in Figure 3. By design, messages 2, 3, and 4 are totally opaque to the user code.
Figure 3 Silverlight Runtime Requests Cross-Domain Policy Automatically for Socket Connections
I need to implement a policy server on the same IP address as the call manager server. I could implement both the servers in a single OS process; however, for simplicity's sake I will implement them in two separate console programs. These console programs can easily be converted to Windows services and made cluster-aware for failover to provide reliability and availability.

Asynchronous I/O Loops
The .NET Framework 3.5 introduced new async programming API to sockets; these are the methods that end with Async(). The methods I will be using on the server are Socket.AcceptAsync, Socket.SendAsync, and Socket.ReceiveAsync. Async methods are optimized for high throughput server applications through the use of I/O Completion Ports and effective receive and send buffer management through the reusable SocketAsyncEventArgs class.
Because of the fact that Silverlight is not allowed to create TCP listeners, its Socket class only supports ConnectAsync, SendAsync, and ReceiveAsync. Silverlight only supports an asynchronous programming model, and that's not just for socket API but for any network interaction.
Since I will be using an async programming model on the server as well as on the client, let's get comfortable with the design patterns. One recurring design pattern is the I/O loop, which is applicable to all the Async operations. First, I will look at the typical synchronous execution of the socket accept loop, as seen here:
_listener.Bind(localEndPoint);
 _listener.Listen(50);
 while (true)
 {
    Socket acceptedSocket = _listener.Accept();
    RepConnection repCon = new
      RepConnection(acceptedSocket);
    Thread receiveThread = new Thread(ReceiveLoop);
    receiveThread.Start(repCon);
 }
The sync accept is intuitive and easy to program and maintain; however, this implementation is not really scalable for servers, as there are dedicated threads for each client connection. This implementation can easily peak at a few connections if the connections are very chatty.
For Silverlight to work well with the browser runtime environment, it has to intrude on the resources as little as possible. All of the calls in the "socket accept" pseudo code shown previously block the threads on which they execute, and thus have a negative impact on scalability. For that reason, Silverlight is very restrictive on blocking calls and, in fact, allows only asynchronous interaction with network resources. Asynchronous loops require adjusting your mental model to envision an invisible message box that should have at least one request inside it all the time for the loop to work.
Figure 4 shows a receive loop (a more complete implementation is in the code download). There are no infinite loop programming constructs like the while (true) loop you saw in the synchronous socket accept pseudo code earlier. Getting used to this kind of programming is vital for a Silverlight developer. In order for the receive loop to continue receiving data, after a message has been received and processed, there should be at least one request in the queue against the I/O Completion Port associated with the connected socket. A typical async loop is illustrated in Figure 5 and is applicable to ConnectAsync, ReceiveAsync, and SendAsync. AcceptAsync can be added to this list on the server where you will use the .NET Framework 3.5.
Figure 4 Async Send/Receive Loops with Silverlight Sockets
public class CallNetworkClient
{
   private Socket _socket;
   private ReceiveBuffer _receiveBuffer;

   public event EventHandler<EventArgs> OnConnectError;
   public event EventHandler<ReceiveArgs> OnReceive;
   public SocketAsyncEventArgs _receiveArgs;
   public SocketAsyncEventArgs _sendArgs;
//removed for space
    public void ReceiveAsync()
    {
       ReceiveAsync(_receiveArgs);
    }

    private void ReceiveAsync(SocketAsyncEventArgs recvArgs)
    {
       if (!_socket.ReceiveAsync(recvArgs))
       {
          ReceiveCallback(_socket, recvArgs);
       }
    }

    void ReceiveCallback(object sender, SocketAsyncEventArgs e)
    {
      if (e.SocketError != SocketError.Success)
      {
        return;
      }
      _receiveBuffer.Offset += e.BytesTransferred;
      if (_receiveBuffer.IsMessagePresent())
      {
        if (OnReceive != null)
        {
           NetworkMessage msg =
                       NetworkMessage.Deserialize(_receiveBuffer.Buffer);
           _receiveBuffer.AdjustBuffer();
           OnReceive(this, new ReceiveArgs(msg));
        }
      }
      else
      {
        //adjust the buffer pointer
        e.SetBuffer(_receiveBuffer.Offset, _receiveBuffer.Remaining);
      }
      //queue an async read request
      ReceiveAsync(_receiveSocketArgs);
    }
    public void SendAsync(NetworkMessage msg) { ... }

    private void SendAsync(SocketAsyncEventArgs sendSocketArgs)
    {
    ...
    }

     void SendCallback(object sender, SocketAsyncEventArgs e)
    {
    ...
    }
   }
Figure 5 Asynchronous Socket Loop Pattern
In the receive loop implementation shown in Figure 4, ReceiveAsync is a wrapper to the reentrant method ReceiveAsync(SocketAsyncEventArgs recvArgs) that will queue the request on the socket's I/O completion port. SocketAsyncEventArgs, introduced in the .NET Framework 3.5, has a similar role in the Silverlight socket implementation and can be reused between multiple requests, avoiding the garbage collection churn. It will be the responsibility of the callback routine to extract the message, trigger a message-processing event, and enqueue the next receive item for continuing the loop.
To handle cases of partial message reception, ReceiveCallback adjusts the buffer before queuing another request. NetworkMessage is wrapped in an instance of ReceiveArgs and passed to the external event handler for processing the received message.
The buffer is reset for every complete receipt of the NetworkMessage after copying the partial message, if any, to the beginning of the buffer. A similar design is used on the server, but real implementations can benefit from circular buffers.
In order to implement the "Call Acceptance" scenario shown in the sequence diagram (see Figure 6), you need to create an extensible message architecture that allows you to serialize and deserialize messages that contain arbitrary content without necessitating the rewrite of the serialization logic for every new message.
Figure 6 Layout of the Serialized NetworkMessage Types
The message architecture is rather simple: each child object of the NetworkMessage declares its signature at the instantiation time with appropriate MessageAction. NetworkMessage.Serialize and Deserialize implementations will work on Silverlight and the .NET Framework 3.5 (on the server) because of the source code-level compatibility. The serialized message will have the layout shown in Figure 6.
Instead of inserting the length at the beginning of the message, you could use the "begin" and "end" markers with appropriate escape sequences. Encoding length into the message is a lot simpler for processing the buffers.
The first four bytes of every serialized message will include the number of bytes of the serialized object following the 4 bytes. Silverlight supports XmlSerializer located inside System.Xml.dll that is part of the Silverlight SDK. The serialization code is included in the code download. You'll note that it does not have any direct dependencies on the child classes, such as RegisterMessage or other messages including UnregisterMessage and AcceptMessage. A series of XmlInclude annotations will help the serializer to properly resolve the .NET types while serializing the child classes.
The use of NetworkMessage.Serlialize and Deserialize is shown in ReceiveCallback and SendAsync in Figure 4. In the receive loop, the actual message processing is done by the event handler attached to the NetworkClient.OnReceive event. I could have processed the message inside CallNetworkConnection, but wiring the receive handler to process the message will help the extensibility by decoupling the handler from CallNetworkConnection at design time.
Figure 7 shows the Silverlight application RootVisual, which starts the CallNetworkClient (shown in Figure 4). All Silverlight controls are attached to a single UI thread, and any UI updates can only be done when the code executes in the context of that UI thread. The asynchronous programming model of Silverlight executes the network access code and the processing handlers on the thread pool's worker threads. All of the FrameworkElement derived classes (such as Control, Border, Panel, and most of the UI elements) inherit the Dispatcher property (from DispatcherObject), which can execute code on the UI thread.
In Figure 7, the case MessageType.RegisterResponse will update the UI with the agent's call center shift details through an anonymous delegate. The updated UI resulting from the delegate execution is shown in Figure 8.
Figure 7 Silverlight UserControl that Processes the Incoming Messages
public partial class Page : UserControl
{
  public Page()
  {
    InitializeComponent();
    ClientGlobals.socketClient = new CallNetworkClient();
    ClientGlobals.socketClient.OnReceive += new
                         EventHandler<ReceiveArgs>(ReceiveCallback);
    ClientGlobals.socketClient.Connect(4530);
    //code omitted for brevity
  }
  void ReceiveCallback(object sender, ReceiveArgs e)
  {
    NetworkMessage msg = e.Result;
    ProcessMessage(msg);
  }
  void ProcessMessage(NetworkMessage msg)
  {
    switch(msg.GetMessageType())
    {
      case MessageAction.RegisterResponse:
           RegisterResponse respMsg = msg as RegisterResponse;
           //the if is unncessary as the code always executes in the
           //background thread
           this.Dispatcher.BeginInvoke(
              delegate()
              {
                 ClientGlobals.networkPopup.CloseDialog();
                 this.registrationView.Visibility = Visibility.Collapsed;
                 this.callView.Visibility = Visibility.Visible;
                 this.borderWaitView.Visibility = Visibility.Visible;
                 this.tbRepDisplayName.Text = this.txRepName.Text;
                 this.tbRepDisplayNumber.Text = respMsg.RepNumber;
                 this.tbCallServerName.Text =
                                      respMsg.CallManagerServerName;
                 this.tbCallStartTime.Text =
                                respMsg.RegistrationTimestamp.ToString();
              });
            break;
      case MessageAction.Call:
           CallMessage callMsg = msg as CallMessage;
       //Code omitted for brevity
           if (!this.Dispatcher.CheckAccess())
           {
              this.Dispatcher.BeginInvoke(
                 delegate()
                 {
                    ClientGlobals.notifyCallPopup.ShowDialog(true);
                 });
           }
           break;
           //
           //Code omitted for brevity
           //
      default:
             break;
    }
  }
}
Figure 8 Registration with the Call Center Server Is in Progress
Figure 9 Rep’s Initial Registration Screen

Modal Dialog Boxes in Silverlight
When a call center rep logs in, he will be asked to start the shift by registering with the call center server. The registration process on the server will save the session indexed by the rep number. This session will be used for subsequent screen pop and other notifications. The screen transition of the call center app for the registration process is shown in Figures 8 and 9. I will use a modal dialog box that shows the progress of the network send. Typical enterprise LOB apps use pop-up dialog boxes, modal and non-modal, quite liberally. As there is no built-in DialogBox in the Silverlight SDK, you will see how to develop one in Silverlight for use in this app.
Until Silverlight, there was no simple way to build a modal dialog, as there was no easy way to prevent keyboard events from being pumped into the UI. Mouse interaction can be disabled indirectly by setting UserControl.IsTestVisible = false. Starting from RC0 setting Control.IsEnabled = false prevents UI controls from receiving any keyboard or mouse events. I will use System.Windows.Controls.Primitives.Popup to display the dialog UI on top of the existing control.
Figure 10 shows a base SLDialogBox control with the abstract methods GetControlTree, WireHandlers, and WireUI. These methods will be overridden by the child classes, as shown in Figure 11. Primitives.Popup requires a control instance that is not part of the control tree to which the Popup will be attached. In the code in Figure 10, the ShowDialog(true) method will disable the entire control tree recursively so that none of the contained controls will receive any mouse or keyboard events. Since my pop-up dialog must be interactive, the Popup.Child should be set from a new control instance. The GetControlTree implementation in child classes will act as a control factory and provide a new instance of a user control appropriate for the UI requirements of the dialog.
Figure 10 Popup DialogBox in Silverlight
using System;
using System.Windows;
using System.Windows.Controls;
using System.Windows.Controls.Primitives;

namespace SilverlightPopups
{
    public abstract class SLDialogBox
    {
        protected Popup _popup = new Popup();
        Control _parent = null;
        protected string _ caption = string.Empty;
        public abstract UIElement GetControlTree();
        public abstract void WireHandlers();
        public abstract void WireUI();

        public SLDialogBox(Control parent, string caption)
        {
            _parent = parent;
            _ caption = caption;
            _popup.Child = GetControlTree();
            WireUI();
            WireHandlers();
            AdjustPostion();

        }
        public void ShowDialog(bool isModal)
        {
            if (_popup.IsOpen)
                return;
            _popup.IsOpen = true;
            ((UserControl)_parent).IsEnabled = false;
        }
        public void CloseDialog()
        {
            if (!_popup.IsOpen)
                return;
            _popup.IsOpen = false;
            ((UserControl)_parent).IsEnabled = true;
        }
        private void AdjustPostion()
        {
            UserControl parentUC = _parent as UserControl;
            if (parentUC == null) return;

            FrameworkElement popupElement = _popup.Child as FrameworkElement;
            if (popupElement == null) return;

            Double left = (parentUC.Width - popupElement.Width) / 2;
            Double top = (parentUC.Height - popupElement.Height) / 2;
            _popup.Margin = new Thickness(left, top, left, top);
        }
    }
}
Figure 11 NotifyCallPopup.xaml Skin
//XAML Skin for the pop up
<UserControl
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  Width="200" Height="95">
   <Grid x:Name="gridNetworkProgress" Background="White">
     <Border BorderThickness="5" BorderBrush="Black">
       <StackPanel Background="LightGray">
          <StackPanel>
             <TextBlock x:Name="tbCaption" HorizontalAlignment="Center"
                        Margin="5" Text="&lt;Empty Message&gt;" />
             <ProgressBar x:Name="progNetwork" Margin="5" Height="15"
                        IsIndeterminate="True"/>
          </StackPanel>
          <StackPanel Orientation="Horizontal" HorizontalAlignment="Center" >
             <Button x:Name="btAccept"  Margin="10,10,10,10"
                      Content="Accept" HorizontalAlignment="Center"/>
             <Button x:Name="btReject"  Margin="10,10,10,10"
                     Content="Reject" HorizontalAlignment="Center"/>
          </StackPanel>
       </StackPanel>
      </Border>
    </Grid>
</UserControl>
GetControlTree can be implemented to instantiate a Silverlight UserControl that is compiled into the application package, or a control can be created from an eXtensible Application Markup Language (XAML) file using XamlReader.LoadControl. Typically, dialog boxes can easily be implemented in skins to which the compiled handlers can be attached at run time. Figure 11 shows a XAML skin that has btAccept and btReject buttons. The LoadControl method will throw an exception if you leave the class attribute (<userControl class="AdvCallCenter.NotifyCallPopup"…>…</UserControl>) in the XAML after your Microsoft Expression Studio or Visual Studio design task. Any UI event handler attributes must be removed for successful parsing using LoadControl.
To create skins, you can add the Silverlight UserControl to the project, design it in Expression, and remove the "class" attribute and event handler names, if any, attached to the controls from the XAML file. The click handlers can be part of the child pop-up class, as shown in Figure 12, or alternatively a separate handler library can be created that can be wired to the controls using reflection.
Figure 12 NotifyCallPopup Implementation
public class NotifyCallPopup : SLDialogBox
{
   public event EventHandler<EventArgs> OnAccept;
   public event EventHandler<EventArgs> OnReject;
   public NotifyCallPopup(Control parent, string msg)
        : base(parent, msg)
   {
   }
   public override UIElement GetControlTree()
   {
      Return SLPackageUtility.GetUIElementFromXaml("NotifyCallPopup.txt");
   }
   public override void WireUI()
   {
      FrameworkElement fe = (FrameworkElement)_popup.Child;
      TextBlock btCaption = fe.FindName("tbCaption") as TextBlock;
      if (btCaption != null)
          btCaption.Text = _caption;
      }
   public override void WireHandlers()
   {
      FrameworkElement fe = (FrameworkElement)_popup.Child;
      Button btAccept = (Button)fe.FindName("btAccept");
      btAccept.Click += new RoutedEventHandler(btAccept_Click);

      Button btReject = (Button)fe.FindName("btReject");
      btReject.Click += new RoutedEventHandler(btReject_Click);
   }

   void btAccept_Click(object sender, RoutedEventArgs e)
   {
      CloseDialog();
      if (OnAccept != null)
          OnAccept(this, null);
   }
   void btReject_Click(object sender, RoutedEventArgs e)
   {
      CloseDialog();
      if (OnReject != null)
         OnReject(this, null);
   }
}
The handlers can be in any Silverlight library project, as they will automatically be compiled into the XAP package as a result of the project dependency. In order for the skin files to be included in the XAP package, add them to the Silverlight project as XML files and change the extension to XAML. The default build action for files with a XAML extension is to compile them into the application DLL. Since I want these files to be packaged as text files, the following attributes from the Properties windows should be set:
  • BuildAction = "Content"
  • Copy to Output Directory = "Do Not Copy"
  • Custom Tool = <clear any existing value>
The XAML parser (XamlReader.Load) does not care about the extension; however, using the .xaml extension will be more intuitive and representative of the content. SLDialogBox is only responsible for showing and closing the dialog. The child implementations will be customized to suit the needs of the application.

Push Notification Implementation
A call center application must be able to do a screen pop with the caller information. A call center workday starts with the rep registering with the call center server. Push notifications are implemented with connection-oriented sockets. The entire call manager server implementation is not shown in the figures but is available in the code download. When the Silverlight client makes a socket connection on the server, a new RepConnection object gets added to the RepList. The RepList is a generic list indexed by a unique Rep Number. When a call comes in, you will use this list to find an available rep and, using the socket connection associated with the RepConnection, notify the agent with the call information. RepConnection uses ReceiveBuffer, as shown in Figure 13.
Figure 13 RepConnection Uses ReceiveBuffer
class SocketBuffer
{
  public const int BUFFERSIZE = 5120;
  protected byte[] _buffer = new byte[BUFFERSIZE]
  protected int _offset = 0;
  public byte[] Buffer
  {
    get { return _buffer; }
    set { _buffer = value; }
  }

 //offset will always indicate the length of the buffer that is filled
  public int Offset
  {
    get {return _offset ;}
    set { _offset = value; }
  }

  public int Remaining
  {
    get { return _buffer.Length - _offset; }
  }
}
class ReceiveBuffer : SocketBuffer
{
  //removes a serialized message from the buffer, copies the partial message
  //to the beginning and adjusts the offset
  public void AdjustBuffer()
  {
    int messageSize = BitConverter.ToInt32(_buffer, 0);
    int lengthToCopy = _offset - NetworkMessage.LENGTH_BYTES - messageSize;
    Array.Copy(_buffer, _offset, _buffer, 0, lengthToCopy);
    offset = lengthToCopy;
  }
  //this method checks if a complete message is received
  public bool IsMessageReceived()
  {
    if (_offset < 4)
       return false;
    int sizeToRecieve = BitConverter.ToInt32(_buffer, 0);
    //check if we have a complete NetworkMessage
    if((_offset - 4) < sizeToRecieve)
      return false; //we have not received the complete message yet
    //we received the complete message and may be more
      return true;
   }
 }
You will use a Silverlight call simulator to drop a call into the CallDispatcher._callQueue to trigger the screen pop process. The CallDispatcher is not shown in any of the figures, but it is available in the downloadable code. It attaches a handler to _callQueue.OnCallReceived and will get notified when the simulator enqueues the message to the _callQueue inside the ProcessMessage implementation. By taking advantage of the pop-up dialogs I discussed previously, the client will display the Accept/Reject notification, as shown in Figure 14. Here's the line of code that is responsible for displaying the actual notification dialog from Figure 8:
ClientGlobals.notifyCallPopup.ShowDialog(true);
Figure 14 Incoming Call Notification

Cross-Domain Access of TCP Services
Unlike the media and the ad display applications, real enterprise-class LOB applications require integration with a variety of service-hosting environments. For instance, the call center application is hosted on a Web site (advcallclientweb hosted at localhost:1041), uses a stateful socket server on another domain (localhost:4230) for screen pop, and reaches LOB data through services hosted on a different domain (localhost:1043). It will use yet another domain for transmitting instrumentation data.
The Silverlight sandbox doesn't, by default, allow network access to any domain other than the domain of origin—advcallclientweb (localhost:1041). When such network access is detected, the Silverlight runtime checks for the opt-in policies established by the destination domain. Here is a typical list of the service-hosting scenarios that need to support cross-domain policy requests by the client:
  • Services hosted in the cloud
  • Web services hosted in a service process
  • Web services hosted in IIS or other Web servers
  • HTTP resources such as XAML markup and XAP packages
  • TCP Services hosted in a service process
While it is straightforward to implement cross-domain policies for HTTP resources and Web service endpoints hosted in IIS, the other cases require the knowledge of the policy request/response semantics. In this section I will briefly implement the policy infrastructure necessary for the TCP screen pop server, which is referred to as Call Manager in Figure 1. The other cross-domain scenarios will be discussed in Part 2 of this article.

Cross-Domain Policies with TCP Services
Any TCP service access in Silverlight is considered a cross-domain request, and the server needs to implement a TCP listener on the same IP address that is bound to port 943. The policy server shown in Figure 3 is the listener implemented for this purpose. This server implements a request/response process for streaming out declarative policies needed by the Silverlight runtime before allowing the networking stack on the client to connect to the screen pop server (Call Manager in Figure 3).
For simplicity's sake, I will host the call manager server in a console application. This console application can easily be converted to a Windows service for real implementations. Figure 3 shows the typical interaction with a policy server; the Silverlight runtime will connect to the server on port 943 and send a policy request that will contain a single line of text: "<policy-file-request/>".
The XML-based policies enable the scenario shown in Figure 3. The socket resource section can specify a group of ports within the allowed range of 4502 to 4534. The rationale for restricting to a range is to minimize the attack vector, thereby mitigating the risk of accidental weaknesses in the firewall configuration. Since the call center server (Call Manager in Figure 1) listens on port number 4530, the socket resource is configured like so:
<access-policy>
   <policy>
     <allow-from> list of URIs</allow-from>
     <grant-to> <socket-resource port="4530" protocol="tcp"/></grant-to>
  </policy>
</access-policy>
You can also configure <socket-resource> to allow all the permitted port numbers by specifying port="4502–4534".
To save time, I'll repurpose the code from the call manager server in implementing the policy server. A Silverlight client connects to the policy server, submits a request, and reads the response. Policy server closes the connection once the policy response is sent successfully. The policy contents are read by the policy server from a local file, clientaccesspolicy.xml, which is included in the download.
The TCP listener implementation for policy server is shown in Figure 15. This uses the same async looping pattern discussed earlier for TCP Accept. Clientaccesspolicy.xml is read into a buffer and reused for sending to all the Silverlight clients. The ClientConnection encapsulates the accepted socket and the receive buffer that will be associated with the SocketAsyncEventArgs.
Figure 15 TCP Policy Server Implementation
class TcpPolicyServer
{
  private Socket _listener;
  private byte[] _policyBuffer;
  public static readonly string PolicyFileName = "clientaccesspolicy.xml";
  SocketAsyncEventArgs _socketAcceptArgs = new SocketAsyncEventArgs();
  public TcpPolicyServer()
  {
    //read the policy file into the buffer
    FileStream fs = new FileStream(PolicyServer.PolicyFileName,
                        FileMode.Open);
    _policyBuffer = new byte[fs.Length];
    fs.Read(_policyBuffer, 0, _policyBuffer.Length);
    _socketAcceptArgs.Completed += new
                 EventHandler<SocketAsyncEventArgs>(AcceptAsyncCallback);

  }
  public void Start(int port)
  {

    IPHostEntry ipHostInfo = Dns.GetHostEntry(Dns.GetHostName());
    //Should be within the port range of 4502-4532
    IPEndPoint ipEndPoint = new IPEndPoint(IPAddress.Any, port);

    _listener = new Socket(AddressFamily.InterNetwork, SocketType.Stream,
                                                       ProtocolType.Tcp);

    // Bind the socket to the local endpoint and listen for incoming connections
    try
    {
      _listener.Bind(ipEndPoint);
      _listener.Listen(50);
      AcceptAsync();
    }
    //code omitted for brevity

   }
   void AcceptAsync()
   {
      AcceptAsync(socketAcceptArgs);
    }

    void AcceptAsync(SocketAsyncEventArgs socketAcceptArgs)
    {
      if (!_listener.AcceptAsync(socketAcceptArgs))
      {
         AcceptAsyncCallback(socketAcceptArgs.AcceptSocket,
                                             socketAcceptArgs);
      }
    }

    void AcceptAsyncCallback(object sender, SocketAsyncEventArgs e)
    {
      if (e.SocketError == SocketError.Success)
      {
        ClientConnection con = new ClientConnection(e.AcceptSocket,
                                               this._policyBuffer);
        con.ReceiveAsync();
      }
      //the following is necessary for the reuse of _socketAccpetArgs
      e.AcceptSocket = null;
      //schedule a new accept request
      AcceptAsync();
     }
   }
The code sample shown in Figure 15 reuses SocketAsyncEventArgs between multiple TCP accepts. For this to work, e.AcceptSocket should be set to null in AcceptAsyncCallback. This approach will prevent GC churn on the server with high scalability requirements.

Wrapping Up
Implementation of the push notifications is an important aspect of a call center application and is an enabler of the screen pop process. Silverlight makes the screen pop implementation a lot easier than the AJAX or similar frameworks. Since the server programming and client programming models are similar, I was able to attain some reusability at the source-code level. In the case of the call center, I could use message definitions and receive buffer abstractions in both the server and client-side implementations.
I will implement on-premise Web service integration, security, cloud-service integration, and application partitioning in Part 2 of this series. I hope this effort resonates well with some of the LOB scenarios you come across, and I look forward to your feedback.
I would like to thank Dave Murray and Shane DeSeranno of Microsoft for offering guidance on the internals of Silverlight socket implementation, along with Robert Brooks, a domain expert on call centers for the screen pop discussion.

Hanu Kommalapati is a Platform Strategy Advisor at Microsoft, and in his current role he advises enterprise customers in building scalable line-of-business applications on Silverlight and Azure Services platforms.


Windows Mobile
Use GPS And Web Maps For Location-Aware Apps
Christopher Mitchell

Code download available from the MSDN Code Gallery
Browse the Code Online
This article discusses:
  • MapPoint Web services
  • Caching tasks and maps
  • Getting nearby points of interest
  • Creating new tasks
This article uses the following technologies:
Windows Mobile 6, MapPoint
Contents
Using MapPoint for Location
Tasks and Pocket Outlook
Application Architecture
Getting Nearby Points
Adding Tasks
Moving on down the Road
I recently moved to a new house, and some kind friends spent a full day helping me pack and unpack boxes, arrange for utility hookups, and all of the other important but tedious tasks of moving. On my way back from retrieving the last box at the old house, I nearly forgot to pick up dinner for my friends. I had set a reminder on my Windows Mobile phone, but it couldn't tell me when I was driving right past the most convenient take-out restaurant.
The solution I needed was a phone that informed me when I was near a place where I could complete said task. A time and location-appropriate reminder would have had us enjoying take-out rather than driving around an unfamiliar neighborhood after a long day of shifting boxes.
Windows Mobile provides many interfaces and capabilities that can help keep the device informed about its environment—where it is, whether it has a signal, how much of a signal, and so on. But how can these features be used in your applications? The most prominent and arguably useful of these capabilities is location awareness. This should enable you to create location-aware applications, ranging from the rather obvious satellite navigation program to the much more sophisticated task list that will be discussed here.
This article will look at the issues involved with using this capability and the additional code you need to write to develop useful applications. In this article, I will discuss the mobile application development environment and utilities, and show you how to build a location-aware task list application that can remind you of a task when and where it's appropriate.

Using MapPoint for Location
My location-aware task list application, named wheretodo, needs to perform several core tasks. It needs to get information about where the phone is located geographically. It also needs to store and monitor tasks. Furthermore, the application needs to know what shops and services are available near the phone's current location that could be used to solve a current task. Finally, it needs to provide alerts to the phone. The interface of the application is shown in Figure 1.
Figure 1 Wheretodo App
The first thing the app requires is geographical data. For the purpose of this article, I chose to use the Microsoft MapPoint Web service. This Web service is the underlying technology for Live Search Maps and Virtual Earth and provides a search nearby service for Europe and the United States. The example code provided uses European map data settings; you will need to change this to make it work elsewhere.
MapPoint takes a GPS latitude and longitude and search codes for specific types of stores as its arguments. The position information can be provided by the FakeGPS utility when installed on an emulator or smartphone. (See the "FakeGPS" sidebar for more information about this utility.)
MapPoint provides an XML Web service with a SOAP API. The Web service is broken down into four main services: a common service, a find service, a render service, and a route service. The main service of interest to this application is the find service, but if you wanted to give directions to the user or a map, the other services could be used to extend the functionality.
The common service (CommonServiceSoap) contains classes, methods, and properties that are common to the find, route, and render services, or are basic utility functions.
You can use the find service (FindServiceSoap) to locate addresses, geographic entities, latitude and longitude coordinates, and points of interest (POI) from MapPoint Web service data. You can also parse addresses and return location information for a specified latitude and longitude.
You can use the render service (RenderServiceSoap) to draw maps of routes and locations, place pushpins, draw polygon regions, set the map size and map view, select points on a map, get location information about points and polygons on a map, and pan and zoom a rendered map.
The route service (RouteServiceSoap) generates routes, driving directions, and calculated route representations (used to render a highlighted route on the map) based on locations or waypoints, sets segment and route preferences, and generates map views of segments and directions.
A full set of object model class diagrams is on MSDN. MapPoint holds the data used for the find, route, and render services in a number of different data sources depending on geographic region or type of information required. You will be able to find a full range of MapPoint technical articles on using the service on MSDN.

Tasks and Pocket Outlook
The Pocket Outlook Object Model (POOM) allows you to add menus and functionality to the Tasks and Contacts applications in Windows Mobile and to manipulate their associated items and data. There are three main interfaces of interest to a location-aware application: IAppointment, ITask, and IContact.
IAppointment represents an appointment in the Calendar folder. An appointment object can represent a meeting, a one-time appointment, or a recurring appointment or meeting.
ITask represents an assigned, delegated, or self-assigned task to be performed within a specified time frame. Task items are contained within the Tasks folder.
IContact represents a contact in the Contacts folder. Its methods can be used to save, delete, duplicate, or display a contact. The IPOutlookItemCollection interface can be used to add new contacts or to retrieve existing ones.
The sample application will be using ITask. (Functionally, it could use IAppointment, but that interface is not as well suited to the immediate needs of the application.) The POOM is similar to the desktop Outlook object model, and you can learn more about it on MSDN in the "Differences Between the Pocket Outlook Object Model and the Outlook Object Model" article.

Application Architecture
The location-aware application is made up of two chains of functions. The first is the system that monitors the current tasks and issues, as shown in Figure 2. Tasks are stored in a SQL Server Compact database table. In addition, a set of auxiliary data is stored within another table called geocache. Using the separate table is a means of limiting the amount of connectivity required, as I will explain later in this article.
Figure 2 Monitoring and Alerting of Location-Aware Tasks
The information for the points nearby is provided by the Microsoft MapPoint Web service. There are different data sources in the MapPoint Web service for different parts of the world, and some have more capabilities than others. The data source used by the sample application is NavTech.EU. Once this information is gathered, some distance calculations can be performed and alerts issued, where applicable.
The second chain of events (see Figure 3) adds tasks to the Tasks list on your Windows Mobile device via the POOM. This framework also allows access to the calendar and SMS functions.
Figure 3 System Flow for Adding Tasks to the Wheretodo Application
Tasks and Cache represents the first part of the monitoring and alerting chain of events. The task manager simply gets a list of tasks from the POOM. It also manages the geocache, which addresses the issues with using Web services on potentially limited connectivity devices. The level of connectivity of the device can range from permanent connectivity to sparsely connected or no connection at all.
To cope with the large range of connectivity strength, you need a decent cache policy for the Web service data and a straightforward system that knows how to make the most of the connectivity. Although very complex scenarios could easily be imagined, a simple radius cache policy was established for the purpose of this article. For example, a query is placed to the MapPoint Web service looking for all Chinese take-out restaurants within a given radius from the current location, say 80 kilometers (km). Once you have gone 40 km in any one direction, a new search is issued. This is based on the principal that every query to the Web service is costly.
It also makes the assumption that the distance you travel away from the point where you entered the task is not likely to be greater than 40 km. As the distance used to control the actions of the cache is likely to vary from person to person and country to country, it is designed to be input by the user.
Logs of where queries have been run are stored in the geocachelog database and checked before any additional queries are made so that duplicate information is not requested from the Web service. It also provides the opportunity for the system to be pre-populated with a considerable volume of information.

FakeGPS
FakeGPS Application
The Windows Mobile 6 SDK contains a utility application called FakeGPS that lets you test your application with simulated GPS data. For information about installing and testing using FakeGPS, see the MSDN Library article "Using the FakeGPS Utility."
FakeGPS reads a pre-recorded set of GPS instructions. The GPS data can be obtained from a real GPS device using a raw GPS data recorder. The resulting file is read by a FakeGPS device and can be re-run, simulating changing location data, for testing purposes. For example, I recorded a trip around London to gather data for my application and then re-enacted the trip through FakeGPS.
Getting Nearby Points
The next component defines how the geocache is populated. When the application needs to find a location, a GetNearByPOI function is called (see Figure 4). The code queries the MapPoint Web service. Note that the code shown here has been simplified for clarity and is different from that in the code download.
Figure 4 Getting Nearby Points of Interest
private void GetNearByPOI(
  string KeyWord, LatLong CurrentPosition) {

  FindServiceSoap findService = new FindServiceSoap();
  FindNearbySpecification findNearBySpec = new FindNearbySpecification();
  findService.Credentials =
    new System.Net.NetworkCredential(myUserName, myPassword);
  findService.PreAuthenticate = true;

  findNearBySpec.Distance = Convert.ToDouble(inputdistance.Text);
  findNearBySpec.LatLong = new LatLong();
  findNearBySpec.LatLong.Latitude = CurrentPosition.Latitude;
  findNearBySpec.LatLong.Longitude = CurrentPosition.Longitude;

  findNearBySpec.Filter = new FindFilter();
  //findNearBySpec.Filter.EntityTypeName = KeyWord;
  findNearBySpec.Filter.EntityTypeName = "FoodType3"; // SIC CODE
  findNearBySpec.DataSourceName = "NavTech.EU";

  FindResults foundResults;
  foundResults = findService.FindNearby(findNearBySpec);

  string connectionString;
  string fileName = System.IO.Path.GetDirectoryName(
    System.Reflection.Assembly.GetExecutingAssembly().GetName().CodeBase) +
    "\\wheretodo.sdf";
  string password = "sa";
  connectionString = string.Format("DataSource=\"{0}\"; Password='{1}'",
    fileName, password);
  SqlCeConnection cn = new SqlCeConnection(connectionString);
  cn.Open();
  SqlCeCommand cmd;

  //Loop Round and add it to the geocache
  foreach (FindResult fr in foundResults.Results) {
    //This needs to include LongLat in geocache
    string sql =
      "insert into geocache2 (POI_Name, POI_Address, POI_Tel, POI_Web, " +
      "POI_POST_ZIP, POI_Lat, POI_Long, POI_KeyWord) " +
      "values (@Name, @Address, @Tel, @Web, @POST_ZIP, '" +
      fr.FoundLocation.LatLong.Latitude.ToString() +
      "', '" + fr.FoundLocation.LatLong.Longitude.ToString() + "', '" +
      KeyWord.ToString() + "')";
    cmd = new SqlCeCommand(sql, cn);
    cmd.Parameters.Add("@Name", SqlDbType.NVarChar, 255, "Name").Value =
      fr.FoundLocation.Entity.Properties[0].Value.ToString();
    cmd.Parameters.Add("@Address", SqlDbType.NVarChar, 255, "Address").Value =
      fr.FoundLocation.Entity.Properties[1].Value.ToString() +
      fr.FoundLocation.Entity.Properties[2].Value.ToString();
    cmd.Parameters.Add("@Tel", SqlDbType.NVarChar, 255, "Tel").Value =
      fr.FoundLocation.Entity.Properties[9].Value.ToString();
    cmd.Parameters.Add("@Web", SqlDbType.NVarChar, 255, "Web").Value =
      "http://m.live.com";
    cmd.Parameters.Add("@POST_ZIP", SqlDbType.NVarChar, 255, "POST_ZIP").Value =
      fr.FoundLocation.Entity.Properties[7].Value.ToString();
    cmd.ExecuteNonQuery();

    //Update geocachelog
    sql = "insert into GeoCodeLog (Keyword, Lat, Long ) values ('" + KeyWord +
      "', '" + fr.FoundLocation.LatLong.Latitude.ToString() + "', '" +
      fr.FoundLocation.LatLong.Longitude.ToString() + "' )";
    cmd = new SqlCeCommand(sql, cn);
    cmd.ExecuteNonQuery();
  }
}
Take a look at the section in which CurrentPosition.Latitude and CurrentPosition.Longitude are called. This is ultimately derived from the GPS Intermediate Driver, a software layer that sits between applications and the device driver for GPS hardware. This layer of abstraction allows applications to be written once and work with multiple GPS devices. The GPS Intermediate Driver API is exposed through a native code library. You can gain access to this library from managed code by using the sample that is included with the Windows Mobile 6 Professional SDK (see "Using the GPS Intermediate Driver from Managed Code)".
Once you have the current location and nearby POI that are relevant to the task, identified here by standard industrial classification (SIC) codes used in the Navtech.EU data source, you need to determine exactly how far these points are from your current location. I used the longitude and latitude values returned from the Web service and the GPS (in this case, FakeGPS); Figure 5 shows how I converted these values to distance.
Figure 5 Computing the Distance
private double GetLatLongTuppleDistance(
  double Lat1, double Long1, double Lat2, double Long2) {

  //Convert Degress to Radians for Calculations
  double Lat1r = ConvertDegreesToRadians(Lat1);
  double Lat2r = ConvertDegreesToRadians(Lat2);
  double Long1r = ConvertDegreesToRadians(Long1);
  double Long2r = ConvertDegreesToRadians(Long2);

  // Spherical law of cosines formula—ignores the effect of hills
  double R = 6371; // Earth's radius (km)
  double d = Math.Acos(Math.Sin(Lat1r) * Math.Sin(Lat2r) +
    Math.Cos(Lat1r) * Math.Cos(Lat2r) *
    Math.Cos(Long2r—Long1r)) * R;
  //Returns distances in km
  return d;
}
A spherical trigonometry function based on the spherical law of cosines (which generalizes the Pythagorean theorem) provides the function that translates latitude and longitude pairs into km distances. This obviously makes some massive assumptions about the things blocking where you are from the location you're trying to find. As the earth is not a perfect sphere, there will be errors in using these formulas. For miles, divide km by 1.609344 and, in case you're nipping out in your boat to get some bread, for nautical miles divide km by 1.852.
The final part of this chain of events is to issue the alert to the user if a solution to the task is found within a certain distance from your current location. As this distance is determined by the mode of transport—driving, walking, cycling, and so on—this has been left as an input for the user.

Adding Tasks
Of course, before the application can find a location, the user needs to add tasks to the mobile device. The POOM mirrors the Outlook object model, but its scope of functionality is reduced to accommodate the practical constraints of mobile devices.
Using the POOM, it is straightforward to modify and display appointment, task, and contact items, as well as manipulate the folders that contain them. Here's the code to create a task item:
OutlookSession outlooksession = new OutlookSession();
Task NewTask = new Task();
NewTask.Body = textBox2.Text.ToString();

string MyString = dateTimePicker2.Value.ToShortDateString() + " " + dateTimePicker1.Text.ToString();
DateTime MyDateTime = new DateTime();
MyDateTime = DateTime.ParseExact(MyString, "M/d/yy h:mm:ss tt", null);
NewTask.DueDate = MyDateTime.ToUniversalTime();
NewTask.Subject = textBox1.Text.ToString();
outlooksession.Tasks.Items.Add(NewTask);
Various parameters can be altered to set the end date and the task body text and then add it to the POOM.
This component makes up the last of the two chains of components used to provide the functionality of the wheretodo application. The system UI itself is very simple and provides a means of setting all the relevant information and determining the SIC codes the application will be searching out. The overall application is simply wrapped in an infinite loop just to illustrate the use of location-aware applications. You could very easily add this as a background service to your mobile device to make it a more efficient and cleaner integration into the Windows Mobile 6 or Windows Mobile 5 environment.

Mobile Data Resources

Write Code Once for Both Mobile and Desktop Apps

Going Places: Adaptable Apps for Windows Mobile

Data Points: Accessing Data from a Mobile Application
Moving on down the Road
You've seen one type of location-aware device and a tool that eases the development of these applications. This basic concept of a location-aware task list could be easily extended for RFID.
There are improvements that can be made to this code that apply to any location-aware application. The biggest collection of these can be summarized by how the location awareness ties in with the other context-aware services that run on the majority of mobile phones. The tasks do not have an idea what else you are trying to do, only where you are. You might target tasks to both a time and a location so that they are triggered by you getting near something at a particular time (say, driving home from work). Even better, the task reminder would know that you have a meeting in 10 minutes, and it would also know that is not enough time to get stamps, even though you are passing the post office on the way to the meeting.
If you are interested in building a custom installer for this or your own newly updated location-aware application, see my article from the October 2007 issue of MSDN Magazine (Adjust Your Ring Volume For Ambient Noise).
I would like to dedicate this article to Tom Passey, a dear friend who, along with his wife, has looked after me on many occasions, providing great memories. Wish you all the best.

Christopher Mitchell finished his PhD in machine learning and music/sound signal processing, was a Kauffman/NCGE fellow, and is currently undertaking a start-up in Cambridge UK. He is a part-time lecturer at Anglia Ruskin University, Cambridge, UK. You can contact Chris at chris.mitchell@anglia.ac.uk.


VSTS 2010
Agile Planning Tools In Visual Studio Team System 2010
Ajoy Krishnamoorthy

This article is based on a prerelease version of Visual Studio Team System (VSTS) 2010. All information is subject to change.

This article discusses:
  • Product and iteration planning
  • Product backlog workbook
  • Capacity planning and reports
  • Iteration backlog workbook
This article uses the following technologies:
VSTS 2010, VSTS Process for Agile Software Development 1.0
Contents
Long-Range Planning
Release and Iteration Planning
VSTS 2010 Excel Workbooks
Product Backlog Workbook
Capacity Planning
Iteration Backlog Workbook
Reports
Is "agile planning" an oxymoron? I hope you don't think so, but in a recent focus group that took place in Los Angeles, one of the attendees indicated that her organization had moved from being agile to adopting more formal practices. When quizzed further, she said her team can no longer make a code fix based on a verbal request from her manager and then immediately deploy it to production. Now she has to follow a formal procedure. To her, this meant moving away from being agile.
However, her understanding of agile development is incorrect, and I am glad her organization instituted a formal process for change. Agile is not about flying blind or being nimble for the sake of speed. Rather, it is a disciplined approach to planning and one that factors in empirical data.
Visual Studio Team System (VSTS) 2010 introduces new features and capabilities to help agile teams with planning. In this article I will introduce you to the new product backlog and iteration backlog workbooks and a set of new reports that will help agile teams plan and manage releases and iterations.

Long-Range Planning
The concern of not having a long-range plan with precision is a common one and the one that is raised as a key barrier to adoption. In the 2008 State of Agile Development Survey, lack of advanced planning was the number-one concern in regard to adopting agile practices in respondents' organizations. For many, I suspect lack of long-range, precision planning is equated to lack of planning altogether. Agile teams opt for multiple levels of planning with course corrections over a waterfall-style plan, and rightly so.
Steve McConnell, discussing his Software Estimation Cone of Uncertainty, suggests that estimates done early in the project could be inaccurate by as much as 400% with the error leaning toward the high side: "Early in a project, specific details of the nature of the software to be built, details of specific requirements, details of the solution, project plan, staffing, and other project variables are unclear. The variability in these factors contributes variability to project estimates."
Of course, none of this means the executive management will go with a strategy of "we don't know when the project will be done and what will it look like when it is done." What it means instead is a change in the approach to how teams go about planning releases and the scope of the work completed in the releases.

Figure 1 Product and Iteration Backlog
Release and Iteration Planning
For agile teams, planning is done at two distinct levels: release planning and iteration planning. Release planning is a high-level planning activity that helps agile teams review the broad set of features or user stories. The items in the backlog are then stack ranked, estimated, and assigned to a set of iterations. Note that at this stage, the estimation is done using a unit such as a T-shirt sizing (small, medium, large). The intent at this point is to get a rough order magnitude of cost for the various items on the backlog and not a precise quote. This also helps the customer stack rank the requirements with the approximate sizing in mind.
In Scrum, this set of user stories is kept in a list called product backlog (see Figure 1). The scope of work to be tackled in each iteration depends largely on team velocity. What defines a release largely depends on when you have a credible set of requirements done according to the customers. For instance, if it takes four iterations to get the first set of features, then the first release will be scheduled after the fourth iteration.
Iteration planning is a much more detailed planning activity and the one takes place prior to the start of every iteration. High-level user stories from the product backlog are reviewed and are broken down in to smaller user stories as appropriate. At this point, the team is ready to break down user stories into smaller stories and define the tasks that are required to complete a user story. These user stories and associated tasks are then estimated in hours. At this point the team can understand the scope of an iteration.
Microsoft Office Excel is a tool you will often find in an agile team's toolbox along with index cards and sticky notes. VSTS 2010 introduces two new Excel workbooks to help agile teams manage the product backlog and iteration backlog. But let's take a quick look at the new Agile process template with VSTS 2010 before jumping right into the workbooks.
A process template in VSTS includes work item types, queries, reports, and textual guidance. Work items are the key entities here. Work items could be a user story, a task, a bug, and so on. To get started, you set up a team project in Team Foundation Server (TFS) and choose the VSTS Process for Agile Software Development v1.0 template in the New Team Project Wizard. This template includes the following Work Item Types:
  • Task
  • User Story
  • Bug
  • Issues
  • Test Case
You can create your own work item type or customize the particular work item itself. Learn more about work item customization in Brian Randell's December 2008 article on using and customizing TFS process templates, "Team System: Streamline Team Projects with Process Templates."
Now let's delve into the Excel workbooks and look at how these work items flow through the development process and aid in planning and managing flow of value.

VSTS 2010 Excel Workbooks
In the paper titled "Tools for Agility," Kent Beck talks about transitions galore in agile teams and the need for tools to take transitions into account. The integration of Excel-based planning workbooks with TFS work item tracking aids in minimizing the transition overhead. From keeping the product backlog and iteration backlog synchronized to automatically capturing the update to the status of a user story or a task work item into the iteration backlog to generating the wealth of reports, many of the mundane activities are either eliminated or optimized.
Here is a suggested flow that an agile team might use to manage the product backlog and iteration backlog with VSTS 2010:
  • Create a new team project using the VSTS 2010 Agile template.
  • Build the product backlog by adding user story to the product backlog worksheet or by adding work items from within Visual Studio.
  • Start assigning the items based on their stack rank in the product backlog to an iteration. By default, Iterations 0, 1, and 2 are created. You can create additional iterations using the team project settings.
  • Set up queries to pull the user stories, tasks, and other items from particular iterations and map them to corresponding iteration backlog worksheets.
The integration between these worksheets and TFS is accomplished using the queries. Figure 2 shows the configuration for the product backlog worksheet. In the Excel ribbon, on the Team tab, in the Work Items group, click on Configure List. This opens the Configure List Properties dialog. In this dialog, you can choose a TFS query and the result of this query is what you see in the spreadsheet.
Figure 2 Query in Excel Worksheet
The queries are created in the team project. By default, when you set up a team project, a folder called Work Items\Team Queries\Workbook Queries is created. Under this folder, you will find the default queries for product backlog and iteration backlog workbooks.
To get a better idea of how the workbooks operate, let's take a look at the DinnerNow sample application included in the October 2008 VSTS 2010 and .NET Framework 4.0 CTP. (You can find the latest CTP download on the Team Suite Developer Center.) Both the product backlog and iteration backlog workbooks can be found in the \DinnerNow\Documents\Shared Documents folder in Team Explorer.

Product Backlog Workbook
Product backlog primarily serves as a list of requirements desired by customers in an application. I have heard terms such as epics or themes used by teams when referring to the set of high level requirements. Collecting this set of requirements in a list, prioritizing them, and estimating them at a high level helps answer two important questions at this stage of the planning:
  1. 1.What are the requirements of the application?
  2. 2.How much it is going to cost? Obviously the answer to this is based on estimates. I have seen teams use story points, T-shirt sizes, or hours to estimate at this stage.
In answering these questions, the teams can get a good handle on what the release or next few releases might look like and when those releases can be scheduled. There is usually a budget or schedule constraint, such as an upcoming ad campaign, a legal requirement, or seasonal activity. This helps to plan the scope of the release as you can manage the scope of the release based on the constraint.
If there is a target date set for a release, the scope of work is managed by deciding which requirements to include in the iterations within the release time frame. For instance, if planning begins in December and the release date is set in June, you are essentially looking at running four to five iterations (assuming one-month iterations) to complete the work.
If there is flexibility with the target date, the release schedule will depend on when a minimum set of requirements could be completed. For instance, if a minimum set of required features could be completed in three iterations, then Release 1 could be set after three iterations. If the next set of features could be completed in five or six iterations, then Release 2 would be set to go after those five or six iterations.
Figure 3 shows the product backlog worksheet for the DinnerNow project. What you are looking at is a backlog of user stories. Several of these user stories are already assigned to a particular iteration and some of them are already completed in Iteration 0 and Iteration 1. Obviously, when you are beginning a new project, you will start with a blank workbook and start building these high-level user stories.
Figure 3 Product Backlog Worksheet
The columns on these spreadsheets are the fields in the work item which, in turn, get stored in the TFS data store. The integration between Excel and TFS adds a Team ribbon in Excel (see Figure 4) that has menu items to publish the items in the backlog into TFS, refresh the backlog with updated work items in TFS, and more.
Figure 4 Team Tab in Excel Ribbon
Each row in the backlog is stored as a work item within TFS as shown in Figure 5. With this type of integration, team members who are using Visual Studio can now update the user story and other work items from within Visual Studio itself. Updating the status of a user story, the estimate, or the remaining work, can now be done without having to switch between different tools.
Figure 5 Work Item in TFS

Capacity Planning
As part of the release planning, agile teams will spend a good deal of time in the spreadsheet adding new user stories, estimating these user stories, and, more importantly, prioritizing them. But it's equally important to keep an eye on the status of the release. The product backlog workbook includes a worksheet for capacity planning. This worksheet helps you get a quick handle on the iterations themselves, based on the estimates of user stories and the iteration in which their work is assigned.
The capacity planning is an important activity in planning releases. This helps in understanding the features that can be completed in the various iterations. The key data point in this calculation is velocity. Velocity is the amount of work being completed by the team in an iteration. If you happen to have data from prior iterations, that is the best place to start.
Figure 6 Using Previous Iterations to Calculate Velocity
This is generally referred to as "prediction based on yesterday's weather." In fact, the capacity planning spreadsheet can pull the historical data from the TFS data warehouse if it is available. As seen in Figure 6, I can select Iteration 1 as the iteration to get historical data from and key in the data for start date, end date, and number of team members. In this case, I get a velocity of 816 hours, which means the team was able to complete 816 hours of work in Iteration 1. Shy of that data, teams can start with an estimate and use the velocity of the first iteration in planning the future iterations.
In the capacity planning spreadsheet, you can specify the date range for iterations, the number of team members, and any interruptions during the iteration, such as holidays. This data, combined with the user story estimates and velocity, creates a chart that gives a sense of what the iteration work load looks like. If you see estimated work exceed the expected capacity bar, then you'll want to move around user stories between iterations to get to a favorable allocation.
In my case, I do not have any work planned in Iteration 2. I can add a few of the remaining user stories in the backlog to Iteration 2. Now the capacity chart looks like Figure 7. This is a good situation to be in—the estimate work has not exceeded the capacity.
Figure 7 Capacity Chart with Work Assigned for Iteration 2
The product backlog worksheet can also be used to get a picture of the overall status of the various user stories once the project is in flight. But you can get a lot more information from reports such as Burndown and Velocity, Remaining Work, and Stories Progress. These reports are included in the Agile template and will be found in the Report folder under the team project. I will cover the reports later in this article.

Iteration Backlog Workbook
Iteration is a key activity for an agile teams. Agile teams practicing Scrum are familiar with this by the term "sprint." The duration of the iteration typically varies. Teams using eXtreme Programming follow a one- or two-week iteration, whereas teams using Scrum typically follow a four-week sprint.
Iteration planning helps define the scope of a particular iteration. During an iteration planning meeting, teams typically analyze the user stories assigned for the particular iteration, gather detailed requirements, add associated tasks, and estimate how long it would take to finish each task. During this meeting, product owners along with the rest of the team prioritize user stories based on several factors: dependencies, cost estimates, detailed requirements, and possible evidence that a particular story is not as important as it was deemed during the release planning.
Let's start by looking at the iteration backlog in the DinnerNow team project. There are folders named Iteration 0, 1, and 2 under the Shared Documents folder in the team project. Within each of these iteration folders you will find the iteration backlog. Each of the iteration backlog worksheets are connected to a specific query that only brings that particular iterations user stories and tasks.
If you have added other work item types such as features, themes, or epics, then you will add them to this query so you get those additional work items pulled in the list. The DinnerNow team project already has several tasks that are added as child items to the user stories in Iteration 2. But typically, as part of the iteration planning meeting, teams will be adding tasks and estimating them to get a good iteration plan for Iteration 2. Figure 8 shows the iteration backlog.
Figure 8 Iteration Backlog with Child Tasks
TFS now supports hierarchical work items and that allows you to create parent/child trees. In this case, the following new tasks are added as child tasks to the user story "Users should be able to use DinnerNow from their cell phones":
  • Identify which parts of the UI to use for the phone
  • Use a card stack architecture for the UI
  • Identify most popular phones
  • Reduce number of keystrokes needed to place an order
At this point, the team is ready to go with the task assignment. How much work each team member picks up depends on factors including team member capacity for that iteration, domain expertise, and how long the team member has been part of the team.
Iteration backlog workbook has additional sheets to help with other aspects of planning and execution. The capacity planning worksheet is similar to the one in the product backlog workbook. You can use this worksheet to get a sense of team's capacity.
The load balancing worksheet comes in handy during planning and during the iteration itself. Agile teams continue their planning throughout the iteration to do course correction as new information comes available about a particular user story, when a technical dependency is discovered on a task, or when a team member becomes unavailable. These occurrences call for updating the task assignments and that's where load balancing worksheet comes to play.
Another interesting worksheet is the one provided for velocity tracking. Those of you familiar with sport of cricket will recognize the terms current run rate and required run rate. These are two statistics used to get a good indication of how a team is doing in a match. In general, if the required run rate is higher than the current run rate, the team that is batting has to pick up the pace in order to avoid a defeat. On the other hand, if your current run rate is higher than required run rate, then the batting team is in good shape.
Before a cricket-savvy reader starts calling me on this, let me say that additional stats such as wickets down and remaining overs are important to get a complete handle on the status. And this is true in an agile project as well. The velocity tracking sheet gives you a quick sense of the current team velocity and the required velocity to complete the user stories in an iteration. Just as with cricket, other stats such as remaining days are important to get a complete status on where you are in the iteration. For instance, the team may have to cut scope if their current velocity is not in pace with required velocity. Again, the key is to be transparent about this to the customers and make the necessary adjustments as a team.

Reports
Of course, you don't have to become an avid fan of cricket to start managing projects with VSTS 2010. You can rely on reports to monitor a project's progress such as burn down and work remaining.
Figure 9 shows the burn down report for Iteration 1. A burn down report shows the hours of work completed, hours of work remaining, and the rate of progress. As you can see from the hours remaining in the chart, the team did not complete all the tasks allocated to this iteration.
Figure 9 Burn Down Chart
The trend lines show you this status as well. As you can see in the chart, the actual trend line indicates that the work won't be completed on time at this team's rate during Iteration 1. During the iteration, the team can watch the burn down and can make necessary course corrections and also let the customers know that the release plan might be affected.
The remaining work report is also quite useful. As you see in Figure 10, the team has been making steady progress in completing work. There is no additional work added to the backlog during this iteration which is a good sign. The team's progress tapered off towards the end of the iteration causing the tasks left unfinished.
Figure 10 Remaining Work
As you've seen, the agile planning tools and worksheets for VSTS 2010 provide the teams with an Excel front end and the integrated data store that captures and mines data from release planning to iteration planning and from test execution to build quality. This provides a great tool for both managers planning and managing agile projects as well as the developers and testers on the team—they can work together, assess progress, make changes as necessary, and manage the entire project. This integration eliminates many of the manual and monotonous activities that agile teams undertake to collect data and generate reports.
For a rather different, more mathematically based approach to advance planning, take a look at Dr. James McCaffrey's Test Run column in this issue.

Ajoy Krishnamoorthy is a Lead Product Planner with the Microsoft patterns and practices team. Prior to this, Ajoy was as a Senior Product Manager for Microsoft Visual Studio Team System. In this role, Ajoy was responsible for product management, strategy, and marketing. He has more than 10 years of consulting experience in a variety of roles including developer, architect, and technical project manager. You can check out his blog at blogs.msdn.com/ajoyk.


ASP.NET Workflow
Web Apps That Support Long-Running Operations
Michael Kennedy

Code download available from the MSDN Code Gallery
Browse the Code Online
This article discusses:
  • Process-independent workflows
  • Synchronous and asynchronous activities
  • Workflows, activities, and persistence
  • Integrating with ASP.NET
This article uses the following technologies:
Windows Workflow Foundation, ASP.NET
Contents
Harnessing Workflows
Synchronous and Asynchronous Activities
What Exactly Do You Mean by Idle?
Making Synchronous Tasks Asynchronous
Workflows and Activities
Persistence
Making It Real
Integrating with ASP.NET
A Few Things to Ponder
Bringing It All Together
Software developers are often asked to build Web applications that support long-running operations. One example is the checkout process of an online store, which might take several minutes to complete. While that is a long-running operation by some standards, in this article I will explore long-running operations of an entirely different scale: operations that can take days, weeks, or even months to complete. One example of such an operation is the application process for employment, which can involve interactions between several people and the exchange of many real documents.
First, let's consider a more benign problem from an ASP.NET standpoint: you need to architect a solution for a checkout operation at an online store. There are special considerations to make for this solution because of its duration. For example, you may choose to store the shopping cart data in an ASP.NET session. You may even choose to move that session state into an out-of-process state server or database to allow for updates to the site and load balancing. Even so, you'll find all the tools you need to solve this problem easily are provided for by ASP.NET itself.
However, when the duration of the operation grows longer than the typical ASP.NET session duration (20 minutes) or requires multiple actors (as in my hiring example), ASP.NET does not offer sufficient support. You may recall that the ASP.NET worker processes automatically shut down on idle and periodically recycle themselves. This will cause big problems for long-running operations, as state held within those processes will be lost.
Imagine for a moment that you were to host these very long running operations inside of a single process. Clearly the ASP.NET worker process is not suitable for them for the reasons just outlined. So maybe it would be possible to create a windows service whose sole responsibility will to be to execute these operations. If you never restart this service, you'll be closer to a solution than if using ASP.NET directly, since simply having a service process that does not automatically restart theoretically ensures that I will not lose the state of my long running operation.
But will this really solve the problem? Probably not. What if the server requires load balancing? That becomes very difficult when you're tied to a single process. Worse, what if you have to reboot the server or the process crashes? Then all of the operations that have been running will be lost.
In fact, when operations take days or weeks to complete, you need a solution that is independent of the lifecycle of the process that is executing it. This is true in general, but it is especially important for ASP.NET Web applications.

Harnessing Workflows
Windows Workflow Foundation (WF) may not be the technology that comes to mind for building Web applications. However, there are several key features provided by WF that make a workflow solution worth considering. WF gives you the ability to achieve process independence for long-running operations by unloading idle workflows entirely from the process space and automatically reloading them into the active process when they're no longer idle (see Figure 1). Using WF, you can rise above the nondeterministic lifecycle of the ASP.NET worker process and provide for long-running operations inside of Web application.
Figure 1 Workflows Preserve Operations across Process Instances
Two key features of WF combine to provide this capability. First, asynchronous activities signal to the workflow runtime that the workflow is idle while waiting on an external event. Second, a persistence service will unload idle workflows from the process, save it to a durable storage location such as a database, and reload the workflows when they are ready to run again.
This process independence has other benefits. It provides a simple means of load balancing as well as durability—fault tolerance in the face of process or server failures.

Synchronous and Asynchronous Activities
Activities are the atomic elements of WF. All workflows are built from activities in a pattern resembling the composite design pattern. In fact, workflows themselves are simply specialized activities. These activities can be classified as either synchronous or asynchronous. A synchronous activity executes all of its instructions from start to end.
An example of a synchronous activity might be one that computes tax given an order in an online store. Let's think about how such an activity might be implemented. Like most WF activities, the majority of work happens in the overridden Execute method. The steps of that method might go something like this:
  1. Obtain the order data from a previous activity. This is usually done through data binding, and you will see an example of this later.
  2. Look up the customer associated with the order from a database.
  3. Look up the tax rate in a database based on the customer's location.
  4. Do some simple math using the tax rate and the order items associated with order.
  5. Store the total tax in a property that subsequent activities can bind to in order to complete the checkout process.
  6. Signal to the workflow runtime that this activity is completed by returning the Completed status flag from the Execute methods.
It is important to recognize that nowhere are you waiting. You are always working. The Execute method simply runs through the steps and finishes in short order. This is the essence of a synchronous activity: all the work is done in the Execute method.
Asynchronous activities are different. Unlike their synchronous counterparts, asynchronous activities execute for some period of time and then wait on an external stimulus. While waiting, the activities become idle. When the event occurs, the activity resumes operation and finishes executing.
An example of an asynchronous activity is a step in the hiring process when an application for a position needs to be reviewed by a manager. Consider what might happen if this manager is on vacation and will not get a chance to review the application until next week. It is completely unreasonable to block in the middle of the Execute method while waiting for this response. When software waits on a person, it might have to wait for a long time. You need to consider this in your design.

What Exactly Do You Mean by Idle?
At this point English semantics and architectural semantics diverge. Let's take a step back from WF and think more generally about what it means to be idle.
Consider the following class that uses a Web services to change a password:
public class PasswordOperation : Operation {
  Status ChangePassword(Guid userId, string pw) {
    // Create a web service proxy:
    UserService svc = new UserService();

    // This can take up to 20 sec for
    // the web server to respond:
    bool result = svc.ChangePassword( userId, pw );

    Logger.AccountAction( "User {0} changed pw ({1}).",
      userId, result);
    return Status.Completed;
  }
}
Is the ChangePassword method ever idle? If so, where?
This method's thread is blocked waiting for an HTTP response from the UserService. So conceptually the answer is yes, the thread is idle waiting on the service response. But can the thread actually do other work while you're waiting on the service? No, not as it's currently utilized. Thus, from a WF perspective, this "workflow" is never idle.
Why is it never idle? Imagine that you had some larger scheduler class that was to run operations like ChangePassword efficiently. By efficiently, I mean running several operations in parallel, using the minimum number of threads required for full parallelism, and so on. It turns out that the key to this efficiency is to know when the operation is executing and when it is idle. Because when an operation becomes idle, the scheduler can use the thread that was running the operation to do other work until the operation is ready to run again.
Unfortunately, the ChangePassword method is completely opaque to the scheduler. Because although there is a period when it is effectively idle, viewed from the outside by the scheduler that method is a single unit of blocking work. The scheduler has no capability of splitting that unit of work apart and reusing the thread during the idle period.

Making Synchronous Tasks Asynchronous
You can add this needed scheduling transparency to the operation by splitting the operation into two pieces: one that executes up to point where the operation is potentially idle and one that executes the code after the idle state.
For the hypothetical example shown earlier, you could use the asynchronous capabilities provided by the Web service proxy itself. Keep in mind that this is a simplification and WF actually works a little bit differently, as you will see in a minute.
In Figure 2, I created an improved version of the password-changing method called ChangePasswordImproved. I create the Web service proxy as before. Then the method registers a callback method to be notified when the server has responded. Next, I asynchronously execute the service call and report back to the scheduler that the operation is idle, but not finished, by returning Status.Executing. This step is important—it is this step that allows other work to be accomplished by the scheduler while my code is idle. Finally, when the completed event occurs, I call the scheduler to signal that the operation is finished and that it can move on.
Figure 2 Simple Password-Changing Service Call
public class PasswordOperation : Operation {
  Status ChangePasswordImproved(Guid userId, string pw) {
    // Create a web service proxy:
    UserService svc = new UserService();

    svc.ChangePasswordComplete += svc_ChangeComplete;
    svc.ChangePasswordAsync( userId, pw );
    return Status.Executing;
  }

  void svc_ChangeComplete(object sender, PasswordArgs e) {
    Logger.AccountAction( "User {0} changed pw ({1}).",
      e.UserID, e.Result );

    Scheduler.SignalCompleted( this );
  }
}

Workflows and Activities
Now I am going to apply the concept of an operation being idle to building activities in WF. This is very similar to what you saw earlier, but now I must work within the WF model.
WF comes with many built-in activities. However, when you first get started building real systems with WF, you will quickly want to start building your own custom reusable activities. This is simple to do. You just define a class that derives from the ubiquitous Activity class. Here is a basic example:
class MyActivity : Activity {
  override ActivityExecutionStatus
    Execute(ActivityExecutionContext ctx) {

    // Do work here.
    return ActivityExecutionStatus.Closed;
  }
}
For your activity to do anything useful, you must override the Execute method. If you're building a short-lived synchronous activity, then you simply implement your activity's operation inside this method and return the status Closed.
There are a few bigger issues that real-world activities are likely to consider. How will your activity communicate with other activities within the workflow and the larger application hosting the workflow? How will it access services such as database systems, UI interaction, and so on? For building synchronous activities, these issues are relatively straightforward.
Building asynchronous activities, on the other hand, can be a more complex undertaking. Fortunately, the pattern employed is repeated across most asynchronous activities. In fact, you can easily capture this pattern in a base class, as I will demonstrate in a moment.
These are the basic steps required for building most asynchronous activities:
  1. Create a class that derives from Activity.
  2. Override the Execute method.
  3. Create a workflow queue that can be used to receive notification that the asynchronous event you have been waiting for has completed.
  4. Subscribe to the queue's QueueItemAvailable event.
  5. Initiate the start of the long running operation (for example, send an e-mail asking a manager to review an application for a job position).
  6. Wait for an external event to occur. This effectively signals that the activity has become idle. You indicate this to the workflow runtime by returning ExecutionActivityStatus.Executing.
  7. When the event occurs, the method handling the QueueItemAvailable event removes the item from the queue, converts it to the expected data type, and processes the results.
  8. Typically this concludes the activity's operation. The workflow runtime is then signaled by returning ActivityExecutionContext.CloseActivity.

Persistence
At the beginning of this article, I said the two fundamental things that needed to achieve process independence via workflow were asynchronous activities and a persistence service. You have just seen the asynchronous activities portion. Now let's dig into the technology behind persistence: workflow services.
Workflow services are a key extensibility point for WF. The WF runtime is a class that you instantiate in your application to host all the running workflows. This class has two opposing design goals that are simultaneously achieved through the concept of workflow services. The first goal is for this workflow runtime to be a lightweight object that can be used in many places. The second goal is for this runtime to provide powerful capabilities to the workflows while they're running. For example, it can provide the ability to automatically persist idle workflows, track workflow progress, and support other custom capabilities.
The workflow runtime remains lightweight by default because only a couple of these capabilities are built in. More heavyweight services such as persistence and tracking are optionally installed through the service model. In fact, the definition of a service is any global capability that you want to provide to your workflows. You install these services in the runtime by simply calling the AddService method on the WorkflowRuntime class:
void AddService(object service)
Because AddService takes a System.Object reference, you can add anything your workflow might need.
I am going to work with two services. First, I will use WorkflowQueuingService for accessing the workflow queues that are fundamental for building asynchronous activities. This service is installed by default and cannot be customized. The other service is SqlWorkflowPersistenceService. This service will, of course, provide the persistence capabilities and it is not installed by default. Luckily, it is included with WF. You just have to add it to the runtime.
With a name like SqlWorkflowPersistenceService, you can bet that a database will be required somewhere. You can create an empty database for this purpose or you can add some tables to an existing database. I personally prefer to use a dedicated database rather than mingle workflow persistence data with my other data. So I create an empty database in SQL Server called WF_Persist. I create the necessary database schema and stored procedures by running a couple of scripts. These are installed as part of the Microsoft .NET Framework and are by default located in this folder:

C:\Windows\Microsoft.NET\Framework\v3.0\Windows Workflow Foundation\SQL\EN\
You will want to run the SqlPersistenceService_Schema.sql script first and then run the SqlPersistenceService_Logic.sql script. Now I can use this database for persistence by passing the connection string to the persistence service:
SqlWorkflowPersistenceService sqlSvc =
    new SqlWorkflowPersistenceService(
  @"server=.;database=WF_Persist;trusted_connection=true",
  true, TimeSpan.FromSeconds(10), TimeSpan.FromSeconds(10));

wfRuntime.AddService(sqlSvc);
This simple AddService method call is all that is required to begin unloading idle workflows and storing them in the database, and then restoring them when they're needed again. The WF runtime takes care of everything else.

Making It Real
Now that you have enough of the technical underpinnings in place, you can weave them together to build an ASP.NET Web site that supports long-running operations. The three major elements that you will see in this example are the construction of asynchronous activities, integrating the workflow runtime into the Web application, and communicating with the workflow from the Web pages.
I am going to be working with a hypothetical .NET consulting company called Trey Research. They would like to automate the recruiting and hiring process for their consultants. So I will be building an ASP.NET Web site to support this hiring process. I am going to keep things very simple, but there are several steps to the process:
  1. A job candidate will visit the Trey Research Web site and express interest in a job.
  2. An e-mail will be sent to the manager noting that there is a new applicant.
  3. This manager will review the application and approve the applicant for a particular position.
  4. An e-mail will be sent the applicant with information about the proposed position.
  5. The applicant will visit the Web site and either accept or reject the position.
While this process is straightforward, there are several steps where the application is waiting for a person to return and fill out some more information. These idle points can take a long time. So they are perfect for demonstrating a long-running process.
This Web application is included in the source code for the article. However, to see the full effect you will need to create the persistence database and configure the sample to use it. I've created a switch to manage whether the persistence service is on or off and have set it to be off by default. To turn it on, set usePersistDB to true in the AppSettings section of web.config. If you want to see it running while you read, you can view the async work viewer running on my website.
Figure 3 The Hiring Process as a Workflow
I will begin by designing the workflow completely independent from ASP.NET. To build the workflow, I will create four custom activities. The first is a send e-mail activity and will be a simple synchronous activity. The other three will represent steps 1, 3, and 5 shown earlier and will be asynchronous activities. These activities are the key to the long running operation's success. I will call these GatherEmployeeInfoActivity, AssignJobActivity, and ConfirmJobActivity, respectively. I will then combine these activities into the admittedly bare bones workflow shown in Figure 3.
The send e-mail activity is straightforward so I won't go into the details of that activity in this article. It is a synchronous activity like the MyActivity class shown earlier. Take a look at the code download for details.
That leaves me with creating the three asynchronous activities. I will save myself a lot of work if I can encapsulate that eight-step process for building an asynchronous activity into a common base class. Toward that end, I will define a class called AsyncActivity (see Figure 4). Note that this listing does not include several internal helper methods or error handling that is present in the real code. Those details the have been left out for the sake of brevity.
Figure 4 AsyncActivity
public abstract class AsyncActivity : Activity {
  private string queueName;

  protected AsyncActivity(string queueName) {
    this.queueName = queueName;
  }

  protected WorkflowQueue GetQueue(
      ActivityExecutionContext ctx) {
    var svc = ctx.GetService<WorkflowQueuingService>();
    if (!svc.Exists(queueName))
      return svc.CreateWorkflowQueue(queueName, false);

    return svc.GetWorkflowQueue(queueName);
  }

  protected void SubscribeToItemAvailable(
      ActivityExecutionContext ctx) {
    GetQueue(ctx).QueueItemAvailable += queueItemAvailable;
  }

  private void queueItemAvailable(
      object sender, QueueEventArgs e) {
    ActivityExecutionContext ctx =
      (ActivityExecutionContext)sender;
    try { OnQueueItemAvailable(ctx); }
    finally { ctx.CloseActivity(); }
  }

  protected abstract void OnQueueItemAvailable(
    ActivityExecutionContext ctx);
}
In this base class, you'll see that I wrapped up several of the tedious and repetitive parts of building an asynchronous activity. Let's run through this class from top to bottom. Starting with the constructor, I pass in a string for the queue name. Workflow queues are the input points for the host application (the Web pages) to pass data into activities while remaining loosely coupled. These queues are referred to by name and workflow instance, so every asynchronous activity needs its own distinct queue name.
Next, I define the GetQueue method. As you can see, accessing and creating workflow queues is easy, but somewhat tedious. I created this method as a helper method for use inside this class and derived classes.
I then define a method called SubscribeToItemAvailable. This method encapsulates the details of subscribing to the event fired when an item arrives in the workflow queue. This almost always represents the completion of a long waiting period where the workflow has been idle. So the use case goes something like this:
  1. Begin long running operation and call SubscribeToItemAvailable.
  2. Tell the workflow runtime that the activity is idle.
  3. The workflow instance is serialized to the database by the persistence service.
  4. When the operation completes, an item is sent to the workflow queue.
  5. This triggers the workflow instance to be restored from the database.
  6. The abstract template method OnQueueItemAvailable is executed by the base AsyncActivity.
  7. The activity finishes its operation.
To see this AsyncActivity class in action, let's implement the AssignJobActivity class. The other two asynchronous activities are similar and are included in the code download.
In Figure 5, you can see how AssignJobActivity uses the template provided by the AsyncActivity base class. I override Execute to do any preliminary work to begin the long running activity, though in this case there really isn't any. Then I subscribe to the event for when more data is available.
Figure 5 AssignJobActivity
public partial class AssignJobActivity : AsyncActivity {
  public const string QUEUE NAME = "AssignJobQueue";

  public AssignJobActivity()
    : base(QUEUE_NAME)
  {
    InitializeComponent();
  }

  protected override ActivityExecutionStatus Execute(
      ActivityExecutionContext ctx) {
    // Runs before idle period:
    SubscribeToItemAvailable(ctx);
    return ActivityExecutionStatus.Executing;
  }

  protected override void OnQueueItemAvailable(
      ActivityExecutionContext ctx) {
    // Runs after idle period:
    Job job = (Job)GetQueue(ctx).Dequeue();

    // Assign job to employee, save in DB.
    Employee employee = Database.FindEmployee(this.WorkflowInstanceId);
    employee.Job = job.JobTitle;
    employee.Salary = job.Salary;
  }
}
There is an implicit contract here that the host application, the Web page, will send in a new Job object into the activity's queue when it has gathered that information from the manager. This will signal the activity that it is OK to continue. It will update the employee in the database. The next activity in the workflow will send an e-mail to the prospective employee informing him that this job is proposed for his position.

Integrating with ASP.NET
That's how it works inside the workflow. But how do you start the workflow? How does the Web page actually gather the job offer from the manager? How does it pass the Job to the activity?
First things first: let's look at how to start the workflow. On the landing page for the Web site there is an Apply Now link. When the applicant clicks this link, it starts both the workflow and the navigation through the user interface in parallel:
protected void LinkButtonJoin_Click(
    object sender, EventArgs e) {
  WorkflowInstance wfInst =
    Global.WorkflowRuntime.CreateWorkflow(typeof(MainWorkflow));

  wfInst.Start();
  Response.Redirect(
    "GatherEmployeeData.aspx?id=" + wfInst.InstanceId);
}
I simply call CreateWorkflow on the workflow runtime and start the workflow instance. After that I keep track of the workflow instance by passing the instance ID to all subsequent Web pages as a query parameter.
How do I send data from the Web page back into the workflow? Let's look at the assigned job page, in Figure 6, where a manager chooses a job for an applicant.
Figure 6 Assigning a Job
public class AssignJobPage : System.Web.UI.Page {
  /* Some details omitted */
  void ButtonSubmit_Click(object sender, EventArgs e) {
    Guid id = QueryStringData.GetWorkflowId();
    WorkflowInstance wfInst = Global.WorkflowRuntime.GetWorkflow(id);

    Job job = new Job();
    job.JobTitle = DropDownListJob.SelectedValue;
    job.Salary = Convert.ToDouble(TextBoxSalary.Text);

    wfInst.EnqueueItem(AssignJobActivity.QUEUE_NAME, job, null, null);

    buttonSubmit.Enabled = false;
    LabelMessage.Text = "Email sent to new recruit.";
  }
}
The assign job Web page is largely just a simple input form. It has a dropdown list of available jobs and a textbox for the proposed salary. It also displays the current applicant, although that code is omitted from the listing. When the manager assigns a position and salary to the applicant, he will click the submit button and run the code in Figure 6.
This page uses the workflow instance ID as a query string parameter to look up the associated workflow instance. Then a Job object is created and initialized with the values from the form. Finally, I send this information back into the activity by enqueuing the job into that activity's queue. This is the key step that reloads the idle workflow and allows it to continue executing. AssignJobActivity will associate this job with the previously collected employee and save them to a database.
These last two code listings emphasize how fundamental workflow queues are to the success of asynchronous activities and workflow communication with the outside host. It is also important to note that the use of workflow here has no impact on page flow. Though I could also use WF to control the page flow, it's simply not the focus of this article.
In Figure 6, you saw that I accessed the workflow runtime via the global application class, as follows:
WorkflowInstance wfInst =
  Global.WorkflowRuntime.GetWorkflow(id);
This brings me to the final point of integrating Windows Workflow into our Web application: all workflows execute inside the workflow runtime. Although you can have as many workflow runtimes as you like in your AppDomain, it most often makes sense to have a single workflow runtime. Because of this, and because the WF runtime object is thread-safe, I made it a public static property of the global application class. Additionally, I start the workflow runtime in the application start event and stop it in the application stop event. Figure 7 is an abbreviated version of the global application class.
Figure 7 Starting the Workflow Runtime
public class Global : HttpApplication {
  public static WorkflowRuntime WorkflowRuntime { get; set; }

  protected void Application_Start(object sender, EventArgs e) {
    WorkflowRuntime = new WorkflowRuntime();
    InstallPersistenceService();
    WorkflowRuntime.StartRuntime();
    // ...
  }

  protected void Application_End(object sender, EventArgs e) {
    WorkflowRuntime.StopRuntime();
    WorkflowRuntime.Dispose();
  }

  void InstallPersistenceService() {
    // Code from listing 4.
  }
}
In the application start event, I create the runtime, install the persistence service, and start the runtime. And in the application end event, I stop the runtime. This is an important step. It will block if there are running workflows until those workflows have been unloaded. After stopping the runtime, I call Dispose. While the call to StopRuntime and then to Dispose may look redundant, it is not. You need to call both methods in that order.

A Few Things to Ponder
Let me give you a few things to think about that I didn't directly touch on, presented in question and answer format. Why didn't I use ManualWorkflowSchedulerService? Often when people talk about integrating WF with ASP.NET they emphasize that you should replace the default scheduler for workflows (which uses the thread pool) with a service called ManualWorkflowSchedulerService. The reason is because it's not required or really appropriate for our long running goals. The manual scheduler is good when you expect to run a single workflow to completion within a given request. It makes less sense when your workflow will execute across process lifetimes, not to mention across requests.
Is there a way to track the current progress of a given workflow instance? Yes, there is an entire tracking service built into WF and it is used in a similar manner as the SQL persistence service. See the March 2007 Foundations column, "Tracking Services in Windows Workflow Foundation" by Matt Milner.

Bringing It All Together
I can summarize the techniques discussed in this article with a couple of steps. I began by outlining why the ASP.NET worker processes and the process model in general are not appropriate for very long running operations. To escape this limitation I took advantage of two features of WF that are combined to achieve process independence: asynchronous activities and workflow persistence.
Because building asynchronous activities can be somewhat tricky, I encapsulated the details into the AsyncActivity base class introduced in this article. Then I expressed the long-running operation as a sequential workflow built with asynchronous activities that I can incorporate that into a Web application and get process-independence for free.
Finally, I demonstrated that integrating the workflow into ASP.NET has two basic parts: communicating with the activities via a workflow queue and hosting the runtime in the global application class.
Now that you have seen WF integrated with ASP.NET to support long running operations, you have one more powerful tool for building solutions on top of the .NET Framework.

Michael Kennedy is an instructor for DevelopMentor where he specializes in core .NET technologies as well as agile and TDD development methodologies. Keep up with Michael via his Web site and blog at michaelckennedy.net.


Editor's Note
A Look Back As We Move Forward
Howard Dierking

Well, here we are again at the start of a new year. You know, it seems like it was just yesterday when I wrote the January 2008 editor's note! This month, I'm going to recap some of the changes we made in 2008 and tell you about a few of the new things you'll see in MSDN Magazine for 2009.
Let's begin with our 2008 reader survey, which revealed the high-level topics you all wanted to see more of in the magazine. By far the most requested topic was on software design patterns and best practices. We responded with our March issue, which carried this theme and contained articles discussing topics such as dependency injection and inversion of control and development lifecycle best practices such as continuous integration. We then welcomed Jeremy Miller as the author of the new Patterns in Practice column. Additionally, you can look forward to reading the next themed issue on software design in February.
Other requested topics from the 2008 survey were service development, database development, and Web app development. Last year, we ran many service-focused articles, and you'll likely notice that service-oriented and distributed architecture is the theme for this month. You can expect to see continued, if not increased, coverage of service-based system architectures. In this regard, we are excited to welcome Jon Flanders as the new regular author for the Service Station column.
While we have covered data access, we have generally stayed away from database development because it was considered within the scope of the DBA. And while this may be true for some organizations, the more common reality is that developers are responsible for building the accompanying databases for their solutions and as such need developer-oriented guidance around building databases. We are therefore happy to welcome Bob Beauchemin as the author of the new database development column, Under the Table.
There are so many facets of Web app development that the topic could be a magazine in and of itself. That said, we have maintained consistent coverage of both the client-side and server-side technologies with columns such as Cutting Edge and Wicked Code. In 2009, we are welcoming Scott Allen as the new author for the Extreme ASP.NET column, which will focus more on the server technologies for building Web apps while the aforementioned columns will focus more on the technologies used to directly create the client experiences. And on that note, I would also like to welcome Ambrose Little and Dr. Charles Kreitzberg to MSDN Magazine Online. Ambrose and Charles have started a monthly online column called Usability in Practice to tackle the psychological and engineering aspects of building great user experiences.
Furthermore, we recently began moving all of our article code samples to MSDN Code Gallery. This will allow our team to be much more agile in managing and updating code samples as needed. In that agility, we are now also able to translate all C# code samples into Visual Basic.
And with a growing number of more diverse and specialized languages, we are happy to add a new column, The Polyglot Programmer, and its author, Ted Neward, to the 2009 lineup. Ted will be focusing on a variety of programming languages and language types, providing guidance on where each type can contribute the most on development projects.
As we continue to grow our coverage to keep pace with the rapidly expanding set of technologies, we will often offer content exclusively online at msdn.microsoft.com/magazine. So please check in frequently!

Visit us at msdn.microsoft.com/magazine. Questions, comments, or suggestions for MSDN Magazine? Send them to the editor: mmeditor@microsoft.com.

Thanks to the following Microsoft technical experts for their help with this issue: Kent Brown, Michele Leroux Bustamante, Mike Calligaro, Sanjeev Garg, T.J. Goltermann, Dana Groff, Phil Haack, Robert Horvick, Michael Howard, Sesha Mani, David Murray, and Matt Winkler.


Toolbox
C# JavaScript, Eric Lippert’s Blog, And More
Scott Mitchell

All prices were confirmed November 19, 2008, and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.

Contents
Create JavaScript for AJAX Apps Using C# and Visual Studio
Blogs of Note
Spell-Check Your Comments, Strings, Markup, and Resource Files
The Bookshelf

Create JavaScript for AJAX Apps Using C# and Visual Studio
In the past, JavaScript was considered a toy language that was useful for simple input validation and Document Object Model (DOM) manipulation. But JavaScript is, in fact, a powerful and feature-rich language that powers today's AJAX-enabled Web sites.
And while the prevalence and role of JavaScript in modern Web applications have expanded dramatically over the past few years, there is still a very noticeable difference in the design-time experience when writing JavaScript code compared to writing C# code.
Visual Studio provides IntelliSense and debugging support for JavaScript, but many JavaScript errors and warnings cannot be detected until run time. On the other hand, C# offers compile-time-type checking; more robust object-oriented semantics; and tools for refactoring, creating technical documentation, and class browsing. Furthermore, most ASP.NET developers are more familiar with C# than with JavaScript.
Wouldn't it be nice if you could write C# code instead of JavaScript? This is possible with Script# (version 0.5.1), a free compiler, runtime, and framework created by Microsoft Software Architect Nikhil Kothari. The Script# compiler takes C# code and translates it into equivalent JavaScript code.
Use Script# to create short, one-off JavaScript functions or event handlers as well as larger JavaScript libraries. You can use it to create ASP.NET AJAX components and behaviors or Windows Vista Sidebar gadgets. The Script# compiler can use .resx files to generate localized script files and can be configured to generate release or debug scripts. (Debug scripts include white space and descriptive variable names for readability, whereas release scripts have unnecessary white space removed and variable renaming for code compactness.) In a nutshell, Script# gives you all the benefits of working with C# code—compile-time errors, familiar C# OOP syntax and semantics, and better tool support—while generating the JavaScript code needed for your Web application or Sidebar gadgets.
In addition to a compiler, Script# includes a runtime and a framework. Both the runtime and framework include a server-side implementation that you program against when writing C# code and a client-side implementation that is used by the JavaScript code generated by the Script# compiler.
The runtime defines the type system and it is mandatory. The framework is optional, but offers APIs for performing common tasks such as HTTP requests, JSON serialization, browser detection, and so forth. The framework also includes user interface controls and APIs to work with the Silverlight control in the browser, to access the RSS store in Internet Explorer 7, and to access Microsoft Virtual Earth from within the browser.
Keep in mind that Script# is not designed to convert existing C# applications into AJAX-enabled Web applications. Rather, Script# gives developers the ability to create Web applications or Sidebar gadgets by writing C# code that is then converted to JavaScript code.
However, because there is not a one-to-one correspondence between C# and JavaScript, not all of the C# language features are available when using Script#. For instance, write-only properties are disallowed, as are certain constructs and keywords, including goto, using, lock, and sizeof.
Price: Free.
projects.nikhilk.net/ScriptSharp

Blogs of Note
The most important ingredients for a really great technical blog include the following: frequent, knowledgeable, on-topic posts; in-depth examinations of important technologies or concepts; and a unique voice or style. You'll find these ingredients in just the right proportions on Eric Lippert's blog.
Eric is a Senior Software Design Engineer at Microsoft who has worked with both the scripting languages and C# teams. His blog offers many great posts that focus on security, performance, and code quality, along with entries on language semantics and features such as lambda expressions, immutability, and covariance and contravariance. He explores many of these topics in great depth, often spanning several blog posts.
There's a 5-part series on lambda expression versus anonymous methods, an 11-part series on immutability in C#, and a 5-part series on protected member semantics. You will also find posts with Eric's insights and opinions on ways to use (or not to use) various language features. Read his take on arrays in his blog article "Fabulous Adventures In Coding."
While many of the entries on the blog examine complex topics, Eric has a talent for explaining the concepts and theory in straightforward, down-to-earth terms.
blogs.msdn.com/ericlippert
Eric Lippert Blogs about Code Security, Performance, and Quality

Spell-Check Your Comments, Strings, Markup, and Resource Files
Compilers serve as a spell-checker of sorts. If you transpose two letters in a method name or mistype a variable name, the compiler reports a build error. Fortunately, these errors are easy to identify, as Visual Studio underlines the misspelling with a squiggly line.
However, note that the compiler does not serve as a spell-checker for source code comments; strings in code or in resource files; or HTML, XML, or XAML markup. It is all too easy for a misspelled comment to make its way into the technical documentation, or worse, for a misspelled string to slip past QA and into production.
The good news is that there are tools to spell-check the strings, comments, markup, and other content that is opaque to the compiler. One such tool is IntelliSpell 2.0 by ComponentOne. IntelliSpell works much like the spell-check in Microsoft Office Word. IntelliSpell underlines any misspelled word with a squiggly line. Right-click on the misspelled word to see a list of possible corrections. These misspellings are also enumerated in the Task List window.
You can correct a misspelling from the Task List or jump to the misspelled word by double-clicking it in the Task List. You can run the spell-checker at any time by selecting the menu option from the Tools menu or from the IntelliSpell toolbar. IntelliSpell can also be configured to check your spelling automatically as you type and to run the spell-checker after each build.
By default, IntelliSpell checks the spelling in all code files, Web pages, XML files (including configuration files), resource files, and text files. It ignores words in mixed case, words with numbers, and Internet and file addresses, although these rules and what types of files are spell-checked are configurable. And IntelliSpell offers spelling dictionaries in 16 different languages; these dictionaries can be used alone or in tandem when you are spell-checking localized projects.
IntelliSpell comes in two editions: a free Community Edition and a Professional Edition that costs $79.99. The Community Edition has the core features—Visual Studio integration, underlining misspelled words, spelling suggestions via a context menu, and so on—but lacks some of the features found in the Professional Edition.
With the Community Edition, you can only spell-check a single file at a time. The Professional Edition, however, is able to spell-check an entire Project or Solution. Also, features such as spell-checking as you type, spell-checking localized projects, and the ability to create custom dictionaries are not included in the Community Edition.
Price: Community Edition is free. Professional Edition is $79.99.
componentone.com/
IntelliSpell Spell-Checks Strings, Comments, and Markup

The Bookshelf
In all professions and hobbies there are certain areas that everyone, regardless of their experience, should study. Every golfer, despite her skill level, can certainly improve by working on her short game, for instance.
For software developers, you can never be too knowledgeable about the syntax and features of SQL. And while there are plenty of online articles that include SQL tips and tricks, I have found that the best way for me to hone my SQL skills is by methodically working through a book on the subject.
I recently finished the book Pro T-SQL 2008 Programmer's Guide (Apress) by Michael Coles, which provides a thorough review of the T-SQL syntax and language features. The book begins with a look at the new features in T-SQL 2008, such as the MERGE statement and new data types. The author then moves on to the core T-SQL constructs—user-defined functions; stored procedures; triggers; XML, XQuery, and XPath support. He then continues with coverage on encryption, common table expressions and windowing functions, SQL CLR programming, and error handling, among others. Each of these topics is given its own chapter with in-depth discussion and examples. And each chapter concludes with a series of exercises whose answers are provided in the appendix.
I especially enjoyed the chapter on performance tuning. This chapter starts with an overview of how SQL Server arranges your data on disk and how the arrangement affects performance, along with tips for saving disk space and for storing sparse data. Next, Coles discusses the performance benefits of indexes and compares and contrasts clustered, non-clustered, and filtered indexes. The chapter concludes with a discussion on analyzing query plans and tips for optimizing queries.
Pro T-SQL 2008 Programmer's Guide assumes that the reader is already familiar with T-SQL basics—there are no beginner-level discussions on SELECT, INSERT, UPDATE, or DELETE statements. Moreover, this book concentrates on T-SQL and does not explore database administration, although there is a short chapter that outlines tools such as SQL Server Management Studio, SQL Profiler, SQLCMD, and BCP. There is also a chapter that shows two different ways to interface with SQL Server from a Microsoft .NET Framework client application: using the ADO.NET SqlClient provider and LINQ to SQL.
Time spent learning new SQL skills and practicing the old ones is always time well spent. Pro T-SQL 2008 Programmer's Guide, with its straightforward prose, countless samples, and end-of-chapter exercises, is a great way to learn about the latest features in SQL Server 2008 and to bolster your existing SQL skills.
Price: $52.99.
apress.com/

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out
Best Practices For Managed And Native Code Interoperability
Jesse Kaplan
Contents
When Is Managed-Native Interop Appropriate?
Interop Technologies: Three Choices
Interop Technologies: P/Invoke
Interop Technologies: COM Interop
Interop Technologies: C++/CLI
Considerations for Your Interop Architecture
API Design and Developer Experience
Performance and the Location of Your Interop Boundary
Lifetime Management
In some ways, it may seem odd to see a column like this one appearing in MSDN Magazine in early 2009—managed and native code interoperability has been supported in the Microsoft .NET Framework, in more or less the same form, since version 1.0 in 2002. In addition, you can readily find detailed API- and tool-level docs and thousands of pages of detailed support documents. Yet the missing piece in all of this is comprehensive, high-level architectural guidance describing when to use interop, what architectural considerations you should take into account, and which interop technology to use. This is the gap I'm going to begin to fill here.

When Is Managed-Native Interop Appropriate?
There has not been a great deal written about when it's appropriate to use managed-native interop, and much of what can be found on the subject is conflicting. Sometimes the guidance is not based on actual practical experience, either. So before I begin, let me state that everything I'm writing about today is guidance that we developed based on the experiences the interop team has had helping internal and external customers of all sizes.
In synthesizing this experience, we put together three products that serve as excellent examples of successful uses of interop and a representative set of the types of uses of interop. Visual Studio Tools for Office is the managed extensibility toolset for Office and is the first application I think of when I think of interop. It represents the classic use of interop—a large native application that wants to enable managed extensions or add-ins. Next on my list is Windows Media Center, an application that was built from the ground up as a mixed managed and native application. Windows Media Center was developed using mostly managed code with some portions—those dealing directly with the TV tuner and other hardware drivers—built in native code. Finally there is Expression Design, an application with a large, preexisting native code base that wants to leverage new managed technologies, in this case Windows Presentation Foundation (WPF), to provide a next-generation user experience.
These three applications address the three most common reasons to use interop: to allow managed extensibility of preexisting native applications, to allow most of an application to take advantage of the benefits of managed code while still writing the lowest-level pieces in native code, and to add a differentiated, next-generation user experience to an existing native application.
In the past, available guidance would have suggested simply rewriting the entire application in managed code in these instances. Trying to follow this advice and seeing how many people simply refuse to follow it has made it clear that this simply isn't an option for most existing applications. Interop is going to be the vital technology that allows developers to maintain their existing investment in native code while still being able to take advantage of the new managed environment. If you are planning to rewrite your application for other reasons, managed code may be a good choice for you. But you generally don't want to rewrite your application merely to use new managed technologies and avoid interop.

Interop Technologies: Three Choices
There are three main interop technologies available in the .NET Framework, and which one you choose will be determined in part by the type of API you are using for interop and in part by your requirements and need to control the boundary. Platform Invoke, or P/Invoke, is primarily a managed-to-native interop technology that allows you to call C-style native APIs from managed code. COM interop is a technology that allows you either to consume native COM interfaces from managed code or export native COM interfaces from managed APIs. Finally there is C++/CLI (formerly known as managed C++), which allows you to create assemblies that contain a mix of managed and native C++ compiled code and is designed to serve as a bridge between managed and native code.

Interop Technologies: P/Invoke
P/Invoke is the simplest of the three technologies, and it was designed primarily for providing managed access to C-style APIs. With P/Invoke you need to wrap each API individually. This can be a very good choice if you have a few APIs to wrap and their signatures are not very complex. P/Invoke gets substantially harder to use, however, if the unmanaged APIs have a lot of arguments that don't have good managed equivalents such as variable-length structures, void *s, overlapping unions, and so on.
The .NET Framework base class libraries (BCLs) contain many examples of APIs that are really just thick wrappers around large numbers of P/Invoke declarations. Nearly all functionality in the .NET Framework that wraps unmanaged Windows APIs is built using P/Invoke. In fact, even Windows Forms is built almost entirely on the native ComCtl32.dll using P/Invoke.
There are a few very valuable resources that can make using P/Invoke significantly easier. First, the Web site pinvoke.net has a wiki, originally set up by Adam Nathan from the CLR interop team, that has a large number of user-contributed signatures for a wide variety of common Windows APIs.
There is also a very handy Visual Studio add-in that makes it easy to consult pinvoke.net from within Visual Studio. For APIs that are not covered on pinvoke.net, whether they are from your own libraries or someone else's, the interop team has released a P/Invoke signature-generating tool called the P/Invoke Interop Assistant, which automatically creates signatures for native APIs based on a header file. The accompanying screenshot shows the tool in action.
Creating Signatures in P/Invoke Interop Assistant

Interop Technologies: COM Interop
COM interop allows you either to consume COM interfaces from managed code or expose managed APIs as COM interfaces. You can use the TlbImp tool to generate a managed library that exposes managed interfaces for talking to a specific COM tlb. And TlbExp performs the opposite task and will generate a COM tlb with interfaces that correspond to the ComVisible types in a managed assembly.
COM interop can be a very good solution for you if you are already using COM inside your application or as its extensibility model. It is also the easiest way to maintain full-fidelity COM semantics between managed and native code. In particular, COM interop is an excellent choice if you are interoperating with a Visual Basic 6.0-based component, as the CLR follows basically the same COM rules Visual Basic 6.0 does.
COM interop is less useful if your application does not already use COM internally, or if you don't need full fidelity COM semantics and its performance is not acceptable for your application.
Microsoft Office is the most prominent example of an application that uses COM interop as its bridge between managed and native code. Office was a great candidate for COM interop, as it has long used COM as its extensibility mechanism and was most commonly used from Visual Basic for Applications (VBA) or Visual Basic 6.0.
Originally Office relied entirely on TlbImp and the thin interop assembly as its managed object model. Over time, however, the Visual Studio Tools for Office (VSTO) product was built into Visual Studio, providing a richer and richer development model that incorporated many of the principles described in this column. When using the VSTO product today, it is sometimes as easy to forget that COM interop is serving as the foundation of VSTO as it is to forget that P/Invoke is the foundation of much of the BCLs.

Interop Technologies: C++/CLI
C++/CLI is designed to be a bridge between the native and managed world, and it allows you to compile both managed and native C++ into the same assembly (even the same class) and make standard C++ calls between the two portions of the assembly. When you use C++/CLI, you choose which portion of the assembly you want to be managed and which you want to be native. The resulting assembly is a mix of MSIL (Microsoft intermediate language, found in all managed assemblies) and native assembly code. C++/CLI is a very powerful interop technology that gives you almost complete control over the interop boundary. The downside is that it forces you to take almost complete control over the boundary.
C++/CLI can be a good bridge if static-type checking is needed, if strict performance is a requirement, and if you need more predictable finalization. If P/Invoke or COM interop meets your needs, they are generally simpler to use, especially if your developers are not familiar with C++.
There are a few things to keep in mind when considering C++/CLI. The first thing to remember is that if you are planning to use C++/CLI to provide a faster version of COM interop, COM interop is slower than C++/CLI because it does a lot of work on your behalf. If you only loosely use COM in your application and don't require full fidelity COM interop, then this is a good trade-off.
If, however, you use a large portion of the COM spec, you'll likely find that once you add back the pieces of COM semantics you need into your C++/CLI solution, you'll have done a lot of work and will have performance no better than what is provided with COM interop. Several Microsoft teams have gone down this road only to realize this and move back to COM interop.
The second major consideration for using C++/CLI is to remember that this is only intended to be a bridge between the managed and native worlds and not intended to be a technology you use to write the bulk of your application. It is certainly possible to do so, but you'll find that developer productivity is much lower than in a pure C++ or pure C#/Visual Basic environment and that your application runs much slower to boot. So when you use C++/CLI, compile only the files you need with the /clr switch, and use a combination of pure managed or pure native assemblies to build the core functionality of your application.

Considerations for Your Interop Architecture
Once you have decided to use interop in your application and decided which technology to use, there are a few high-level considerations when architecting your solution, including your API design and the developer experience for those coding against the interop boundary. Also take into account where you place your native managed transitions and the performance impact this can have on your application. And finally, you need to consider lifetime management and whether you need to do anything to bridge the gap between the garbage-collected world of the managed environment and the manual/deterministic lifetime management of the native world.

API Design and Developer Experience
When thinking about your API design, there are several questions you need to ask yourself: Who is going to be coding against my interop layer, and should I optimize for improving his experience or for minimizing the cost of building the boundary? Are the developers coding against this boundary the same ones writing the native code? Are they other developers in your company? Are they third-party developers extending your application or using it as a service? What is their sophistication level? Are they comfortable with native paradigms or are they only happy when writing managed code?
Your answers to these questions will help determine where you end up on the continuum between a very thin wrapper over the native code and a rich managed object model that uses native code under the covers. With a thin wrapper, all the native paradigms will bleed through, and developers will be very aware of the boundary and the fact that they are coding against a native API. With a thicker wrapper you can almost completely hide the fact that native code is in the picture—the file system APIs in the BCL are an excellent example of a very thick interop layer that provides a first-class managed-object model.

Performance and the Location of Your Interop Boundary
Before you spend too much time optimizing your application, it's important that you determine whether or not you have an interop performance problem. Many applications use interop in performance-critical sections and need to pay close attention here. But many others are going to be using interop in response to user mouse clicks and will not see tens, hundreds, or even thousands of interop transitions causing delays to their users. That said, when you do take a look at the performance of your interop solution, you should have two goals in mind: reducing the number of interop transitions you make and reducing the amount of data passed on each transition.
A given interop transition with a given amount of data crossing between the managed and native world is basically going to have a fixed cost. This fixed cost will be different depending upon your choice of interop technology, but if you made that choice because you needed the features of that technology, then you won't be able to change it. This means your focus should be reducing the chattiness of the boundary and then reducing the amount of data crossing it.
How you accomplish this depends largely upon your application. But a common, and adaptable, strategy with which many have been successful is to move the isolation boundary by writing a bit of code on the side of the boundary that defined the busy and data-heavy interface. The basic idea is to write an abstraction layer that batches calls into the very busy interface or, even better, to move the portion of your application logic that needs to interact with this API across the boundary and only pass inputs and results across the boundary.

Lifetime Management
The differences in lifetime management between the managed and native worlds is often one of the biggest challenges for interop customers. The fundamental difference between the garbage collection-based system in the .NET Framework and the manual and deterministic system in the native world can often manifest itself in surprising ways that can be difficult to diagnose.
The first problem you might notice in an interop solution is the lengthy amount of time some managed objects hold on to their native resources even after the managed world is finished using them. This often causes problems when the native resource is very scarce and depends upon being released as soon as its callers are finished using it (database connections are a great example of this).
When these resources are not scarce, you can simply rely on the garbage collector calling an object's finalizer and letting that finalizer release the native resources (either implicitly or explicitly). When resources are scarce, the managed dispose pattern can be very useful. Instead of exposing native objects directly to managed code, you should put at least a thin wrapper around them that implements IDisposable and follows the standard dispose pattern. This way, if you find resource exhaustion to be a problem, you can explicitly dispose of these objects in your managed code and release the resources as soon as you're done with them.
The second lifetime management problem that commonly impacts applications is one that developers often perceive as a stubborn garbage collection: their memory use keeps rising, but for some reason the garbage collector is running infrequently and objects are kept alive. Often they will keep adding calls to GC.Collect to force the issue.
The root of this problem is usually that there are a lot of very small managed objects that hold onto, and keep alive, very large native data structures. What happens is that the garbage collector is self-tuning and tries to avoid wasting time doing unnecessary or unhelpful collections. And, in addition to looking at the current memory pressure of the process, it looks at how much memory each garbage collection frees when deciding whether to do another one.
When it runs in this scenario, though, it sees that each collection only frees a small amount of memory (remember it only knows about how much managed memory is freed) and doesn't realize that freeing those small objects can significantly decrease overall pressure. This leads to a situation where fewer and fewer collections happen even as memory use keeps increasing.
The solution to this problem is to give hints to the garbage collector as to the real memory cost of each of these small managed wrappers over the native resources. We added a pair of APIs in the .NET Framework 2.0 that allow you to do just that. You can use the same type of wrappers you used to add the dispose patterns to scarce resources but repurpose them to provide hints to the garbage collector instead of explicitly having to free the resources yourself.
In the constructor for this object you simply call the method GC.AddMemoryPressure and pass in the approximate cost in native memory of the native object. You can then call GC.RemoveMemoryPressure in the object's finalizer method. This pair of calls will help the garbage collector understand the true cost of these objects and the real memory freed when releasing them. Note that it is important to make sure you perfectly balance your calls to Add/RemoveMemoryPressure.
The third common disconnect in lifetime management between the managed and native worlds is not so much about the management of individual resources or objects but rather of whole assemblies or libraries. Native libraries can be easily unloaded when an application is done with them, but managed libraries cannot be unloaded on their own. Instead, the CLR has isolation units called AppDomains that can be individually unloaded and will clean up all assemblies, objects, and even threads running in that domain when unloaded. If you are building a native application and are accustomed to unloading your add-ins when you are done with them, you'll find that using different AppDomains for each of your managed add-ins will give you the same flexibility you had in unloading individual native libraries.

Send your questions and comments to clrinout@microsoft.com.

Jesse Kaplan currently is the Program Manager of Managed/Native Interoperability for the CLR team at Microsoft. His past responsibilities include compatibility and extensibility.


Basic Instincts
Inspecting COM Objects With Reflection
Lucian Wischik
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Type Libraries and Runtime Callable Wrappers
When a Type Lacks an RCW
Using ITypeInfo
Chasing Down Type References
Getting the Members
Primitive and Synthetic Types
The COM Representation of Value
Dumping the Properties of a COM Object
Using IDispatch.Invoke
Discussion
Many of you have felt the frustration of trying to get COM to work. You've also felt the joy that comes when you're successful. A common trick for understanding how an object works is to inspect it using the reflection capabilities of the Microsoft .NET Framework. In some cases, .NET reflection also works on COM objects. Look at the following code to see what I mean. The code uses .NET reflection to obtain and display a list of members in the object
Dim b As New SpeechLib.SpVoice
Console.WriteLine("GETTYPE {0}", b.GetType())
For Each member In b.GetType().GetMembers()
    Console.WriteLine(member)
Next
and produces this output to the console:
GETTYPE SpeechLib.SpVoiceClass
Void Speak(System.String, UInt32, UInt32 ByRef)
Void SetVoice(SpeechLib.ISpObjectToken)
Void GetVoice(SpeechLib.ISpObjectToken ByRef)
Int32 Volume
...
But this code doesn't work for all COM objects. For that, you have to use COM reflection. This column explains why and how.
Why might you want to use reflection on an object? I've found reflection to be useful for debugging and logging; you can use it to write a general-purpose "dump" routine that prints out everything it can about an object. The code in this column will be enough for you to write your own "dump" routine. Once it's done, you could even invoke it from the Immediate window while debugging. That's especially useful since the Visual Studio debugger doesn't always provide much information about COM objects.
For production use, reflection is useful if you've written an application that takes plug-in components where users drop their components into a directory or list them in the registry, and your application has to examine these components and find which classes and methods they expose. For instance, Visual Studio uses reflection in this way to populate IntelliSense.

Type Libraries and Runtime Callable Wrappers
Let's build a project for illustration. First, create the project and add a COM reference via Project>AddReference. For this column I'll use the "Microsoft Speech Object Library" SpeechLib. Figure 1 shows the relevant entities and files that are examined when you run the reflection code you saw earlier.
Figure 1 Reflecting on SpeechLib
Sapi.dll is the DLL that contains SpeechLib. It happens to live in %windir%\system32\speech\common\sapi.dll. This DLL contains both the implementation of the SpVoice COM class and a TypeLibrary that contains all reflection information for it. TypeLibraries are optional, but almost every COM component on your system will have one.
Interop.SpeechLib.dll was automatically generated by Visual Studio through Project>AddReference. The generator reflects upon the TypeLibrary and produces an Interop Type for SpVoice. This type is a managed class that contains a managed method for every native COM method found in the TypeLibrary. You can also generate interop assemblies yourself using the tlbimp.exe command-line tool from the Windows SDK. An instance of an interop type is called a runtime callable wrapper (RCW), and it wraps a pointer to an instance of a COM class.
Running the following Visual Basic command creates an RCW (an instance of the interop type) and also an instance of the SpVoice COM class:
Dim b As New SpeechLib.SpVoice
The variable "b" references the RCW, so when the code reflected upon "b" it was really reflecting upon the managed equivalent that had been constructed from the TypeLibrary.
Users who deploy their ConsoleApplication1.exe will also have to deploy Interop.SpeechLib.dll. (However, Visual Studio 2010 will allow the interop type to be copied directly inside ConsoleApplication1.exe. This will greatly simplify deployment. The feature is called "No-Primary-Interop-Assembly," or "No-PIA" for short.)

When a Type Lacks an RCW
What happens if you don't have an interop assembly for a COM object? For example, what if you created the COM object itself via CoCreateInstance, or if, as often happens, you call a method on a COM object and it returns a COM object whose type isn't known in advance? What if you've written a managed plug-in for an unmanaged application, and the application gave you a COM object? What if you discovered which COM object to create by looking through the registry?
Every one of these things will give you an IntPtr reference to the COM object rather than an Object reference to its RCW. When you ask for an RCW around that IntPtr, you get what is illustrated in Figure 2.
Figure 2 Getting a Runtime Callable Wrapper
In Figure 2 you'll see that the CLR provided a default RCW, an instance of the default interop type "System.__ComObject". If you reflect on this like so
Dim b = CoCreateInstance(CLSID_WebBrowser, _
                   Nothing, 1, IID_IUnknown)
Console.WriteLine("DUMP {0}", b.GetType())
For Each member In b.GetType().GetMembers()
    Console.WriteLine(member)
Next
you'll find that it doesn't have any members that are useful to you; it only has these:
DUMP System.__ComObject
System.Object GetLifetimeService()
System.Object InitializeLifetimeService()
System.Runtime.Remoting.ObjRef CreateObjRef(System.Type)
System.String ToString()
Boolean Equals(System.Object)
Int32 GetHashCode()
System.Type GetType()
To get useful reflection on such a COM object, you have to reflect on its TypeLibrary yourself. You can do so using ITypeInfo.
But first, a brief note: if a method gives you back an Object or IDispatch or ITypeInfo or other .NET class or interface, then it has given you a reference to the RCW, and .NET will take care of releasing it for you. But if the method gives you back an IntPtr, it means you have a reference to the COM object itself, and you almost always have to call Marshal.Release on it (this depends on the precise semantics of whatever method gave you that IntPtr). Here's how:
Dim com As IntPtr = ...
   Dim rcw = Marshal.GetObjectForIUnknown(com)
   Marshal.Release(com)
But it is far more common to declare the function with marshaling so that the marshaler calls GetObjectForIUnknown and Release automatically, as in the declaration of CoCreateInstance that you see in Figure 3.
Figure 3 CoCreateInstance
<DllImport("ole32.dll", ExactSpelling:=True, PreserveSig:=False)> _
Function CoCreateInstance( _
    ByRef clsid As Guid, _
    <MarshalAs(UnmanagedType.Interface)> ByVal punkOuter As Object, _
    ByVal context As Integer, _
    ByRef iid As Guid) _
    As <MarshalAs(UnmanagedType.Interface)> Object
End Function

Dim IID_NULL As Guid = New Guid("00000000-0000-0000-C000-000000000000")
Dim IID_IUnknown As Guid = New _
    Guid("00000000-0000-0000-C000-000000000046")
Dim CLSID_SpVoice As Guid = New _
    Guid("96749377-3391-11D2-9EE3-00C04F797396")

Dim b As Object = CoCreateInstance(CLSID_SpVoice, Nothing, 1, _
    IID_IUnknown)

Using ITypeInfo
ITypeInfo is the equivalent of System.Type for COM classes and interfaces. With it you can enumerate the members of a class or interface. In this example, I'm going to print them out; however, you could use ITypeInfo to look up members at run time and then invoke them or get their property values through IDispatch. Figure 4 shows how ITypeInfo fits in, as well as all the other structures you'll need to use.
Figure 4 ITypeInfo and Type Information
The first step is getting the ITypeInfo for a given COM object. It would be nice if you could use rcw.GetType(), but alas this returns the System.Type information about the RCW itself. It would also be nice if you could use the built-in function Marshal.GetITypeInfoForType(rcw), but unfortunately this only works for RCWs that come from interop assemblies. Instead, you will have to get the ITypeInfo manually.
The following code will work for both of these cases, whether the RCW came from the stub in mscorlib, or from a proper interop assembly:
Dim idisp = CType(rcw, IDispatch)
Dim count As UInteger = 0
idisp.GetTypeInfoCount(count)
If count < 1 Then Throw New Exception("No type info")
Dim _typeinfo As IntPtr
idisp.GetTypeInfo(0, 0, _typeinfo)
If _typeinfo = IntPtr.Zero Then Throw New Exception("No ITypeInfo")
Dim typeInfo = CType(Marshal.GetTypedObjectForIUnknown(_typeinfo, _
                     GetType(ComTypes.ITypeInfo)), ComTypes.ITypeInfo)
Marshal.Release(_typeinfo)
This code uses the interface IDispatch. The interface isn't defined anywhere in the .NET Framework, so you have to define it yourself, as you see in Figure 5. I left the function GetIDsOfNames empty because it's not needed for the current purposes; you'll need to include an entry for it, though, because the interface has to list the right number of methods in the right order.
Figure 5 Defining the IDispatch Interface
''' <summary>
''' IDispatch: this is a managed version of the IDispatch interface
''' </summary>
''' <remarks>We don't use GetIDsOfNames or Invoke, and so haven't
''' bothered with correct signatures for them.</remarks>
<ComImport(), Guid("00020400-0000-0000-c000-000000000046"), _
 InterfaceType(ComInterfaceType.InterfaceIsIUnknown)> _
 Interface IDispatch
    Sub GetTypeInfoCount(ByRef pctinfo As UInteger)
    Sub GetTypeInfo(ByVal itinfo As UInteger, ByVal lcid _
      As UInteger, ByRef pptinfo As IntPtr)
    Sub stub_GetIDsOfNames()
    Sub Invoke(ByVal dispIdMember As Integer, ByRef riid As Guid, _
               ByVal lcid As UInteger, ByVal dwFlags As UShort, _
               ByRef pDispParams As ComTypes.DISPPARAMS, _
               ByRef pVarResult As [VARIANT], ByRef pExcepInfo As IntPtr, _
               ByRef pArgErr As UInteger)
End Interface
You might wonder why IDispatch has its InterfaceType attribute set to ComInterfaceType.InterfaceIsUnknown rather than set to ComInterfaceType.InterfaceIsIDisapatch. That's because the InterfaceType attribute says what the interface inherits from, not what it is.
You have an ITypeInfo. Now it's time to start reading from it. Take a look at Figure 6 because the function I'm going to implement to dump the type information is shown there. For GetDocumentation, the first parameter is a MEMBERID, that is, GetDocumentation is meant to return information about each member of the type. But you can also pass in MEMBERID_NIL, which has the value -1, to get information about the type itself.
Figure 6 DumpTypeInfo
''' <summary>
''' DumpType: prints information about an ITypeInfo type to the console
''' </summary>
''' <param name="typeInfo">the type to dump</param>
Sub DumpTypeInfo(ByVal typeInfo As ComTypes.ITypeInfo)

    ' Name:
    Dim typeName = "" : typeInfo.GetDocumentation(-1, typeName, "", 0, "")
    Console.WriteLine("TYPE {0}", typeName)

    ' TypeAttr: contains general information about the type
    Dim pTypeAttr As IntPtr : typeInfo.GetTypeAttr(pTypeAttr)
    Dim typeAttr = CType(Marshal.PtrToStructure(pTypeAttr, _
                         GetType(ComTypes.TYPEATTR)), ComTypes.TYPEATTR)
    typeInfo.ReleaseTypeAttr(pTypeAttr)
    ...

End Sub
Note how the marshaling works. When you call typeInfo.GetTypeAttr, it allocates an unmanaged block of memory and returns to you the pointer pTypeAttr. Then Marshal.PtrToStructure copies from this unmanaged block into a managed block (which will then be garbage collected). So it's fine to immediately call typeInfo.ReleaseTypeAttr.
As shown previously, you need typeAttr to know how many members and implemented interfaces there are (typeAttr.cFuncs, typeAttr.cVars, and typeAttr.cImplTypes).

Chasing Down Type References
The first task that must be completed is to get a list of implemented/inherited interfaces. (In COM, one never inherits from another class). Here's the code:
' Inheritance:
For iImplType = 0 To typeAttr.cImplTypes - 1
    Dim href As Integer
    typeInfo.GetRefTypeOfImplType(iImplType, href)
    ' "href" is an index into the list of type descriptions within the
    ' type library.
    Dim implTypeInfo As ComTypes.ITypeInfo
    typeInfo.GetRefTypeInfo(href, implTypeInfo)
    ' And GetRefTypeInfo looks up the index to get an ITypeInfo for it.
    Dim implTypeName = ""
    implTypeInfo.GetDocumentation(-1, implTypeName, "", 0, "")
    Console.WriteLine("  Implements {0}", implTypeName)
Next
There's a layer of indirection here. GetRefTypeOfImplType doesn't give you the ITypeInfo of the implemented types directly: instead, it gives you a handle to an ITypeInfo. The function GetRefTypeInfo is what looks up that handle. Then you can use the familiar GetDocumentation(-1) to get the name of that implemented type. I'll discuss handles to ITypeInfo again later.

Getting the Members
For reflecting on field members, each field has a VARDESC to describe it. Once again, the typeInfo object allocates an unmanaged memory block pVarDesc, and you marshal it into a managed block varDesc and release the unmanaged block:
' Field members:
For iVar = 0 To typeAttr.cVars - 1
    Dim pVarDesc As IntPtr : typeInfo.GetVarDesc(iVar, pVarDesc)
    Dim varDesc = CType(Marshal.PtrToStructure(pVarDesc, _
                        GetType(ComTypes.VARDESC)), ComTypes.VARDESC)
    typeInfo.ReleaseVarDesc(pVarDesc)
    Dim names As String() = {""}
    typeInfo.GetNames(varDesc.memid, names, 1, 0)
    Dim varName = names(0)
    Console.WriteLine("  Dim {0} As {1}", varName, _
                      DumpTypeDesc(varDesc.elemdescVar.tdesc, typeInfo))
Next
The function "GetNames" is curious. Conceivably each member might have several names. It's enough just to get the first.
The code for reflecting on function members is generally similar (see Figure 7). The return type is funcDesc.elemdescFunc.tdesc. The count of formal parameters is given by funcDesc.cParams, and formal parameters are stored in the array funcDesc.lprgelemdescParam. (It's not pleasant accessing an unmanaged array like this from managed code because you have to do pointer arithmetic.)
Figure 7 Reflecting on Function Members
For iFunc = 0 To typeAttr.cFuncs - 1

   ' retrieve FUNCDESC:
   Dim pFuncDesc As IntPtr : typeInfo.GetFuncDesc(iFunc, pFuncDesc)
   Dim funcDesc = CType(Marshal.PtrToStructure(pFuncDesc, _
                         GetType(ComTypes.FUNCDESC)), ComTypes.FUNCDESC)
   Dim names As String() = {""}
   typeInfo.GetNames(funcDesc.memid, names, 1, 0)
   Dim funcName = names(0)

   ' Function formal parameters:
   Dim cParams = funcDesc.cParams
   Dim s = ""
   For iParam = 0 To cParams - 1
        Dim elemDesc = CType(Marshal.PtrToStructure( _
                  New IntPtr(funcDesc.lprgelemdescParam.ToInt64 + _
                  Marshal.SizeOf(GetType(ComTypes.ELEMDESC)) * iParam), _
                  GetType(ComTypes.ELEMDESC)), ComTypes.ELEMDESC)
        If s.Length > 0 Then s &= ", "
        If (elemDesc.desc.paramdesc.wParamFlags And _
            Runtime.InteropServices.ComTypes.PARAMFLAG.PARAMFLAG_FOUT) _
             <> 0 Then s &= "out "
        s &= DumpTypeDesc(elemDesc.tdesc, typeInfo)
   Next

   ' And print out the rest of the function's information:
   Dim props = ""
   If (funcDesc.invkind And ComTypes.INVOKEKIND.INVOKE_PROPERTYGET) _
      <> 0 Then props &= "Get "
   If (funcDesc.invkind And ComTypes.INVOKEKIND.INVOKE_PROPERTYPUT) _
      <> 0 Then props &= "Set "
   If (funcDesc.invkind And ComTypes.INVOKEKIND.INVOKE_PROPERTYPUTREF) _
     <> 0 Then props &= "Set "
   Dim isSub = (FUNCDESC.elemdescFunc.tdesc.vt = VarEnum.VT_VOID)
   s = props & If(isSub, "Sub ", "Function ") & funcName & "(" & s & ")"
   s &= If(isSub, "", " as " & _
     DumpTypeDesc(funcDesc.elemdescFunc.tdesc, typeInfo))
   Console.WriteLine("  " & s)
   typeInfo.ReleaseFuncDesc(pFuncDesc)
Next
There are other flags as well as PARAMFLAG_FOUT—flags for in, retval, optional, and so on. Type information for both fields and members was stored in a TYPEDESC structure, and I invoked a function DumpTypeDesc to print it out. It might seem surprising that TYPEDESC is used instead of ITypeInfo. I'll elaborate next.

Primitive and Synthetic Types
COM uses TYPEDESC to describe some types and ITypeInfo to describe others. What's the difference? COM uses ITypeInfo only for classes and interfaces defined in type libraries. And it uses TYPEDESC for primitive types such as Integer or String, and also for compound types such as Array of SpVoice or IUnknown Reference.
This is different from .NET: first, in .NET even the primitive types such as Integer and String are represented by classes or structures through System.Type; second, in .NET the compound types such as Array of Integer are represented through System.Type.
The code that you need to dig through a TYPEDESC is quite straightforward (see Figure 8). Note that the case VT_USERDEFINED again uses a handle to a reference, which you must look up through GetRefTypeInfo.
Figure 8 Looking at TYPEDESC
Function DumpTypeDesc(ByVal tdesc As ComTypes.TYPEDESC, _
  ByVal context As ComTypes.ITypeInfo) As String
    Dim vt = CType(tdesc.vt, VarEnum)
    Select Case vt

        Case VarEnum.VT_PTR
            Dim tdesc2 = CType(Marshal.PtrToStructure(tdesc.lpValue, _
                          GetType(ComTypes.TYPEDESC)), ComTypes.TYPEDESC)
            Return "Ref " & DumpTypeDesc(tdesc2, context)

        Case VarEnum.VT_USERDEFINED
            Dim href = CType(tdesc.lpValue.ToInt64 And Integer.MaxValue, Integer)
            Dim refTypeInfo As ComTypes.ITypeInfo = Nothing
            context.GetRefTypeInfo(href, refTypeInfo)
            Dim refTypeName = ""
            refTypeInfo.GetDocumentation(-1, refTypeName, "", 0, "")
            Return refTypeName

        Case VarEnum.VT_CARRAY
            Dim tdesc2 = CType(Marshal.PtrToStructure(tdesc.lpValue, _
                          GetType(ComTypes.TYPEDESC)), ComTypes.TYPEDESC)
            Return "Array of " & DumpTypeDesc(tdesc2, context)
            ' lpValue is actually an ARRAYDESC structure, which also has
            ' information on the array dimensions, but alas .NET doesn't
            ' predefine ARRAYDESC.

        Case Else
            ' There are many other VT_s that I haven't special-cased,
            ' e.g. VT_INTEGER.
            Return vt.ToString()
    End Select
End Function

The COM Representation of Value
The next step is to actually dump a COM object, that is, print out the values of its properties. This task is easy if you know the names of those properties since you can just use a late-bound call in Visual Basic:
Dim com as Object : Dim val = com.SomePropName
The compiler translates this into a runtime call of IDispatch::Invoke to fetch the value of the property. But in the case of reflection, you may not know the property name. Perhaps all you have is MEMBERID, so you have to call IDispatch::Invoke yourself. This isn't very pretty.
The first headache comes from the fact that COM and .NET represent values in very different ways. In .NET you use Object to represent arbitrary values. In COM you use the VARIANT structure, as shown in Figure 9.
Figure 9 Using VARIANT
''' <summary>
''' VARIANT: this is called "Object" in Visual Basic. It's the universal ''' variable type for COM.
''' </summary>
''' <remarks>The "vt" flag determines which of the other fields have
''' meaning. vt is a VarEnum.</remarks>
<System.Runtime.InteropServices.StructLayoutAttribute( _
           System.Runtime.InteropServices.LayoutKind.Explicit, Size:=16)> _
Public Structure [VARIANT]
    <System.Runtime.InteropServices.FieldOffsetAttribute(0)> Public vt As UShort
    <System.Runtime.InteropServices.FieldOffsetAttribute(2)> _
      Public wReserved1 As UShort
    <System.Runtime.InteropServices.FieldOffsetAttribute(4)> _
      Public wReserved2 As UShort
    <System.Runtime.InteropServices.FieldOffsetAttribute(6)> _
      Public wReserved3 As UShort
    '
    <System.Runtime.InteropServices.FieldOffsetAttribute(8)> Public llVal As Long
    <System.Runtime.InteropServices.FieldOffsetAttribute(8)> Public lVal As Integer
    <System.Runtime.InteropServices.FieldOffsetAttribute(8)> Public bVal As Byte
    ' and similarly for many other accessors
    <System.Runtime.InteropServices.FieldOffsetAttribute(8)> _
      Public ptr As System.IntPtr

    ''' <summary>
    ''' GetObject: returns a .NET Object equivalent for this Variant.
    ''' </summary>
    Function GetObject() As Object
        ' We want to use the handy Marshal.GetObjectForNativeVariant.
        ' But this only operates upon an IntPtr to a block of memory.
        ' So we first flatten ourselves into that block of memory. (size 16)
        Dim ptr = Marshal.AllocCoTaskMem(16)
        Marshal.StructureToPtr(Me, ptr, False)
        Try : Return Marshal.GetObjectForNativeVariant(ptr)
        Finally : Marshal.FreeCoTaskMem(ptr) : End Try
    End Function
End Structure
A COM value uses the vt field to indicate what type it is. It might be VarEnum.VT_INT, or VarEnum.VT_PTR, or any of the 30 or so VarEnum types. Knowing its type, you can figure out which of the other fields to look up in a giant Select Case statement. Luckily, that Select Case statement has already been implemented in the Marshal.GetObjectForNativeVariant function.

Dumping the Properties of a COM Object
You'll want to dump the properties of your COM object, more or less like the "Quick Watch" window in Visual Studio:
DUMP OF COM OBJECT #28114988
ISpeechVoice.Status = System.__ComObject   As Ref ISpeechVoiceStatus
ISpeechVoice.Rate = 0   As Integer
ISpeechVoice.Volume = 100   As Integer
ISpeechVoice.AllowAudioOutputFormatChangesOnNextSet = True   As Bool
ISpeechVoice.EventInterests = 0   As SpeechVoiceEvents
ISpeechVoice.Priority = 0   As SpeechVoicePriority
ISpeechVoice.AlertBoundary = 32   As SpeechVoiceEvents
ISpeechVoice.SynchronousSpeakTimeout = 10000   As Integer
The thing is, there are lots of different types in COM. It would be exhausting to write code to handle every single case correctly and hard to assemble enough test cases to test them all. Here I'll settle for dumping a small set of types that I know I can handle correctly.
And beyond that, what would be useful to dump? Besides the properties, it would also be useful to get a dump of anything exposed through pure (side-effect-free) functions such as IsTall(). But you wouldn't want to invoke functions such as AddRef(). To discriminate between these two, I reckon that any function name such as "Is*" is fair game for dumping (see Figure 10). It turns out that COM programmers seem to use Is* functions a lot less frequently than do programmers using .NET!
Figure 10 Looking at Get* and Is* Methods
' We'll only try to retrieve things that are likely to be side-effect-
' free properties:

If (funcDesc.invkind And ComTypes.INVOKEKIND.INVOKE_PROPERTYGET) = 0 _
   AndAlso Not funcName Like "[Gg]et*" _
   AndAlso Not funcName Like "[Ii]s*" _
   Then Continue For
If funcDesc.cParams > 0 Then Continue For
Dim returnType = CType(funcDesc.elemdescFunc.tdesc.vt, VarEnum)
If returnType = VarEnum.VT_VOID Then Continue For
Dim returnTypeName = DumpTypeDesc(funcDesc.elemdescFunc.tdesc, typeInfo)

' And we'll only try to evaluate the easily-evaluatable properties:
Dim dumpableTypes = New VarEnum() {VarEnum.VT_BOOL, VarEnum.VT_BSTR, _
           VarEnum.VT_CLSID, _
           VarEnum.VT_DECIMAL, VarEnum.VT_FILETIME, VarEnum.VT_HRESULT, _
           VarEnum.VT_I1, VarEnum.VT_I2, VarEnum.VT_I4, VarEnum.VT_I8, _
           VarEnum.VT_INT, VarEnum.VT_LPSTR, VarEnum.VT_LPWSTR, _
           VarEnum.VT_R4, VarEnum.VT_R8, _
           VarEnum.VT_UI1, VarEnum.VT_UI2, VarEnum.VT_UI4, VarEnum.VT_UI8, _
           VarEnum.VT_UINT, VarEnum.VT_DATE, _
           VarEnum.VT_USERDEFINED}
Dim typeIsDumpable = dumpableTypes.Contains(returnType)
If returnType = VarEnum.VT_PTR Then
    Dim ptrType = CType(Marshal.PtrToStructure( _
      funcDesc.elemdescFunc.tdesc.lpValue, _
                        GetType(ComTypes.TYPEDESC)), ComTypes.TYPEDESC)
    If ptrType.vt = VarEnum.VT_USERDEFINED Then typeIsDumpable = True
End If
In this code, the final kind of dumpable type you consider is a VT_PTR to a VT_USERDEFINED type. This covers the common case of a property that returns a reference to another object.

Using IDispatch.Invoke
The final step is to read the property you've identified by its MEMBERID or invoke the function. You can see the code to do it in Figure 11. The key method here is IDispatch.Invoke. Its first argument is the member id of the property or function that you're invoking. The variable dispatchType is either 2 for a property-get or 1 for a function-invoke. If you were invoking a function that took arguments, then you'd also set up the dispParams structure. Finally, the result comes back in varResult. As before, you can just call GetObject on it to convert the VARIANT into a .NET object.
Figure 11 Read the Property or Invoke the Function
' Here's how we fetch an arbitrary property from a COM object,
' identified by its MEMBID.
Dim val As Object
Dim varResult As New [VARIANT]
Dim dispParams As New ComTypes.DISPPARAMS With {.cArgs = 0, .cNamedArgs = 0}
Dim dispatchType = If((funcDesc.invkind And _
   ComTypes.INVOKEKIND.INVOKE_PROPERTYGET)<>0, 2US, 1US)
idisp.Invoke(funcDesc.memid, IID_NULL, 0, dispatchType, dispParams, _
   varResult, IntPtr.Zero, 0)
val = varResult.GetObject()
If varResult.vt = VarEnum.VT_PTR AndAlso varResult.ptr <> IntPtr.Zero _
   Then
   Marshal.Release(varResult.ptr)
End If
Note the call to Marshal.Release. It's a universal pattern in COM that if a function hands someone a pointer, it first calls AddRef on it, and it's the caller's responsibility to call Release on it. This makes me even happier that .NET has garbage collection.
Incidentally, I could have used ITypeInfo.Invoke instead of IDispatch.Invoke. But it gets a little confusing. Suppose you have a variable, "com," which points to the IUnknown interface of a COM object. And suppose that com's ITypeInfo is a SpeechLib.SpVoice that happens to have a property with member-id 12. You can't call ITypeInfo.Invoke(com,12) directly; you first must call QueryInterface to get com's SpVoice interface, and then call ITypeInfo.Invoke on that. In the end, it's easier to use IDispatch.Invoke.

Now you've seen how to reflect on COM objects through ITypeInfo. This is useful for COM classes that lack interop types. And you've seen how to use IDispatch.Invoke to retrieve values from COM, stored in a VARIANT structure.
I did wonder about creating a full wrapper around ITypeInfo and TYPEDESC, one that inherits from System.Type. With this, users could use the same code for reflection on COM types as .NET types. But in the end, at least for my project, this kind of wrapper was too much work for negligible gain.
For more on what reflection can do for you, see "Dodge Common Performance Pitfalls to Craft Speedy Applications" and "CLR Inside Out: Reflections on Reflection."
Sincere thanks to Eric Lippert, Sonja Keserovic, and Calvin Hsia for their help with this column.

Send your questions and comments to instinct@microsoft.com.

Lucian Wischik is the Visual Basic Specification Lead. Since joining the Visual Basic compiler team he has worked on new features relating to type inference, lambdas, and generic covariance. He has also worked on the Robotics SDK and concurrency, and has published several academic papers on the subject. Lucian holds a PhD in concurrency theory from the University of Cambridge.


Cutting Edge
Managing Dynamic Content Delivery In Silverlight, Part 1
Dino Esposito
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
The Size of Silverlight Applications
Dynamically Generated XAML
Dynamically Generated XAP
On-Demand Content
Caching Downloaded Content
A Tool for Downloads
Downloading XAML-Only Data
Working with XAP Packages
Processing XAP Content
Summing It Up
Users of any rich Internet application (RIA) have valid concerns about security and download size. Silverlight applications are backed by a subset of the full Microsoft .NET Framework and, as such, have the potential to take harmful actions against the local user machine. For this reason, the Silverlight team developed a new security model that makes it impossible for applications to call into any security-critical class in the Core CLR (the Silverlight edition of the .NET Framework). You can read more about the Silverlight security model in the article "Program Silverlight with the CoreCLR" and more about building Silverlight apps in "Get Started Building a Deeper Experience across the Web."
Downloading the Silverlight plug-in is not really an issue, as it takes just a few seconds and is an operation that you run only once. But what about the size of downloaded applications?
In this month's column, I'll tackle the question of Silverlight application downloads. First, I'll illustrate how you can generate XAML dynamically. Next, I'll discuss how to enable application features dynamically on a strictly per-user request basis. Some application-specific features that are heavy to implement with the main stream of code can then be implemented separately and, more important, downloaded separately and still be easily integrated with the primary user interface.

The Size of Silverlight Applications
Once a user has installed the Silverlight plug-in, he has in place all system assemblies that a Silverlight application may require. This means that the download is limited to the assembly that contains the application, plus any referenced custom assemblies. In the end, the size of the application download is often in the tens of kilobytes. Note that this estimation applies to the RTM version of Silverlight 2 and to code compiled in release mode.
Of course, the application might have a much larger footprint, especially if it contains long algorithms, graphic and media content, or animations. To deal with large applications where the download time becomes a problem, you have two main options. One is to stream the Silverlight content as discussed at silverlight.live.com. The other option is to split the application into independent pieces that can be downloaded separately and on demand.

Dynamically Generated XAML
The Silverlight plug-in is essentially designed to display XAML content. If the XAML comes with some codebehind, then the plug-in processes the code to produce the user interface and support any coded behavior or effects. The XAML can be pointed to directly via a URL if it is all that you want to be downloaded; otherwise, you can reference a Silverlight package through the XAP extension.
A XAP package contains a manifest and one or more assemblies. One of the assemblies contains the entry point of the application; other assemblies are just referenced assemblies. The XAML for the user interface is stored in the resources of the entry point assembly. A XAP package is created by the Visual Studio 2008 extension for Silverlight 2 when you create and build the project.
XAML and XAP streams are all that the Silverlight plug-in knows how to handle. To download such content, though, you don't necessarily have to point the plug-in to a physical XAML or XAP resource on the server. You can, for instance, point the plug-in to a URL that can return dynamically generated XAML or XAP content.
Figure 1 shows a sample ASP.NET HTTP handler that returns some XAML created on the fly. The ProcessRequest method sets the content type on the Response object and then writes out some XAML content, such as XAML composed dynamically, based on configuration data, parameters, or runtime conditions. By setting the Expires property on the Response object, you can also prevent caching of the resource on the client. This may be helpful if the content you're serving changes periodically and needs refreshing.
Figure 1 An HTTP Handler that Returns XAML
<%@ WebHandler Language="C#" Class="XamlGenHandler" %>

using System;
using System.Web;

public class XamlGenHandler : IHttpHandler
{
    public void ProcessRequest (HttpContext context)
    {
        // Prevent caching of the response
        context.Response.Expires = -1;

        // Set the type of data we're returning
        context.Response.ContentType = "text/xaml";

        // Create some XAML and return it down the wire
        context.Response.Write("<Canvas xmlns=
            'http://schemas.microsoft.com/client/2007' " +
            "xmlns:x='http://schemas.microsoft.com/winfx/2006/xaml'>" +
            "<TextBlock Foreground='black' Padding='10' FontSize='20'>
             <Run>XAML content</Run><LineBreak/>" +
            "<Run>[generated " + DateTime.Now.ToLongTimeString() + "]</Run>" +
            "</TextBlock></Canvas>");
    }

    public bool IsReusable
    {
        get {return true;}
    }

}

Dynamically Generated XAP
Returning a dynamically generated XAP package is not much different from returning raw XAML text, except that a XAP package is not a plain text file. A XAP package is a ZIP file that contains an XML manifest and one or more assemblies. By using a package format, the team minimized the number of round-trips necessary to download the entire content required by a Silverlight application. Figure 2 shows an ASP.NET HTTP handler that writes the content of a XAP file out to the HTTP response stream.
Figure 2 An HTTP Handler that Returns a XAP Package
<%@ WebHandler Language="C#" Class="XapGenHandler" %>

using System;
using System.Web;

public class XapGenHandler : IHttpHandler
{
    public void ProcessRequest (HttpContext context)
    {
        // XAP file to return
        string xapFile = "...";

        // Set the type of data we're returning
        context.Response.ContentType = "application/octet-stream";

        // Create some XAML and return it down the wire
        content.Response.WriteFile(xapFile);
    }

    public bool IsReusable
    {
        get {return true;}
    }

}
The sample code reads the XAP data from an existing file. It goes without saying that if you embed a ZIP library in your project, you could easily assemble the package on the fly by combining different DLLs and then creating a proper XML manifest file.
If you're returning XAP content, you set the content type of the response to application/octet-stream—the MIME type that commonly identifies generic binary content.
To associate the plug-in with an HTTP handler, or any other endpoint you choose, you use the usual Silverlight programming techniques. For example, you can use the Silverlight server control in an ASP.NET page:
<asp:Silverlight ID="Xaml1" runat="server"
    Source="~/xap.ashx"
    MinimumVersion="2.0.30523"
    Width="100%"
    Height="100%" />
In both examples, the factory for the Silverlight application lives on the Web server. This is an excellent approach if the host page needs to figure out dynamically which content to download.
However, this is just one possible scenario. There's another scenario that is probably more common—the need to download optional components for the current Silverlight application. In this case, the logic to select and download external content is all running on the client in the Silverlight plug-in.

On-Demand Content
Silverlight 2 provides a rich and powerful API to download code and/or XAML on demand that you can use for downloading content and inserting it into the existing XAML document object model.
All visual elements of a XAML tree feature a property called Children that you can use to programmatically add or remove child elements of any size. For example, you can append an entire user control downloaded from the same server or even from a trusted, opt-in remote server. The following line shows an example:
StackPanel1.Children.Add(downloadedContent);
The user control represented by the argument is added to the Children collection of a StackPanel XAML element. The rendering is immediate, and the user interface is updated in real time.
You can do much more than just download content and attach it to the existing document object model. You can, for example, cache it locally in the local storage of the application and check your own cache for on-demand content before you go for a new request to the server.
This approach gives you permanent storage of the downloaded content. However, in some cases this may be overkill. Another, simpler option exists that doesn't require any extra work: letting the browser cache the XAP resource for you.

Caching Downloaded Content
The XAP package that you get from the Web server has no special meaning to the browser. The browser, therefore, will cache it as it caches anything else it gets from a Web server, adhering to the request cache policies determined by the cache-control and "expires" HTTP header in the request or similar meta tags in the host HTML page.
Note that when you have a XAP resource to be downloaded in the browser, you can control caching via the settings in the page that you typically insert using meta tags or ASP.NET directive attributes. If the XAP resource is to be downloaded via an HTTP handler as in the previous example, then you can control caching for the specific request.
Another point that is worth noting is that it is the original XAP content, including assemblies and XAML, that is cached. Therefore, the running application can programmatically modify the original XAML. However, such changes won't be cached automatically, and likewise, any resource you may extract from a XAP package (media, images, and so on) is not cached separately. Thus, whenever the user visits the page, the XAP package is not downloaded again (unless it is expired), but any resources will be extracted again. Furthermore, any changes you may have made to these resources in previous sessions are lost. To persist changes to the XAML document object model, you have to arrange your own tailor-made cache. (This is a cool technique that I will cover in Part 2 of this topic.)
Finally, note that the XAP package saved in the browser's cache is at the mercy of the user. Should the user decide to clear the cache at some time, anything located there will be lost—including your XAP packages. To permanently store Silverlight XAP packages, you have to resort to isolated storage (this topic is also slated for Part 2).

A Tool for Downloads
When you write Silverlight applications, remember that every resource you need that is not already packed in the application's XAP must be explicitly downloaded from the server. The WebClient class is the primary Silverlight tool you want to use to arrange downloads of additional resources. The class provides a few asynchronous methods for sending data to, and receiving data from, a Web resource. Here's how it works:
WebClient wc = new WebClient();
wc.DownloadStringCompleted +=
     new DownloadStringCompletedEventHandler(callback);
wc.DownloadStringAsync(address);
The DownloadStringAsync method operates an HTTP GET and captures the URL response as a string. The string is received by the associated callback, as shown here:
void callback(object sender, DownloadStringCompletedEventArgs e)
{
  if (e.Error != null)
     return;

  string response = e.Result;
...
}
As you'll see in a moment, this method is perfect for downloading plain XAML with no attached codebehind. To programmatically download binary data, such as a XAP package, you need a stream and a slightly different approach. The WebClient class is still helpful, as it provides an apt method in OpenReadAsync:
WebClient wc = new WebClient();
wc.OpenReadCompleted +=
     new OpenReadCompletedEventHandler(callback);
wc.OpenReadAsync(address);
The structure of the associated callback is the same in the previous case. In the end, you use the DownloadStringAsync method to get a simple string; you use the OpenReadAsync method to get a stream of any data. Whether you decide to download a string or a stream is mostly a matter of preference and depends essentially on how you plan to use the data you're receiving.
Note also that WebClient provides a pair of methods for writing to a remote URL: UploadStringAsync for posting a string and OpenWriteAsync for uploading any data to a URL using a stream.
You can use the Headers property to specify additional headers because the class, by default, doesn't specify any. You should note, though, that some of the headers you may set are stripped off by the Framework and managed internally. These include Referer, Connection, and User-Agent. The Content-Type header, if set, is preserved.
When it comes to downloading a resource, the WebClient class transparently uses the browser's cache before attempting a connection. If the XAML or XAP resource is not in the cache, then the class proceeds with the download. The process of downloading content from a Silverlight application results from the combined efforts of the Silverlight runtime and the internal API that the host browser exposes to a plug-in. This means that under the hood of WebClient, the Silverlight runtime talks to the browser sandbox to check whether the requested resource is already in the cache. If not, Silverlight proceeds with its own security policies to authorize the request. When data is finally returned from the endpoint, the Silverlight runtime caches it locally through the browser's services and in full respect of ongoing caching policies.
A number of security restrictions apply to the WebClient class and other HTTP classes in the Silverlight System.Net namespace. In particular, the WebClient class supports only HTTP and HTTPS schemes when downloading via stream and the FILE scheme when downloading pure XAML. Cross-scheme access is always prohibited, so you can't point WebClient to, say, an HTTPS resource if the host page downloaded via HTTP (and vice versa). A WebClient request can, in general, reach a URL in a different browser's zone, but not if it attempts to move from an Internet zone to a more restrictive zone. Silverlight currently supports zones only on a Windows operating system.
Finally, cross-domain access is supported only if the remote site opted in by hosting a proper XML file in its root directory. Note also that cross-domain access doesn't work in an HTTPS-to-HTTPS scenario.
The same WebClient object can't handle multiple requests simultaneously. You should check the IsBusy property—a Boolean value—to determine whether it is safe for your code to place a new request via the same WebClient instance. If you use multiple WebClient objects—perhaps on different threads—you can start multiple downloads at the same time.

Downloading XAML-Only Data
Let's see how to use the WebClient to download XAML data and integrate it in the visual tree. In Silverlight 2 applications, the dynamic download of plain XAML data doesn't necessarily give you the programming power you need. The XAML string must be plain XAML with no references to be resolved at run time ,such as bindings or references to styles, resources, and events.
Once the XAML string is downloaded, you use the XamlReader class to turn it into a UI element that can be added to the existing document object model. The following code shows how to programmatically download a XAML string from a URL. Note that you need to provide the URL as a Uri object:
WebClient client = new WebClient();
client.DownloadStringCompleted +=
   new DownloadStringCompletedEventHandler(OnDownloadCompleted);
Uri uri = new Uri("xaml.ashx", UriKind.Relative);
client.DownloadStringAsync(uri);
The URL can point to a plain XAML resource or to an endpoint that returns a text/xaml response. Here's the code to process the downloaded XAML and append it to a placeholder in the visual tree:
void OnDownloadCompleted(object sender, DownloadStringCompletedEventArgs e)
{
    // Parse XAML to a UI element
    string xaml = e.Result;
    UIElement dom = XamlReader.Load(xaml) as UIElement;

    // Append to the DOM
    Placeholder.Children.Clear();
    Placeholder.Children.Add(dom);
}
As mentioned, a placeholder may be any element in the document object model currently rendered in the plug-in. Note that the children of a UI element form a collection, and they are rendered as a sequence. This means that to avoid undesired overlapping of elements, in the case of updates you should first remove elements and then add.
XAML serialization, which is performed through the XamlReader and XamlWriter classes, is based on the principle that extension references are de-referenced and runtime values are saved over design-time settings. What if you want to download XAML content and customize it before display, such as through dynamic data binding? You can't embed bindings in the XAML source, but you can define placeholders in the downloaded XAML, retrieve them with parsing, and set them to any value you want programmatically. However, in Silverlight 2, downloading a XAP package is definitely a better solution.

Working with XAP Packages
A XAP package contains an entire Silverlight application whose user interface consists essentially of a user control, which is really just a container of XAML markup and code.
As mentioned, a XAP package includes one or more assemblies tracked by a manifest file. Processing a XAP package entails two extra steps beyond download. You need to extract the main assembly and then instantiate the entry point class that starts up the downloaded application. Needless to say, in this case you can use binding, styles, events, and whatever else you want in the XAML. When you work with a XAP package, it's the Silverlight runtime—not the serialization API—that processes the XAML and then resolves references. This makes a huge difference in terms of programming power.
Downloading and processing a XAP package requires a bit of more work than just building an object model out of a string. Some of this work—typically the content download and assembly extraction—can be moved to a reusable downloader class (see Figure 3).
Figure 3 Downloading a XAP Package Dynamically
public partial class Page : UserControl
{
    private UIElement content = null;
    private TabItem item = null;

    public Page()
    {
        InitializeComponent();
    }

    private void chkNewContent_Click(object sender, RoutedEventArgs e)
    {
        bool shouldDisplay = (sender as CheckBox).IsChecked.Value;

        if (shouldDisplay)
        {
            if (!IsContentAvailable())
                DownloadContent();
            else
                ShowContent();
        }
        else
        {
            HideContent();
        }
    }

    private bool IsContentAvailable()
    {
        return (content != null);
    }

    private void DownloadContent()
    {
        Downloader dl = new Downloader();
        dl.XapDownloaded +=
            new EventHandler<XapEventArgs>(OnPackageDownload);
        dl.LoadPackage("more.xap", "more.dll", "More.ExtraTab");
    }

    void OnPackageDownload(object sender, XapEventArgs e)
    {
        content = e.DownloadedContent;
        ShowContent();
    }

    private void HideContent()
    {
        this.TabList.Items.Remove(item);
    }

    private void ShowContent()
    {
        item = new TabItem();
        item.Header = "Extra tab";
        item.Content = content;
        this.TabList.Items.Add(item);
    }
}
The code in Figure 3 shows a sample tab-based application that loads a new tab the first time the user clicks on a checkbox. In this case, a new XAP package is downloaded, and the user control contained in its user interface is inserted in a newly created TabItem. Hiding the content from the newly downloaded package is a simple client operation. And showing it again doesn't require a second round-trip for two good reasons: the content is cached in memory, and the package from which it was built is cached in the browser's cache.
Figure 4 shows the user interface of this sample application. The content inserted in the additional tab comes from a new Silverlight application project. In terms of deployment, all you have to do is store the XAP package in the ClientBin folder of the host Web site (the default location where WebClient looks for relative content) or in any other location that is safe for WebClient to reach.
Figure 4 An Application Capable of Downloading Parts of UI Dynamically
Now, let's explore the code of the helper Downloader class. Note that the Downloader class used in this example is an internal class and has nothing to do with the Silverlight Downloader object available to JavaScript callers. The JavaScript downloader object is essentially a script-callable wrapper for WebClient.

Processing XAP Content
The Downloader class uses WebClient to download a XAP package and then extracts the assembly that contains the application from the zipped stream and instantiates the specified root class. All this logic is hidden behind a relatively simple programming facade made of a LoadPackage method and a XapDownloaded event. The method has the following signature:
public void LoadPackage(
    string xapURL, string assemblyName, string className);
It receives the URL to the package, the name of the assembly in the package to extract, and the name of the class to instantiate. As mentioned, this class is the codebehind of the XAML interface file.
The XapDownloaded event is fired when the downloader class has finished with the XAP processing and has the user control ready for the client application. Here's the definition of the event:
public event
    EventHandler<XapEventArgs> XapDownloaded;
The event argument class returns the following information:
public class XapEventArgs : EventArgs
{
  public UserControl DownloadedContent;
}
Let's dig out the logic of the LoadPackage method. The method employs the binary interface of WebClient to point to the XAP package:
void LoadPackage(string package, string asm, string cls)
{
    assemblyName = asm;
    className = cls;

    Uri address = new Uri(package, UriKind.RelativeOrAbsolute);
    WebClient client = new WebClient();
    client.OpenReadCompleted +=
       new OpenReadCompletedEventHandler(OnCompleted);
    client.OpenReadAsync(address);
}
The download proceeds asynchronously and fires the OpenReadCompleted event when it is finished. Figure 5 shows the implementation for the handler of the event.
Figure 5 Download Complete
void OnCompleted(object sender, OpenReadCompletedEventArgs e)
{
    if (PackageDownloaded == null)
        return;

    if (e.Error != null)
        return;

    // Load a particular assembly from the XAP package
    Assembly a = GetAssemblyFromPackage(assemblyName, e.Result);

    // Get an instance of the XAML object
    object page = a.CreateInstance(className);

    // Fire the event
    XapEventArgs args = new XapEventArgs();
    args.DownloadedContent = page as UserControl;
    XapDownloaded(this, args);
}
If no error occurred, a helper function extracts the specified assembly from the package and gets an instance of the user control class in the codebehind of the XAML block to add. Next, a custom event is fired to the caller so that the XAML subtree can be processed. At this point, one thing remains to expand on: the details of how an assembly is extracted from the zipped stream downloaded from the URL. This code is shown in Figure 6. This is boilerplate code you use to extract an assembly from a XAP package. StreamResourceInfo and Application.GetResourceStream are also common tools for extracting resource packages such as images or media content from the current XAP.
Figure 6 Extracting an Assembly from a XAP Package
Assembly GetAssemblyFromPackage(string assemblyName, Stream xap)
{
    // Local variables
    Uri assemblyUri = null;
    StreamResourceInfo resPackage = null;
    StreamResourceInfo resAssembly = null;
    AssemblyPart part = null;

    // Initialize
    assemblyUri = new Uri(assemblyName, UriKind.Relative);
    resPackage = new StreamResourceInfo(xap, null);
    resAssembly = Application.GetResourceStream(resPackage, assemblyUri);

    // Extract an assembly
    part = new AssemblyPart();
    Assembly a = part.Load(assemblySri.Stream);
    return a;
}

Summing It Up
Silverlight applications are expected to download and install quickly and run fast on the user's machine. Managed code outperforms interpreted JavaScript, so performance will indeed be snappy. But because downloading large Silverlight applications, or applications that require large external graphic and media content, can create undesired latency, breaking the download into steps helps. It is even better if you program these steps to occur on demand.
The WebClient class offers an effective and asynchronous programming API to download any sort of resources you can access over the Internet. The extensible model for the Silverlight object model allows you to incorporate any changes in the existing structure quickly. Finally, the stream-based API centered around the StreamResourceInfo class makes it easy to extract content from the resources of assemblies and XAP packages.
Any downloaded resource is cached by the browser. This sort of caching is not permanent, though. In Part 2 of this column, I'll use isolated storage to make any downloaded content persist longer on the user's machine, thus saving many potential round-trips. For more Silverlight goodies, see Jeff Prosise's Wicked Code column "Silverlight Tips, Tricks, and Best Practices."

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is an architect at IDesign and the coauthor of Microsoft .NET: Architecting Applications for the Enterprise (Microsoft Press, 2008). Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


Test Run
Analyzing Project Exposure And Risk Using PERIL
Dr. James McCaffrey
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Meta-Risks
Risk Identification
Risk Analysis
Wrapping Up
All software projects face risks. A risk is an event, which may or may not occur and which causes some sort of loss. The relationship between risk and software testing is straightforward. Because, except in rare situations, you cannot exhaustively test a software system, risk analysis reveals issues that can cause the most loss. You can use this information to help prioritize your testing effort. In this month's column, I present practical techniques you can use to identify and analyze the risks involved with a software project. Let's begin.
Imagine a hypothetical situation in which you are developing an ASP.NET Web-based application of some sort. Figure 1 illustrates some of the key ideas and problems associated with software risk analysis. The overall process of risk analysis involves identifying your risks, estimating how likely each risk is, determining the loss associated with each risk, and combining likelihood and loss information into a value called risk exposure.
According to the example data in Figure 1, the risk that "User IDs can be viewed" has the highest exposure, so other factors being equal, you would likely want to make it a top priority to test for this to prevent the risk from becoming a reality. But how can you identify risks? How can you estimate risk probability and loss? Can you still perform risk analysis if you cannot estimate risk probability and loss?
Although there have been several efforts to formalize and standardize risk analysis terminology, in practice different terms tend to be used in different problem domains. I will use the term "risk analysis" to mean computing risk exposure by multiplying the risk probability or likelihood by the risk loss, or to mean the overall process of identifying, analyzing, and managing software project risk.
In spite of the fact that risk analysis is a critically important part of software development, based on my experience, many risk analysis techniques are not widely known in the software testing community. If you search the Web, you will find tens of thousands of references on software risk analysis. However, the majority of these references either address risk analysis at a very high level and do not present practical techniques or present just one specific risk analysis technique and do not explain how the technique fits into an overall risk analysis framework. I'll present you with both an overview of risk analysis and useful techniques you can employ immediately in your software development environment.
In the sections of this column, I describe two meta-risks that are common to all software projects. Then I present three ways you can identify specific risks associated with your software project and three ways to analyze risk. In particular, I will introduce you to an interesting new risk analysis technique called Project Exposure using Ranked Impact and Likelihood (PERIL) that is especially useful in a software development environment. I conclude with a brief discussion of risk management. I believe that the techniques presented here will become an extremely valuable addition to your software testing, development, and management tool kit.

Meta-Risks
Two special types of software project risk analyses are what I call time and cost meta-risk analyses. Traditional project management defines a concept which has different names including "the project management triple constraint" and "the project management triangle." The idea is, in short, that virtually every project has three limiting factors: cost, schedule, and scope. Cost is how much money you have to spend on the project, schedule is how long you have to complete the project, and scope is the set of required project features and their quality.
The three project constraints have a variety of aliases. For example, cost is also referred to as budget or money. Schedule is often called time or duration. And scope is sometimes called features, or quality, or even features/quality. Notice that this last features/quality constraint can be (and often is, in fact) considered to be two distinct constraints.
A key notion is that a change in any one of the constraints will likely cause a change in one or both of the other two constraints. For example, if you are developing a software application, and you are suddenly required to finish the project in a shorter amount of time than originally planned, you probably will have to spend more money (for example, to buy additional resources or outsource part of the project) or cut some features or quality. If the budget for your project is cut, then you will likely have to extend the time to finish the project, remove some features, or lower the quality of your project. Using the project management triangle paradigm, because software testing is designed to improve the quality of a system, it follows that the two highest levels of risk in a software project are that the project does not finish on time and that the project runs over budget.
There is a relatively simple but effective way to estimate the overall schedule and cost risks to a software project. Let's look just at the time/schedule meta-risk (analyzing budget/cost risk works in exactly the same way). The first step in a high-level meta-risk analysis is to break your overall project into smaller, more manageable chunks of activities.
For example, suppose you are working on a small Web application project, and the project must be completed in 30 working days. You begin the meta-risk analysis by listing all the activities involved in the project. The most common approach here is to create what is called a work breakdown structure (WBS), which you can see in Figure 2. You create a top-level task that consists of the entire project. Then you break that task into a number of smaller sub-tasks, typically about three to seven tasks. You repeat the process, decomposing each sub-task into smaller sub-tasks until you reach the appropriate level of granularity for your environment.
Figure 2 Work Breakdown Structure
The bottom level, leaf node tasks are sometimes called work packages. Exactly how you decompose your tasks and how granular you make your WBS depends on a number of factors. For example, in an agile development environment, you may well decide that a simple two-level deep work breakdown structure meets your needs. Or, if you are working on a very large, complex software project using a traditional software development lifecycle methodology, you may have tens of thousands of work packages.
You can create a small WBS by hand, use generic productivity tools such as Microsoft Office Excel, or use sophisticated tools such as Microsoft Office Project. A WBS does not contain sequencing, time, or cost information. In other words, a WBS tells you what must be done, but not in what order, and does not tell you how long each task will take or how much each will cost. After you create your work breakdown structure, typically the next step is to use the work packages to create what is called a precedence diagram.
Figure 3 Precedence Diagram
The precedence diagram adds sequencing information. The diagram shown in Figure 3 indicates that the Requirements task must be completed before the Database Back End task, which in turn must be completed before both the Middle-Tier task and the Front-End task can begin. These last two tasks can be performed in parallel according to the precedence diagram. Finally, both the Middle-Tier task and the Front-End task must be completed before the Deploy Application task can begin.
After creating a precedence diagram with its sequencing information, the next step in a time meta-risk analysis is to estimate the time required for each individual work package. Although you can estimate each time as a single data point, a better approach is to supply three estimates—an optimistic estimate, a best-guess estimate, and a pessimistic estimate.
OK, but just where do such estimates come from? Determining time and cost estimates is by far the most difficult part of software project meta-risk analyses. There are many ways to estimate activity time and cost. You can use historical experience, educated guesses, sophisticated math models, and so on. The techniques you use will depend upon your particular situation. Regardless of the method you use, estimating the time and cost of a set of smaller activities is much easier than estimating the time and cost of one monolithic activity. The table in Figure 4 shows an example time risk meta-analysis.
When you are analyzing optimistic, best-guess, and pessimistic time data, you usually use a simple mathematical distribution called the Beta distribution. The mean, or average, of a Beta distribution is computed as this:
(optimistic + (4 * best-guess) + pessimistic) / 6
So for the Deploy Application task, the mean estimated time for completion is:
mean = (3 + 4*8 + 13) / 6
     = 48 / 6
     = 8.0 days.
Notice that a Beta mean is just a weighted average with weights 1, 4, and 1. Therefore, the variance of a Beta distribution is given by this formula:
((pessimistic - optimistic)/6)²
So for the Deploy Application task the variance is:
variance = ((13 - 3) / 6)²
         = (10/6)²
         = (1.6667)²
         = 2.78 days²
The overall standard deviation for the project is the square root of the sum of the activity variances. Thus, in this example, the equation looks like the following:
std. deviation = sqrt(5.44 + 1.78 + 2.25 + 2.78)
               = sqrt(12.25)
               = 3.50 days
Notice that my calculations do not use the data for the Design and Code Logic Middle-Tier activity. Because the Logic Middle-Tier activity can be performed in parallel with the Database Back-End activity, and the Front-End activity cannot begin until both parallel activities have finished, the shorter parallel activity (Logic Middle-Tier) does not explicitly contribute to the overall time to complete the project.
This type of analysis is called the critical path method (CPM) and is a standard project management technique. With the schedule mean and variance data computed, you can now compute the probability that the entire project will take longer than 30 days to complete:
z = (30.00 - 28.83) / 3.50 = 0.33
p(0.33) = 0.6293
p(late) = 1.0000 - 0.6293 = 0.3707
First, you compute the so-called z-value, which is equal to the amount of time you have scheduled to finish the project (in this case, 30 days) minus the estimated time to completion (28.83 days), divided by the total task standard deviation (3.50 days). Then you take the z-value (0.33) and look up the corresponding p-value in a Standard Normal Distribution table or use the Excel NORMSDIST function (0.6293). Finally, you subtract the p-value from 1.0000 to get the probability that your project will go over schedule.
You are performing a one-tail analysis here because you only care if you take longer than scheduled and don't care if you take less time. With this example data, the probability that the Web application takes longer than 30 days to complete is 0.3707, or nearly 40%—a rather risky situation. If you think about it for a moment, this result should make sense. Your planned schedule of 26 days of development is too close to the project limit of 30 days and so you may not have enough wiggle room to account for schedule variances.
Obviously, your meta-risk results are only as good as your input data—your time estimates in this case. If your input estimates are wrong, then no amount of statistical analysis can produce meaningful results. You can compute the meta-risk probability that your project will go over budget using the same technique as I've just demonstrated for schedule. Once you have the probability that your project will go over schedule, you can compute the risk exposure for the meta-risk if you can estimate the monetary loss due to being late.
In some situations, you may be under a contract to create a software system, and the contract may have well-defined and significant late penalties. For example, suppose your contract states that there is a $10,000.00 penalty for late delivery. Your meta-risk exposure is $10,000.00 * 0.3707 = $3,707. In other cases, the cost of a late software project is too difficult to estimate beyond "very, very expensive."
But notice that even without computing a risk exposure, your time meta-risk analysis yields useful information. If you examine the data in Figure 4, you can see that Determine Requirements task has the greatest schedule variance. Therefore, simply applying additional resources early on in the project can reduce your task variance, which in turn will reduce the probability of going over schedule.

Risk Identification
Unlike time and cost meta-risks, where the risk events can be determined in a somewhat step-by-step (although by no means easy) way by iteratively decomposing tasks into smaller subtasks, risk identification in the general case is much less mechanical. In a software development and testing environment, there are three main approaches to risk identification: taxonomy-based, scenario-based, and specification-based.
A taxonomy is just a classification list. Consider the following analogy. You are going to take a trip on an airplane, so you use a standard reminder list that you use before every trip. The list contains statements or questions such as, "Do I have my ID?" and "Have I checked to see if my flight is on time?"
Over the years, many people and organizations have created software risk taxonomies. One such list was created by Barry Boehm, an early pioneer and a well-known researcher in the area of software project risk. In 1989 Boehm identified a top 10 software risk taxonomy, and he updated the list in 1995. The 1995 version of the top 10 software project risk taxonomy is listed here:
  1. Personnel shortfalls
  2. Schedules, budgets, process
  3. Commercial off-the-shelf software, external components
  4. Requirements mismatch
  5. User interface mismatch
  6. Architecture, performance, quality
  7. Requirements changes
  8. Legacy software
  9. Externally performed tasks
  10. Straining computer science
It should be apparent to you that Boehm's top 10 risk list does not immediately identify risks. Rather, the taxonomy merely provides you with a starting point to begin thinking about risks that apply to your software project. For example, the first risk, "Personnel shortfalls," encompasses have many different possible risks related to staffing. Your project simply may not have enough engineers to create your application or system. Or a key engineer may leave the project halfway through the project's schedule. Or the engineering staff may not have the technical skills needed for the project. And so on.
Most of the top 10 risk categories should be familiar to you, except for perhaps the 10th risk category, "straining computer science." This is somewhat of a catch-all category and covers tasks related to things such as technical analysis, cost-benefit analysis, and prototyping.
Another commonly used software risk taxonomy list was created by the Software Engineering Institute (SEI). The SEI is one of 36 federally funded research and development centers in the U.S. These research centers are rather strange hybrid organizations which are funded by public money, but sell products and services. The SEI software risk taxonomy was created in 1993 and consists of approximately 200 questions. For example, question #1 is, "Are the requirements stable? If no, what is the effect on the system (quality, functionality, schedule, integration, design, testing)?" Question #16 is, "How do you determine the feasibility of algorithms and designs (prototyping, modeling, analysis, simulation)?" You can find the SEI risk taxonomy in an appendix to the document.
In scenario-based software risk identification, you imagine yourself in different roles, create scenarios for those roles, and identify what could go wrong in each scenario. Using the airplane trip analogy I described previously, you might mentally trace the steps you will be taking on your trip. For example, "First I drive to the airport. Then I park my car. Next, I check in at the airline counter." This scenario process could reveal many risks including traffic delays due to road construction or an accident, parking unavailability, forgetting your ID, and so on.
In a software project environment, some common roles used for scenario-based risk identification are users, developers, testers, sales people, software architects, and project managers. A user scenario might be something along the lines of, "First, I install the application. Next, I launch the application." In many cases a risk identification scenario maps directly to a test case.
Scenario-based risk identification roles are not necessarily people. Roles can be software modules or subsystems too. For example, suppose you have some C# object that performs encryption and decryption. You can imagine that the object is the role and create scenarios such as, "First I accept some input and instantiate myself. Next I accept some input and pass it to my encrypt method." There has been less research on scenario-based software risk identification than on taxonomy-based identification. The research paper found at Risk Identification Patterns for Software Projects presents a good overview of the field and proposes an interesting, theoretical, pattern-based approach to risk identification.
In addition to taxonomy-based and scenario-based risk identification strategies, a third approach is a specification-based strategy. In this approach you closely examine each feature and process in your product or system specification documents and attempt to identify what can go wrong. Using the airplane trip analogy, you might carefully examine a detailed trip itinerary which was created by a travel agent. Imagine that one of your specification documents for a Web application states that you intend to use an outside contractor to produce the various Help files for the application. An external project dependency can give rise to a long list of risks. What if the contractor fails to deliver on time? What if the contractor's work quality does not meet your subjective standards?
There is no single, optimal risk identification strategy, as each has pros and cons. Risk taxonomies are an excellent way to begin the process of identifying the risks in your software project. They provide a somewhat mechanical way to get started in the sense that you simply start examining each question or statement in the taxonomy. Taxonomies also help you to distribute the risk identification process among several people by assigning different people to different taxonomy questions. On the negative side, using taxonomies for risk identification can be very time consuming. Also, taxonomies are, by their nature, generic and so they cannot identify risks that are specific to your software system unless you put in the effort to discover these specific risks.
Compared to taxonomy-based risk identification, an advantage of a scenario-based approach is that it tends to be less generic and forces you from the beginning to be more definite. On the other hand, scenario-based risk identification is somewhat more art than science and you can easily miss a key scenario. Specification-based risk identification is usually the least generic, most specific approach. However, a specification-based approach will yield results only as good as your specification documents. When used together, the three approaches give you a good chance of accurately identifying your software risks.

Risk Analysis
Risk analysis is the process of combining the probability (or likelihood) of a risk event with the monetary loss (or negative effect) that occurs if the event happens, to produce a value that can be used to compare and prioritize the risk against other risks. In this section, I present two older approaches to risk analysis (the expected value technique and the categorical technique), and one new approach called PERIL. Let's look at the expected value technique first.
Take a look at the example shown in Figure 5. Suppose you have identified four risk events. Let's call them Risk A, Risk B, Risk C, and Risk D. You assign probabilities to each risk event. A probability is a number between 0.00 (meaning impossible) and 1.00 (meaning certainty) that indicates how likely the event is. Next, you assign a monetary loss value to each risk event, which is the cost to you if the risk event occurs. Now for each risk event you simply multiply the risk's probability and the risk's loss to get the risk exposure.
Using this method, risk exposure is just a form of expected value. Obviously, there are several major problems with the expected value approach. How can you estimate risk probabilities? How can you estimate a risk loss? In some situations you may have good historical data or experience to base your estimates upon, but this is generally a rare situation when creating software. Based on my experience, the expected value approach to risk analysis is often not feasible in a software development environment.
Because it is difficult or even impossible in many software development environments to estimate the probability of a risk event or its associated loss, a common alternative is to use categorical scales for both risk probability and risk loss. This is the categorical technique. An example will make the idea clear. Suppose you have identified four risks, A, B, C, and D. Now instead of guessing at a probability and a loss for each risk, you generate a categorical risk exposure table like the one shown in the top part of Figure 6.
As you can see, I have a total of nine categories of risk exposure. There are three categories of risk probability—Low, Medium, and High. There are three categories of loss—also Low, Medium, and High. The cross product of probability category and loss category yields nine risk exposure categories, from Low-Low (low probability of a low loss) through High-High (high probability of high loss). Now I can look at each of my four risk events, assign a Low, Medium, or High probability, and then a Low, Medium, or High loss, to yield a nine-point risk exposure. The idea is that it is often more reasonable to assign a probability value of "Low" instead of an exact numeric value like 0.05 for example.
The hypothetical data in the table in the bottom part of Figure 6 suggest that Risk B has the highest exposure and may warrant more attention or resources (including testing) than Risk A, which has the lowest exposure. Although a categorical risk analysis approach somewhat eases the problem of assigning difficult or impossible-to-determine probabilities and loss information, the technique introduces new problems.
Notice that I arbitrarily use three categories for both probability and loss. This is a very coarse approach. But suppose I decide to improve my risk analysis by using five categories for both the probability factor and the loss factor: Very Low, Low, Medium, High, and Very High. Now I would end up with a total of 25 risk exposure categories—(Very Low + Very Low) through (Very High + Very High). How would I rank or compare these 25 exposure values? Just how does a (Very Low + High) risk exposure compare to a (High + Medium) exposure? If multiple people are evaluating your categorical risk exposure data, would they interpret the exposure data in the same way?
To address the problems with a purely categorical risk analysis approach, several years ago I developed a technique I call Project Exposure using Ranked Impact and Likelihood (PERIL). The essence of the idea is to use categories (as in the categorical approach) but convert them into a quantitative scale so they can be easily combined (as in the expected value approach) to produce numeric exposure metrics.
Let me show you an example. Suppose you have identified four risks: A, B, C, and D. Now suppose you decide that trying to assign meaningful numeric values to each risk's probability and loss is just not feasible. Additionally, you decide that in your particular environment it makes sense to categorize risk likelihood into five categories: Very Low, Low, Medium, High, and Very High. Next, you determine that you will categorize loss/impact on a four-point scale: Very Low, Low, High, and Very High. The PERIL technique maps categorical data onto a quantitative scale using a simple mathematical construct called rank order centroids. The mapping technique is best explained by example. For the five-category likelihood scale my five rank order centroid mappings are shown in Figure 7.
Similarly, my four-category impact mappings are computed, as shown in Figure 8. Now I can combine each risk's likelihood and impact centroid value to compute the risk's exposure by multiplying. For example, look at Figure 9. Here, Risk D has High likelihood, which maps to 0.25667, and Low impact, which maps to 0.14583, so the exposure is 0.25667 * 0.14583 = 0.03743. From this data I conclude the Risk C clearly has the highest exposure, and I would look at ways to prevent the risk from occurring and create a contingency plan if the risk event does occur.

Instead of computing each risk's exposure individually, I can construct a complete PERIL exposure lookup table for five likelihood levels and four impact levels, and then simply read PERIL exposure values from the table, as illustrated in Figure 10. The PERIL technique generalizes to any number of likelihood and impact categories.
Rank order centroids map ranks (such as first, second, third) to numeric values (such as 0.61111, 0.27778, 0.11111). Notice that rank order centroid values are normalized in the sense that they sum to 1.0 (subject to rounding error). Expressed in sigma notation, if N is the number of categories then the numeric value corresponding to kth category is:
There are many other mathematical mappings between categories and numeric values, but there is some research which suggests that using rank order centroids is a very good way to map rankings such as the ones used in risk analysis. A complete discussion of rank order centroids is outside the scope of this column, but consider this informal argument. Suppose you are dealing with just two categories of risk event likelihood: High and Low. Presumably the High-risk event likelihood has a probability of occurring that is greater than 0.5 and therefore the Low-risk event likelihood has a probability of less than 0.5. Without any additional information, you can assume that the High event likelihood is halfway between 0.5 and 1.0 and so equals 0.75. In the same way, the Low event likelihood is halfway between 0.0 and 0.5 which is 0.25. These two values, 0.75 and 0.25, are rank order centroids for N = 2 categories (as shown in Figure 11).
Notice that when using PERIL, I use the terms likelihood and impact rather than probability and loss. PERIL likelihood and impact are relative, normalized values. Even though PERIL likelihood values sum to 1.0 just as a probability set does, let me emphasize that PERIL likelihood values are not probabilities. Similarly, PERIL impact values have meaning only when compared with each other and are not monetary loss values.
The three techniques to determine risk exposure—the expected value technique, the categorical technique, and the PERIL technique—have strengths and weaknesses. If you have solid historical data that allows you to estimate the probabilities of risk events and the monetary loss associated with each event, then the expected value technique is usually the best approach. However, in a software development and testing environment you rarely have enough data to make meaningful probability and loss estimates.
At the other extreme, if you have virtually no historical risk data, then the categorical technique, with two or three categories of risk probability and associated risk loss, is a reasonable approach. In situations where you are able to categorize risk event likelihood and associated risk impact (which can be monetary or non-monetary loss such as morale effect) into roughly five categories, then the PERIL technique is often a great choice.
Regardless of which of the risk analysis techniques you decide to use in your environment, you must take care to interpret your results cautiously. Keep in mind that risk analysis almost always has very large amounts of variability in the input estimates. In other words, risk analysis gives you guidelines, not rules, to prioritize your software testing efforts.
Risk Resources

For more on risk, see these MSDN Magazine columns:
"Test Run: Competitive Analysis Using MAGIQ"
"Test Run: The Analytic Hierarchy Process"

Wrapping Up
One part of the overall risk analysis process that I did not discuss in this column is risk management. Risk management entails activities such as establishing a system for entering and storing risk data plus monitoring risk information over time as your software project progresses. Risk management systems can range from an informal system based on e-mail, through a lightweight approach based on Excel spreadsheets, up to a sophisticated approach based on using Risk Items in a Microsoft Team Foundation Server system.
It is important to understand that risk analysis should be an ongoing, iterative process, regardless of how you decide to manage your risk effort. Because software project development is such a dynamic activity, you must revise your risk data and results as the project evolves.
Common sense suggests that software risk analysis should be a part of all software projects. Projects ranging from tiny, one-developer, one-week efforts up to huge, multi-year projects with hundreds or even thousands of developers should have some form of risk analysis. However, there is quite a bit of survey research which shows that risk analyses are often not performed, especially on medium and small software projects.
There are several likely explanations. I suspect that one reason risk analyses are rarely performed is that they require techniques and aptitudes which are nearly opposite of the skills and aptitudes required for coding. Let me explain. Most software development activities are relatively well-defined, based on a closed-system, are micro-goal oriented, and usually provide immediate feedback. For example, when you write some code as a developer, you can get instant feedback when you compile and then execute your code. If your code runs as expected, you typically get a certain amount of satisfaction. Performing software risk analysis is a much different kind of activity. You don't get any of the types of feedback or satisfaction you're used to getting.
My point is that software risk analysis is very different from coding. Hopefully this column has convinced you of the importance of performing risk analysis and shown you some neat techniques to create better, more reliable software.

Send your questions and comments for James to testrun@microsoft.com.

Dr. James McCaffrey works for Volt Information Sciences, Inc., where he manages technical training for software engineers working at Microsoft in Redmond, Washington. He has worked on several Microsoft products, including Internet Explorer and MSN Search, and is the author of .NET Test Automation: A Problem-Solution Approach (Apress, 2006). James can be reached at jmccaffrey@volt.com or v-jammc@microsoft.com.


Service Station
An Introduction To RESTful Services With WCF
Jon Flanders
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
Learning about REST
An Abstract Example
Why Should You Care about REST?
WCF and REST
WebGetAttribute and WebInvokeAttribute
UriTemplate and UriTemplateTable
WebHttpBinding and WebHttpBehavior
WebServiceHost and WebServiceHostFactory
Using the Example Code
This is the first in a series of columns about building Windows Communication Foundation (WCF) services using the architectural style known as Representational State Transfer (REST). You could say that 2009 will be the year of REST here in the Service Station column, and it's long overdue in my opinion. REST has been a popular style for many years since its introduction in 2000.
In this first column, I want to discuss some of the basic tenets of REST, as well as present an implementation of a RESTful service using WCF. In future columns, I'll dig into more details of the base ideas of REST, as well as technologies that are built on that base.

Learning about REST
An architectural style is a set of constraints that can be applied when building something. And an architectural style of software is something that describes the features that can be used to guide the implementation of a software system. REST is an architectural style that can be used to build software in which clients (user agents) can make requests of services (endpoints). REST is one way to implement a client-server architectural style—in fact, REST explicitly builds on the client-server architectural style.
A man named Roy Thomas Fielding first coined the term REST as a concept in his PhD dissertation ("Architectural Styles and the Design of Network-based Software Architectures"). He was one of the people who worked on the specification that drives most of the Internet today: Hypertext Transfer Protocol (HTTP). Normally the background of the people describing an architectural style isn't relevant to a discussion of the style, but here I think it is important because I believe one of the best ways to get a basic understanding of REST is to think about the Web and how it works.
I have to assume you, as a developer, are familiar with (and probably, like me, are a heavy daily addict/user of) the Web. Arguably, the Web can be regarded as the largest, most scalable, and most popular distributed application of all time. The constraints of REST are based on the same underlying principles that govern the Web. Those principles are:
  • User agents interact with resources, and resources are anything that can be named and represented. Each resource can be addressed via a unique Uniform Resource Identifier (URI).
  • Interaction with resources (located through their unique URIs) is accomplished using a uniform interface of the HTTP standard verbs (GET, POST, PUT, and DELETE). Also important in the interaction is the declaration of the resource's media type, which is designated using the HTTP Content-Type header. (XHTML, XML, JPG, PNG, and JSON are some well-known media types.)
  • Resources are self-descriptive. All the information necessary to process a request on a resource is contained inside the request itself (which allows services to be stateless).
  • Resources contain links to other resources (hyper-media).

An Abstract Example
A service that uses the architectural style of REST is generally referred to as a RESTful service or endpoint. Just to get a feeling for the ideas behind this architectural style, I'm going to present a small example. Imagine I needed to build a service to work with the data behind MSDN Magazine—a service that could tell me all of the years MSDN Magazine has been published and each of the articles in each issue. Let's say the requirement is that the editors of the magazine could use this service to add new articles and manage the data for upcoming issues.
When building RESTful services, you can go through a very simple set of basic steps to design your service:
  1. What are your resources going to be?
  2. What are the URIs that are going to be used to represent those resources?
  3. What parts of the uniform interface (HTTP verbs) is each URI going to support?
These are the basic building blocks of a RESTful endpoint: resources and their representations, URIs for those resources, and the parts of the uniform interface to which each URI will respond. There are more advanced features you can take advantage of, such as more explicit use of status codes and using hyperlinks to manage resource state, but for this example I'm going to stick to the basics.
Now I can use these steps to design my hypothetical service. The resources are all of the years that MSDN Magazine has been published, all of the issues published each year, and all of the articles published in each magazine. For the purposes of this particular example, I'm going to use a media-type application/xml (XML) to represent these resources, but it's important to remember that RESTful services are not in any way limited to XML as the media type.
Next, I need to determine the URIs for each resource. Right now I only need to determine the relative URIs since the absolute URI will be determined by where I host the endpoint. The list of years will be the root URI of the service (/). Using this syntax, /{year} will return all of the issues for each year; /{year}/{issue} will be the URI for each issue (I'll identify each issue by its month of publication); and /{year}/{issue}/{article} will represent each article (I'll assume each article is numbered from 1 to n in each issue).
Next comes the mapping of URIs to the uniform interface. Since the magazine's history should really be read-only, the root resource will only expose GET. A new year can be added by doing a PUT to the URI /{year}. PUT is used to create new resources when the URI of the new resource is known by the client, as it would be in this case. PUT can also be used to update existing resources when the URI is known. POST is used to create a resource when the URI of the new resource isn't known by the client, so POST will be the verb I'll use when adding a new article resource, which would be sent to the /{year}/{issue} URI.
I can go on with each resource and each verb, but hopefully you've gotten a feel for the steps you go through to determine the design of a RESTful endpoint. You can see the full list of resources, URIs, and verbs in Figure 1.
Figure 1 Operating system support
Resource | URI | Verbs
All years | "/ " | GET
A particular year's issues | "/{year}" | GET, PUT
A particular issue | "/{year}/{issue}" | GET, PUT
An article | "/{year}/{issue}/{article}" | GET, POST (the article number will be assigned by the system), PUT, DELETE (delete would be turned off once an issue has been published)
If I wanted to consume the January 2006 articles as a client, I'd do an HTTP GET to /2006/January. If I were the editor and wanted to add an article to December 2008, the client would POST an article resource to /2008/December and a new article would be added to that issue. This is the pattern I would continue to use over and over to be able to consume this service as a client.

Why Should You Care about REST?
So now that I've explained a little about REST, you are probably wondering why you should care. As a developer, you need motivation to learn and adopt any style, technology, or pattern. If you are reading this magazine, you likely are a developer in the Microsoft technology stack. And for implementing client-server applications, you likely have used another architectural style: remote procedure call (RPC). Whether you have used proprietary RPC systems such as DCOM or .NET Remoting, or used interoperable RPC technologies such as SOAP using ASMX or WCF, these are the implementations of the client-server style we've had on the Microsoft platform. So why learn or use REST?
In my mind, there are two main reasons. First, REST offers some significant features and benefits over RPC technologies in many cases. Second, Microsoft is moving many of its own implementations away from RPC technologies (such as SOAP) and toward REST. This means that even if you aren't convinced or motivated to use REST to build your own systems, as more frameworks and technologies from Microsoft (and others) move to REST, you'll need to know how to interact with them.
The following is a list of other advantages (but don't consider this list exhaustive):
Caching When RESTful endpoints are asked for data using HTTP, the HTTP verb used is GET. Resources returned in response to a GET request can be cached in many different ways. Conditional GET, which is a way that a client can check with the service if his version of the data is still the current version, is an optional implementation detail a RESTful endpoint can implement that can further improve speed and scalability.
Scale-Out REST encourages each resource to contain all of the states necessary to process a particular request. RESTful services are much easier to scale out when they fulfill this constraint and can be stateless.
Side Effects RESTful services should have no side effects when you ask for a resource using GET (unfortunately, this constraint is easier to break than some of the other REST constraints).
Idempotent The other two main HTTP verbs typically used as part of the uniform interface are PUT and DELETE. PUT is most often used when a user agent wants to modify a resource, and DELETE is self-descriptive. The important bit (and what the word idempotent describes) is that you can use these two verbs on a particular resource more than once, and the effect will be the same as the first time you used them—or at least there won't be any further effect. This is reassuring when building reliable distributed systems in which errors, network failures, or latency might cause code to execute multiple times.
Interoperability Many people tout SOAP as being the most interoperable way to implement client-server programs. But some languages and environments still don't have SOAP toolkits. And some that do have toolkits are based on older standards that can't always communicate with toolkits that implement newer standards. REST only requires an HTTP library to be available for most operations (an XML library, of course, is often useful as well), and it is certainly more interoperable than any RCP technology (including SOAP).
Simplicity This advantage is more subjective than the others and can mean different things to different people. To me, the simplicity of using REST relates to URIs representing resources and the uniform interface. As an accomplished Web surfer, I understand typing different URIs into my browser to get different resources (this is sometimes known as URI or URL hacking, but it is not nefarious in any way). Because of this experience in using URIs for many years, designing URIs for resources seems very natural to me. Using the uniform interface simplifies development by freeing me from having to build an interface, contract, or API for each service I need to build. The interface (how clients will interact with my service) is set by the constraints of the architecture.
As I said, this is not an exhaustive list nor should you take it as conclusive evidence that REST is the one and only true technology to use always. You should be aware of the advantages so you can leverage them when appropriate.

WCF and REST
WCF is the Microsoft framework for building applications that communicate over a network, regardless of the style or protocol. The concept behind WCF was to create a framework that was extensible and pluggable so that developers could learn one programming and configuration model and be able to apply those skills to many different kinds of distributed systems.
While it is true that much of WCF is geared toward RPC (using SOAP), it actually has had the ability to expose and consume REST services since it was first released as part of the .NET Framework 3.0. What it lacked was a programming model needed to make using REST with WCF easy. There also were some pieces of infrastructure that you had to build to make REST work with the .NET Framework 3.0. Both a programming model and those pieces of infrastructure were added to WCF in the .NET Framework 3.5 in the System.ServiceModel.Web assembly. And the .NET Framework 3.5 SP1 added a few small improvements as well.
The programming model centers around two new attributes, WebGetAttribute and WebInvokeAttribute, and a URI template mechanism that enables you to declare the URI and verb to which each method is going to respond. The infrastructure comes in the form of a binding (WebHttpBinding) and a behavior (WebHttpBehavior) that provide the correct networking stack for using REST. Also, there is some hosting infrastructure help from a custom ServiceHost (WebServiceHost) and a ServiceHostFactory (WebServiceHostFactory).

WebGetAttribute and WebInvokeAttribute
One of the ways that WCF simplifies building connected systems is by routing network messages to methods on instances of the classes you define as implementations of your service. This allows you to concentrate on the logic of the code in your services rather than on the infrastructure necessary to process network traffic.
By default, WCF does this routing (also known as dispatching) based on the concept of action. For this dispatching to work, an action needs to be present in every message that WCF receives on your behalf. Each unique action is mapped to a particular Action method.
The value of Action is based either on the name of your method (plus the namespace of your service) or a custom value (set via the OperationContractAttribute.Action property). This system of routing is tightly coupled to SOAP since the SOAP specification's Action header is used by default. Luckily, like almost everything else in WCF, this default dispatching infrastructure is replaceable.
When you use the REST infrastructure with WCF, the default dispatcher is replaced by one that routes based not on Action, but instead based on the URI of the incoming request and the HTTP verb being used. This routing (done by a class called WebHttpDispatchOperationSelector) enables you to easily implement a RESTful endpoint. This dispatcher is configured on each endpoint by a behavior named WebHttpBehavior, which must be added to each endpoint from which you want to use this programming model (though you don't often have to do this manually, as you'll see later).
The key to making this work is for the WebHttpDispatchOperationSelector to know how to map different URIs and verbs to your methods. For this, the WebGetAttribute and WebInvokeAttribute must be added to the methods on your WCF ServiceContract type.
WebGetAttribute tells the dispatcher that the method should respond to HTTP GET requests. WebInvokeAttribute is mapped to HTTP POST by default, but the WebInvokeAttribute.Method property can be set to support any of the other HTTP verbs (PUT and DELETE being the two most common). By default, the URI is determined by the name of the method (added onto the base URI of the endpoint).
This is not really very RESTful, since method names are not what you want to deal with in REST because they represent verbs. What you want to expose as URIs are nouns. For this reason, the WCF REST programming model allows customization of URIs for each method by using templates that can be set via the UriTemplate property on WebGetAttribute or WebInvokeAttribute.

UriTemplate and UriTemplateTable
To enable customization of the URI for each method and verb combination, WCF added the ability to define the URI for each resource by using a special template syntax, such as the one I used earlier in this column for describing the MSDN Magazine service endpoint. This syntax allows you to define, with replaceable tokens, the URI structure you'd like each method to represent in conjunction with the HTTP verb (via the WebGetAttribute or WebInvokeAttribute). I'll get into the syntax in more detail in a future column.
Figure 2 shows the WCF ServiceContract definition for the MSDN Magazine service (with the appropriate attributes and UriTemplate customizations) and applies the features I mentioned earlier. It extends the existing WCF contract definition system with the WebGetAttribute for those operations that should respond to GET. It also adds WebInvokeAttribute to the operation in order to respond to any other verbs.
Figure 2 WCF ServiceContract Definition
[ServiceContract]
public interface IMSDNMagazineService
{
    [OperationContract]
    [WebGet(UriTemplate="/")]
    IssuesCollection GetAllIssues();
    [OperationContract]
    [WebGet(UriTemplate = "/{year}")]
    IssuesData GetIssuesByYear(string year);
    [OperationContract]
    [WebGet(UriTemplate = "/{year}/{issue}")]
    Articles GetIssue(string year, string issue);
    [OperationContract]
    [WebGet(UriTemplate = "/{year}/{issue}/{article}")]
    Article GetArticle(string year, string issue, string article);
    [OperationContract]
    [WebInvoke(UriTemplate = "/{year}/{issue}",Method="POST")]
    Article AddArticle(string year, string issue, Article article);

}
In this case, in the AddArticle method I added Method="POST" for readability, as the default verb for WebInvokeAttribute is POST. Both the GET and POST methods have URI customization using the UriTemplate attribute. Notice that the UriTemplate syntax allows for multiple variable path segments, and that each of those path segments are passed to the methods as arguments.

WebHttpBinding and WebHttpBehavior
In WCF, a binding determines how WCF is going to communicate. A binding is really the configuration that tells WCF how to build what is known as the channel stack, which is the set of objects that will work together to provide the type of communication you want for a particular endpoint.
For a RESTful endpoint, the binding you use is WebHttpBinding. Unlike many other bindings, WebHttpBinding is fairly simple, containing only two components: the HTTP transport and the text message encoder (set to not expect or use SOAP, just plain XML).
As I mentioned earlier, WebHttpBehavior is the object that causes the URI-plus-verb dispatcher to be used, so the WebHttpBinding and the WebHttpBehavior are almost always used together. Here is the code for creating such an endpoint if you are self-hosting a WCF RESTful endpoint:
ServiceHost sh =
  new ServiceHost(typeof(MSDNMagazineServiceType));
string baseUri = "http://localhost/MagazineService";
ServiceEndpoint se =
  sh.AddServiceEndpoint(typeof(IMSDNMagazineService),
  new WebHttpBinding(), baseUri);
se.Behaviors.Add(new WebHttpBehavior());
sh.Open();
Notice that not only do I have to specify the WebHttpBinding when adding the endpoint to the ServiceHost, I also have to explicitly add the WebHttpBehavior to the endpoint to make the URI-plus-verb dispatching system work. Of course, I can do this with configuration as well (see Figure 3).
Figure 3 URI-Plus-Verb Dispatching in Configuration
<configuration>
  <system.serviceModel>
    <services>
      <service name="MSDNMagazine.MSDNMagazineServiceType">
        <endpoint
          address="http://localhost/MagazineService"
          binding="webHttpBinding"
          contract="MSDNMagazine.IMSDNMagazineService"
          behaviorConfiguration="webby"/>
      </service>
    </services>
    <behaviors>
      <endpointBehaviors>
        <behavior name="webby">
          <webHttp/>
        </behavior>
      </endpointBehaviors>
    </behaviors>
  </system.serviceModel>
</configuration>

WebServiceHost and WebServiceHostFactory
One of the complaints about WCF is that it is sometimes too complex, especially in terms of configuration. To alleviate this concern with RESTful endpoints (again, simplicity is one of the oft-stated advantages of REST), Microsoft added two new types, WebServiceHost and WebServiceHostFactory, to the .NET Framework 3.5.
WebServiceHost is a ServiceHost-derived type, which simplifies self-hosting scenarios of RESTful endpoints. The code to self-host with WebServiceHost looks like this:
string baseUri = "http://localhost/MagazineService";
WebServiceHost sh =
  new WebServiceHost(typeof(MSDNMagazineServiceType),
  new Uri(baseUri));
sh.Open();
This is a nice optimization, as it avoids the repetitive code of adding the WebHttpBinding and WebHttpBehavior manually. The WebServiceHost class automatically creates the endpoint and configures it with the WebHttpBinding and WebHttpBehavior.
In the managed hosting scenario with WCF, inside Internet Information Services (IIS), WCF normally requires an .svc file that points to the service type, plus entries in the web.config file to inform WCF about the endpoint (the binding and behaviors among other configuration).
To simplify the managed hosting scenario, Microsoft added WebServiceHostFactory, which uses an open WCF extensibility point (using a custom ServiceHostFactory type) in the managed hosting scenario to create a configuration-free experience for many RESTful services. The .svc file looks like this:
<%@ ServiceHost Factory=
  "System.ServiceModel.Activation.WebServiceHostFactory"
  Service="MSDNMagazine.MSDNMagazineServiceType" %>
WebServiceHostFactory creates an instance of the WebServiceHost, and since the WebServiceHost will auto-configure the endpoint using WebHttpBinding and WebHttpBehavior, there doesn't need to be any configuration for this endpoint in the web.config at all. (Of course, if you need to customize the binding, you have to use the configuration system or build a class that derives from WebServiceHost/WebServiceFactory). If I did need to customize the binding, I could still add the appropriate entries in the configuration file. Figure 4 shows a configuration file that would turn on HTTP basic authentication on my service endpoint.
Figure 4 Turn on HTTP Basic Authentication
<configuration>
<system.serviceModel>
  <services>
    <service name="MSDNMagazine.MSDNMagazineServiceType">
      <endpoint
        address="http://localhost/MagazineService"
        binding="webHttpBinding"
        contract="MSDNMagazine.IMSDNMagazineService"
        behaviorConfiguration="webby"/>
    </service>
  </services>
  <bindings>
    <webHttpBinding>
      <binding name="secure">
        <security mode="Transport">
          <transport clientCredentialType="Basic"/>
        </security>
      </binding>
    </webHttpBinding>
  </bindings>
  <behaviors>
    <endpointBehaviors>
      <behavior name="webby">
        <webHttp/>
      </behavior>
    </endpointBehaviors>
  </behaviors>
</system.serviceModel>
</configuration>

Using the Example Code
Through the explanation of the features of WCF in relation to REST, I've laid out most of the implementation of the service I proposed at the beginning of the column. Let me walk through interacting with this service.
Once I have the service up and running, I can hit its root URI with any client, including a Web browser such Internet Explorer. Being able to run quick tests against RESTful endpoints with a browser is, most would agree, a pretty useful consequence of the REST architectural style being based on the way the Web works. You can see the result in Figure 5.
Figure 5 Root Resource URI
In this case, I am hosting in the Visual Studio 2008 Web Development Server, which dictates the base URI. The Issues.svc file is the necessary file for WCF in the managed hosting scenario. If I want to see the result for a particular year, I can just add that year to the address (the URI) in the browser (see Figure 6).
Figure 6 The Resource Representing the Year 2007
If I asked for October of 2008, the URI would be localhost:1355/Issues.svc/2008/October, which, at this point, would be an empty set. And if I wanted to add an article, I'd do an HTTP POST to that URI with the XML representation of an article as the HTTP request body.
Another nice feature of HTTP is all of the tools that are available to interact with it. One of my favorites is Fiddler, which allows me to "spy" on HTTP requests and responses. But it also has a feature that allows me to do arbitrary HTTP operations. So I can use the Fiddler Request Builder tab to do my HTTP POST (see Figure 7). You can see a request for the October 2008 resource after the POST in Figure 8.
Figure 7 Making an HTTP POST Request to Create a New Article Resource with Fiddler
Figure 8 Article Resource Added
Although it can be argued that Internet Explorer and Fiddler are real clients, building an actual client-server implementation is an extension of these simple steps I just went through (a more complex example of building a RESTful client will come in a later column). A client needs to know the URI of each resource and what parts of the uniform interface to use against each URI. The client can then interact with the service by using these pieces of information to build out its functionality.
What I've shown you in this first column is the base set of WCF features that enable the REST architectural style to be used in your .NET applications with ease. This base enables other interesting technologies, including things such as Web Feeds (RSS and Atom) and support for JSON-encoded resources for interaction with AJAX.
In the next few columns, I'm going to build on this base knowledge of REST and WCF and delve into the details of several other features in the Microsoft platform that build on this style and technology. I'll also be dealing with some of the more commonly asked REST questions, for example, about security and about the idea of implementing processing endpoints.

Send your questions and comments to sstation@microsoft.com.

Jon Flanders is an independent consultant, speaker, and trainer for Pluralsight. He specializes in BizTalk Server, Windows Workflow Foundation, and Windows Communication Foundation. You can contact Jon at masteringbiztalk.com/blogs/jon.


Security Briefs
Getting Started With The SDL Threat Modeling Tool
Adam Shostack
Contents
Starting the Threat Modeling Process
Analyzing Threats
Environment Screen
Keeping Track with Reports
Actions Menu
Threat Modeling Meetings
Thinking about Assets
Figure 1 The Threat Modeling Process
In November 2008, Microsoft announced the general availability of the Security Development Lifecycle (SDL) threat modeling tool as a free download from MSDN. This column follows a team through the process of getting started with the SDL threat modeling approach and shows you how to use the new tool to develop great threat models as a backbone of your security process.
This column isn't a primer on SDL threat modeling. For that, see the article I co-authored in the November 2006 issue of MSDN Magazine on using the STRIDE approach, "Threat Modeling: Uncover Security Design Flaws Using the STRIDE Approach." Figure 1 provides a quick overview of the process.

Starting the Threat Modeling Process
When you launch the SDL threat modeling tool, you'll see that the lower left-hand corner looks quite a bit like Microsoft Office Outlook with four screens: diagram, analyze, environment, and reports (see Figure 2 for details). Note that these screens are slightly different from the outline shown in Figure 1 because it makes sense to consider threats and mitigations together since they are closely related.
In this section, I will follow Deb (a developer), Paul (a program manager), and Tim (a tester) through the process of developing their first threat model, and I'll also discuss each screen of the tool.
"Hi Deb, I've been working on that threat model diagram, and wanted to walk through it with you to make sure we've gotten the details right."
"Sure thing, Paul! Come on in."
Paul brings out a print-out of a diagram that he's already made from the threat model tool's "Diagrams only" report, shown in Figure 3.
"Paul, I haven't seen these diagrams before. It looks pretty simple, but can you walk me through what the different shapes mean?"
"The way this works is Carl, our normal customer persona, is drawn as an outside entity—a rectangle. He's sending commands to our Web server—the circle is any running code, and the arrow gives us the direction of communication. The Web server is consulting a database, which, just as with anywhere we store data, is two parallel lines. The system is called a data flow diagram (DFD). There's a good Wikipedia article on DFDs. The only bit that's not covered there is these trust boundary dotted lines between where different people are in control. For example, you know the IT pros require that we use their Active Directory system for logon information, and so the Active Directory is shown as outside our control."
Figure 3 Paul’s DFD Diagram
When the tool starts, the diagram screen is displayed. This is where Paul used the Visio tools and the provided stencil to draw his DFD (see Figure 4). Even though this was his first time, he was comfortable because the validator on the left gave him feedback, based on his experience using threat modeling as part of the SDL. As he found himself drawing more complexity, he added additional details by right-clicking on the context folder in the upper- right and was able to create a complex, layered diagram.
Figure 4 The Diagrams Screen

Analyzing Threats
Paul was a little hesitant when he opened the Analyze screen (see Figure 5). There was a long list of threats there—where did they all come from? The tool had constructed them, using the SDL approach called "STRIDE per element." The idea is that software generally comes under a predictable set of threats (those shown in Figure 5). Some security experts like to chase after the hacker first because the chase itself can be fun. I think it makes sense to start securing your house by making sure each door and window has some sort of lock on it, and only then wondering about an alarm system. So start with STRIDE per element by clicking on any of the lines of the Analyze screen .
Figure 5 The Analyze Screen
Paul started out by selecting the database in the list of elements. He read at the top of the screen that "database" is a data store, therefore subject to tampering, information disclosure, and denial of service threats. As he read down, the questions helped him think about how people might tamper with the data, and he realized that no one had specified who was able to connect to the database. A whiteboard diagram and some simple rules revealed the first threat! Score one for threat modeling.
A few minutes of discussion led to a realization that they needed to think about access control and roles. Paul filled in some quick notes in two threats. The first note said "No access control plan." He also filed a work item in their Team Foundation Server (TFS) database. The second note said "Access control plan requires a role list." Paul then went into TFS and created a second bug that was dependent on the first.
As Paul went into information disclosure, he realized that their access control plan required some read-only accounts for audit and report generation. He wondered whether this was a new threat, then decided it wasn't because the mitigation is the same, and he edited the bug in TFS. He then decided to certify the threat as mitigated elsewhere, and wrote "covered in TFS bug #235." He wasn't quite sure whether that was OK, but that's what the certification feature is for (see Figure 6).
Figure 6 Certifying That Threats Don’t Apply
He also thought about information disclosure a bit more and realized that the backup tapes were going to need encryption, but that's an operations job. (I'll cover how he tracked that in just a minute, after covering one related feature: the "auto-generate threats for this element" checkbox at the top.)
The auto-generation feature is designed for large teams that have a lot of threat models and that also have a way to ensure that the testers and program managers are all talking about what's in the threat models. So, for this situation, Paul might say that Phil is responsible for several elements that he wants to show for context and how they interact with his features. The auto-generate box is checked by default, but Paul can uncheck it and state that he thinks this is Phil's feature.

Environment Screen
Worried about operations encrypting backup tapes, Paul opened the environment screen and saw a section for external security notes (see Figure 7). There, he made a note that operations was going to have to handle tape backup. He would have to make sure operations had a copy of the tool.
Figure 7 External Security Notes
While he was there, he wondered what the document header section was and was relieved to see that there was more guiding text up there, explaining that this was where he identified who owned the threat model, what it was for, and so on. He filled it in and wished that he could include the Contoso project tracking number.
Moving systematically through the elements of the tree, Paul noted that there were dependencies on SQL Server and the Fabrikam Foxy Web Widgets 2.3 widget library. Paul added a note to have Tim investigate, make sure they were up-to-date, and that they were getting security notifications from Fabrikam.

Keeping Track with Reports
There are five threat modeling reports available:
Analysis Report This report is designed for a security advisor or consultant to review a threat model, although anyone can use it to see what diagram validation issues are open, what empty threats haven't been filled out, what threats have no mitigations, what threats have been certified or marked as not generating threats.
Threat Model Report This report contains the information entered into the threat model, presented in a single page view.
Diagrams Only This report is designed to make it easy to print diagrams. Some people like working on paper, but don't need to print entire reports when all they want is the diagram.
Bug Report This report shows the bugs which have been filed from this threat model and their status
Fuzzing Report The fuzzing report uses the architectural information provided in the diagramming step to offer a prioritized list of fuzzing targets. Fuzzing is a testing technique that involves generating random input for a program. It is surprising how good fuzzing can be at making things crash, and many of those crashes are exploitable. (Please see Create a Custom Test Interface Provider for Team System or Fuzz Testing at Microsoft and the Triage Process for more on fuzz testing.)

Actions Menu
There are a couple of useful features tucked away under the Action menu: Thumbnail view, Bug Tracking Settings, and Team Lead Mode. Thumbnail view gives you easy access to the diagrams when you're on other screens. This is useful when you have a complex diagram and you'd like to have it on screen while analyzing your model. It automatically sizes the thumbnail to take up most of the window, keeping the whole diagram in view as you resize.
If you try to file a bug without entering any bug information, the bug tracking dialog will come up to prompt you, but you can bring it up anytime via the actions menu. There's a very simple XML file that you can use to define fields to be filled in, or you can just edit the fields (assuming the "use template" box isn't checked). Bugs automatically get a title of "TM: [threat] affects [element]" and content is pre-filled from the threat and mitigation information. Fields can be deleted by selecting them and hitting the Delete key.
Team Lead Mode displays a new section in the describe environment screen, called Template Settings. This allows a team leader to change the guiding questions and set a default location to save threat models. The team leader can also edit the fields in document header information, adding and removing things to fit the environment.
As he had wanted to do earlier, Paul added the Contoso project tracking number as a new field. Any threat model saved from team lead mode can function as a template. (Actually, any threat model at all can function as a template for additional work.)
Changing the guiding questions involves editing an XML file that starts in the SDL threat modeling tool's \Data folder. The format is quite easy to follow.

Threat Modeling Meetings
When Paul sent his threat model around, Tim, the tester, was pretty underwhelmed. All sorts of things popped out at him, and he asked Paul: "You program managers always assume everything is going to work, huh?"
It may surprise you to know that testers and their skepticism can function as a great complement to threat models. As a result, many teams ask their testers to lead the threat modeling process. In this scenario, after Tim took over the threat model, he called for two threat modeling meetings: one meeting to synchronize on the process and walk through the diagrams and then a second meeting for threat review and sign-off.
In the first meeting, Tim spent 10 minutes walking everyone through the SDL threat modeling process. He then pulled up the threat model diagram and started explaining it in detail. Within five minutes, an important missing component had been identified.
A few minutes later, Tim and Paul got into an extended discussion of how the Web server was actually built. It was not the ideal way for a meeting to proceed, but everyone eventually agreed that discovering the discrepancy early was going to save them a great deal of time later on.
In the second meeting, the team walked through the threats, discussed some ways to address them, and signed off on the threat model. They checked the document into source control and continued with development.

Thinking about Assets
Some readers who have threat modeled may notice that we haven't talked about assets at all. We've discovered that many software engineers understand their software better than they understand the concept of assets and what assets an attacker may be interested in.
If you're going to threat model a house, you might start by thinking about your family or irreplaceable photos or valuable artwork. Perhaps you might start by thinking about who might break in and the current security system. Or you might start by considering the physical features, like the pool or the front porch. These are analogous to thinking about assets, attackers, or software design. Any of these three approaches will work.
The approach to threat modeling we've presented here is substantially simpler than what Microsoft has done in the past. The Microsoft SDL team has found that the software design approach works really well for a lot of teams. We hope that will include yours.

Send your questions and comments to briefs@microsoft.com.

Adam Shostack is a Program Manager on the Security Development Lifecycle (SDL) team at Microsoft. He is responsible for the threat modeling component of the SDL.


Extreme ASP.NET
Routing with ASP.NET Web Forms
Scott Allen
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
What Is Routing?
A Brief History of URL Rewriting
Of Routes and Route Handlers
Configuring ASP.NET for Routing
Configuring Routes
The Recipe Routing Handler
Routing and Security
URL Generation
Wrapping Up with Routes
Service Pack 1 for the Microsoft .NET Framework 3.5 introduced a routing engine to the ASP.NET runtime. The routing engine can decouple the URL in an incoming HTTP request from the physical Web Form that responds to the request, allowing you to build friendly URLs for your Web applications. Although you've been able to use friendly URLs in previous versions of ASP.NET, the routing engine provides an easier, cleaner, and more testable approach.
The routing engine began as a part of the ASP.NET Model View Controller (MVC) framework, which is in a preview stage as of this writing. However, Microsoft packaged the routing logic into the System.Web.Routing assembly and released the assembly with SP1. The assembly currently provides routing for Web sites using ASP.NET Dynamic Data features (which were also released with SP1), but in this column I will demonstrate how to use the routing functionality with ASP.NET Web Forms.

What Is Routing?
Imagine you have an ASP.NET Web Form named RecipeDisplay.aspx, and this form lives inside a folder named Web Forms. The classic approach to viewing a recipe with this Web Form is to build a URL pointing to the physical location of the form and encode some data into the query string to tell the Web Form which recipe to display. The end of such a URL might look like the following: /WebForms/RecipeDisplay.aspx?id=5, where the number 5 represents a primary key value in a database table full of recipes.
Routing is fundamentally about decomposing a URL endpoint into parameters and then using those parameters to steer the HTTP request processing to a specific component. Let's take the URL /recipe/5 as an example. With the proper routing configuration, you can still respond to this URL with the Web Form RecipeDisplay.aspx.
The URL no longer represents a physical path. Instead, the word recipe represents a parameter that the routing engine can use to locate a component for processing recipe requests. The number 5 represents a second parameter that you'll need during processing to display a specific recipe. Instead of encoding database keys into the URL, a better idea might be to use a URL like /recipe/tacos. This URL not only includes enough parameters to display a specific recipe, but is also human readable, reveals its intent to end users, and includes important keywords for search engines to see.

A Brief History of URL Rewriting
In ASP.NET, using a URL ending with /recipe/tacos traditionally required one to work with a URL rewriting scheme. For detailed information on URL rewriting, see Scott Mitchell's definitive article "URL Rewriting in ASP.NET." The article describes the common implementation of URL rewriting in ASP.NET using an HTTP module and the static RewritePath method of the HttpContext class. Scott's article also details the benefits of friendly, hackable URLs.
Those of you who have used the RewritePath API in the past are probably familiar with some of the quirks and weaknesses in the rewriting approach. The primary problem with RewritePath is how the method changes the virtual path used during the processing of a request. With URL rewriting, you needed to fix up the postback destination of each Web Form (often by rewriting the URL a second time during the request) to avoid postbacks to the internal, rewritten URL.
In addition, most developers would implement URL rewriting as a one-way translation because there was no easy mechanism to let the URL rewriting logic work in two directions. For example, it was easy to give the URL rewriting logic a public-facing URL and have the logic return the internal URL of a Web Form. It was difficult to give the rewriting logic the internal URL of a Web Form and have it return the public URL required to reach the form. The latter is useful when generating hyperlinks to other Web Forms that hide behind rewritten URLs. As you'll see in the rest of this column, the URL routing engine circumvents these problems.
Figure 1 Routes, Route Handlers, and the Routing Module
Of Routes and Route Handlers
There are three fundamental players in the URL routing engine: routes, route handlers, and the routing module. A route associates a URL with a route handler. An instance of the Route class from the System.Web.Routing namespace represents a route during run time and describes the route's parameters and constraints. A route handler inherits from the System.Web.Routing.IRouteHandler interface. This interface requires the route handler to implement a GetHttpHandler method that returns an object implementing the IHttpHandler interface. The IHttpHandler interface has been a part of ASP.NET since the beginning, and a Web Form (a System.Web.UI.Page) is an IHttpHandler. When using routing with Web Forms, your route handlers need to locate, instantiate, and return the proper Web Form. Finally, the routing module plugs into the ASP.NET processing pipeline. The module will intercept incoming requests, examine the URL, and discover if there are any matching routes defined. The module will retrieve the associated route handler for a matching route and ask the routing handler for the IHttpHandler that will process the request.
The three primary types I've mentioned are shown in Figure 1. In the next section, I'll put these three players to work.

Configuring ASP.NET for Routing
To configure an ASP.NET Web site or Web application for routing, you first need to add a reference to the System.Web.Routing assembly. The SP1 installation for the .NET Framework 3.5 will install this assembly into the global assembly cache, and you can find the assembly inside the standard "Add Reference" dialog box.
You'll also need to configure the routing module into the ASP.NET pipeline. The routing module is a standard HTTP module. For IIS 6.0 and earlier and for the Visual Studio Web development server, you install the module using the <httpModules> section of web.config, as you see here:
<httpModules>
    <add name="RoutingModule"
         type="System.Web.Routing.UrlRoutingModule,
               System.Web.Routing,
               Version=3.5.0.0, Culture=neutral,
               PublicKeyToken=31bf3856ad364e35"/>

    <!-- ... -->

</httpModules>
To run a Web site with routing in IIS 7.0, you need two entries in web.config. The first entry is the URL routing module configuration, which is found in the <modules> section of <system.webServer>. You also need an entry to handle requests for UrlRouting.axd in the <handlers> section of <system.webServer>. Both of these entries are shown in Figure 2. Also, see the sidebar "IIS 7.0 Configuration Entries."
Figure 2 URL Routing Module Configuration
<system.webServer>
   <modules runAllManagedModulesForAllRequests="true">

      <add name="UrlRoutingModule"
             type="System.Web.Routing.UrlRoutingModule,
                   System.Web.Routing, Version=3.5.0.0,
                   Culture=neutral,
                   PublicKeyToken=31BF3856AD364E35" />
      <!-- ... -->

    </modules>
    <handlers>

      <add name="UrlRoutingHandler"
            preCondition="integratedMode"
            verb="*" path="UrlRouting.axd"
            type="System.Web.HttpForbiddenHandler,
                  System.Web, Version=2.0.0.0, Culture=neutral,
                  PublicKeyToken=b03f5f7f11d50a3a" />
      <!-- ... -->

    </handlers>
</system.webServer>
Once you've configured the URL routing module into the pipeline, it will wire itself to the PostResolveRequestCache and the PostMapRequestHandler events. Figure 3 shows a subset of the pipeline events. URL rewriting implementations typically perform their work during the BeginRequest event, which is the earliest event to fire during a request. With URL routing, the route matching and selection of a route handler occurs during the PostResolveRequestCache stage, which is after the authentication, authorization, and cache lookup stages of processing. I will need to revisit the implications of this event timing later in the column.
Figure 3 HTTP Request

Configuring Routes
Routes and route handlers go hand in hand, but I'll look at the code to configure routes first. The routing engine's RouteTable class exposes a RouteCollection via its static Routes property. You need to configure all of your custom routes into this collection before the application begins to execute the first request, which means you need to use a global.asax file and the Application_Start event.
Figure 4 shows the route registration code you need to use for "/recipe/brownies" to reach the RecipeDisplay.aspx Web Form. The parameters for the Add method on the RouteCollection class include a friendly name for the route, followed by the route itself. The first parameter to the Route constructor is a URL pattern. The pattern consists of the URL segments that will appear at the end of a URL pointing to this application (after any segments required to reach the application's root). For an application rooted at localhost/food/ then, the route pattern in Figure 4 will match localhost/food/recipe/brownies.
Figure 4 Route Registration Code for /recipe/brownies
protected void Application_Start(object sender, EventArgs e)
{
    RegisterRoutes();
}

private static void RegisterRoutes()
{
    RouteTable.Routes.Add(
        "Recipe",
        new Route("recipe/{name}",
                  new RecipeRouteHandler(
                      "~/WebForms/RecipeDisplay.aspx")));
}
IIS 7.0 Configuration Entries

The runAllManagedModulesForAllRequests attribute requires a value of true if you want to use the extensionless URLs as I've done in this sample. Also, it might seem strange to configure an HTTP handler for UrlRouting.axd. This is a small workaround that the routing engine requires in order for routing to work under IIS 7.0. The UrlRouting module actually rewrites the incoming URL to ~/UrlRouting.axd, which will rewrite the URL back to the original, incoming URL. It's likely that a future version of IIS will integrate perfectly with the routing engine and not require this workaround.
Segments enclosed inside curly braces denote parameters, and the routing engine will automatically extract the values there and place them into a name/value dictionary that will exist for the duration of the request. Using the previous example of localhost/food/recipe/brownies, the routing engine will extract the value "brownies" and store the value in the dictionary with a key of "name". You'll see how to use the dictionary when I look at the code for the route handler.
You can add as many routes as you need into the RouteTable, but the ordering of the routes is important. The routing engine will test all incoming URLs against the routes in the collection in the order in which they appear, and the engine will select the first route with a matching pattern. For this reason, you should add the most specific routes first. If you added a generic route with the URL pattern "{category}/{subcategory}" before the recipe route, the routing engine would never find the recipe route. One additional note—the routing engine performs the pattern matching in a case-insensitive manner.
Overloaded versions of the Route constructor allow you to create default parameter values and apply constraints. Defaults allow you to specify default values for the routing engine to place into the name/value parameter dictionary when no value exists for the parameter in an incoming URL. For example, you could make "brownies" the default recipe name when the routing engine sees a recipe URL without a name value (like localhost/food/recipe).
Constraints allow you to specify regular expressions to validate parameters and fine-tune the route pattern matching on incoming URLs. If you were using primary key values to identify recipes in a URL (like localhost/food/recipe/5), you could use a regular expression to ensure the primary key value in the URL is an integer. You can also apply constraints using an object that implements the IRouteConstraint interface.
The second parameter to the Route constructor is a new instance of my route handler, which I'll look at in Figure 5.
Figure 5 RecipeRouteHandler
public class RecipeRouteHandler : IRouteHandler
 {
     public RecipeRouteHandler(string virtualPath)
     {
         _virtualPath = virtualPath;
     }

     public IHttpHandler GetHttpHandler(RequestContext requestContext)
     {
         var display = BuildManager.CreateInstanceFromVirtualPath(
                         _virtualPath, typeof(Page)) as IRecipeDisplay;

         display.RecipeName = requestContext.RouteData.Values["name"] as string;
         return display;
     }

     string _virtualPath;
 }

The Recipe Routing Handler
The following code snippet shows a basic implementation of a route handler for recipe requests. Since the route handler ultimately has to create an instance of an IHttpHandler (in this case, RecipeDisplay.aspx), the constructor requires a virtual path that points to the Web Form the route handler will create. The GetHttpHandler method passes this virtual path to the ASP.NET BuildManager in order to retrieve the instantiated Web Form:
interface IRecipeDisplay : IHttpHandler
{
    string RecipeName { get;  set; }
}
Notice how the route handler can also pull data from the routing engine's parameter dictionary, which is the RouteData property of the RequestContext class. The routing engine sets up the RequestContext and passes an instance when it invokes this method. There are many options available for getting the route data into the Web Form. You could pass the route data along in the HttpContext Items collection, for instance. In this example, you've defined an interface for your Web Form to implement (IRecipeDisplay). The route handler can set strongly typed properties on the Web Form to pass along any information the Web Form requires, and this approach will work with both the ASP.NET Web site and ASP.NET application compilation models.

Routing and Security
When you're using ASP.NET routing, you can still use all the ASP.NET features you've come to love—Master Pages, output caching, themes, user controls, and more. There is one notable exception, however. The routing module works its magic using events in the pipeline that occur after the authentication and authorization stages of processing, meaning that ASP.NET will be authorizing your users using the public, visible URL and not the virtual path to the ASP.NET Web Form that the route handler selects to process the request. You need to pay careful attention to the authorization strategy for an application using routing.
Let's say you wanted to only allow authenticated users to view recipes. One approach would be to modify the root web.config to use the authorization settings here:
<location path="recipe">
  <system.web>
    <authorization>
      <deny users="?"/>
    </authorization>
  </system.web>
</location>
Although this approach will prevent anonymous users from viewing /recipe/tacos, it does have two fundamental weaknesses. First, the setting doesn't prevent a user from directly requesting /WebForms/RecipeDisplay.aspx (although you could add another authorization rule that prevents all users from directly requesting resources from the Web Forms folder). Second, it is easy to change the route configuration in global.asax.cs without changing the authorization rules and leave your secret recipes open to anonymous users.
An alternate approach to authorization would be to protect the RecipeDisplay.aspx Web Form based on its physical location, which is to place web.config files with <authorization> settings directly into the protected folder. However, since ASP.NET is authorizing users based on the public URL, you'll need to make the authorization checks manually on the virtual path that your route handler uses.
You'll need to add the following code to the beginning of your route handler's GetHttpHandler method. This code uses the static CheckUrlAccessForPrincipal method of the UrlAuthorizationModule class (the same module that performs authorization checks in the ASP.NET pipeline):
if (!UrlAuthorizationModule.CheckUrlAccessForPrincipal(
    _virtualPath, requestContext.HttpContext.User,
                  requestContext.HttpContext.Request.HttpMethod))
{
    requestContext.HttpContext.Response.StatusCode =
        (int)HttpStatusCode.Unauthorized;
    requestContext.HttpContext.Response.End();
}
In order to access the HttpContext members via the RequestContext, you'll need to add a reference to the System.Web.Abstractions assembly.
With a secure routing handler in place, you can now turn your attention to the page that needs to generate hyperlinks for each recipe in your database. It turns out the routing logic can help you build this page, too.

URL Generation
To generate the hyperlink to any given recipe, I will once again turn to the collection of routes configured during application startup. As shown here, the RouteCollection class has a GetVirtualPath method for this purpose:
VirtualPathData pathData =
    RouteTable.Routes.GetVirtualPath(
                  null,
                  "Recipe",
                  new RouteValueDictionary { { "Name", recipeName } });

return pathData.VirtualPath;
You need to pass in the desired route name ("Recipe") along with a dictionary of the required parameters and their associated values. This method will use the URL pattern you created earlier (/recipe/{name}) to construct the proper URL.
The following code uses this method to generate a collection of anonymously typed objects. The objects have Name and Url properties that you can use with data binding to generate a list or table of available recipes:
var recipes =
    new RecipeRepository()
        .GetAllRecipeNames()
        .OrderBy(recipeName => recipeName)
        .Select(recipeName =>
          new
          {
            Name = recipeName,
            Url = GetVirtualPathForRecipe(recipeName)
          });
The ability to generate URLs from your routing configuration means you can change the configuration without the fear of creating broken links inside your application. Of course, you still might break your user's favorite links and bookmarks, but having the ability to change is a tremendous advantage when you are still designing the application's URL structure.

Wrapping Up with Routes
The URL routing engine does all of the dirty work of URL pattern matching and URL generation. All you need to do is configure your routes and implement your route handlers. With routing, you are truly isolated from file extensions and the physical layout of your file system, and you don't need to deal with the quirks of using a URL rewriter. Instead, you can concentrate on the optimum URL design for your end users and for search engines. In addition, Microsoft is working on making URL routing with Web Forms even easier and more configurable in the upcoming ASP.NET 4.0.

Send your questions and comments to xtrmasp@microsoft.com.
Scott Allen is a founder of OdeToCode and a member of the Pluralsight technical staff. You can reach Scott at scott@odetocode.com, or read his blog at OdeToCode.com/blogs/scott.


Foundations
Easily Apply Transactions To Services
Juval Lowy
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
State Management and Transactions
Per-Call Transactional Services
Instance Management and Transactions
Session-Based Services and VRMs
Transactional Durable Services
Transactional Behavior
Adding Context to the IPC Binding
InProcFactory and Transactions
A fundamental problem in programming is error recovery. After an error, your application must restore itself to the state it had before the error took place. Consider an application that tries to perform an operation comprising several smaller operations, potentially concurrently, where each of the individual operations can fail or succeed independently of the others. An error in any one of the smaller operations means the system is at an inconsistent state.
Take a banking application, for example, that transfers funds between two accounts by crediting one account and debiting the other. Successfully debiting one account but failing to credit the other is an inconsistent state, because the funds cannot be in both places at the same time, and failing to debit while successfully crediting results with an equally inconsistent state in which the money is gone. It is always up to the application to recover from the error by restoring the system to the original state.
This is far easier said than done for a number of reasons. First, for a large operation, the sheer number of permutations of partial success and partial failure gets quickly out of hand. This results in fragile code that is very expensive to develop and maintain and quite often does not really work, since developers often deal only with the easy recovery cases, which are the cases that they are both aware of and know how to handle. Second, your composite operation could be part of a much larger operation, and even if your code executed flawlessly, you still may have to undo it if something else outside your control encounters an error. This implies tight coupling between the participating parties on the management and coordination of the operations. Finally, you also need to isolate what you do from whomever else is required to interact with the system, because if you later on recover from an error by rolling back some of your actions, you will put someone else implicitly in an error state.
As you can see, it is practically impossible to write robust error-recovery code by hand. This realization is not new. Ever since software was used in business contexts (in the 1960s), it was clear there had to be a better way of managing recovery. There is a better way: transactions. A transaction is a set of operations where the failure of any individual operation causes the entire set to fail, as one atomic operation. When using transactions, there is no need to write recovery logic, since there is nothing to recover. Either all operations succeeded so there is nothing to recover, or they all failed, and failed to affect the system's state, so there is also nothing to recover.
When using transactions, it is essential to use transactional resource managers. The resource manager is capable of rolling back all changes made during the transaction if the transaction aborts and persisting the changes if the transaction commits. The resource manager also provides isolation; that is, while a transaction is in progress, the resource manager prevents all other parties (besides the transaction) from accessing it and seeing the changes, which could still roll back. This also means that the transaction should never access non-resource managers, since any changes made to those will not roll back if the transaction aborts, and thus recovery will be necessary.
Traditionally, resource managers were durable resources such as databases and message queues. However, in the article from the May 2005 issue of MSDN Magazine titled "Can't Commit? Volatile Resource Managers in .NET Bring Transactions to the Common Type", I presented my technique for implementing a general-purpose volatile resource manager (VRM) called Transactional<T>:
public class Transactional<T> : ...
{
   public Transactional(T value);
   public Transactional();
   public T Value
   {get;set;}
   /* Conversion operators to and from T */
}
By specifying any serializable type parameter (such as an int or a string) to Transactional<T>, you turn that type into a full-blown volatile resource manager that auto-enlists in the ambient transaction, commits or rolls back the changes according to the outcome of the transaction, and isolates the current changes from all other transactions.
Figure 1 demonstrates the use of Transactional<T>. Since the scope is not completed, the transaction aborts, and the values of number and city revert to their pre-transaction state.
Figure 1 Using Transactional<T>
Transactional<int> number = new Transactional<int>(3);
Transactional<string> city = new Transactional<string>("New York, ");

city.Value += "NY"; //Can use with or without transactions
using(TransactionScope scope = new TransactionScope())
{
   city.Value = "London, ";
   city.Value += "UK";
   number.Value = 4;
   number.Value++;
}
Debug.Assert(number == 3); //Conversion operators at work
Debug.Assert(city == "New York, NY");
In addition to Transactional<T>, I have also provided a transactional array as well as transactional versions for all of the collections in System.Collections.Generic, such as TransactionalDictionary<K,T>. These collections are polymorphic with their non-transactional cousins and are used exactly the same way.

State Management and Transactions
The sole purpose of transactional programming is to leave the system in a consistent state. In the case of Windows Communication Foundation (WCF), the state of the system consists of the resource managers plus the in-memory state of the service instances. While the resource managers will automatically manage their state as a product of the transaction's outcome, that is not the case with in-memory objects or static variables.
The solution to this state management problem is to develop a service as a state-aware service and proactively manage its state. Between transactions, the service should store its state in a resource manager. At the beginning of each transaction, the service should retrieve its state from the resource, and by doing so enlist the resource in the transaction. At the end of the transaction, the service should save its state back to the resource manager. This technique elegantly provides for state auto-recovery. Any changes made to the instance state will commit or roll back as part of the transaction.
If the transaction commits, the next time the service gets its state it will have the post-transaction state. If the transaction aborts, the next time it will have its pre-transaction state. Either way, the service will have a consistent state ready to be accessed by a new transaction.
There are two remaining problems when writing transactional services. The first is how the service can know when transactions start and end, so that it can get and save its state. The service may be part of a much larger transaction that spans multiple services and machines. At any moment between calls, the transaction might end. Who will call the service, letting it know to save its state? The second problem has to do with isolation. Different clients might call the service concurrently, on different transactions. How can the service isolate from one transaction change made to its state by another transaction? If the other transaction was to access its state and operate based on its values, that transaction would be inconsistent if the original transaction aborted and the changes rolled back.
The solution to both problems is to equate method boundaries with transaction boundaries. At the beginning of every method call, the service should read its state from the resource manager, and at the end of each method call, the service should save its state to the resource manager. Doing so ensures that if a transaction ends between method calls, the service's state will either persist or roll back with it. In addition, reading and storing the state in the resource manager in each method call addresses the isolation challenge because the service simply lets the resource manager isolate access to the state between concurrent transactions.
Since the service equates method boundaries with transaction boundaries, the service instance must also vote on the transaction's outcome at the end of every method call. From the service perspective, the transaction completes once the method returns. In WCF, this is done automatically via the TransactionAutoComplete property of the OperationBehavior attribute. When this property is set to true, if there were no unhandled exceptions in the operation, WCF will automatically vote to commit. If there was an unhandled exception, WCF will vote to abort. Since TransactionAutoComplete defaults to true, any transactional method will use auto-completion by default, like so:
//These two definitions are equivalent:
[OperationBehavior(TransactionScopeRequired = true,
                   TransactionAutoComplete = true)]
public void MyMethod(...)
{...}

[OperationBehavior(TransactionScopeRequired = true)]
public void MyMethod(...)
{...}
For more on WCF transactional programming, see my Foundations column "WCF Transaction Propagation" in the May 2007 issue.

Per-Call Transactional Services
With a per-call service, once the call returns, the instance is destroyed. Therefore, the resource manager used to store the state between calls must be outside the scope of the instance. The client and the service must also agree on which operations are responsible for creating or removing the instance from the resource manager.
Because there could be many instances of the same service type accessing the same resource manager, every operation must contain some parameter that allows the service instance to find its state in the resource manager and bind against it. I call that parameter the instance ID. The client must also call a dedicated operation to remove the instance state from the store. Note that the behavioral requirements for a state-aware transactional object and a per-call object are the same: both retrieve and save their state at method boundaries. With a per-call service, any resource manager can be used to store the service state. You might use a database or you might use a VRM, as shown in Figure 2.
Figure 2 Per-Call Service Using a VRM
[ServiceContract]
interface IMyCounter
{
   [OperationContract]
   [TransactionFlow(TransactionFlowOption.Allowed)]
   void Increment(string instanceId);

   [OperationContract]
   [TransactionFlow(TransactionFlowOption.Allowed)]
   void RemoveCounter(string instanceId);
}
[ServiceBehavior(InstanceContextMode = InstanceContextMode.PerCall)]
class MyService : IMyCounter
{
   static TransactionalDictionary<string,int> m_StateStore =
                               new TransactionalDictionary<string,int>();

   [OperationBehavior(TransactionScopeRequired = true)]
   public void Increment(string instanceId)
   {
    if(m_StateStore.ContainsKey(instanceId) == false)
      {
         m_StateStore[instanceId] = 0;
      }
      m_StateStore[instanceId]++;
      Trace.WriteLine(m_StateStore[instanceId]);
   }
   [OperationBehavior(TransactionScopeRequired = true)]
   public void RemoveCounter(string instanceId)
   {
    if(m_StateStore.ContainsKey(instanceId))
      {
         m_StateStore.Remove(instanceId);
      }
   }
}

//Client side:
MyCounterClient proxy = new MyCounterClient();

using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment("MyInstance");
   scope.Complete();
}

//This transaction will abort since the scope is not completed
using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment("MyInstance");
}

using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment("MyInstance");
   proxy.RemoveCounter("MyInstance");
   scope.Complete();
}

proxy.Close();

//Traces:
1
2
2

Instance Management and Transactions
WCF forces the service instance to equate method boundaries with transaction boundaries and to be state-aware, which means to purge all its instance state at method boundaries. By default, once the transaction completes, WCF destroys the service instance, ensuring there are no leftovers in memory that might jeopardize consistency.
The lifecycle of any transactional service is controlled by the ReleaseServiceInstanceOnTransactionComplete property of the ServiceBehavior attribute. When ReleaseServiceInstanceOnTransactionComplete is set to true (the default), it disposes of the service instance once the method completes the transaction, in effect turning any WCF service into a per-call service as far as the instance programming model is concerned.
This heavy-handed approach did not originate with WCF. All distributed transactional programming models on the Microsoft platform, ever since MTS, through COM+ and Enterprise Services, equated a transactional object with a per-call object. The architects of these technologies simply did not trust developers to properly manage the state of the object in the face of transactions, something that is both intricate and a non-intuitive programming model. The main disadvantage is that all developers that want to benefit from transactions have to adopt the non-trivial per-call programming model (see Figure 2), while most developers feel much more at ease with the familiar session-based stateful programming model of regular Microsoft .NET Framework objects.
I personally have always felt that equating transactions with per-call instantiation is a necessary evil, and yet, conceptually, it is distorted. One should only choose the per-call instancing mode when scalability is required, and, ideally, transactions should be separated from the object instance management and the application's regard for scalability.
If your application is required to scale, then choosing per-call and using transactions will work very well together. However, if you do not need scalability (which is probably the common case with most applications) your services should be allowed to be session-based, stateful, and transactional. The rest of this column presents my solution to the problem of enabling and preserving the session-based programming model while using transactions with common services.

Session-Based Services and VRMs
WCF does allow you to maintain the session semantic with a transactional service by setting ReleaseServiceInstanceOnTransactionComplete to false. In this case WCF will stay out of the way and will let the service developer worry about managing the state of the service instance in the face of transactions. The per-session service still must equate method boundaries with transaction boundaries because every method call may be in a different transaction and a transaction may end between method calls in the same session. While you could manage that state manually just as with a per-call service (or use some other advanced WCF features outside the scope of this column), you could use VRMs for the service members, as shown in Figure 3.
Figure 3 Using VRMs by Per-Session Transactional Service
[ServiceBehavior(ReleaseServiceInstanceOnTransactionComplete = false)]
class MyService : IMyContract
{
   Transactional<string> m_Text = new Transactional<string>("Some initial value");
   TransactionalArray<int> m_Numbers = new TransactionalArray<int>(3);

   [OperationBehavior(TransactionScopeRequired = true)]
   public void MyMethod()
   {
      m_Text.Value = "This value will roll back if the transaction aborts";

      //These will roll back if the transaction aborts
      m_Numbers[0] = 11;
      m_Numbers[1] = 22;
      m_Numbers[2] = 33;
   }
}
The use of VRMs enables a stateful programming model: the service instance simply accesses its state as if no transactions were involved. Any changes made to the state will commit or roll back with the transaction. However, I find Figure 3 to be an expert programming model, and thus it has its own pitfalls. It requires familiarity with VRM, meticulous definition of members, as well as the discipline to always configure all operations to require transactions, and to disable releasing the instance upon completion.

Transactional Durable Services
In the October 2008 installment of this column ("Managing State With Durable Services"), I presented the support that WCF offers for durable services. A durable service retrieves its state from the configured store and then saves its state back into that store on every operation. The state store may or may not be a transactional resource manager.
If the service is transactional, it should of course use only transactional storage and enlist it in each operation's transaction. That way, if a transaction aborts, the state store will roll back to its pre-transaction state. However, WCF does not know whether a service is designed to propagate its transactions to the state store, and by default it will not enlist the storage in the transaction even if the storage is a transactional resource manager, such as SQL Server 2005 or SQL Server 2008. To instruct WCF to propagate the transaction and enlist the underlying storage, set the SaveStateInOperationTransaction property of the DurableService attribute to true:
[Serializable]
[DurableService(SaveStateInOperationTransaction = true)]
class MyService: IMyContract
{...}
SaveStateInOperationTransaction defaults to false, thus state storage will not participate in the transaction. Since only a transactional service could benefit from having SaveStateInOperationTransaction set to true, if it is true then WCF will insist that all operations on the service either have TransactionScopeRequired set to true or have mandatory transaction flow. If the operation is configured with TransactionScopeRequired set to true, the ambient transaction of the operation will be the one used to enlist the storage.

Transactional Behavior
In the case of the DurableService attribute, the word durable is a misnomer, since it does not necessarily indicate a durable behavior here. All it means is that WCF will automatically deserialize the service state from a configured storage and then serialize it back again on every operation. Similarly, the persistence provider behavior does not necessarily mean persistence, since any provider that derives from the prescribed abstract provider class will suffice.
The fact that the durable service infrastructure is, in reality, a serialization infrastructure enabled me to leverage it into a technique for managing service state in the face of transactions, while relying underneath on a volatile resource manager, without having the service instance do anything about it. This further streamlines the transactional programming model of WCF and yields the benefit of the superior programming model of transactions for mere objects and common services.
The first step was to define two transactional in-memory provider factories called TransactionalMemoryProviderFactory and TransactionalInstanceProviderFactory. The TransactionalMemoryProviderFactory uses a static TransactionalDictionary<ID,T> to store the service instances. The dictionary is shared among all clients and sessions. As long as the host is running, TransactionalMemoryProviderFactory allows clients to connect and disconnect from the service. When using TransactionalMemoryProviderFactory you should designate a completing operation that removes the instance state from the store using the CompletesInstance property of the DurableOperation attribute.
TransactionalInstanceProviderFactory, on the other hand, matches each session with a dedicated instance of Transactional<T>. There is no need for a completing operation since the service state will be garbage collected after the session is closed.
Next, I defined the TransactionalBehavior attribute, shown in Figure 4. TransactionalBehavior is a service behavior attribute that performs these configurations. First, it injects into the service description a DurableService attribute with SaveStateInOperationTransaction set to true. Second, it adds the use of either TransactionalMemoryProviderFactory or TransactionalInstanceProviderFactory for the persistent behavior according to the value of the AutoCompleteInstance property. If AutoCompleteInstance is set to true (the default) it will use TransactionalInstanceProviderFactory. Finally, if the TransactionRequiredAllOperations property is set to true (the default), TransactionalBehavior will set TransactionScopeRequired to true on all the service operation behaviors, thus providing all operations with an ambient transaction. When it is explicitly set to false, the service developer can choose which operations will be transactional.
Figure 4 The TransactionalBehavior Attribute
[AttributeUsage(AttributeTargets.Class)]
public class TransactionalBehaviorAttribute : Attribute,IServiceBehavior
{
   public bool TransactionRequiredAllOperations
   {get;set;}

   public bool AutoCompleteInstance
   {get;set;}

   public TransactionalBehaviorAttribute()
   {
      TransactionRequiredAllOperations = true;
      AutoCompleteInstance = true;
   }
   void IServiceBehavior.Validate(ServiceDescription description,
                                  ServiceHostBase host)
   {
      DurableServiceAttribute durable = new DurableServiceAttribute();
      durable.SaveStateInOperationTransaction = true;
      description.Behaviors.Add(durable);

      PersistenceProviderFactory factory;
      if(AutoCompleteInstance)
      {
         factory = new TransactionalInstanceProviderFactory();
      }
      else
      {
         factory = new TransactionalMemoryProviderFactory();
      }

      PersistenceProviderBehavior persistenceBehavior =
                                new PersistenceProviderBehavior(factory);
      description.Behaviors.Add(persistenceBehavior);

      if(TransactionRequiredAllOperations)
      {
         foreach(ServiceEndpoint endpoint in description.Endpoints)
         {
            foreach(OperationDescription operation in endpoint.Contract.Operations)
            {
               OperationBehaviorAttribute operationBehavior =
                  operation.Behaviors.Find<OperationBehaviorAttribute>();
               operationBehavior.TransactionScopeRequired = true;
            }
         }
      }
   }
   ...
}
When using the TransactionalBehavior attribute with the default values, the client is not required to manage or interact in any way with the instance ID. All that is necessary for the client to do is use the proxy over one of the context bindings and let the binding manage the instance ID, as shown in Figure 5. Note that the service is interacting with a normal integer as its member variable. The interesting thing is that because of the durable behavior, the instance is still, of course, deactivated like a per-call service on method boundaries, yet the programming model is that of a common .NET object.
Figure 5 Using the TransactionalBehavior Attribute
[ServiceContract]
interface IMyCounter
{
   [OperationContract]
   [TransactionFlow(TransactionFlowOption.Allowed)]
   void Increment();
}

[Serializable]
[TransactionalBehavior]
class MyService : IMyCounter
{
   int m_Counter = 0;

   public void Increment()
   {
      m_Counter++;
      Trace.WriteLine(m_Counter);
   }
}
//Client side:
MyCounterClient proxy = new MyCounterClient();

using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment();
   scope.Complete();
}

//This transaction will abort since the scope is not completed
using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment();
}

using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment();
   scope.Complete();
}

proxy.Close();

//Traces:
1
2
2

Adding Context to the IPC Binding
TransactionalBehavior requires a binding that supports the context protocol. While WCF provides context support for the basic, Web services (WS), and TCP bindings, missing from that list is the inter-process communication (IPC; also called pipes), binding. It would be valuable to have that support for the IPC binding since that would enable the use of TransactionalBehavior over IPC, yielding the benefits of IPC for intimate calls. To that end, I defined the NetNamedPipeContextBinding class:
public class NetNamedPipeContextBinding : NetNamedPipeBinding
{
   /* Same constructors as NetNamedPipeBinding */

   public ProtectionLevel ContextProtectionLevel
   {get;set;}
}
NetNamedPipeContextBinding is used exactly like its base class. You can use this binding programmatically like any other built-in binding. However, when using a custom binding in the application .config file, you need to inform WCF where the custom binding is defined. While you can do this on a per-application basis, the easier option is to reference the helper class NetNamedPipeContextBindingCollectionElement in machine.config to affect every application on the machine, as shown here:
<!--In machine.config-->
<bindingExtensions>
   ...
   <add name = "netNamedPipeContextBinding"
        type = "ServiceModelEx.                NetNamedPipeContextBindingCollectionElement,
                ServiceModelEx"
   />
</bindingExtensions>
You can use NetNamedPipeContextBinding also in your Workflow applications.
Figure 6 lists an excerpt from the implementation of NetNamedPipeContextBinding and its supporting classes (the full implementation can be found in this month's code download). The constructors of NetNamedPipeContextBinding all delegate the actual construction to the base constructors of NetNamedPipeBinding, and the only initialization they do is to set the context protection level to default to ProtectionLevel.EncryptAndSign.
Figure 6 Implementing NetNamedPipeContextBinding
public class NetNamedPipeContextBinding : NetNamedPipeBinding
{
   internal const string SectionName = "netNamedPipeContextBinding";

   public ProtectionLevel ContextProtectionLevel
   {get;set;}

   public NetNamedPipeContextBinding()
   {
      ContextProtectionLevel = ProtectionLevel.EncryptAndSign;
   }
   public NetNamedPipeContextBinding(NetNamedPipeSecurityMode securityMode) :
                                                      base(securityMode)
   {
      ContextProtectionLevel = ProtectionLevel.EncryptAndSign;
   }
   public NetNamedPipeContextBinding(string configurationName)
   {
      ContextProtectionLevel = ProtectionLevel.EncryptAndSign;
      ApplyConfiguration(configurationName);
   }

   public override BindingElementCollection CreateBindingElements()
   {
      BindingElement element = new ContextBindingElement(ContextProtectionLevel,
                            ContextExchangeMechanism.ContextSoapHeader);

      BindingElementCollection elements = base.CreateBindingElements();
      elements.Insert(0,element);

      return elements;
   }

   ... //code excerpted for space
}
The heart of any binding class is the CreateBindingElements method. NetNamedPipeContextBinding accesses its base binding collection of binding elements and adds to it the ContextBindingElement. Inserting this element into the collection adds support for the context protocol.
The rest of the implementation is mere bookkeeping to enable administrative configuration. The ApplyConfiguration method is called by the constructor, which takes the binding section configuration name. ApplyConfiguration uses the ConfigurationManager class to parse out of the .config file the netNamedPipeContextBinding section, and from it an instance of NetNamedPipeContextBindingElement. That binding element is then used to configure the binding instance by calling its ApplyConfiguration method.
The constructors of NetNamedPipeContextBindingElement add to its base class Properties collection of configuration properties a single property for the context protection level. In OnApplyConfiguration (which is called as a result of NetNamedPipeContextBinding.ApplyConfiguration calling ApplyConfiguration), the method first configures its base element and then sets the context protection level according to the configured level.
The NetNamedPipeContextBindingCollectionElement type is used to bind NetNamedPipeContextBinding with the NetNamedPipeContextBindingElement. This way, when adding NetNamedPipeContextBindingCollectionElement as a binding extension, the configuration manager knows which type to instantiate and provide with the binding parameters.

InProcFactory and Transactions
The TransactionalBehavior attribute allows you to treat almost every class in your application as transactional without compromising on the programming model of familiar .NET. The downside is that WCF was never designed to be used at a very granular level—you will have to create, open, and close multiple hosts, and your application .config file will become unmanageable with scores of service and client sections. To address these issues, in my book Programming WCF, 2nd Edition I defined a class called the InProcFactory, which lets you instantiate a service class over WCF:
public static class InProcFactory
{
   public static I CreateInstance<S,I>() where I : class
                                         where S : I;
   public static void CloseProxy<I>(I instance) where I : class;
   //More members
}
When using InProcFactory, you utilize WCF at the class level without ever resorting to explicitly managing the host or having client or service .config files. To make the programming model of TransactionalBehavior accessible at every class level, the InProcFactory class uses NetNamedPipeContextBinding with transaction flow enabled. Using the definitions of Figure 5, InProcFactory enables the programming model of Figure 7.
Figure 7 Combining TransactionalBehavior with InProcFactory
IMyCounter proxy = InProcFactory.CreateInstance<MyService,IMyCounter>();

using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment();
   scope.Complete();
}

//This transaction will abort since the scope is not completed
using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment();
}
using(TransactionScope scope = new TransactionScope())
{
   proxy.Increment();
   scope.Complete();
}

InProcFactory.CloseProxy(proxy);

//Traces:
Counter = 1
Counter = 2
Counter = 2
The programming model of Figure 7 is identical to that of plain C# classes, without any ownership overhead, and yet the code fully benefits from transactions. I see it as a fundamental step toward the future, where memory itself will be transactional and it will be possible for every object to be transactional.
Figure 8 shows the implementation of the InProcFactory, with some code removed for brevity. InProcFactory's static constructor is called once per app domain, allocating in each a new unique base address using a GUID. This lets InProcFactory be used multiple times on the same machine, across app domains and processes.
Figure 8 The InProcFactory Class
public static class InProcFactory
{
   struct HostRecord
   {
      public HostRecord(ServiceHost host,string address)
      {
         Host = host;
         Address = new EndpointAddress(address);
      }
      public readonly ServiceHost Host;
      public readonly EndpointAddress Address;
   }
   static readonly Uri BaseAddress = new Uri("net.pipe://localhost/" +
                                             Guid.NewGuid().ToString());
   static readonly Binding Binding;
   static Dictionary<Type,HostRecord> m_Hosts = new Dictionary<Type,HostRecord>();

   static InProcFactory()
   {
      NetNamedPipeBinding binding = new NetNamedPipeContextBinding();
      binding.TransactionFlow = true;
      Binding = binding;
      AppDomain.CurrentDomain.ProcessExit += delegate
                                             {
                         foreach(HostRecord hostRecord in m_Hosts.Values)
                                                {
                                                 hostRecord.Host.Close();
                                                }
                                             };
   }

public static I CreateInstance<S,I>() where I : class
                                         where S : I
   {
      HostRecord hostRecord = GetHostRecord<S,I>();
      return ChannelFactory<I>.CreateChannel(Binding,hostRecord.Address);
   }
   static HostRecord GetHostRecord<S,I>() where I : class
                                          where S : I
   {
      HostRecord hostRecord;
      if(m_Hosts.ContainsKey(typeof(S)))
      {
         hostRecord = m_Hosts[typeof(S)];
      }
      else
      {
         ServiceHost host = new ServiceHost(typeof(S),BaseAddress);
         string address = BaseAddress.ToString() + Guid.NewGuid().ToString();
         hostRecord = new HostRecord(host,address);
         m_Hosts.Add(typeof(S),hostRecord);
         host.AddServiceEndpoint(typeof(I),Binding,address);
         host.Open();
      }
      return hostRecord;
   }
   public static void CloseProxy<I>(I instance) where I : class
   {
      ICommunicationObject proxy = instance as ICommunicationObject;
      Debug.Assert(proxy != null);
      proxy.Close();
   }
}
InProcFactory internally manages a dictionary that maps service types to a particular host instance. When CreateInstance is called to create an instance of a particular type, it looks in the dictionary, using a helper method called GetHostRecord. If the dictionary does not already contain the service type, this helper method creates a host instance for it and adds an endpoint to that host, using a new GUID as the unique pipe name. CreateInstance then grabs the address of the endpoint from the host record and uses ChannelFactory<T> to create the proxy.
In its static constructor, which is called upon the first use of the class, InProcFactory subscribes to the process exit event to close all hosts when the process shuts down. Finally, to help the clients close the proxy, InProcFactory provides the CloseProxy method, which queries the proxy to ICommunicationObject and closes it. To learn how you can take advantage of transactional memory, see the Insights sidebar "What Is Transactional Memory?."
What Is Transactional Memory?
You may have heard of transactional memory, the new technology for managing shared data that many claim will solve all the problems you encounter when authoring concurrent code. You may have also heard that transactional memory promises more than it can deliver and is nothing more than a research toy. The truth lies somewhere between these two extremes.
Transactional memory allows you to avoid the management of individual locks. Instead, you can structure your program in well-defined sequential blocks—units of work, or transactions, as they're called in the database world. You can then let the underlying runtime system, compiler, hardware, or a combination provide the desired isolation and consistency guarantees.
Typically, the underlying transactional memory system provides optimistic concurrency control on a rather finely grained basis. Instead of always locking a resource, the transactional memory system assumes that there is no contention. It also detects when these assumptions were incorrect and then rolls back any tentative changes made in the transaction. Depending on the implementation, the transactional memory system may then attempt to re-execute the block of code until it can complete without contention. Again, the system is able to detect and manage contention without requiring you to specify or code creative back-off and retry mechanisms yourself. When you have optimistic, fine-grained concurrency control, contention management, and no need to specify and manage specific locks, you are able to think about solving your problem in a serial way while using components that take advantage of concurrency.
Transactional memory promises to provide composition, a feat that existing locking mechanisms cannot easily perform. To compose multiple operations or multiple objects together, you generally need to increase the granularity of the lock—usually by wrapping these operations or objects together under one lock. Transactional memory automatically manages fine-grained locking on behalf of your code while providing deadlock avoidance, such that composition is provided without hurting scalability or introducing deadlocks.
There are no large-scale commercial implementations of transactional memory today. Experimental software approaches using libraries, language extensions, or compiler directives have been published in academia and to the Web. Hardware that can provide limited transactional memory does exist in some high-end, highly concurrent environments, but the software that leverages this hardware hides its explicit use. The research community is very excited about transactional memory, and you should expect to see some of this research making it into more accessible products over the next decade.
The accompanying column describes the creation of volatile resource managers that you can use with current transaction technologies to provide atomicity—all or nothing execution characteristics—and the subsequent manageability, quality, and other advantages that transactional programming provides. Transactional memory provides similar functionality, but for any arbitrary data type, using fairly lightweight runtime software or hardware primitives, and focuses on providing scalability, isolation, and composition as well as atomicity without the need to create your own resource manager. When transactional memory is broadly available, programmers will gain not only the benefits of volatile resource managers such as a simpler programming model but will also realize the scalability gains brought about by a transactional memory manager.
— Dana Groff, Senior Program Manager, Microsoft Parallel Computing Platform Team

Send your questions and comments to mmnet30@microsoft.com.

Juval Lowy is a software architect with IDesign, providing WCF training and architecture consulting. His recent book is Programming WCF Services, 2nd Edition (O'Reilly, 2008). He is also the Microsoft Regional Director for the Silicon Valley. Contact Juval at www.idesign.net.


Net Matters
Round-Robin Access To The ThreadPool
Stephen Toub

Q I'm currently using the Microsoft .NET Framework ThreadPool, and I've run into a situation I'm not sure how to solve. I start off with a large batch of work items that get queued, and then a second (smaller batch) arrives after the first batch starts its processing. Initially, some of the work in the large batch will be dispatched to all of the worker threads in the ThreadPool. However, when the second batch arrives, I want the distribution to be fair, with each batch getting serviced equally, rather than the first batch getting full attention due to it arriving first.
When one of the batches finishes, I'd like whichever one still needs processing to then get the attention of all the worker threads. Is there anything I can do to layer such batching functionality on top of the ThreadPool?
A In past columns, I've showed how to layer various types of functionality on top of the existing .NET ThreadPool. In the October 2004 issue of MSDN Magazine, I showed how to add support onto the ThreadPool for waiting on queued work items (see "ThreadPoolWait and HandleLeakTracker"). In the November 2004 issue, I showed how to add support for work item priorities (see "ThreadPoolPriority, and MethodImplAttribute"). And in the March 2006 issue, I showed how to add cancellation support (see "Abortable Thread Pool"). In the future, I'll be able to look back at the January 2009 issue and say that there I showed how to add round-robin scheduling support on top of the ThreadPool.
The problem you're looking to solve first requires an understanding of how the ThreadPool dispatches work. Internally, it maintains a queue of the work items that have been queued to it. When a thread in the pool is available to execute work, it goes back to the work queue and pulls off the next item. The order in which this processing happens is undocumented and should most definitely not be relied upon (as it could and likely will change in future versions).
Today it's implemented in a very simple manner: a first-in-first-out (FIFO) queue. Thus, the first work to be queued will be the first work to be picked up by a thread. In your batching scenario, this means all of the work from the first batch will be in the queue in front of all of the work from the second batch. And thus all of the first batch's work will be dispatched before that of the second batch. For some scenarios, this is fine and optimal. For your scenario, you need more control.
One of the easiest ways to get that control from the ThreadPool is to substitute your own delegate for the one the user really wants executed. As an example, let's say you wanted to catch all unhandled exceptions thrown out of queued work and raise an event for each. To do that, you could write code such as that in Figure 1. Then, instead of using ThreadPool.QueueUserWorkItem, you'd use ExceptionThreadPool.QueueUserWorkItem. The work would still be executed by the ThreadPool, but the pool's threads would actually be executing the delegate you queued rather than the one the user provided. Invoking your delegate would then invoke the user-provided delegate, catching any exceptions and raising the target event.
Figure 1 Shimming the ThreadPool
public static class ExceptionThreadPool {
  public static void QueueUserWorkItem(
      WaitCallback callback, object state) {
    ThreadPool.QueueUserWorkItem(delegate {
      try { callback(state); }
      catch (Exception exc) {
        var handler = UnhandledException;
        if (handler != null)
          handler(null,
            new UnhandledExceptionEventArgs(exc, false));
      }
    });
  }

  public static event
    UnhandledExceptionEventHandler UnhandledException;
}
Note that this technique, while powerful, does come at a cost: an extra delegate needs to be allocated, an extra delegate needs to be invoked, and so forth. Whether that cost is prohibitive can be determined only by you and your scenarios, but this kind of layering is typically more cost-effective than writing your own thread pool from scratch.
This is, of course, a very simple example, but you can do more complicated things. The priority pool example to which I referred earlier stores the user-provided delegates in its own data structures. It then queues to the pool a substitute delegate that comes back and searches those data structures for the right delegate to execute, preferring to execute first those that have higher priorities. You can employ a similar technique to address your batching predicament.
Take a moment to imagine that rather than having a single queue, you had one queue per batch. Each batch would put work into its own relevant queue. You'd then use the pool's threads to round-robin between all of the queues. The delegate you queue to the real ThreadPool comes back into your data structures, and starting with the next queue to be examined, looks for work. If it finds some, it executes from that queue. If not, it goes on to looking at the next queue.
In this manner, you are able to provide fair scheduling between the batches. If there is only one batch, the threads are going to always pull work from that one queue. If there are multiple batches, the threads will visit each of the queues, giving them approximately equal attention. Figure 2 provides you with an overview of what this solution will look like.
Figure 2 RoundRobinThreadPool Approach
To get things going, you first need a data structure to store the user-provided delegate. My representation is shown in Figure 3. This data structure contains three properties. The first two should look familiar; they are the state the user provides to QueueUserWorkItem, so of course you need to cache that away. The third property might be unfamiliar, however. Associated with every thread of execution in .NET is a System.Threading.ExecutionContext that represents information such as the current user, any state associated with the logical thread of execution, code-access security information, and so forth. It's important that this context flow across asynchronous points of execution.
Figure 3 Capturing a Work Item
internal class WorkItem {
  public WaitCallback WaitCallback;
  public object State;
  public ExecutionContext Context;

  private static ContextCallback _contextCallback = state => {
    var item = (WorkItem)state;
    item.WaitCallback(item.State);
  };

  public void Execute() {
    if (Context != null)
      ExecutionContext.Run(Context, _contextCallback, this);
    else WaitCallback(State);
  }
}
For example, if you're impersonating a specific Windows identity and you call ThreadPool.QueueUserWorkItem, the work you queued should execute under that same Windows identity. If it doesn't, that's a potential security hole. Starting with the .NET Framework 2.0, this context flows automatically by default across all async points in your code: ThreadPool.QueueUserWorkItem, creating a new Thread, asynchronous delegate invocation, and so on.
However, you're playing by a different set of rules with the implementation being discussed here. By changing the order in which delegates are queued from the order in which they're executed, there's no longer a direct correspondence between this ExecutionContext flow and the work items the user provided. As such, your implementation needs to correctly cache the ExecutionContext with the user-provided delegate and then use that captured context to run that delegate.
Now that you have a work item, let's take a look at the queue in which it will be held (shown in Figure 4). The RoundRobinThreadPool.Queue data structure itself is fairly simple. Internally, it contains a Queue<WorkItem> to store all of the work items provided to it, a reference to the RoundRobinThreadPool instance with which the queue is associated, and a Boolean value that denotes whether the Dispose method has been called on the queue. It then provides QueueUserWorkItem methods with the same signature as the ThreadPool's.
Figure 4 A Round-Robin Queue
public sealed class RoundRobinThreadPool {
  private List<Queue> _queues;
  ...

  public sealed class Queue : IDisposable {
    internal Queue(RoundRobinThreadPool pool) { _pool = pool; }

    internal Queue<WorkItem> _workItems = new Queue<WorkItem>();
    private RoundRobinThreadPool _pool;
    internal bool _disposed;

    public void QueueUserWorkItem(WaitCallback callback) {
      QueueUserWorkItem(callback, null);
    }

    public void QueueUserWorkItem(WaitCallback callback, object state) {
      if (_disposed)
        throw new ObjectDisposedException(GetType().Name);
      var item = new WorkItem {
        Context = ExecutionContext.Capture(),
        WaitCallback = callback, State = state };
      lock (_pool._queues) _workItems.Enqueue(item);
      _pool.NotifyNewWorkItem();
    }

    public void Dispose() {
      if (!_disposed)  {
        lock (_pool._queues) {
          if (_workItems.Count == 0)
            _pool.RemoveQueueNeedsLock(this);
        }
        _disposed = true;
      }
    }
  }
}
When QueueUserWorkItem is called, the callback and state provided by the user (along with the current ExecutionContext) are captured into a WorkItem. This work is then stored into the generic queue, and the relevant pool is notified that new work has arrived. It is important for you to note that a lock is used to protect the work item queue, as QueueUserWorkItem may be called concurrently from multiple threads, and you need to ensure that invariants are held.
Note, too, that the object being locked is a global queues list coming from the pool. I'm using one fairly course-grained lock for the whole implementation. A more efficient implementation would likely use more fine-grained locking, such as using individual locks per queue rather than one for the whole all-up RoundRobinThreadPool. For ease of implementation and simplicity I've opted for the single lock.
The Dispose method is used when this queue is no longer needed. In a typical batching scenario, a queue will be created, work queued to it, and then the queue will be disposed. If the Dispose method were to simply remove this queue from the pool, it's likely that it would be removed while there were still work items in it to be processed.
As such, Dispose does two things. First, it checks whether or not there are any work items remaining. If there aren't, the queue calls into the pool to remove itself. Second, it marks itself as having been disposed. You will see in a moment how the pool handles a situation where it comes across a disposed pool that has not been removed.
Figure 5 shows the rest of the implementation, that of the RoundRobinThreadPool class itself. The pool contains four fields:
  • A list of the individual queues maintained by the pool (which also serves as the lock previously mentioned).
  • A default queue for the pool.
  • An integer representing the next queue to be searched for work.
  • The callback delegate that's actually queued to the ThreadPool.
Figure 5 RoundRobinThreadPool
public sealed class RoundRobinThreadPool {
  private List<Queue> _queues;
  private Queue _defaultQueue;
  private int _nextQueue;
  private WaitCallback _callback;

  public RoundRobinThreadPool() {
    _queues = new List<Queue>();
    _callback = DequeueAndExecuteWorkItem;
    _nextQueue = 0;
    _defaultQueue = CreateQueue();
  }

  public Queue CreateQueue() {
    var createdQueue = new Queue(this);
    lock (_queues) _queues.Add(createdQueue);
    return createdQueue;
  }

  public void QueueUserWorkItem(WaitCallback callback)  {
    QueueUserWorkItem(callback, null);
  }

  public void QueueUserWorkItem(WaitCallback callback, object state) {
    _defaultQueue.QueueUserWorkItem(callback, state);
  }

  private void RemoveQueueNeedsLock(Queue queue) {
    int index = _queues.IndexOf(queue);
    if (_nextQueue >= index) _nextQueue--;
    _queues.RemoveAt(index);
  }

  private void NotifyNewWorkItem() {
    ThreadPool.UnsafeQueueUserWorkItem(_callback, null);
  }

  private void DequeueAndExecuteWorkItem(object ignored) {
    WorkItem item = null;

    lock (_queues) {
      var searchOrder =
        Enumerable.Range(_nextQueue, _queues.Count - _nextQueue).
        Concat(Enumerable.Range(0, _nextQueue));

      foreach (int i in searchOrder) {
        var items = _queues[i]._workItems;
        if (items.Count > 0) {
          item = items.Dequeue();
          _nextQueue = i;
          if (queue._disposed &&
              items.Count == 0) RemoveQueueNeedsLock(_queues[i]);
          break;
        }
      }
      _nextQueue = (_nextQueue + 1) % _queues.Count;
    }
    if (item != null) item.Execute();
  }

  ... // RoundRobinThreadPool.Queue and .WorkItem, already shown
}
When a RoundRobinThreadPool is initialized, all of this state is configured. In particular, the default queue is initialized by calling the CreateQueue method. This CreateQueue method is the same method that's exposed publicly to allow a developer to add another queue to the pool (such as when a new batch of work arrives that needs its own isolated queue). It simply instantiates a new RoundRobinThreadPool.Queue instance (the type explored in Figure 3), adds it to the list of queues, and returns it.
For ease of use, RoundRobinThreadPool exposes its own QueueUserWorkItem methods; these simply target the default queue that was created when the pool was instantiated.
Next to examine is the NotifyNewWorkItem method. You'll remember that when QueueUserWorkItem was called on a queue, after storing the work item, the queue called to this NotifyNewWorkItem method on the pool. This method simply delegates to the real ThreadPool, submitting a delegate that will come back to the DequeueAndExecuteWorkItem method (to be examined shortly) that will, as its name aptly implies, dequeue and execute a work item from the round-robin pool.
Notice that NotifyNewWorkItem is calling ThreadPool.UnsafeQueueUserWorkItem rather than ThreadPool.QueueUserWorkItem. The "Unsafe" prefix simply implies that it's not flowing ExecutionContext; not doing so has a performance benefit. And as this implementation is already handling ExecutionContext flow manually, there is no need for the ThreadPool to attempt to do so as well.
DequeueAndExecuteWorkItem is where the real magic happens. This method first generates an order in which to search the queues. The search order goes from the next queue to be examined up to the end of the list and then circles back around, starting at the beginning of the list and going up to the queue at which it started. To simplify the implementation, the LINQ Enumerable.Range method is being used to generate the two lists, which are then concatenated together using the LINQ Enumerable.Concat method.
Once it has the search order, it goes hunting for work items. Each queue is examined in the specified order, and as soon as a work item is found, the work item is removed and the next pointer is updated. The work item is then invoked using the Execute method shown in Figure 3.
There's one particularly interesting line of code here, and that's the check to see whether the queue from which an item was just retrieved is both disposed and empty. If the pool finds such a queue, it knows that it won't ever have more items added to it (as it's been disposed) and, thus, doesn't need to keep it around anymore. At that point, RemoveQueueNeedsLock is used to remove the target queue from the queues list and potentially update the next queue pointer in case it's now out of range.
Note that this method does not use a lock internally but accesses shared state; hence I've named the method with a "NeedsLock" suffix to remind myself that it needs to be called while the lock is held. You'll notice that both call sites to RemoveQueueNeedLock—one in the queue's Dispose method and one in the pool's DequeueAndExecuteWorkItem method—call this while holding the queues lock.
With the implementation complete, you can now test this in your code. In the following example, I've created a single static instance of the RoundRobinThreadPool. When a batch of work arrives to be processed, a new queue is created, all of the work is queued to this queue, and the queue is disposed:
private static RoundRobinThreadPool _pool =
  new RoundRobinThreadPool();
...
private void ProcessBatch(Batch b) {
  using(var queue = _pool.CreateQueue()) {
    foreach(var item in b) {
      queue.QueueUserWorkItem(() => ProcessItem(item));
    }
  }
}
Even though all of the work for the first batch to arrive will be scheduled first, as soon as the second batch arrives, it will start getting an approximately equal share of the processing resources.
There are more bells and whistles that could be added to this implementation, and the implementation could likely be improved from a performance perspective. However, with very little code, you've been able to create an abstraction that takes advantage of the .NET ThreadPool and all of the capabilities and functionality it provides, yet still get support for the batch-based fairness you were seeking.

Send your questions and comments for Stephen to netqa@microsoft.com.

Stephen Toub is a Senior Program Manager on the Parallel Computing Platform team at Microsoft. He is also a Contributing Editor for MSDN Magazine.


Usability in Practice
When Things Go Wrong
Dr. Charles Kreitzberg and Ambrose Little
Code download available from the MSDN Code Gallery
Browse the Code Online
Contents
The Usability Challenge
Error Messages Matter
Key Questions to Ask at the Design Stage
Formatting Error Messages
Programming Techniques
Consistent Handling
A Well-Constructed Error Message
Frameworks
Customizing Exceptions (with Enterprise Library)
Charles Kreitzberg
THE USABILITY CHALLENGE
From a usability perspective, error messages are often nightmares. Something has gone wrong with the program and it's up to the user to decide what to do next. Ideally, the program would produce an error message to communicate to the user what went wrong and how to correct the problem. Unfortunately, many error messages fall far short of this goal.
Consider the message in Figure 1, which appeared on my PC shortly after the boot sequence. Imagine the impact this message has on a non-technical user who has no idea what the problem is. The message implies that security has been breached and the situation is desperate. In fact, the situation was not quite so dire; I later identified the error message as coming from my video editor software and everything continued to run just fine. But it's amazing how many design flaws are contained in this message:

  1. There is no indication which program produced the error message.
  2. The message fails to explain why the program is being terminated.
  3. The message is very general. It says "the security information" without any reference to what the information is.
  4. The message does not indicate how severe the problem is and whether the user's computer is at risk.
  5. The user has no idea how to correct the problem or where to get more help. All he can do is press OK (which it certainly is not).
Figure 1 Wow, Termination Sounds Painful
Figure 2 Hmm, What If I’m Not Home When Windows Contacts Me?

No one is immune from confusing error messages. Even Microsoft, which has produced standards to suggest how to create good messages, ends up with an occasional clunker like the one in Figure 2, which I receive from time to time.
The Windows Vista User Experience Guide provides guidelines for error messages that state that a good error message is built around three components:

  • States that a problem occurred.
  • Explains the cause.
  • Provides a solution so that users can fix the problem.
The guidelines also suggest that a good error message is presented in a way that is:
  • Relevant—the message presents a problem that users care about.
  • Actionable—users should either perform an action or change their behavior as a result of the message.
  • User-centered—the message describes the problem in terms of target user actions or goals, not in terms of what the code is unhappy with.
  • Brief—the message is as short as possible, but no shorter.
  • Clear—the message uses plain language so that the target users can easily understand the problem and solution.
  • Specific—the message describes the problem using specific language, giving specific names, locations, and values of the objects involved.
  • Courteous—users shouldn't be blamed or made to feel stupid.
  • Rare—displayed infrequently. Frequently displayed error messages are a sign of bad design.
The error message in Figure 2 does not meet most of these criteria.

Error Messages Matter
Because end-user efficiency is so important, quality error messages are not simply a nicety. Errors bring processing to a halt, degrade the user experience, and carry a price tag. The cost of supporting users can become significant. Good error messages that enable the user to identify and correct problems can save a great deal of time, and money and minimize the impact on the user experience. But it's not always easy to produce good error messages.
By definition, error messages are produced when the program encounters an unexpected situation. The source of the problem may be far removed from the point at which the error is detected, and it may be impossible to figure out the original cause. While you cannot always control the environment, you can improve the situation by taking the time to design a useful message and translate the technical problem into user-centric and actionable terms.

Key Questions to Ask at the Design Stage
Here are some key user-centric questions you may want to consider as you think about how you will manage exceptions:
Who Is Your Audience? Are they developers, trained users, casual users, or the public?
What Types of Exceptions Will You Report? How do you want to present them?
What Support Will the User Have? Will there be Help Desk and Knowledge Base support, for example?
How Will You Get the User Back on Track? Simply notifying the user that a problem has occurred without offering an actionable way to resolve it will invariably lead to a negative user experience.
Do You Need Logging and Notification of Exception Occurrences? If so, you may need to consider how users or IT personnel will view, search, sort, filter, and maintain the logs and what the format and content of notifications will be. These should all be highly usable.
Should You Personalize the View to Different Audiences? Would, for example, a developer view and a public view be helpful? Does the use of external policy files to control the display open any security holes?
Can You Increase the Usability of Your Messages? And can you also preserve the link to the basic technical information by wrapping the exception in a new exception you create? Does passing technical information across an application boundary potentially carry any security risks?
These questions should be considered early—not at the end of the project.

Formatting Error Messages
The goals of error messages are to be as understandable, accurate, and actionable as possible. Strive to deliver:
  • As complete a picture as possible.
  • As much context as possible in terms that are meaningful to the user.
  • The suggested user actions in clear terms. Help the user be as comfortable as possible with the decision by making certain that the user understands the consequences of each choice.
Here are some suggestions to help you create good error messages:
Identify the Cause Be sure to indicate the site or application from which the error message originated. It may also be useful to indicate more in-depth information that can help locate the area of code where the error occurred, but be careful in exposing any internal details to users as such information could be used in malicious attacks. Separate additional diagnostic and lookup information from the main message so that it does not appear too technical for the end user. Consider showing details on demand (in a collapsible region, for instance) so the non-technical user does not need to deal with a technical presentation.
Use Plain Language Explain what happened using the least-technical language that you can. This can be difficult because you may not know what the user was attempting to do when the error occurred. However, the more you can relate the problem to what the user was trying to accomplish and the data involved, the better the user will be able to understand and potentially act upon it to resolve the situation. Again, be sensitive here not to expose information that could be used in an attack or to breach privacy. Explain the severity of the error and, if possible, explain the consequences of the problem.
Provide Details and Guidance For exceptions that are high level, such as "HTML Error -500 -Internal server error," attempt to provide more detail and specific guidance when you can. Explain the actions that the user can take to resolve the error and make certain that the choices and consequences are clear. If possible, provide a link to more information that can help the user make the appropriate decision. If the only option is to suggest that the user contact support, make certain that the contact information is accurate and easily updatable.
Orient the User If you use a custom error page in ASP.NET, make it visually integrated with your site and provide navigation to related pages (or at least the home page). Also, if you point users to a Web site with additional information do not dump them unceremoniously on the home page and leave it to them to locate the information; take them directly to the appropriate page.

Ambrose Little
PROGRAMMING TECHNIQUES
I agree that error messages can be difficult, and all of the suggestions that Charles makes are very useful. So how do you actually implement them in your code?
For programming purposes, it is useful to divide error messages into two categories. Programmatic errors are exceptions that occur in code due to unforeseen circumstances; when a programmatic error occurs, there may be no option except to terminate the program gracefully. User input or validation exceptions are situations where the user may be able to correct the problem; for this type of error, a well-constructed error message may enable recovery. In this column, I am focusing on the first—programmatic errors—and I'll reserve user input and validation errors for another day.

Consistent Handling
Even in the best-written program, errors will occur. The best way to deal with them is to create classes for error messages that can be used throughout the program. By channeling all error message processing through a single point, you can do the following:
  1. Ensure that all messages are presented in a complete and consistent format.
  2. Use custom exception types and assign unique error codes for each message so that it becomes easy to produce a dictionary of error messages, expand it with additional information, and localize it if appropriate.
  3. If applicable, link error messages to a Web site that provides user support or at least provides some way for the user to contact application support with the error message.
  4. Create logs that can be automatically, or with user permission, shipped to a server and used to analyze problems and improve the software.
A Well-Constructed Error Message
The dialog boxes in Figures 3 and 4 show elements of a well-structured error message. If the error message is simple, you can use a box like the one in Figure 3, adapted from the Windows Vista Guidelines. This error message will be most useful for a user-correctable error. A programmatic error may require a more comprehensive message like the one shown in Figure 4.
Figure 3 Format of a Simple Error Message
Figure 4 Message for a More Complex Error

A good source of information on handling exceptions is the MSDN Library article ".NET Framework Developer's Guide: Design Guidelines for Exceptions." While you are at the MSDN Library, take a look at the article "Exception Handling Application Block." It has the capability to wrap an exception, nesting it inside of an exception that you create with more information. This can be used to add additional information that captures the context in which the exception occurred. You can use this information to create more user-centric error messages.
If security is a concern and you want to avoid passing on technical information, you can replace the exception with a more user-centric one, rather than wrapping it. Finally, you can use the Exception Handling Block to log the error and notify appropriate stakeholders by e-mail, Windows Management Instrumentation (WMI), or some other custom mechanism.
If you are looking for a lightweight error logging and reporting facility for ASP.NET, take a look at Error Logging Modules and Handlers (ELMAH). It offers logging, several log storage options, the ability to view the logs on a Web page, e-mail, and RSS notifications. And you can create an ELMAH exception handler for EntLib, such as the one freely available on dotNetTemplar. That way you get the infrastructure that EntLib provides and the reporting services that ELMAH provides together.
If you are building applications for a Windows XP or Windows Vista environment, you may want to consider the Windows Error Reporting Service (Dr. Watson). When a user submits crash data, the Windows Error Reporting Service checks to see if there is a user message associated with it and presents the message at the time of the crash.
While these frameworks are excellent solutions for managing exception handling, using them does not guarantee usability. You still need to view the situation from the user's perspective by trying to get an understanding of the user's technical understanding and crafting the most clear, accurate, and actionable presentation.
Now let's take a look at an example and some sample code that demonstrates the concepts discussed thus far.

Customizing Exceptions (with Enterprise Library)
The following outlines an example of how to leverage the Enterprise Library Exception Handling Block to customize exceptions for a Windows Forms application; a similar technique could be employed for other Microsoft .NET Framework UI technologies.
First, centralize exception handling through some kind of ExceptionHandling class, as seen in Figure 5. This is a static class with methods that map directly to defined exception policies in your Enterprise Library configuration.
Figure 5 The ExceptionHandling Class
public static class ExceptionHandling
{
    public static Exception UiUnknown(Exception exception)
    {
        try
        {
            if (Microsoft.Practices.EnterpriseLibrary.
                ExceptionHandling.ExceptionPolicy
                  .HandleException(exception, "UiUnknown"))
                return exception;
        }
        catch (Exception ex)
        {
            return ex;
        }
        return null;
    }
}
Configure that policy in the Enterprise Library Configuration tool, as shown in Figure 6. Specify two type handlers, much like you might in a try/catch statement. The more specific handler is chosen. For both, you can first log the thrown exception using the built-in XML exception formatter with an XML logging trace listener. For the general exception, you can wrap the source exception with a new exception that has a user-centric message. You could alternatively replace it, but just in case the UI wants to have the details of the actual exception, it may be better to wrap it.
Figure 6 The Enterprise Library Configuration Tool
One important little trick you need to do when wrapping or replacing is to set the PostHandlingAction on the exception types to be ThrowNewException—this way it will throw the new, wrapped exception.
If you want the calling code to do more with the exception, the other option is to choose NotifyRethrow, which causes the EntLib ExceptionPolicy.HandleException to return true. As you'll see, for the DatabaseConnectivityException, you can handle the user-centric stuff in your custom exception type, so all the previous policy does for that is log it and notify that it should be rethrown.
From the call to ExceptionHandling.UiUnknown you get back the new, wrapped exception, the original exception, or none if the policy chooses not to throw a new one or notify to rethrow—that's a key benefit of the exception handling block—you can configure your policies outside of the application code itself (a good way to keep that cross-cutting concern out of application code). Then pass that into the ErrorMessage.Show method, which is defined in Figure 7 on the ErrorMessage form.
Figure 7 The ErrorMessage Form
public static void Show(Exception relatedException)
{
    ErrorMessage em = new ErrorMessage();
    if (relatedException == null)
    {
        em.problemDetailsContainer.Visible = false;
    }
    else
    {
        em.problemDescription.Text = relatedException.Message;
        IUIExceptionDetail detail =
          relatedException as IUIExceptionDetail;
        if (detail == null)
        {
            em.errorCode.Text = "500";
            em.problemDetails.Visible = false;
        }
        else
        {
            em.errorCode.Text = detail.ErrorCode.ToString();
            em.problemDetails.Text = detail.DetailMessage;
        }
        em.problemType.Text =
          em.GetMeaningfulExceptionType(relatedException).Name;
        em._SearchText = em.errorCode.Text + " " + em.problemType.Text;
    }
    em.ShowDialog();
}
The ErrorMessage form is a friendly screen that has company branding, contact information, and offers the ability to report the problem to the company as well as search online for a solution.
If no exception is given, it hides the details container—no need to mislead users into thinking there are details if there aren't any. (If that's the case, you may not want to even show a message.) If you do have an exception, set the problem description to the exception message. Then check to see if the exception type implements the custom IUIExceptionDetail interface.
This interface allows you to intelligently provide more meaningful exception information to users when you can. The interface looks like this:
public interface IUIExceptionDetail
{
    int ErrorCode { get; }
    string DetailMessage { get; }
}
Using this, you can create custom, meaningful, user-centric exceptions that can help users potentially solve the problems on their own or at a minimum provide a good error code that serves as an easy reference key for searching and support. The sample implementation of this interface is the DataConnectivityException type shown in Figure 8.
Figure 8 DataConnectivityException
public class DatabaseConnectivityException
  : Exception, IUIExceptionDetail
{
    public DatabaseConnectivityException(Exception innerException)
     : base(Resources.Exceptions.DatabaseConnectivityException_Message,
                innerException)
    {
        int.TryParse(
          Resources.Exceptions.DatabaseConnectivityException_Code,
          out _ErrorCode);
        _DetailMessage =
           Resources.Exceptions
             .DatabaseConnectivityException_DetailMessage;
    }

    #region IUIExceptionDetail Members
    int _ErrorCode;
    public int ErrorCode
    {
        get { return _ErrorCode; }
    }

    string _DetailMessage;
    public string DetailMessage
    {
        get { return _DetailMessage; }
    }
    #endregion
}
The most interesting thing here is the use of a RESX file to store the message, error code, and details. This allows for localization, but it also serves as a handy XML source for error message documentation—you could easily apply XSLT to the RESX format to produce some HTML or other rich text markup.
You may find this approach preferable to the more basic wrap/replace facilities that EntLib provides, and that's fine—you don't have to use EntLib for providing user-centric messages, but it is still good to externalize your exception handling into policies for logging and at least giving you the ability to customize exception handling outside of the application itself.
Looking back to the ErrorMessage.Show method, you can see the check for the IUIExceptionDetail interface. If it is found, it uses the extra supplied information to show to the user. Otherwise, it hides the details box and shows a generic error code of 500 (similar to HTTP 500).
The result for a generic/unknown error is shown in Figure 9. For a more specific error, like the DataConnectivityException, you get a message like Figure 10.
Figure 9 Generic Error Message
Figure 10 More Specific Error Message

There are a few things to note here. First, this is following the Windows Vista guidelines for error messages fairly closely. The window is a modal dialog. The dialog title tells users clearly what is generating the error so that they don't, for instance, confuse it with system errors; this could also be a specific feature of the application. There is a prominent indicator that this is an error with the large icon, which is overlaid on the application logo, serving the dual-purpose of reinforcing the source of the problem while clearly communicating it is an error. The title text also prominently tells the user that there is a problem with the app.
Right below the title shows the exception message. This should be concise but courteous and written in terms the user would understand. If possible, it should also offer solutions. In the case of the general error, you might not know, so the best approach in that case is to instruct the user to report the error or search online. In the case of the connectivity, you can supply some extra suggestions if you know likely solutions. The ideal here is to try to empower the user to resolve the problem on his own, which is why very specific messages are preferable to general messages.
Figure 11 Tell Us about the Problem Is the Featured Button Ambrose Little
The "Search for Solutions Online" link button will launch a search using "MSDN Magazine" plus the error and exception type. This is a nice feature, as users might not know what to put into the search box. If you have a user support section on your site, you could also limit the search to your site. And of course, if you have a specific lookup service (based on code), you should prefer that over a general keyword search.
The "Tell Us About the Problem" button should ship the XML log to a Web service. You might want give the user a chance to provide any contextual notes and/or contact info, and the service should return some tracking info that the user could use to reference if they choose to call in.
An alternative display you could use is shown in Figure 11. The value here is that the Tell Us About the Problem is the most prominent command on the screen. One thing to think about in your UI design is guiding users based on visual hierarchy; in other words, if something is more important or, in this case, is the command that you want to encourage users to use, ensure that it is the most prominent, obvious, and easy to use. You have to decide based on your context which actions are more important and, if there is a hierarchy, communicate that visually.
Once you have this in place, you can add an application-level handler in the Program.cs Main method like so:
Application.ThreadException += Application_ThreadException;
That method looks like this:
static void Application_ThreadException(object sender,
  System.Threading.ThreadExceptionEventArgs e)
{
    ErrorMessage.Show(ExceptionHandling.UiUnknown(e.Exception));
}
This catches all unexpected UI exceptions and routes them through the "UiUnknown" policy in your centralized exception handler class, which in this sample delegates handling the Enterprise Library, though it could be whatever exception handling framework you have. That method will potentially return an exception to show via the more user-friendly ErrorMessage form that you designed. And, of course, you can use a similar approach for more specific exception policies throughout your application.
As you see, creating good error messages is not easy, but taking the time to carefully structure them can save a great deal of user effort and reduce frustration.

Dr. Charles Kreitzberg is CEO of Cognetics Corporation, which offers usability consulting and user experience design services. His passion is creating intuitive interfaces that engage and delight users while supporting the product's business goals. Charles lives in central New Jersey where he moonlights as a performing musician.

Ambrose Little lives with his wife and four children in central New Jersey. He's been designing and developing software for more than 10 years and is honored to be an INETA speaker and Microsoft MVP. Lately, he's shifted from technical design to designing for people and is now a user experience designer for Infragistics.


{ End Bracket }
The Emergence Of Machine Translation
Vikram Dendi
Historians and futurologists alike have been fascinated with the idea of communication without language barriers. Science fiction luminaries have given flight to their thoughts in creating the Babel Fish or the Universal Translator, while history tells us about the legendary Rosetta Stone on which one text was translated into three languages. In an increasingly flattened world, it is more important than ever before to be able to get a message across, irrespective of language.
There have been many approaches adopted over the last few decades to solve this problem. Machine translation (MT) is one that's been around for decades yet has failed to become ubiquitous even after years of investment in research. Back in the 1960s there was an acronym coined for what researchers wanted to achieve with translation: FAHQT—Fully Automatic High-Quality Translation of General Text. This idealistic approach to MT proved to be unrealistic, and over time a perhaps more accurate acronym was coined: FAUT—Fully Automatic Useful Translation. The goal was not to compete with a human translator but to create a system that was sufficiently accurate to deliver real-time translations that were useful to the average user.
At Microsoft, researchers have been working toward this goal for more than a decade. The approach that the researchers here adopt combines rule-based logic with statistical methods, creating a hybrid statistical syntactic system. For language pairs (original and translated text) where we can make use of substantial linguistic information, we utilize grammar and syntax knowledge in pre- and post-processing around a statistical core engine. Where we do not have as much information, we resort to a purely statistical model that scales well to a large number of language pairs.
The machine translation technology that we developed has already proven itself to be very valuable within Microsoft, having been used since 2003 to translate nearly 140,000 Knowledge Base articles into nine major languages. Many other teams within the company have been using the technology to lower the cost and improve the coverage of their localization efforts. In 2005, the MT team was asked to broaden its scope, and since then we have been focused on expanding the use of this technology outside the company. Our general domain translation Web service has been exposed through search (providing translation functionality to search results), Microsoft Office (providing snippet and document translations), Windows Live Messenger (as a translation bot), and others (see microsofttranslator.com).
Efforts to provide general domain translation services for the Web are gaining momentum. The true value of MT is not just in the quality delivered by the translation engine but in how these translations are delivered within scenarios and the means to address any issues of quality. Unlike search, news, entertainment, or gaming, the perceived value of translation has been limited due to historical inconsistency in translation quality. It is critical for product builders to understand how to maximize the potential of MT.
Unlike today, where most of the translation offerings are portals and translation sites, the greatest value of machine translation will be as a basic and essential ingredient for scenarios that target a linguistically diverse audience. We believe that it is important to provide the means for developers, communities, and content creators to integrate translation into their workflows and use it as a means to harness the power of community.
The MSDN Translation Wiki is a good example of these principles in practice. The community is empowered to help improve the quality of the translation and also to contribute new content—a combination of core technology and the power of community.
Machine translation is an "imperfect" technology and, not unlike evaluating search results, puts the onus on the user to apply her judgment toward the appropriateness of the results delivered. Also, like search, it has the possibility of being surprisingly accurate at times and displays the scope to continually improve with new data. Microsoft is making significant investments in increasing the quality of translations.
I predict that this year will be a significant one for machine translation. In conjunction with the massive power that can be harnessed from the increasingly social Web, machine translation is beginning to deliver on its potential. In the coming months, watch for new and exciting approaches with machine translation that will bridge language gaps across the world.

Vikram Dendi is a Senior Product Manager on the Microsoft Translator team. He is responsible for business strategy and product planning and is working toward empowering developers to help build a Web without walls. He blogs at viks.org and at blogs.msdn.com/translation.