 MSDN Magazine September 2007

MSDN Magazine September 2007

ASP.NET: ScriptManager Enables AJAX In Your Web Apps

Learn how to use the ScriptManager control, which provides much of the magic behind ASP.NET AJAX.

Ben Rush Find It: Integrate Search Into Your Site With ASP.NET

We implement search providers for both Live.com and SharePoint so you can take advantage of either in your own applications.

Marco Bellinaso Networking: Get Connected With The .NET Framework 3.5

This article covers three core changes to networking support in Visual Studio 2008 and the upcoming release of the .NET Framework.

Mariya Atanasova, Larry Cleeton, Mike Flasko, and Amit Paka IIS 7.0: Extend Your WCF Services Beyond HTTP With WAS

Learn about a new IIS feature called the Windows Process Activation Service (WAS) that makes it possible to host Web services beyond HTTP.

Dominick Baier, Christian Weyer, and Steve Maine Unit Testing: Exploring The Continuum Of Test Doubles

Creating and using mock component servers simplifies unit testing. Use these examples to get started.

Mark Seemann Editor's Note: Summer in New York

As the thermometer rises, we cool down with a look at AJAX technologies.

Joshua Trupin Toolbox: ASP.NET Controls, FTP, and More

ASP.NET controls, a comprehensive FTP library, rich imaging support, and a book about SQL Server 2005.

Scott Mitchell CLR Inside Out: COM Connection Points

We create an Active Template Library (ATL) connection point server.

Thottam R. Sriram Basic Instincts: Lambda Expressions

Sample the basic concepts of lambda expressions, explore their benefits, and witness how to use them to write more expressive programs.

Timothy Ng Cutting Edge: AJAX Application Architecture, Part 1

In the first of a two-part column, Dino explains AJAX from an architectural standpoint to help developers, architects, designers, and administrators better understand the issues that affect their sites.

Dino Esposito Team System: Work Items and Undo Support

Add support for work items to the Team Foundation Server version control add-in.

Brian A. Randell Foundations: Workflow Communications

Matt Milner introduces the Windows Workflow Foundation (WF) communication architecture and covers sending data out of a workflow and submitting data to running workflow instances.

Matt Milner Security Briefs: Exploring Claims-Based Identity

Keith Brown introduces you to the new identity model in the Microsoft .NET Framework 3.0.

Keith Brown .NET Matters: Tales from the CryptoRandom

Stephen Toub and Shawn Farkas discuss creating an adapter that takes the functionality of RNGCryptoServiceProvider and adapts it to the interface of Random.

Stephen Toub and Shawn Farkas { End Bracket }: Turning the Pages with WPF

The British Library is digitizing some of its collection to reach a broad audience. Even better, the digitized versions are being turned into a rich interactive experience that brings the books to life.

Tim Sneath


ASP.NET: ScriptManager Enables AJAX In Your Web Apps

ASP.NET

ScriptManager Enables AJAX In Your Web Apps

Ben Rush
This article discusses:
  • The role of ScriptManager in ASP.NET AJAX
  • Implicit and explicit use of ScriptManager
  • Web services support in ASP.NET AJAX
  • The ASP.NET AJAX page lifecycle
This article uses the following technologies:
ASP.NET AJAX
Contents
Scripting with ScriptManager
AJAX and ScriptManager
Web Services and ScriptManager
Authentication and Personalization
How It All Works
Registering Objects with ScriptManager
Putting the AJAX in ASP.NET AJAX
Follow the Script

T
oday the consumer of a Web site can be as responsible for its content, direction, and success as the publisher. Social networking sites, blogs, online photo journals, and wikis are just a few of the new styles of Web sites springing up every day, and it's just the beginning. Your Web site can offer the best content available, but if it stands in the way of letting your users participate in its growth and development, your site will quickly fall into uselessness.

As a developer, it's your job to use the tools at your disposal to meet the demands of the average consumer. The most natural place to start when it comes to making a site more compelling is the Web browser itself, as it is the closest part of the site to the consumer. Unfortunately, it is not always easy to take advantage of many features found in current Web browsers. Writing code that interacts with the browser requires a significant effort because myriad browser and operating system combinations exist, making consistency painfully absent. It would help to have a single platform for building rich client and Web applications in parallel so that work spent perfecting the end-user experience for one browser environment doesn't need to be duplicated for another.

ASP.NET AJAX was released by Microsoft to answer this need in Web application development. My goal in writing this article is to expand your knowledge of a central component of ASP.NET AJAX called the ScriptManager control and to show how advanced programming of ASP.NET AJAX is achieved using it. ScriptManager is a server-side control that sits on your Web Form and enables the core of ASP.NET AJAX. Its primary role is the arbitration of all other ASP.NET AJAX controls on the Web Form and the addition of the right scripting libraries to the Web browser so that the client portion of ASP.NET AJAX can function. Often you will find yourself using the ScriptManager to register other controls, Web services, and client scripts.

As a server-side control, ScriptManager reacts to events in the ASP.NET page lifecycle and uses those events to coordinate the activities of all the controls, options, and code employed by ASP.NET AJAX. ScriptManager will hook a particular event, get notified when it occurs, and configure a few settings depending on the environment; this process will repeat itself several times through the rendering cycle of your ASP.NET page. The settings it configures, however, are often the exact settings needed to make your use of ASP.NET AJAX seamless.

I will first go over a few of the main features of ASP.NET AJAX that the ScriptManager control enables for you, and then begin an exploration of the lifecycle of the control on the server. By understanding the internals of ScriptManager, you will gain a deeper appreciation for the options it provides for Web application development and learn how to get the most out of it.

Let's start with scripting, as it is a central element to ASP.NET AJAX. In fact, all of the functionality of ASP.NET AJAX depends upon its scripting libraries. I will then run through some features of AJAX support in ASP.NET AJAX, how to interact with Web services, and finally talk a bit about authentication. During each discussion I'll show you how to tweak options through ScriptManager.

Scripting with ScriptManager

The block of code in Figure 1 demonstrates the standard way of defining a class in ASP.NET AJAX. The internals of the client script library are beyond the scope of this article, but in summary the common steps necessary to create a class based on the ASP.NET AJAX script extensions are as follows:

  1. Register the namespace with ASP.NET AJAX.
  2. Create a constructor method.
  3. Create the class prototype by filling in the member methods and their functionality.
  4. Register the class with ASP.NET AJAX.
  5. Notify the client script added by the ScriptManager that you have reached the end of the type definition (the call to Sys.Application.notifyScriptLoaded).

This class exposes functionality to the client alone. However, through the ScriptManager control you can take advantage of a much more interesting side of scripting in ASP.NET AJAX where your class can expose functionality to both JavaScript on the client and Microsoft® .NET Framework code on the server. For example, if a consumer of your control was to set a property on an instance of it like this

public partial class _Default : System.Web.UI.Page
{
    protected void Page_Load(object sender, EventArgs e)
    {
        this.MyCustomButton.AlertMessage = "Hey, stop clicking me!";
    }
}

it would be intuitive to also be able to interact with that control instance through script in the browser like this:

<script type="text/javascript">
    function get_text()
    {
        var mb = $find("MyCustomButton");
        var text = mb.get_AlertMessage();
        // do something with the text
    }
</script>

Note that I use the same name, MyCustomButton, as a reference to the control on both the server and the client; I also interact with the property, AlertMessage, as if its value seamlessly crossed the server/client boundary. This feels natural, but until ASP.NET AJAX this type of unified server/client programming model was not easy to access and would have required a great deal of custom code. Today, in ASP.NET AJAX, this unified server/client paradigm is integrated and fully supported.

The magic of this tight server/client integration comes through two new interfaces, IScriptControl and IExtenderControl. To use these interfaces, you inherit your ASP.NET Web control from them, implement the required interface methods, and then register the control with the page's ScriptManager control. For example, the following skeleton of server-side control code implements the required methods of IScriptControl:

class CustomButton : Button, IScriptControl
{
    IEnumerable<ScriptReference> IScriptControl.GetScriptReferences()
    {
        ...
    }

    IEnumerable<ScriptDescriptor> IScriptControl.GetScriptDescriptors()
    {
        ...
    }
}

The methods GetScriptDescriptors and GetScriptReferences will return to the ScriptManager control all the information it needs to logically present your control code as a server and client object. The end result will be an experience like you saw previously, where object instances transcend the server/client boundary.

GetScriptReferences will return a list of script files your control code will need. One of the script files it must return contains the definition for your control as a script class—in other words, the client script version of your server-side control.

GetScriptDescriptors, on the other hand, returns what are known as ScriptDescriptors, or objects that describe the properties and events of your client class. ScriptDescriptors can be seen as holding onto metadata for your client class, including properties and their associated values.

Figure 2 shows a more complete example of the server control sketched out earlier. Here you can see the bodies of the GetScriptDescriptors and GetScriptReferences methods filled in.

Figure 2 Using GetScriptDescriptors and GetScriptReferences (Close Figure 2)
private string _alertMessage = null;

public string AlertMessage
{
    get { return _alertMessage; } set { alertMessage = value; }
}

public IEnumerable<ScriptDescriptor> GetScriptDescriptors()
{
    ScriptControlDescriptor descriptor = new ScriptControlDescriptor(
        "CustomButton", this.ClientID);
    descriptor.AddProperty("AlertMessage", this._alertMessage);
    return new ScriptDescriptor[] { descriptor };
}

public IEnumerable<ScriptReference> GetScriptReferences()
{
    ScriptReference reference = new ScriptReference(
        "MyCustomContent.JScript1.js", "MyCustomContent");
    return new ScriptReference[] { reference };
}

In the body for GetScriptDescriptors, I instantiate a ScriptDescriptor object. The constructor for the ScriptDescriptor takes the name for the control I am describing, CustomButton. I then add the AlertMessage property and its value (held by the private member _alertMessage) to ScriptDescriptor. The property that I want to interact with from the client code, AlertMessage, has now been described to ASP.NET AJAX.

In the body of GetScriptReferences, notice that I am creating and returning a ScriptReference object pointing to a .js file. This .js file maintains the client-side code for that control—and thus all its client functionality.

Now I can register the control with the ScriptManager so that it is aware the control exists (I'll explore that step later). When the page is rendered, the ScriptManager control will react to various events for which it gets notified and call GetScriptDescriptors and GetScriptReferences. These two methods will then return to the ScriptManager control ScriptDescriptors and ScriptReferences containing all the necessary information to hook up the client object to its server counterpart. In the example here, ScriptDescriptor describes a custom control named CustomButton with a property AlertMessage. Also included in the ScriptDescriptor will be the value of the property set from within GetScriptDescriptors.

The ScriptManager control will also get a reference to an external .js file called MyCustomContent.JSScript1.js. The reference to the script file will be in the form of a ScriptReference object returned from GetScriptReferences. The prototype and footer for the control's client class in the .js file is defined as shown in Figure 3.

Figure 3 Prototype for the Control Client (Close Figure 3)
CustomButton.prototype = {
    initialize : function() {
        // initialize the base
        CustomButton.callBaseMethod(this,'initialize');

        this._onclick =
            Function.createDelegate(this, this._onclick);

        Sys.UI.DomEvent.addHandler(
            this.get_element(), 'click', this._onclick);
    },
    dispose : function() {
        // release the handlers
        $clearHandlers(this.get_element());

        // call to the base to do its dispose
        CustomButton.callBaseMethod(this,'dispose');
    },
    get_AlertMessage: function(){
        return this._alertMessage;
    },
    set_AlertMessage: function(value){
        this._alertMessage = value;
        return;
    },
    _onclick: function(e){
        alert(this._alertMessage);
        return;
    }
}
CustomButton.registerClass("CustomButton",Sys.UI.Control);
if(typeof(Sys)!=='undefined') Sys.Application.notifyScriptLoaded();

The first thing you should notice is that this client class has get_ and set_ methods for the AlertMessage property. The ScriptManager control will set the value of this property to whatever you specified on the server through the ScriptDescriptor object. Since I exposed AlertMessage from the server code, whatever that property was set to on the server will now be reflected in the AlertMessage property exposed on the client object.

AJAX and ScriptManager

Many developers, when first using ASP.NET AJAX, start with the UpdatePanel control. If a ScriptManager control is included on the page and the UpdatePanel contains any controls, the controls within UpdatePanel can be asynchronously updated through the facilities of AJAX. On the designer surface of Visual Studio®, the setup typically looks like Figure 4.

Figure 4 A Simple Asynchronous Control
Figure 4 A Simple Asynchronous Control

It really could not get much more elementary than this. If someone clicks the Button in this setup, the Button control will raise a postback event that will be caught by the UpdatePanel control. The UpdatePanel then resubmits the postback event as a partial postback and its content will be asynchronously updated (without the browser fully reloading the page).

However, there are many interesting scenarios where your expectations will fail, leaving you with a mess on your hands. For example, there is a new and fascinating way to break script references using the UpdatePanel control and AJAX. In days of old you would have added script to the page as a result of a full postback such as this:

protected void Button1_Click(object sender, EventArgs e)
{
    Page.ClientScript.RegisterStartupScript(
        this.GetType(),"myscript","alert('hello world!');");
}

But this new method can break under ASP.NET AJAX. Why? Because ClientScriptManager is not a control that understands partial postbacks and partial rendering, and so script code registered with it will not be included in the page response for a partial postback.

Instead, if you wanted the button to introduce client script into the page output, you would leverage the ScriptManager:

protected void Button1_Click(object sender, EventArgs e)
{
    ScriptManager.RegisterStartupScript(
        this,this.GetType(),"myscript","alert('hello world!');",true);
}

The result is the same, but the method is slightly different. In fact, the methods of the ClientScriptManager control are now included in the ScriptManager control as static methods (RegisterStartupScript, for example). Whenever you are utilizing ASP.NET AJAX and want to work with script through partial postbacks, you must now use methods exposed by the ScriptManager control instead.

As another advanced case scenario, consider the control setup in Figure 5. Under normal circumstance the LinkButton, once clicked, would cause a full postback of the Web Form. But what if you wanted to click the LinkButton and have it refresh the UpdatePanel asynchronously? In other words, you want to click the LinkButton and make the UpdatePanel behave as if the LinkButton were inside it. To do this, use a method on the ScriptManager control called RegisterAsyncPostBackControl:

Figure 5 Using LinkButton Asynchronously
Figure 5 Using LinkButton Asynchronously

protected void Page_Load(object sender, EventArgs e)
{
    // register the LinkButton1 control as one that can
    // cause partial postbacks.
    ScriptManager1.RegisterAsyncPostBackControl(LinkButton1);
}

Clicking the LinkButton will now cause the UpdatePanel to refresh asynchronously, just as if the LinkButton control were actually in the UpdatePanel.

As an example of inverse behavior, you could also make an element inside an UpdatePanel cause the whole page to refresh—a full postback. To do so, you just use a different method on the ScriptManager called RegisterPostBackControl:

protected void Page_Load(object sender, EventArgs e)
{
    // Button1 will cause a full post-back no matter
    // where it is on the page.
    ScriptManager1.RegisterPostBackControl(Button1);
}

This code will cause the control, Button1, to do a full postback of the page, even if Button1 exists within an UpdatePanel.

Let's go even further now. You have seen how to tweak the controls that trigger the UpdatePanel control's partial postback event using ScriptManager, but what if you wanted to update a control somewhere on the page—totally outside of the UpdatePanel—when the UpdatePanel refreshes? This is also possible through the ScriptManager control.

ScriptManager's RegisterDataItem method makes it easy to refresh controls or data outside an UpdatePanel. RegisterDataItem allows extra data of your choosing to be sent down to the client when an UpdatePanel control posts back; the data can be consumed by script you write on the client.

For example, say you have a situation like the controls in Figure 6. In this example, I want to update the Label with whatever value has been clicked in the Calendar control. This may seem simple, but the Calendar control is within the UpdatePanel while the Label is not. How can I get the UpdatePanel to refresh the Label? The answer is simple: if I use RegisterDataItem from my server-side code, I can get extra data sent to my client-side code. The client can consume the data sent from RegisterDataItem and refresh the Label with it:

Figure 6 Updating Controls Outside of an UpdatePanel
Figure 6 Updating Controls Outside of an UpdatePanel

protected void Calendar1_SelectionChanged(object sender, EventArgs e)
{
    ScriptManager1.RegisterDataItem(
        Label1, Calendar1.SelectedDate.ToShortDateString());
}

RegisterDataItem takes the control you are planning to update as the first argument and the raw data you wish to update the control with as the second argument. The ScriptManager control will take the data you pass, wrap it up, and send it down to the client as part of its response to the partial postback event. In your client code, you would retrieve the data sent from the ScriptManager control after the event has completed, like this:

<script type="text/javascript">
    Sys.WebForms.PageRequestManager.getInstance().add_pageLoading(
        PageLoadingHandler);
    function PageLoadingHandler(sender,args){
        var dataItems = args.get_dataItems();
        if($get('Label1')!==null){
            $get('Label1').innerHTML = dataItems['Label1'];
        }
        return;
    }
</script>

Look at the script code. It does a number of things. It registers for the pageLoading event through the PageRequestManager client class. Next, it implements the event handler for the pageLoading event, PageLoadingHandler. It gets a name/value collection of data items from the second parameter, args. Finally, it retrieves the value you want using the name of the control you supplied as the first argument to RegisterDataItem on the server.

Web Services and ScriptManager

The ease with which you can now asynchronously interact with Web services from script in ASP.NET AJAX, handle the response (including errors), and process the response gives you a great deal of power to really make your pages useful and unique. From point to point—browser to server—you are utilizing the latest technologies of the new Web in one of the most intuitive manners possible through ASP.NET AJAX.

The ScriptManager control acts as a global registrar of sorts for all the services you need in your ASP.NET AJAX application. Figure 7 shows the properties menu for the ScriptManager control as seen through Visual Studio.

Figure 7 ScriptManager Properties
Figure 7 ScriptManager Properties

You can see that I have the Scripts collection highlighted, and below that is a collection for Services (Web services) as well. You need to register with ScriptManager any services you want to interact with from your client code. To add a service reference to the ScriptManager control, you would simply expand the Services collection and add a reference as shown in Figure 8.

Figure 8 Adding a Web Service Reference
Figure 8 Adding a Web Service Reference(Click the image for a smaller view)
Figure 8 Adding a Web Service Reference
Figure 8 Adding a Web Service Reference (Click the image for a larger view)

What exactly does this do? The complete details will be spelled out shortly, but if you were to view the source for a page referencing a service through the ScriptManager, you might see something like this in its content:

<script src="WebService.asmx/jsdebug" type="text/javascript"></script>

This external reference was added to the page output by ScriptManager because the Web service is registered with it. Note that /jsdebug is appended to the name of the Web service; if this were a release build, there would only be /js appended to it. When the ASP.NET AJAX pipeline sees a request for this service with the /js text appended, it will return a script class wrapping the methods of the Web service (this is known as proxy script). The body of each method on this proxy script class will perform an asynchronous call to its corresponding Web service method. Therefore, to call the Web service methods, you simply call the corresponding method on the script class built for you by the ASP.NET AJAX Web service framework. It's really that easy.

For example, if a service named WebService exposed a method called Add, like this

[WebMethod]
public int Add(int a, int b) { return a + b; }

you could call it from this script:

function CallAdd()
{
    // method will return immediately
    // processing done asynchronously
    WebService.Add(0,6, OnMethodSucceeded, OnMethodFailed);
}

In this example, the auto-generated script proxy class handed to you by the framework wraps the Web service method calls so that you may interact with it from script. WebService.Add will thus call the corresponding Web service method, Add.

Two callbacks are typically defined when calling into a Web service: one for a success case and another for a failure case. Since the Web service call is executed asynchronously, callbacks are required to know how the call actually completed. For example, the following methods implement a success callback and failure callback, respectively:

function OnMethodSucceeded(result, eventArgs)
{
    var label = Sys.UI.DomElement.getElementById('Label1');
    var sb = new Sys.StringBuilder();
    sb.append('You got back a value of ');
    sb.append(result.toString());
    label.innerHTML = sb.toString();
}

function OnMethodFailed(error)
{
    alert(error.get_message());
}

The request is sent and the result is received asynchronously from the Web service so the end effect is similar to using the UpdatePanels control. Content can be refreshed with the data received from the Web service without a full browser postback. In the sample code shown earlier, I asynchronously update Label1 with the result of the Web service call to WebMethod Add. The end result might look like Figure 9. No full postback will occur and the end result will be absolutely seamless.

Figure 9 The Updated Label
Figure 9 The Updated Label

Authentication and Personalization

Advanced features of ASP.NET AJAX, such as authentication and personalization, use Web services, too. The authentication service in ASP.NET AJAX implements two methods: one for logging a user in and one for logging the user out:

Sys.Services.AuthenticationService.login
Sys.Services.AuthenticationService.logout

ASP.NET AJAX employs forms authentication, meaning that the user is logged in when a valid forms authentication cookie is inserted into the browser's session by the Web server. The manner by which the cookie data is inserted into the session in ASP.NET AJAX is the same as it would be via a full postback, so physically there is no difference in the mechanisms being used. Once the cookie has already been inserted into the browser's session, the client is then authenticated with the server and can view restricted pages and content.

Being able to authenticate and log in or log out a particular user from client script yields interesting possibilities for highly interactive user-based systems. An example of logging in a user through AuthenticationService and script might look like this:

<script type="text/javascript">
    function MyMethod(username, password)
    {
        Sys.Services.AuthenticationService.login(username,
            password,false,null,null,null,null,"User Context");
    }
</script>

The steps for enabling authentication in your AJAX application are well documented (see ajax.asp.net), so they needn't be covered here. But as you can clearly see, once it is enabled, forms authentication is easily implemented through script.

When the standard ASP.NET authentication service just doesn't fit the bill, you can create your own and register it with the ScriptManager control. The methods required by a Web service implementing authentication functionality are shown here:

[WebMethod]
public bool Login(string userName,
    string password, bool createPersistentCookie)
{
    ... // code to check user credentials and log user in
}

[WebMethod]
public void Logout()
{
    ... // code to log user out.
}

After you have filled in your implementation code, you must register your new authentication service with the ScriptManager control. With your new authentication service registered, any code you previously had on the client interacting with the default authentication services of ASP.NET AJAX will now utilize your service instead. Therefore, from the client script side, no change will be required to use your own custom authentication Web service.

Here's an example of registering as an authentication service through ScriptManager declaratively:

<asp:ScriptManager ID="ScriptManager1"
  runat="server" >
    <AuthenticationService
  Path="WebService.asmx" />
</asp:ScriptManager>

Without going into the implementation at this point, you may also take advantage of your own profile services in ASP.NET AJAX. A profile service can be registered in the same way through the ScriptManager control as an authentication service. The profile services give you the ability to tailor a Web site to a particular user. Since it's done from client script, the effect to the user will be intuitive and seamless. Examples are available through ajax.asp.net.

How It All Works

The ScriptManager control has two basic phases to its existence. In the first phase it verifies that the environment can support all the rich features of ASP.NET AJAX and sets up all the necessary pieces to support it. In the second phase, it actually performs asynchronous communication with the code running on the client so that the script can perform the necessary page updates. Because it is a server-side control and Web programming in ASP.NET is event-driven, the core of the ScriptManager control is in how it registers for and responds to events. An overview of the events to which it responds is shown in Figure 10.

Figure 10 Defining a Class in ASP.NET AJAX
Figure 10 Defining a Class in ASP.NET AJAX

Registering Objects with ScriptManager

You may add script and service references declaratively with ScriptManager. However, you may also add them programmatically through the Services and Scripts properties on the ScriptManager control. You also can choose special data to be sent to the client, and you can design controls to work with ScriptManager as special script controls. All of these features require that you reference your object with the ScriptManager control.

So what is ScriptManager doing with all of these references? When a page hosting a ScriptManager control and ASP.NET AJAX functionality is first requested by a browser, the initialization stage of the page's child controls will invoke ScriptManager's OnInit method. OnInit carries out a number of important steps, including a check that only one ScriptManager control resides on the page and whether the page is currently undergoing a partial postback. The most important step OnInit performs, however, is to register for a series of events generated by the host page such as InitComplete, PreRenderComplete, and PreRender. The ScriptManager control needs to know when these events occur on its parent page because ASP.NET AJAX operates by tapping into these page events.

The most crucial page event for loading script and service references into the browser is the page's PreRenderComplete event. The handler for this event is called OnPagePreRenderComplete (see Figure 11) and it is a method of the ScriptManager control itself.

Figure 11 ScriptManager's OnPagePreRenderComplete (Close Figure 11)
private void OnPagePreRenderComplete(object sender, EventArgs e)
{
    if (!IsInAsyncPostBack)
    {
        if (SupportsPartialRendering)
        {
            IPage.ClientScript.GetPostBackEventReference(
                new PostBackOptions(this, null, null, false,
                false, false, false, true, null));
        }

        RegisterGlobalizationScriptBlock();
        RegisterScripts();
        RegisterServices();
    }
    else RegisterScripts();
}

OnPagePreRenderComplete exists to process all the scripts and services that you registered with the ScriptManager control up to the PreRenderComplete event of the page (including scripts needed by script controls, scripts referenced by ScriptManagerProxy controls, and so on). If the page is not undergoing a partial postback, a globalization script block is registered, along with all your scripts and services. If, on the other hand, a partial postback is underway, only scripts are registered. Scripts need to be registered for both partial and full postbacks because the ScriptManager control offers the ability to include script at any time.

All of your scripts and services are registered during this phase, but what does that mean? How is the registration of the script or reference translated into some sort of output onto the page?

Remember that you are still processing the first page request and so you're not dealing with partial postbacks. Because of this, you can still use ClientScriptManager to register scripts. RegisterScripts, for example, iterates every script that was registered with the ScriptManager control and then hands it off to the page's default ClientScriptManager instance. When the page is rendered, ClientScriptManager will be called and the script references will be added to the page output, just like in pre-ASP.NET AJAX days.

RegisterScripts will be called during an asynchronous postback as well (due to the else clause shown in Figure 11). The method still calls into ClientScriptManager, but since ClientScriptManager won't know what to do in an asynchronous postback, nothing will happen. Instead, RegisterScripts will call an internal method, RegisterScriptIncludeInternal, that will tuck away the script reference in an internal array for later use.

What about Web service references? Recall that by adding a Web service reference to your ScriptManager control, a reference to script will be added to the page. This script reference will cause the browser to make a request at the specified URL. The ASP.NET AJAX framework will inspect the class implementing your Web service and return a proxy script class you may consume from client code. The browser will download this automatically generated proxy class, and this will be your Web service reference.

To generate the script reference to the auto-generated proxy class, RegisterScripts will iterate over every Web service reference you have and eventually call into the following method:

private string GetProxyPath(Control containingControl, bool debug)
{
    if (debug)
        return GetServicePath(containingControl, true) +
            RestHandlerFactory.ClientDebugProxyRequestPathInfo;
    else
        return GetServicePath(containingControl, true) +
            RestHandlerFactory.ClientProxyRequestPathInfo;
}

The script reference generated by GetProxyPath will be added to the page in the same manner as regular script references: by utilizing the ClientScriptManager for the first page request and by "tucking away" the reference into some internal array.

What is this internal array? It's a number of arrays, actually. Recall that for the first page request, the ScriptManager leans on the traditional method of simply adding script and service references to the ClientScriptManager class for the page. But using ClientScriptManager will not work during a partial postback. Instead, the ScriptManager must cook up some other method, and this is what the internal arrays are for. The response for a partial postback, as you will see, is actually a well-formed, easily parsable block of data that the client framework will inspect and consume. The ScriptManager control's Render event will be called and it, in turn, will call a member method, ProcessScriptRegistration, of yet another class, PageRequestManager (see Figure 12).

Figure 12 PageRequestManager's ProcessScriptRegistration (Close Figure 12)
private void ProcessScriptRegistration(HtmlTextWriter writer)
{
    owner.ScriptRegistration.RenderActiveArrayDeclarations(
        updatePanelsToRefresh, writer);
    owner.ScriptRegistration.RenderActiveScripts(
        updatePanelsToRefresh, writer);
    owner.ScriptRegistration.RenderActiveSubmitStatements(
        updatePanelsToRefresh, writer);
    owner.ScriptRegistration.RenderActiveExpandos(
        updatePanelsToRefresh, writer);
    owner.ScriptRegistration.RenderActiveHiddenFields(
        updatePanelsToRefresh, writer);
    owner.ScriptRegistration.RenderActiveScriptDisposes(
        updatePanelsToRefresh, writer);
}

This is where the script reference will be rendered to the page output for ASP.NET AJAX. Each method will take the HtmlTextWriter handed to it for the Render phase and write its necessary content to it. Each method (for example, RenderActiveScripts) will refer to each respective internal array that was filled previously.

Putting the AJAX in ASP.NET AJAX

If you use the Fiddler HTTP debugger proxy app (fiddlertool.com/fiddler), you can trace all Web traffic that moves through Internet Explorer® on your machine. Browse to the AJAX samples on ajax.asp.net and watch what happens through Fiddler; you can see that an actual HTTP POST occurs each time you invoke a partial postback event. The HTTP POST contains the usual HTTP headers, but it also includes one that you may have never seen before:

x-microsoftajax: Delta=true

This header is key. Once the ScriptManager control identifies this header, instead of passively injecting references into the page output, it will inspect the data sent in the form POST and render a response back to the client in a format the client script understands.

Along with writing service and script references to the initial page output, the ScriptManager control will also initialize the client-side functionality. When it renders the first time, the ScriptManager control makes sure that two startup scripts are registered with the ClientScriptManager. One script calls into a method that initializes the client-side runtime. The other initializes the client-side version of the PageRequestManager class discussed earlier. For an understanding of AJAX, the second initialization script—the one that initializes PageRequestManager—is the most important.

This client script is written to the page output through a method called RenderPageRequestManagerScript. I removed a lot of the plumbing code from the method so it's easier to read, but in general it looks like this:

internal void RenderPageRequestManagerScript(HtmlTextWriter writer)
{
    ...
    writer.Write(_owner.UniqueID);
    ...
    writer.Write(_owner.IPage.Form.ClientID);
    ...
    RenderUpdatePanelIDsFromList(writer, _allUpdatePanels);
    ...
    writer.Write(GetAsyncPostBackControlIDs(true));
    ...
    writer.Write(GetPostBackControlIDs(true));
    ...
}

Notice how it is writing out UpdatePanels, PostBackControl IDs, and AsyncPostBackControl IDs to the page; these are all controls that need to take part in the ASP.NET AJAX experience. The client-side PageRequestManager is responsible for tracking all events generated by the controls registered with ScriptManager; the RenderPageRequestManagerScript method is what initializes the PageRequestManager operating on the client with the exact controls it should watch.

When a postback event is fired on the client, PageRequestManager will identify whether it was caused by any of the controls identified by the script written in RenderPageRequestManagerScript. If it was, PageRequestManager will cancel the postback event and repackage it. The newly packaged data from the postback event will then be transmitted to the server using the client class Sys.Net.WebRequest (which is public and can be used from your client-side code). The header x-microsoftajax: Delta=true will be set in the POST request sent to the server through Sys.Net.WebRequest.

On the server side, the ScriptManager control has now been instantiated and loaded into the control tree of the page. ScriptManager is a server-side control and so will be notified of the data in the form POST through LoadPostData, a method of ASP.NET that lets individual controls filter through the form POST for pertinent information. (Note that this is a standard event not specific to ASP.NET AJAX.) If you refer to the event diagram in Figure 10, you'll see that LoadPostData occurs immediately after InitComplete on the page. Within LoadPostData, the ScriptManager control identifies the controls that caused the form POST (including the UpdatePanel the control resided in, if applicable). The identity of the control that caused the postback event is tucked away for later use.

So far the ScriptManager control recognizes that a partial postback is occurring and has identified the controls that caused the postback. Now you can build the response to the client. The ScriptManager control completely overwrites the default Render method for its host page, taking over the rendering of your ASP.NET page. Realize that, up until now, you have simply been seeing the ScriptManager control as something to tweak various settings in your ASP.NET AJAX application. Now you can finally see that you give it these various options because it, arguably, becomes your new page class, complete with its own format for rendering controls to the page output stream.

In fact, ScriptManager will see to it that the default rendering for two objects are overwritten: first for the Page itself, and then for the Web Form on the page. Therefore, when the ASP.NET page framework asks the page to render, the ScriptManager's own internal implementation will be called. By knowing what controls caused the postback and their associations—is it in an UpdatePanel with other controls, or is the source of the postback event somehow linked to an UpdatePanel?—the ScriptManager control knows which child controls should be asked to render and which should be ignored. Where the default behavior for a Page class in ASP.NET is to ask each child control to render, ScriptManager will only ask those it deems necessary.

It is also worth noting that within the override for the Form objects' rendering, the ScriptManager will process and send down to the client all of the extra data you registered with it via the RegisterDataItem method. Therefore, it is important that you have your data ready to be sent down to the client by the time the Form object is asked to render itself to the client. The data to be sent down to the client will be encoded in a raw or JavaScript Object Notation (JSON) serialized format.

Finally, the client framework gets the asynchronous response from the server and parses out data. The ScriptManager control has packed into the response all the control IDs and new markup so the client framework can simply perform scripting operations on the browser's document object model to update the page content. Given that this whole process occurred asynchronously, the browser is quickly and silently updated and the user of the Web page gains a better experience.

Follow the Script

ASP.NET AJAX is a powerful technology. To harness many of the ASP.NET AJAX features in your Web app, you will need to employ the ScriptManager control as I've demonstrated in this article.

The ScriptManager control handles many details of the ASP.NET AJAX implementation for you. By now you should be noticing that the ScriptManager's presence can be felt frequently in instances where the default behavior of a control such as the UpdatePanel isn't exactly what you need. In addition to scripting features and AJAX, the ScriptManager control also supports such high-end features as authentication and personalization.

Scripts and services must be registered by the page's PreRenderComplete event with the ScriptManager control. You may add a script reference to the page during either full or partial postbacks. Other objects are asked to hand over to the real ScriptManager all of their script and service references when scripts are being registered.

Finally, ASP.NET AJAX is implemented on the client by overriding the default rendering implementation for its host page. The ScriptManager will take control of the page rendering cycle from then on and will send down to the client a response it can easily parse through and use to update elements within the browser. By overriding the default page rendering, the right controls can be rendered to the client in a specialized format, including with it data registered with the ScriptManager.

Ben Rush is a Microsoft .NET consultant in the Midwest specializing in ASP.NET technologies. You can learn more about Ben at ben-rush.net/blog.

© 2007 Microsoft Corporation and CMP Media, LLC. All rights reserved; reproduction in part or in whole without permission is prohibited.


Find It: Integrate Search Into Your Site With ASP.NET

Find It

Integrate Search Into Your Site With ASP.NET

Marco Bellinaso
Code download available at: Search2007_09.exe (164KB)
This article discusses:
  • Choosing a search solution
  • Building a search framework for ASP.NET
  • Implementing a Live Search provider
  • Implementing a SharePoint provider
This article uses the following technologies:
ASP.NET, Live Search, SharePoint
Contents
Building a Search Infrastructure
Building the Live Search Provider
Building the Provider for SharePoint 2007
Building the Custom Search Page
A Look to the Future: the SearchDataSource Control

N
o matter how much content you provide or how good it is, if your customers can't find what they're looking for they'll go elsewhere. It's no wonder then that there are hundreds of search technology vendors. Which solution should you choose?

Windows Live™ Search crawls content on the Internet and also exposes search functionalities to developers through a Web service that allows the search to be scoped down to just your site. You can then render the results however you see fit.

Microsoft® Office SharePoint® Server 2007 is another good choice. It's a complete site-building and management solution that also provides robust search functionality. Figure 1 shows the results of a SharePoint Search integrated into a custom ASP.NET page.

Figure 1 SharePoint Search
Figure 1 SharePoint Search(Click the image for a smaller view)
Figure 1 SharePoint Search
Figure 1 SharePoint Search (Click the image for a larger view)

If you're only interested in enterprise search, you may opt for the Microsoft Office SharePoint Server (MOSS) 2007 for Search edition (rather than the Standard or Enterprise versions of MOSS). The main differences are the maximum number of documents they can index and the ability to index advanced content such as external databases. However, even with the Standard version of the Search edition you can index file shares, Exchange public folders, Windows SharePoint Services (WSS) sites, and external Web sites.

Building a Search Infrastructure

With so many search providers to choose from, it's important that your site uses an abstracted implementation that doesn't rely on specifics of the underlying search engine. That way, you can decide later whether you'll use Live Search or SharePoint or some other provider as your search solution. In this article, we're going to build a pluggable search infrastructure and toolset that you can add to your Web site today (the code is available for download from the MSDN® Magazine Web site); as part of this, we'll implement search providers for both Live Search and SharePoint so you can take advantage of either in your own applications.

The implementation will include the following features:

  • Support for advanced queries with multiple words and AND/OR operators, exact phrases, words exclusion, and wildcard characters where supported by the query engine.
  • Pagination of results, with a configurable number of items per page.
  • Support for multiple languages, including the ability to filter or rank by language.
  • Highlighting of the searched words in the found items' summary or description text.

It will also include the ability to configure (in the web.config file) which search engine you want to run your query with. To do this we'll build an object model that implements a simplified provider model design pattern—that is, multiple provider classes that actually implement the search using a specific search engine (think of these as the data-access layer classes), and a SearchEngine class (the business-logic layer class) that dynamically instantiates and uses the provider specified in the configuration settings.

Figure 2 shows the code for the high-level SearchEngine class, which exposes properties such as ResultsPerPage, QueryText, and Highlight, and has a parameterless method called ExecuteQuery that delegates to the underlying ExecuteQuery method of the concrete provider class initialized in the class's static constructor (so that it's created only once for the application's lifetime).

Figure 2 SearchEngine Class (Close Figure 2)
public class SearchEngine
{
    private static ISearchProvider _provider = null;

    static SearchEngine()
    {
        if (!string.IsNullOrEmpty(
            WebConfigurationManager.AppSettings["Search.Provider"]))
        {
            _provider = (ISearchProvider)
                Activator.CreateInstance(Type.GetType(
                    WebConfigurationManager.AppSettings[
                        "Search.Provider"]));
        }
        else _provider = new LiveSearchProvider();
    }

    public SearchEngine()
    {
        if (!string.IsNullOrEmpty(WebConfigurationManager.AppSettings[
                "Search.ResultsPerPage"]))
            this.ResultsPerPage = Convert.ToInt32(
                WebConfigurationManager.AppSettings[
                    "Search.ResultsPerPage"]);

        if (!string.IsNullOrEmpty(
            WebConfigurationManager.AppSettings[
                "Search.Culture"]))
            this.Culture = WebConfigurationManager.AppSettings[
                "Search.Culture"];

        if (!string.IsNullOrEmpty(
            WebConfigurationManager.AppSettings[
                "Search.HighlightEnabled"]))
            this.Highlight = Convert.ToBoolean(
                WebConfigurationManager.AppSettings[
                    "Search.HighlightEnabled"]);
    }

    public string ProviderName
    {
        get { return _provider.GetType().Name; }
    }

    private int _resultsPerPage = 10;
    public int ResultsPerPage
    {
        get { return _resultsPerPage; }
        set { _resultsPerPage = value; }
    }

    private int _pageIndex = 0;
    public int PageIndex
    {
        get { return _pageIndex; } set { _pageIndex = value; }
    }

    private string _culture = "en-US";
    public string Culture
    {
        get { return _culture; } set { _culture = value; }
    }

    private string _queryText = "";
    public string QueryText
    {
        get { return _queryText; } set { _queryText = value; }
    }

    private bool _highlight = true;
    public bool Highlight
    {
        get { return _highlight; } set { _highlight = value; }
    }

    public SearchResults ExecuteQuery()
    {
        return _provider.ExecuteQuery(this.QueryText,
            this.Culture, this.ResultsPerPage,
            this.PageIndex, this.Highlight);
    }
}

The provider name and other settings are located in the web.config file's <appSettings> section, as follows:

<add key="Search.Provider" value="LiveSearchProvider" />
<add key="Search.ResultsPerPage" value="5"/>
<add key="Search.Culture" value="en-US"/>
<add key="Search.HighlightEnabled" value="True"/>

The provider object returned by the Activator.CreateInstance call is saved into a static private reference of type ISearchProvider, the interface that all provider classes implement. The interface has a single method, ExecuteQuery, which takes as input all the information needed by the providers to run the query. It is defined as follows:

public interface ISearchProvider
{
    SearchResults ExecuteQuery(string queryText, string culture,
        int resultsPerPage, int resultsPageIndex, bool hightlight);
}

The ExecuteQuery method returns an instance of the SearchResults class, which includes static fields to return the total number of results (TotalResults), the list of results (WebResults), and the list of spelling suggestions for the query text. Here's the class code:

public class SearchResults
{
    public int TotalResults = 0;
    public List<string> Suggestions = new List<string>();
    public List<ResultItem> WebResults = new List<ResultItem>();
}

Suggestions is a collection of strings and WebResults is a collection of ResultItem objects. The ResultItem class exposes properties that return the result's URL, Title, and Description. You can see the actual code in Figure 3.

Figure 3 ResultItem Class (Close Figure 3)
public class ResultItem
{
    public ResultItem(string url, string title, string description)
    {
        this.Url = url;
        this.Title = title;
        this.Description = description;
    }

    private string _url;
    public string Url
    {
        get { return _url; } set { _url = value; }
    }

    private string _title;
    public string Title
    {
        get { return _title; } set { _title = value; }
    }

    private string _description;
    public string Description
    {
        get { return _description; } set { _description = value; }
    }
}

Now that the basic architecture is ready we can start implementing provider classes.

Building the Live Search Provider

The Live Search Web service allows you to programmatically run the same kinds of search queries you can run on live.com. The Web service is located at soap.search.msn.com/webservices.asmx (with the WSDL for the service available at soap.search.msn.com/webservices.asmx?wsdl), and by using the Visual Studio® 2005 Add Web Reference option, you can create a proxy class that you use in your code to execute a search. Here's how you instantiate it:

MSNSearchService searchEngine = new MSNSearchService();

After creating an instance of the search engine class, you create the search request like so:

SearchRequest searchRequest = new SearchRequest();

Be aware that with the free service you're limited to 25,000 queries a day. Live counts your queries by tracking you through the unique "application ID" that you must get at search.msn.com/developer and use as the value for the SearchRequest's AppID property:

searchRequest.AppID = "185A8FADA7931D06D119C2FC1FF104";

The other essential property to set for the SearchRequest object is Query, which contains the search text as you would specify it on live.com. You can specify multiple words (the AND operator is implicit; you can explicitly use OR to search for any of the specified words), you can search for an exact phrase by wrapping it in quotation marks, you can use parentheses to group words, and you can use the minus sign to exclude words. The following example looks for documents that contain the word "ajax", the exact phrase "code example", either "Marco" or "Francesco", and that do not contain the word "xml" or the phrase "Visual Basic":

searchRequest.Query =
    "ajax AND \"code example\" AND (Marco OR Francesco)" +
    " –(xml Visual Basic)";

This query searches the Web as a whole, just as it would if you ran the query from live.com. If you want to restrict the results to only pages on your site, you can add site:www.yoursite.com to the end of the query, like so:

searchRequest.Query =
    "ajax site:msdn.microsoft.com/msdnmag";

Please refer to help.live.com for a complete list of search keywords. The best way to experiment with search strings and combining multiple filters and keywords, however, is to use the query builder tool you'll find at dev.live.com/livesearch/sdk (see Figure 4). This tool even generates the C# code necessary to invoke the Web service with the query, so it's really a must for your toolbox.

Figure 4 Live Search SDK's Query Builder
Figure 4 Live Search SDK's Query Builder(Click the image for a smaller view)
Figure 4 Live Search SDK's Query Builder
Figure 4 Live Search SDK's Query Builder (Click the image for a larger view)

There are other SearchRequest properties that allow you to set the SafeSearch level (use Moderate if you want to exclude lewd images and texts from the results), flags (such as whether you want to highlight query words in the results), and the CultureInfo for the current culture:

searchRequest.SafeSearch = SafeSearchOptions.Moderate;
searchRequest.Flags = SearchFlags.MarkQueryWords;
searchRequest.CultureInfo = "en-US";

Now you must specify which type of results you want to have returned by Live Search: Web pages, spelling suggestions (for example, to suggest "membership" when you type "membershipa"), addresses and phone numbers, news, inline answers or images. For a Web search that returns pages from your site, you usually just want to include Web results and spelling suggestions; you do that by creating an array of two SourceRequest objects as the value for the SearchRequest's property:

SourceRequest[] sourceRequests = new SourceRequest[2];
searchRequest.Requests = sourceRequests;

Then you actually define the two source requests, creating the SourceRequest objects, specifying their type (Web and Spelling), the fields you want to get in the results (only the title for the spelling suggestions, and all typical fields such as title, Url and description for the Web results). You can also specify the number of results you want (three is often sufficient for spelling suggestions, while 10-20 is typically appropriate for Web results) and the offset of the first result (the index to start from among the total results, which is important for pagination):

sourceRequests[0] = new SourceRequest();
sourceRequests[0].Source = SourceType.Web;
sourceRequests[0].ResultFields =
    ResultFieldMask.All | ResultFieldMask.SearchTagsArray;
sourceRequests[0].Count = 20;
sourceRequests[0].Offset = 0;

sourceRequests[1] = new SourceRequest();
sourceRequests[1].Source = SourceType.Spelling;
sourceRequests[1].ResultFields = ResultFieldMask.Title;
sourceRequests[1].Count = 3;
sourceRequests[1].Offset = 0;

You run the query by calling the MSNSearchService Search method with the SearchRequest object defined in input:

SearchResponse searchResponse = searchEngine.Search(searchRequest);

The call to Search returns an object of type SearchResponse. Its Responses property is an array of objects of type SourceResponse, and there's one for each SourceRequest defined above. In the sample, this means there's an array of two objects, one for Web results and the other for spelling suggestions. The SourceResponse object has a property named Total that's the total number of results found for the query executed. You should note that Total isn't the number of returned results (which is usually a subset according to the current index of a pageable report), but rather the overall total count. The actual results are accessible through the SourceResponse's Results property, which is a collection of Result objects. The Result object has several properties including Title, Url, and Description, which may be filled or empty according to what you've specified to return in the corresponding SourceRequest definition:

// Process the Web results
int total =
  searchResponse.Responses[0].Total;
foreach (Result result in
  searchResponse.Responses[0].Results)
{
    // access the result.Title, result.
    //Url, and other values...
}

// Process the spelling results
foreach (Result result in
  searchResponse.Responses[1].Results)
{
    // print the result.Title value...
}

That's all there is to it. To recap, you defined a SearchRequest with two SourceRequest objects, specifying to return Web results and spelling suggestions, with highlighting and moderation enabled; then you called the search engine's Search method for that SearchRequest and processed the resulting SourceResponses' Results collections. Figure 5 lists the complete code for the LiveSearchProvider class, which implements the ISearchProvider interface defined earlier. The code is similar to what you've seen so far in this section almost line by line, but of course much of the required information is not hardcoded into the class but is instead taken as input parameters or read from the configuration file.

Figure 5 ISearchProvider for Live Search (Close Figure 5)
public class LiveSearchProvider : ISearchProvider
{
    public SearchResults ExecuteQuery(string queryText, string culture,
        int resultsPerPage, int resultsPageIndex, bool highlight)
    {
        SearchResults results = new SearchResults();

        // create the search engine and search request
        MSNSearchService searchEngine = new MSNSearchService();
        SearchRequest searchRequest = new SearchRequest();

        // define the search requests: one for web results...
        SourceRequest[] sourceRequests = new SourceRequest[2];
        sourceRequests[0] = new SourceRequest();
        sourceRequests[0].Source = SourceType.Web;
        sourceRequests[0].ResultFields =
            ResultFieldMask.All | ResultFieldMask.SearchTagsArray;
        sourceRequests[0].Count = resultsPerPage;
        sourceRequests[0].Offset = resultsPageIndex * resultsPerPage;

        // ...and one for spelling suggestions
        sourceRequests[1] = new SourceRequest();
        sourceRequests[1].Source = SourceType.Spelling;
        sourceRequests[1].ResultFields = ResultFieldMask.Title;
        sourceRequests[1].Count = 3;
        sourceRequests[1].Offset = 0;

        // set the text to be searched, the App ID, and more
        searchRequest.Requests = sourceRequests;
        string siteFilter = " site:" +

            WebConfigurationManager.AppSettings[
                "LiveSearch.TargetSite"];
            searchRequest.Query = queryText;
        if (!string.IsNullOrEmpty(
            WebConfigurationManager.AppSettings[
                "LiveSearch.TargetSite"]))
            searchRequest.Query += siteFilter;
        searchRequest.SafeSearch = SafeSearchOptions.Moderate;
        searchRequest.AppID =
            WebConfigurationManager.AppSettings["LiveSearch.AppID"];
        searchRequest.Flags = highlight ?
            SearchFlags.MarkQueryWords : SearchFlags.None;
        searchRequest.CultureInfo = culture;

        // run the query and get the response
        SearchResponse searchResponse =
            searchEngine.Search(searchRequest);

        // get the spelling suggestions
        foreach (Result result in searchResponse.Responses[1].Results)
        {
            results.Suggestions.Add(result.Title.Trim()
                .Replace(siteFilter, "")
                .Replace("\xe000", "").Replace("\xe001", ""));
        }

        // get the web results from the Web SourceResponse
        results.TotalResults = searchResponse.Responses[0].Total;
        foreach (Result result in searchResponse.Responses[0].Results)
        {
            results.WebResults.Add(new ResultItem(result.Url,
                result.Title.Replace("\xe000", "")
                    .Replace("\xe001", ""),
                result.Description.Replace("\xe000", "<b>")
                    .Replace("\xe001", "</b>")));
        }

        return results;
    }
}

Building the Provider for SharePoint 2007

SharePoint Portal 2007 search capabilities are very powerful. We'll explore just the features that are most important for our specific goal, but you may want to deepen your knowledge by reading some of the many guides available for system administrators and developers.

Let's start with the SharePoint Web-based administrative application, in the Search Settings section under Shared Services, and create a new search Content Source. This defines what you want to index, and how, and when. SharePoint Portal allows you to index typical Internet Web sites, SharePoint Web sites (which it handles in a specific way because it takes the content directly from the database), documents in a file share, custom databases, and more. In this case, you'll select the first option and specify the root URL of the site you want to index. SharePoint will start indexing from that URL and continue by following the links recursively (you can specify how deep it can go or that it should index all pages it finds on the same domain). Refer to Figure 6 to see the page where you create the content source.

Figure 6 Adding a Search Content Source
Figure 6 Adding a Search Content Source(Click the image for a smaller view)
Figure 6 Adding a Search Content Source
Figure 6 Adding a Search Content Source (Click the image for a larger view)

While creating the content source you can also tell SharePoint (at the bottom of the page) to start the indexing process immediately after confirming the new source, and set up a schedule for the incremental indexing (for example, to run the process every day at midnight, when there's a lighter load on the servers). You can see these configurations in Figure 7.

Figure 7 Configuring the Indexing Schedule
Figure 7 Configuring the Indexing Schedule(Click the image for a smaller view)
Figure 7 Configuring the Indexing Schedule
Figure 7 Configuring the Indexing Schedule (Click the image for a larger view)

The last configuration step consists of creating a search scope so that when you run the search you can limit it to the desired source. This is particularly important if you have multiple Web sites, each indexed separately, and you want to search content from one site and not from the others. To complete this operation, just fill out the fields of the wizard in Figure 8 (note that you may also create a scope that includes everything except a specific URL or content source).

Figure 8 Search Scope Wizard
Figure 8 Search Scope Wizard(Click the image for a smaller view)
Figure 8 Search Scope Wizard
Figure 8 Search Scope Wizard (Click the image for a larger view)

When the new content source is defined, from the Search Settings page click the command to update the content sources so that the number of documents they include is recalculated. This useful information can then be reviewed from the page that lists all content sources.

Now it's time to write some code to query the index. In SharePoint this can be done in one of two ways: through its local object model (the SharePoint DLL assemblies) or through SharePoint Web services. The first approach is only possible when your site runs on the same server as SharePoint, which is quite rare. Therefore, it's much better to use the Web service, which you can call whether or not you're running on the same server. Add a Web Reference in Visual Studio to YourSharePointServerName/_vti_bin/search.asmx so that it creates the QueryService proxy class. The most important part of the coding work is crafting a correct query, which uses a combination of a SQL-like language in combination with XML. Here's an example of the SQL portion of a query:

string sqlQuery = "SELECT Path, Rank, Title, HitHighlightedSummary
    FROM MSDNMagazine..scope() WHERE FREETEXT('ajax asynchronous')
    ORDER BY Rank DESC";

You see that it's very similar to a normal SQL query that uses the SQL Server Full Text Search, even though indexes are stored in files and not in a SQL Server database. The FROM clause does not specify a table name, but rather the search scope name, followed by "..scope()". The WHERE filter uses the FREETEXT predicate, so you can specify advanced multiple conditions. The query will return the path, title, rank, and description of the found results, as specified in the SELECT. The SQL is not passed in to the query engine as is, but rather is enclosed as part of an XML request that defines other information, such as the number of results you want to retrieve, the starting offset (1 being the minimum, instead of 0 like it is for Live Search), the culture, and more. Figure 9 shows a complete request (notice how the SQL query is inserted into the surrounding XML).

Figure 9 SharePoint XML Search Request (Close Figure 9)
string xmlQuery = string.Format(@"
<QueryPacket urn:Microsoft.Search.Query"">
  <Query>
    <SupportedFormats>
      <Format>urn:Microsoft.Search.Response.Document:Document</Format>
    </SupportedFormats>
    <Context>
      <QueryText type=""MSSQLFT"" language=""en-US"">{0}</QueryText>
    </Context>
    <Range>
      <StartAt>1</StartAt>
      <Count>20</Count>
    </Range>
    <EnableStemming>true</EnableStemming>
    <TrimDuplicates>true</TrimDuplicates>
    <IgnoreAllNoiseQuery>true</IgnoreAllNoiseQuery>
    <ImplicitAndBehavior>true</ImplicitAndBehavior>
    <IncludeRelevanceResults>true</IncludeRelevanceResults>
    <IncludeSpecialTermResults>true</IncludeSpecialTermResults>
    <IncludeHighConfidenceResults>true</IncludeHighConfidenceResults>
  </Query>
</QueryPacket>", sqlQuery);

Actually running the query and retrieving the results is just a matter of instantiating a QueryService object, settings its credentials to appropriate credentials (SharePoint filters the results according to the permissions associated with these credentials, but since we're targeting a public Web site's search scope, no result will be filtered out), and calling the Query method with the XML request in input:

QueryService queryService = new QueryService();
queryService.UseDefaultCredentials = true;
string xml = queryService.Query(xmlQuery);

Figure 10 shows a partial XML response.

Figure 10 Sample XML Response Returned by the SharePoint Search Web Service (Close Figure 10)
<ResponsePacket xmlns="urn:Microsoft.Search.Response">
  <Response>
    <Range>
      <StartAt>1</StartAt>
      <Count>10</Count>
      <TotalAvailable>11</TotalAvailable>
      <Results>
        <Document xmlns="urn:Microsoft.Search.Response.Document">
          <Action>
            <LinkUrl>
              http://msdn.microsoft.com/msdnmag/issues/07/02/cuttingedge
            </LinkUrl>
          </Action>
          <Properties xmlns=
              "urn:Microsoft.Search.Response.Document.Document">
            <Property>
              <Name>PATH</Name>
              <Type>String</Type>
              <Value>
                http://msdn.microsoft.com/msdnmag/issues/07/02/cuttingedge
              </Value>
            </Property>
            <Property>
              <Name>RANK</Name>
              <Type>Int64</Type>
              <Value>419</Value>
            </Property>
            <Property>
              <Name>TITLE</Name>
              <Type>String</Type>
              <Value>Cutting Edge: Perspectives on ASP.NET AJAX –
                     MSDN Magazine, February 2007</Value>
            </Property>
            <Property>
              <Name>HITHIGHLIGHTEDSUMMARY</Name>
              <Type>String</Type>
              <Value>Ways to &lt;c0&gt;AJAX&lt;/c0&gt; in ASP.NET
                    &lt;ddd/&gt; First, there's the acronym-it stands for
                    &lt;c1&gt;Asynchronous&lt;/c1&gt; JavaScript and XML.
                    &lt;ddd/&gt;  with ASP.NET &lt;c0&gt;AJAX&lt;/c0&gt;,
                    but it can be used as the core code to add basic
                    &lt;c0&gt;AJAX&lt;/c0&gt; capabilities to ASP.NET 1.
                    &lt;ddd/&gt; </Value>
            </Property>
          </Properties>
        </Document>
        <Document>
            ...
        </Document>
        <!-- more Document nodes -->
      </Results>
    </Range>
    <Status>SUCCESS</Status>
  </Response>
</ResponsePacket>

The rest of the code would just parse the XML text through an XmlDocument. In Figure 11 you find the complete implementation of the SharePointSearchProvider.

Figure 11 ISearchProvider for SharePoint (Close Figure 11)
public class SharePointSearchProvider : ISearchProvider
{
    public SearchResults ExecuteQuery(string queryText, string culture,
        int resultsPerPage, int resultsPageIndex, bool highlight)
    {
        SearchResults results = new SearchResults();

        // prepare the SharePoint XML query string
        string sqlQuery = string.Format(@"
                SELECT Path, Rank, Title, HitHighlightedSummary
                FROM {0}..scope() WHERE FREETEXT('{1}')
                ORDER BY Rank DESC",
            WebConfigurationManager.AppSettings[
                "SharePointSearch.TargetScope"],
            queryText.Replace("'","''").Replace("\"","\"\""));

            string xmlQuery = string.Format(@"
<QueryPacket urn:Microsoft.Search.Query"">
  <Query>
    <SupportedFormats>
      <Format>urn:Microsoft.Search.Response.Document:Document</Format>
    </SupportedFormats>
    <Context>
      <QueryText type=""MSSQLFT"" language=""{1}"">{0}</QueryText>
    </Context>
    <Range>
      <StartAt>{2}</StartAt>
      <Count>{3}</Count>
    </Range>
    <EnableStemming>true</EnableStemming>
    <TrimDuplicates>true</TrimDuplicates>
    <IgnoreAllNoiseQuery>true</IgnoreAllNoiseQuery>
    <ImplicitAndBehavior>true</ImplicitAndBehavior>
    <IncludeRelevanceResults>true</IncludeRelevanceResults>
    <IncludeSpecialTermResults>true</IncludeSpecialTermResults>
    <IncludeHighConfidenceResults>true</IncludeHighConfidenceResults>
  </Query>
</QueryPacket>",
            sqlQuery, culture,
            (resultsPageIndex * resultsPerPage) + 1, resultsPerPage);

        // run the query and get the response
        QueryService queryService = new QueryService();
        queryService.UseDefaultCredentials = true;
        string xml = queryService.Query(xmlQuery);

        // if there were no errors in the query, parse the results
        if (xml.IndexOf("<Status>SUCCESS</Status>") > -1)
        {
            XmlDocument xmlResults = new XmlDocument();
            xmlResults.LoadXml(xml.Replace(
                " xmlns=\"urn:Microsoft.Search.Response\"",
                "").Replace("xmlns=" +
                "\"urn:Microsoft.Search.Response.Document.Document\"",
                "").Replace(" xmlns=" +
                "\"urn:Microsoft.Search.Response.Document\"", ""));

            results.TotalResults = Convert.ToInt32(
                xmlResults.SelectSingleNode(
                    "//Response/Range/TotalAvailable").InnerText);

            foreach (XmlNode resNode in xmlResults.SelectNodes(
                "//Response/Range/Results/Document/Properties"))
            {
                string url = "";
                string title = "";
                string description = "";

                foreach (XmlNode propNode in resNode.ChildNodes)
                {
                    if (propNode.ChildNodes[0].InnerText == "PATH")
                        url = propNode.ChildNodes[2].InnerText;
                    else if (propNode.ChildNodes[0].InnerText ==
                            "TITLE")
                        title = propNode.ChildNodes[2].InnerText;
                    else if (propNode.ChildNodes[0].InnerText ==
                        "HITHIGHLIGHTEDSUMMARY")
                        description =
                            propNode.ChildNodes[2].InnerText;
                }

                // if hightlighting is enabled, replace all
                // <c?></c?> instances with <b></b>, otherwise
                // with an empty string
                if (highlight)
                {
                    description = Regex.Replace(
                        description, @"<c\d+>", "<b>");
                    description = Regex.Replace(
                        description, @"</c\d+>", "</b>");
                }

                results.WebResults.Add(
                    new ResultItem(url, title, description));
            }
        }

        return results;
    }
}

It's worth noting that the QueryService also exposes a QueryEx method in addition to Query. This method returns the results in a DataTable instead of as raw XML. This makes things easier for the developer because no parsing would be required. However, it doesn't return the total number of results, which is necessary to implement the pagination feature in our search page (unless you want to always retrieve all results, and then do the pagination in-memory, but I don't recommend this approach), so we'll use the basic Query method instead.

Building the Custom Search Page

Now that we have the high-level interface and the underlying provider code complete, what's left is just the custom Search.aspx. You'll find the full implementation in the downloadable source code, so I'll simply highlight the most important part. The search box is made up by a textbox and a submit button. For the results reporting section of the page, a Repeater control lists the spelling suggestions as hyperlinks that point to the page itself, with the suggestion text passed in the querystring's "q" parameter:

<asp:Repeater runat="server" ID="rptSuggestions">
    <ItemTemplate>
        <asp:HyperLink runat="server" ID="lnkSuggestionUrl"
            NavigateUrl='<%# "~/Search.aspx?q=" +
                DataBinder.Eval(Container, "DataItem") %>'
            Text='<%# DataBinder.Eval(Container, "DataItem") %>' />
    </ItemTemplate>
    <SeparatorTemplate>, </SeparatorTemplate>
</asp:Repeater>

In case you're wondering why LinkButton controls were not used in place of the hyperlinks, it's because they would have caused postbacks instead of again requesting the page with an updated querystring, with the result that the user could not save a shortcut to the page and later come back to get the same results.

Likewise, the same principle is true for pagination bars that have hyperlinks that point to the same page, with the same "q" parameter for the query text, and with a "p" parameter that specifies the index of the page of results to show. The actual Web results are displayed through a DataList control that renders each result's title and a hyperlink to the actual found page, along with some description text like so:

<asp:DataList ID="dlstResults" runat="server" ...styles here...>
    <ItemTemplate>
        <asp:HyperLink runat="server" ID="lnkResultUrl"
            NavigateUrl='<%# Eval("Url") %>'
            Text='<%# Eval("Title") %>' /><br /><%# Eval("Description") %>
    </ItemTemplate>
</asp:DataList>

The codebehind is also simple. The developer of the UI doesn't need to know anything about the search engine that's used under the covers. In fact, she actually doesn't even need to know which one is used. The reason for this is because all this logic is wrapped up in the SearchEngine class. The following code shows how the class is used, and the results are bound to the UI controls:

SearchEngine search = new SearchEngine();
search.PageIndex = this.PageIndex;
search.QueryText = txtQueryText.Text;
SearchResults results = search.ExecuteQuery();

rptSuggestions.DataSource = results.Suggestions;
lblTotal.Text = results.TotalResults.ToString();
dlstResults.DataSource = results.WebResults;
Page.DataBind();

The result is the same as you saw in Figure 1.

A Look to the Future: the SearchDataSource Control

The May 2007 release of ASP.NET Futures (asp.net/downloads/futures) introduces a new data source control named SearchDataSource that—like all other data source controls, such as ObjectDataSource and SqlDataSource)—can be bound to controls such as Repeater, DataList or GridView, and automatically provide the data to be rendered. Of course, the provided data is the result of a Web search, run by one of the provider classes registered and configured in the web.config file:

<search enabled="true">
   <providers>
      <add name="WindowsLiveSearchProvider"
         type="Microsoft.Web.Preview.Search.WindowsLiveSearchProvider,
              Microsoft.Web.Preview"
         appID="185A8FADA7931D06D119C2FC1FF104"
         siteDomainName="msdn.microsoft.com/msdnmag" />
   </providers>
</search>

The data source control itself can be added to the page either visually through the Visual Studio designer, or with the following markup:

<asp:TextBox ID="txtQuery" runat="server" />
<asp:Button ID="btnSearch" runat="server"  Text="Search" />

<asp:SearchDataSource ID="SearchDataSource1" runat="server" >
   <SelectParameters>
       <asp:ControlParameter ControlID="txtQuery" Name="query"
                             PropertyName="Text" />
   </SelectParameters>
</asp:SearchDataSource>

You see that there's a control parameter, query, that's bound to the Text property of the txtQuery TextBox. During postbacks, the data source control is automatically refreshed (you can also force that programmatically by calling the control's DataBind method), the query is run with the value contained in that TextBox, and the results are bound to the associated data control (the one with DataSourceID="SearchDataSource1"). If you used a DataList or Repeater control to display the results, its template definition would be the same as the code shown earlier for our custom solution, since the results bound to the control are of type SearchResult and contain the Title, Url and Description properties, as our WebResult class does. So the binding expressions would be identical.

If you want to create your own provider, you define a class that inherits from SearchProviderBase, overrides its Search method to provide the specific search engine's implementation, and the Initialize method to read the configuration settings:

public class SharePointSearchProvider : SearchProviderBase
{
    public SharePointSearchProvider() : base() { }

    public override SearchResult[] Search(SearchQuery searchQuery)
    { ... }

    public override void Initialize(string name,
        NameValueCollection config) { ... }
}

The SearchDataSource control approach is very elegant, since it fully implements the provider model design pattern (not a simplified version as the solution developed in this article), and the ability to visually drop this control on the form and bind it to a data control is appealing also since it doesn't require you to write any code. However, at the time of this writing the control is also very limited. I expect that Microsoft will soon extend the base provider class and the input SearchQuery type with more features and members. If this happens, you will be able to easily port the solution presented here to the SearchDataSource model: all you will need to do is move the search implementation from the class that implements the ISearchProvider interface to a class that inherits from SearchProviderBase, and read the input parameters from the provider's properties or the SearchQuery's properties, rather than from the ExecuteQuery's input values. The markup code will remain practically unchanged, since the SearchResult and the WebResult types expose the same properties.

Marco Bellinaso is the founder of www.bytecommerce.com, a free online service to sell shareware software and digital goods. He is also a frequent speaker at Microsoft conferences in Italy and is the author of the Wrox book, ASP.NET 2.0 Website Programming / Problem – Design – Solution. Marco works for Code Architects Srl, consulting for clients and developing commercial developer tools. You can reach him at mbellinaso@gmail.com.


Networking: Get Connected With The .NET Framework 3.5

Networking

Get Connected With The .NET Framework 3.5

Mariya Atanasova and Larry Cleeton and Mike Flasko and Amit Paka
This article discusses:
  • Socket class performance
  • Internationalized URLs
  • The System.Net.PeerToPeer namespace
This article uses the following technologies:
.NET Framework
Contents
Socket Class Performance
International Resource Identifier Support
The System.Net.PeerToPeer Namespace
Wrapping Up

I
n the Microsoft® .NET Framework, the System.Net namespace exposes the functionality of many networking protocols, such as HTTP and SMTP. The upcoming release of the .NET Framework 3.5 (which will ship with Visual Studio® 2008, formerly code-named "Orcas") includes a number of performance and functional enhancements to these core networking layers. In this article, we'll look at three key changes provided by the System.Net team:

  • High performance Socket API
  • International Resource Identifier support for URIs
  • Peer-to-Peer (P2P) namespace

The upcoming release of the .NET Framework 3.5 will introduce changes to the Framework itself. The features described in this article are available in the Beta 1 release of Visual Studio 2008, available for download from MSDN®.

Socket Class Performance

In version 2.0 of the .NET Framework, the System.Net.Sockets namespace provides a Socket class that offers virtually all the functionality of the Windows® WinSock Win32® APIs. This functionality comes in a class with methods and properties designed for developers of managed code. On Socket, there is a set of synchronous methods, including Send and Receive, with various parameter overloads for a variety of scenarios. These synchronous methods are easy to use and are very suitable for simple networking tasks using sockets. There is also a set of asynchronous methods on Socket that are based on the Asynchronous Programming Model (APM) that is prevalent throughout the .NET Framework (see msdn.microsoft.com/msdnmag/issues/07/03/ConcurrentAffairs for more information). These asynchronous methods make the Socket class relatively easy to use asynchronously and provide for a way to handle a lot of sockets or many send and receive operations on a number of sockets.

The Socket class in version 2.0 is suitable for a variety of client applications that need to use network sockets as well as for some server and service type applications. Unfortunately, there are some service application scenarios that are not possible with the Socket class in version 2.0, but they are possible with native languages that directly use the Windows WinSock APIs. The primary issue with the version 2.0 Socket class is that it consumes excess CPU cycles to perform a single socket I/O operation as well as when allocating the necessary underlying objects to maintain I/O operations on a large number of sockets simultaneously.

With the .NET Framework 3.5, the common language runtime (CLR) can now more efficiently manage a large number of Overlapped objects simultaneously. Overlapped objects in the CLR effectively wrap the native Windows OVERLAPPED structure used to manage asynchronous I/O operations. There is one Overlapped object instance for every Socket asynchronous I/O operation in progress. And it is now possible to have 60,000 or more connected sockets while maintaining a pending asynchronous receive I/O operation on each socket.

The Socket class in version 2.0 uses Windows I/O Completion Ports for asynchronous I/O operation completion. This allows applications to easily scale to a large number of open sockets. The .NET Framework implements the System.Threading.ThreadPool class that provides the completion threads that read Completion Ports and complete asynchronous I/O operations. In developing the upcoming 3.5 version of the .NET Framework, we put a significant focus on removing overhead in the code paths between reading a Completion Port and calling an application's completion delegate or signaling the I/O completion event object in an IAsyncResult object.

The APM in the .NET Framework is also referred to as the Begin/End pattern. This is because a Begin method is called to initiate an asynchronous operation and returns an IAsyncResult object. A delegate can optionally be provided as a parameter to the Begin method that will be called when the asynchronous operation completes. Alternatively a thread can wait on the IAsyncResult.AsyncWaitHandle. When the callback is called or the wait signaled, the End method is called to get the results of the asynchronous operation. This pattern is flexible, relatively easy to use, and common throughout the .NET Framework.

It is important for you to note, though, that there is a price when doing a significant number of asynchronous socket operations. For each operation, an IAsyncResult object must be created—and it cannot be reused. This can impact performance by heavily exercising object allocation and garbage collection. To work around this issue, the new release offers a different pattern of methods to do asynchronous I/O with sockets. This new pattern doesn't require the allocation of an operation context object for every socket operation.

Rather than create an entirely new pattern, we took an existing pattern and made one fundamental change. There are now methods in the Socket class that use a variation of the Event-based completion model. In version 2.0, you would use the following code to initiate an asynchronous send on a Socket:

void OnSendCompletion(IAsyncResult ar) { }

IAsyncResult ar = socket.BeginSend(buffer, 0, buffer.Length,
    SocketFlags.None, OnSendCompletion, state);

In the new version, you can also do this:

void OnSendCompletion(object src, SocketAsyncEventArgs sae) { }

SocketAsyncEventArgs sae = new SocketAsyncEventArgs();
sae.Completed += OnSendCompletion;
sae.SetBuffer(buffer, 0, buffer.Length);
socket.SendAsync(sae);

There are a few notable differences here. The object that wraps the context of the operation is a SocketAsyncEventArgs object instead of an IAsyncResult object. The application creates and manages (and can even reuse) SocketAsyncEventArgs objects. All the parameters to the socket operation are specified by properties and methods of the SocketAsyncEventArgs object. Completion status is provided by properties of the SocketAsyncEventArgs object as well. And finally, the use of an event handler callback completion method is required.

All of these changes result in significantly improved performance and scalability of the System.Net.Sockets class in the .NET Framework 3.5. Two of these improvements will be automatically realized by existing applications. The third improvement, the new Socket methods, can only be used by modifying an application, but these methods provide increased scalability for the most demanding socket-based applications.

International Resource Identifier Support

Web addresses are typically expressed using Uniform Resource Identifiers (URIs) that consist of a very restricted set of characters. Basically, they contain only upper- and lowercase letters from the English alphabet, digits from 0 to 9, and a small number of other ASCII symbols including commas and hyphens.

This syntax is not very convenient for parts of the world that rely on a character set other than a Latin alphabet—say, Japanese or Hebrew. Consider an address like www.BaldwinMuseumOfScience.com. If you are an English speaker, this address is easy to understand and remember. However, if you don't speak English, this URL looks like little more than a random assortment of symbols. If you only speak English, could you remember a long address in Chinese?

International Resource Identifiers (or IRIs) support non-ASCII characters—or more precisely Unicode/ISO 10646 characters. This means a domain name can contain Unicode characters, which means you can end up with a URL that looks like this: http://微軟香港.com.

We have extended the existing System.Uri class to provide IRI support based on RFC 3987 (see faqs.org/rfcs/rfc3987.html). Current users will not see any change from the .NET Framework 2.0 behavior unless they specifically opt for IRI capabilities to be enabled. The reason for this is to ensure application compatibility between the 3.5 release and prior versions.

To opt in, you need to make two changes. First, add the following element to the machine.config file:

<section name="uri" type="System.Configuration.UriSection, System,
                          Version=2.0.0.0, Culture=neutral,
                          PublicKeyToken=b77a5c561934e089" />

Then specify whether Internationalized Domain Name (IDN) parsing should be applied to the domain name and whether IRI parsing rules should be applied. This can be done in the machine-wide machine.config or in an individual application's app.config. For example:

<configuration>
   <uri>
      <idn enabled="All" />
      <iriParsing enabled="true" />
   </uri>
</configuration>

Enabling IDN will convert all Unicode labels in a domain name to their Punicode equivalents. Punicode names contain only ASCII characters and always start with the prefix "xn--". This is because most DNS servers currently deployed on the Internet only support ASCII characters. Enabling IDN only affects the value of the Uri.DnsSafeHost property. For 微軟香港.com, it will contain xn--g5tu63aivy37i.com, while Uri.Host will contain the Unicode characters.

There are three possible values for IDN you can use in the idn element's enabled attribute depending on the DNS servers you're using:

  • "All" will use IDN names (Punicode) for all domains.
  • "AllExceptIntranet" will use IDN names for all external domains and Unicode names for all internal domains. This case only applies when the intranet DNS servers support Unicode names.
  • "None", which is the default, is consistent with the .NET Framework 2.0 behavior.

Enabling IRI parsing (iriParsing enabled = "true") causes normalization and character checking according to the latest IRI rules in RFC 3987. The default value is false and will do normalization and character checking according to RFC 2396 (see faqs.org/rfcs/rfc2396.html). To learn more about the uniform resource identifiers and the Uri class, see the online documentation at msdn2.microsoft.com/system.uri.

The System.Net.PeerToPeer Namespace

An exciting new namespace added to the .NET Framework 3.5, the System.Net.PeerToPeer namespace, is located in the System.Net.dll assembly and provides the core building blocks needed to easily create a peer-to-peer (P2P) application. The namespace was designed in accordance with the three phases of a typical P2P application: discover, connect, and communicate. The discover phase is about dynamically locating peers and their associated network locations. The connect phase deals with establishing a network connection between peers. And the communicate phase is when data is passed back and forth among peers.

The features in the System.Net.PeerToPeer namespace provide a number of options that facilitate the discover and connect phases. Meanwhile, complementary technologies—such as Sockets, HTTPWebRequest, and the Window Communication Foundation Peer Channel—provide solutions for the communicate phase.

Before communication can occur between peers, they must be able to discover each other and resolve their network locations (addresses, protocols, and ports) from names or other types of identifiers. How peers discover each other and resolve locations is complicated by transient connectivity, the lack of address records in the DNS, and dynamic IP addresses. The Peer Name Resolution Protocol (PNRP) feature, supported in the .NET Framework 3.5, enables discovery and makes communication among peers possible by enabling serverless name resolution of any resource to a set of network endpoints. PNRP is responsible for two core tasks—publishing a PeerName for others to resolve and resolving a PeerName that was published by another peer and retrieving the associated metadata. (To learn more about how the PNRP protocol works, see microsoft.com/p2p.)

In the System.Net.PeerToPeer namespace, a PeerName represents an endpoint of communication, which can be anything (such as a computer, service, or application) you wish to associate metadata with. PeerNames come in two forms: secured and unsecured. A secured PeerName is backed by a public\private key pair and, when registered with PNRP, it can't be spoofed. Each PeerName string has an Authority section followed by a period and then a Classifier section. The Authority is machine-generated based on the type (secured or unsecured) of the PeerName, whereas the classifier is a user-defined string.

For a secured PeerName, the Framework automatically creates a 40-character hex string as the Authority. This hex string is a hash of the public key associated with the PeerName and is used to ensure registrations of such PeerNames are not spoofable. Here's how you create a secure PeerName:

PeerName p = new PeerName("My PeerName", PeerNameType.Secured);

For unsecured PeerNames, the Authority component is always the character 0. An unsecured PeerName is just a string and does not offer security guarantees:

PeerName p = new PeerName("My PeerName", PeerNameType.UnSecured);

After creating a PeerName, the next step is to associate the name with relevant metadata by instantiating a PeerNameRegistration object. A PeerName is typically associated with an IP address but, as you'll see, a PeerName can also be associated with a comment string and binary data blob. In the following code snippet, an IPAddress is explicitly associated with the PeerName by adding a PeerEndPoint instance to the registrations endpoint collection:

PeerName peerName = new PeerName("My PeerName", PeerNameType.Secured);

PeerNameRegistration pnReg = new PeerNameRegistration();
pnReg.PeerName = peerName;
pnReg.EndPointCollection.Add(new IPEndPoint(
    IPAddress.Parse("<ip address to associate with the name>"), 5000));

pnReg.Comment = "up to 39 unicode char comment";
pnReg.Data = System.Text.Encoding.UTF8.GetBytes(
    "A data blob up to 4K associated with the name");

While this is a valid use of the PeerNameRegistration class, we have found a common scenario is to associate all addresses assigned to the local machine with the PeerName. To do this, you simply ensure that the PeerNameRegistration.UseAutoEndPointSelection property is set to true and refrain from adding to the PeerNameRegistration.EndPointCollection.

Now that all the relevant metadata is assigned to the PeerName by virtue of the PeerNameRegistration object, the final step is to publish the PeerName to a cloud so that other peers can resolve the name. In PNRP, a cloud is simply a set of computers participating in PNRP and defines the scope to which names are published to or resolved from. When publishing a name, you need to determine which cloud (or scope) you want to publish the name to.

PNRP currently uses two clouds: link-local and global. A PeerName published to the link-local cloud means only other peers on the same link can resolve the name. A name published to the global cloud enables anyone on the Internet to resolve the PeerName. To publish a PeerName into the global cloud, you simply assign the global cloud, via the Cloud.Global enumeration, to the Cloud property on the PeerNameRegsitration object and then call the Start method on the registration object. Once the call to start completes, the name is published and can be resolved by remote peers. Figure 1 shows the code used to create and publish a PeerName.

Figure 1 Create and Publish a PeerName (Close Figure 1)
using System;
using System.Collections.Generic;
using System.Text;
using System.Net.PeerToPeer;

namespace CreateAndPublish
{
    class Program
    {
        static void Main(string[] args)
        {
            // Creates a secured PeerName.
            PeerName peerName = new PeerName(
                "MyWebServer", PeerNameType.Secured);

            PeerNameRegistration pnReg = new PeerNameRegistration();
            pnReg.PeerName = peerName;
            pnReg.Port = 80;

//Starting the registration means the name is published
//for other peers to resolve.
            pnReg.Start();

            Console.WriteLine("Registration of Peer Name: {0} complete.",
                peerName.ToString());

            Console.WriteLine("Press any key to stop the registration " +
                             "and close the program");
            Console.ReadKey();

            pnReg.Stop();
        }
    }
}

Now that you've seen how to create and publish a PeerName, you need to know how to resolve a PeerName. You start by instating an instance of the PeerNameResolver class, which is then used to resolve a name synchronously (see Figure 2) or asynchronously. If you are resolving names asynchronously, it is important to note that the same PeerNameResolver can be used to resolve multiple PeerNames. That is, you can start multiple asynchronous resolve operations for different PeerNames before the first operation completes, thus eliminating the need to instantiate a new resolver object for each resolve operation happening in parallel.

Figure 2 Synchronous Name Resolution (Close Figure 2)
using System;
using System.Collections.Generic;
using System.Text;
using System.Net.PeerToPeer;
using System.Net;

namespace SyncResolve
{
    class Program
    {
        // The application accepts the peer name to resolve as the first
        // and only command line parameter.
        static void Main(string[] args)
        {
            // Create a resolver object to resolve a peername.
            PeerNameResolver resolver = new PeerNameResolver();
            PeerName peerName = new PeerName(args[0]);
            // Resolve the PeerName - this is a network operation and will
            // block until the resolve request is completed.
            PeerNameRecordCollection results = resolver.Resolve(peerName);

            // Show the data returned by the resolve operation.
            Console.WriteLine("Records from resolution of PeerName: {0}",
                peerName);
            Console.WriteLine();
            int count = 1;
            foreach (PeerNameRecord record in results)
            {
                Console.WriteLine("Record #{0} results...", count);

                Console.WriteLine("Comment:");
                if (record.Comment != null)
                {
                    Console.WriteLine(record.Comment);
                }

                Console.WriteLine("Data:");
                if (record.Data != null)
                {
                    //Assumes the data is an ASCII formatted string
                    Console.WriteLine(
                        System.Text.Encoding.ASCII.GetString(record.Data));
                }

                Console.WriteLine("Endpoints:");
                foreach (IPEndPoint endpoint in record.EndPointCollection)
                {
                    Console.WriteLine("\t Endpoint:{0}", endpoint);
                    Console.WriteLine();
                }

                count++;
            }

            Console.ReadKey();
        }
    }
}

At this point, you can use the code shown in Figure 1 and Figure 2 to try creating and resolving PeerNames. But note that when trying out the PNRP functionality, you can't resolve a PeerName from the same process that published the name. We should also mention that the PNRP APIs discussed here use the Windows PNRP infrastructure and are supported on Windows XP (with the PNRP update available at support.microsoft.com/kb/920342 installed), Windows Vista®, and Windows Server 2008.

The .NET Framework 3.5 also introduces the System.Net.PeerToPeer.Collaboration namespace. Collaboration works in two contexts. The first, "People Near Me", deals with users who are logged into the collaboration infrastructure and located on the same subnet. (For more details about this, see microsoft.com/technet/network/p2p/pnm.mspx.) The other context, referred to as "Contacts", involves people who have been added into your Windows Address book (this can be found in your Users directory in the Contacts folder). In the Contacts context, users do not have to be on the same subnet.

Both of these contexts allow a user to share applications and objects, send invites, and get notice of events involving other users. Consider a person in sales who needs to "meet" with her colleagues while waiting for a flight at the airport. Ideally, this person should be able to fire up her laptop, launch her company's main line of business (LOB) application, and have the application dynamically find her colleagues whether they are sitting with her in the airport or remain back in the office. The application will initially have to sign into the Collaboration infrastructure and specify the scope of the sign-in. In this example, PeerScope.All includes both the Internet and peer-near-me scope:

PeerCollaboration.Signin(PeerScope.All);

Once signed in, the application looks up all people near this salesperson using:

PeerNearMeCollection peers = PeerCollaboration.GetPeersNearMe();

The PeerNearMeCollection contains an instance of the PeerNearMe class for each peer located on the same subnet as the caller. Thus, the application now has a list of all the "People Near Me" in the form of PeerNearMe instances. A PeerNearMe instance includes a property that specifies the network location or IPEndPoint (IP Address + Port) of the remote peer.

To look up all the contacts stored in the salesperson's address book, the application needs to get all the contacts with the following lines of code:

ContactManager contactManager = PeerCollaboration.ContactManager();
PeerContactCollection contacts = contactManager.GetContacts();

The ContactManager represents the Windows Address Book, which is the contact store used by the System.Net.PeerToPeer.Collaboration infrastructure.

Now that a list of all peers (consisting of PeerNearMe and PeerContact) has been generated, this salesperson can choose which colleagues to interact with and quickly send invitations to each. An invitation, in the context of System.Net.PeerToPeer.Collaboration, is a mechanism to request a remote peer to start a particular application. To establish a network connection between two peer computers, one of the peers needs to be actively listening for incoming data. This mechanism allows one peer to tell another which application it needs to be running.

Say the salesperson in our example wants to interact with another colleague using the company LOB application. The salesperson needs to ensure that her colleague is running the LOB application. To do this, she sends an invitation to the colleague. When the colleague's system receives the invitation, a dialog box is presented to the user stating that a particular person wants them to start a specific application (see Figure 3). The dialog gives the option to accept or decline the invitation. If the recipient clicks the Accept button, the application stated in the invitation automatically launches on the recipients PC—the LOB app in this case—assuming the application is already installed.

Figure 3 Invitation to Launch an App from a Peer
Figure 3 Invitation to Launch an App from a Peer(Click the image for a smaller view)
Figure 3 Invitation to Launch an App from a Peer
Figure 3 Invitation to Launch an App from a Peer (Click the image for a larger view)

Now both colleagues have the necessary application running on their systems and are ready to collaborate. At this point, the LOB application uses another networking technology (such as Windows Communication Foundation Peer Channel, Sockets, or HTTP) to communicate.

Now that we've described invitations, let's look at the details and associated code needed to enable invitations using the System.Net.PeerToPeer.Collaboration namespace. As far as the P2P infrastructure is concerned, an application is any executable on your machine. To support invitations, the application has to be registered with the peer collaboration infrastructure using the same GUID on both the inviter and the invitee machines. This code shows how to create and register an application based on an executable installed on the local machine:

PeerApplication application = new PeerApplication(
    appGuid, "Collaboration Application", bytes, pathToApp,
    arguments, PeerScope.Internet);
PeerCollaboration.RegisterApplication(
    application, PeerApplicationRegistrationType.AllUsers);

Registered applications that get launched via invitations can query which contact or endpoint sent the invitation using PeerCollboration.ApplicationLaunchInfo. This allows the application that just launched to know it was launched via an invitation from a remote peer and thus it knows to connect back to the peer that sent the invitation. An invitation can be sent by calling Invite on the peer object returned from GetPeersNearMe or to a contact returned from GetContacts with Invite as follows:

PeerInvitationResponse pir = peerNearMe.Invite(
    app, "Hello World", data);

PeerInvitationResponse pir = contact.Invite(
    app, "Hello World", data);

Say the salesperson is communicating with a colleague at the airport and is planning to follow later when the two are in different locations. She should add that person as a Contact on her machine. This is done using the peer class returned from the GetPeersNearMe call with peer.AddToContactManager. Once the salesperson and her colleague have added each other as Contacts, they can look up and interact with each other from wherever they may be.

The collaboration APIs in the .NET Framework 3.5 also give you the ability to share data. For example, if all the sales people from the company want to share their business cards electronically, this can be done using peer objects. Peer objects are simply blobs of data (such as an image file) that can be viewed by remote peers. Objects can be shared by scope, specifying People Near Me to allow users on the same subnet to see the objects or in the Internet scope so your contacts can see the objects regardless of where the users are. For example, to create an object and share it in the People Near Me scope, you simply do this:

PeerObject object = new PeerObject(
    objectGuid, bytes, PeerScope.NearMe);
PeerCollaboration.SetObject(object);

Presence and change notifications are also an important element of P2P collaboration. Say, for instance, our salesperson wants to chat with her colleague Steve at the airport as soon as Steve is signed in and available. An event called PeerNearMeChanged is provided on the PeerNearMe class to update peers on the same subnet. Any changes to the peer will trigger the attached delegate to be called with the change information. This way, the salesperson can be notified when Steve enters (or leaves) the network. The necessary code looks like this:

PeerNearMe.PeerNearMeChanged += PeerNearMeChangedCallback;
...
void PeerNearMeChangedCallback(
    object sender, PeerNearMeChangedEventArgs args)
{
        // Check which PeerNearMe has changed and what the change was
        // from the args parameter.
}

Change notifications for Contacts in another location entail a few more steps. The process requires reciprocal trust, meaning you must choose to watch the Contact and that user must choose to allow you to watch him. So first you elect to watch the contact—by calling the Subscribe method on the Contact. Then the peer you want to watch must add you as a Contact and allow you to watch—by setting the SubscribeAllowed property on your Contact to SubscriptionType.Allowed. When these steps have been done, you can track specific changes to names, objects, applications, and so on for the peer you are watching.

For example, suppose our salesperson wants to chat with a customer contact as soon as he advertises himself as being online. The following code will allow the salesperson to get the necessary status changes of the customer:

custContact.PresenceChanged += ContactPresenceChangedCallback;
...
void ContactPresenceChangedCallback(
    object sender, PresenceChangedEventArgs args)
{
    if (args.PeerPresenceInfo.PresenceStatus ==
        PeerPresenceStatus.Online)
    {
        // Start chatting with the customer
    }
}

Note that change information can be for a particular object or application as well. Attaching a delegate to the ObjectChanged event in the PeerObject class will provide information about any changes to that object and where the changed object came from, such as from a Contact or a PeerNearMe. In a similar manner, adding a delegate to the ApplicationChanged event in PeerApplication will provide information about any changes to that application.

For example, to monitor whether a particular application changed from the list of applications exposed by any contact in your contact list, a delegate needs to be added to the ApplicationChanged event on PeerApplication:

peerApplication.ApplicationChanged += AppChangedCallback;
...
void AppChangedCallback(
    object sender, ApplicationChangedEventArgs args)
{
    // Check what the change was and which contact and endpoint it
    // originated from the args parameter.
}

Now let's consider a situation in which the salesperson is going to be occupied and therefore her status needs to be changed to alert peers that she is not immediately available. This is done by modifying the data associated with the PeerCollaboration.LocalPresence property, like so:

PeerCollaboration.LocalPresenceInfo = new PeerPresenceInfo(
    PeerPresenceStatus.Away, "Talking with Customer");

Wrapping Up

This article provides only a brief look at the new core networking features being added to the .NET Framework 3.5. If you can't wait, you can download the latest CTP release at msdn2.microsoft.com/aa700831 to give these new features a try. To keep up with the latest news regarding System.Net, as well as Windows Networking technologies, visit the Windows Network Developer Platform team blog at blogs.msdn.com/wndp.

Mariya Atanasova is a Software Design Engineer in Test on the System.Net team. She can be reached via her blog at blogs.msdn.com/mariya.

Larry Cleeton is a Software Design Engineer on the System.Net team. He can be reached via his blog at blogs.msdn.com/lcleeton.

Mike Flasko is the Program Manager for the System.Net, Winsock, and Winsock Kernel teams. He can be reached via his blog at blogs.msdn.com/mflasko.

Amit Paka is a Software Design Engineer on the System.Net team. He can be reached via his blog at blogs.msdn.com/amitpaka.


IIS 7.0: Extend Your WCF Services Beyond HTTP With WAS

IIS 7.0

Extend Your WCF Services Beyond HTTP With WAS

Dominick Baier and Christian Weyer and Steve Maine
This article is based on a prerelease version of Windows Server 2008. All information herein is subject to change.
Code download available at: WAS2007_09.exe (178KB)
This article discusses:
  • The architectures and process models of IIS 6.0 and IIS 7.0
  • How Web services were hosted in IIS 6.0
  • Hosting robust WCF services with IIS 7.0
  • How Windows Process Activation Service (WAS) works
  • Support for non-HTTP protocols
This article uses the following technologies:
IIS, WAS
Contents
The IIS 6.0 Architecture and Process Model
IIS 7.0 and WAS
Inside WAS
Configuration and Multi-Protocol Addressing
How Listeners Know to Listen
Message-Based Activation over Non-HTTP Protocols
Worker Process Initialization
Getting Data from the Listener to the Worker
Hosting a WCF Service in WAS
Lifetime Management of WAS-Hosted Services
Automating Setup of WAS-hosted Services
Extending WAS

A
ll the talk about service-oriented applications we've been hearing for the past few years has given birth to real frameworks, runtimes, and other bona fide tools for designing, building, and deploying service-oriented connected systems. A good example of this is Windows® Communication Foundation (WCF), which allows you to create services and service consumers in managed code.

One strength of WCF is that you can host WCF-based services in any Windows process, including a console application, a Windows Forms app, or a Windows Presentation Foundation (WPF) UI application. You can even self-host WCF services in long-running Windows NT® services that run in the background working on behalf of a configured identity. WCF services with HTTP-based endpoints can also be hosted inside IIS, much like the traditional Web services as implemented by ASP.NET and ASMX.

If you want to provide robust WCF services through IIS 7.0, it is essential that you understand a new IIS feature called the Windows Process Activation Service (WAS). WAS is a fundamental underlying component of IIS 7.0 that makes it possible to host WCF services beyond HTTP and without having to install the whole IIS package.

Before we dive into WAS and show you how to use it in your own applications, let's take a look at how IIS 6.0 hosting works and why the new process in IIS 7.0 is better.

The IIS 6.0 Architecture and Process Model

Understanding the new process activation mechanism for Windows Vista® and Windows Server® 2008 requires a familiarity with the IIS 6.0 architecture and process model. The architecture for Windows Server 2003 and IIS 6.0 is divided into two basic parts—a listener process and a set of worker processes. The IIS 6.0 listener process is implemented in the w3svc service, which is a long-running Windows NT service activated by the Windows Service Control Manager (SCM). The listener process waits for messages to arrive via HTTP and then dispatches them to the appropriate worker process (w3wp.exe), which hosts the application code that will finally process the request.

When a request arrives on the network, it is processed by the kernel-mode HTTP stack (http.sys) and delivered to the listener process. The w3svc process then looks at the request URI and uses it to map the request to a specific IIS application living inside a particular IIS application pool that is, in turn, hosted inside an instance of the worker process (w3wp), as you see in Figure 1. The mapping between the request URI and the application is based on configuration information stored in the IIS metabase, which can be found in metabase.xml and mbschema.xml, respectively, both of which are stored in the %windir%\system32\inetsrv folder. Once w3svc has determined the destination application, it can look it up in some internal data structures to resolve the destination worker process and application pool.

Figure 1 IIS 6.0 Basic Architecture
Figure 1 IIS 6.0 Basic Architecture(Click the image for a smaller view)
Figure 1 IIS 6.0 Basic Architecture
Figure 1 IIS 6.0 Basic Architecture (Click the image for a larger view)

Depending on the state of the targeted IIS server, one of two things can happen at this point. If the worker process and application pool are already running because a previous request has been received for that pool, then no activation is necessary. Therefore, the request is simply dispatched to the waiting worker process. If no worker process exists to handle the current request, the listener process must create a new instance of w3wp before dispatching the request.

The IIS 6.0 worker process is a lightweight executable. When it starts up in response to an activation request, the first thing it does is load a simple unmanaged shim DLL (w3wphost.dll) that enables w3svc to communicate with the worker process. This shim is also responsible for loading aspnet_isapi.dll, which implements the interface between the managed components of ASP.NET and the unmanaged components of IIS.

Aspnet_isapi.dll is responsible for loading the common language runtime (CLR) into the worker process and creating a default application domain, which is where the managed hosting components of ASP.NET will live. These managed hosting components are responsible for creating additional application domains (one per IIS application) on demand and routing requests to them based on the request's URL.

IIS 7.0 and WAS

If you understand the IIS 6.0 process model, you'll understand IIS 7.0 and WAS. All of the major architectural components of IIS 6.0 (listeners, workers, and application managers) are also present in WAS. The difference is that the IIS 7.0 and WAS implementation supports non-HTTP scenarios, which may work well for your service-based application architectures.

Manageability, reliability, and scalability are exactly the same with WAS as they were for IIS 6.0 on Windows Server 2003. But now, all of the great IIS features that address these issues (on-demand activation, process health monitoring, enterprise-class manageability, rapid failure protection) that were available only to HTTP-based apps in IIS 6.0 will now be available to non-HTTP-based applications and services as well.

Inside WAS

In IIS 6.0, the w3svc service really did double duty. It acted as the HTTP listener because it registered with http.sys and was the direct recipient of incoming HTTP traffic. It also owned the process activation components responsible for starting new instances of w3wp and dispatching requests appropriately. In IIS 7.0, these two responsibilities have been refactored into separate Windows NT services. The w3svc process retains its role as the HTTP listener, but the components responsible for configuration and process activation have been factored into WAS, which has three parts: the configuration manager, the process manager, and the unmanaged listener adapter interface.

The configuration manager reads application and application pool configuration from applicationhost.config (the IIS 7.0 replacement for the metabase). The process manager maps application pools to existing worker processes and is the one responsible for spawning new instances of w3wp to host new application pools in response to activation requests. The unmanaged listener adapter interface defines how external listeners communicate activation requests they receive to WAS.

The w3svc service owns the communication with kernel-level http.sys and communicates HTTP activation requests to WAS across the listener adapter interface.

WCF uses the listener adapter interface to communicate activation requests that are received over the supported non-HTTP protocols (namely TCP, Named Pipes, and MSMQ). The WCF plumbing that actually receives requests over non-HTTP protocols is hosted inside of SMSvcHost.exe, which hosts the following four long-running Windows NT services: NetTcpPortSharing, NetTcpActivator, NetPipeActivator, and NetMsmqActivator (see Figure 2).

Figure 2 Basic Architecture of IIS 7.0 and WAS
Figure 2 Basic Architecture of IIS 7.0 and WAS(Click the image for a smaller view)
Figure 2 Basic Architecture of IIS 7.0 and WAS
Figure 2 Basic Architecture of IIS 7.0 and WAS (Click the image for a larger view)

NetTcpPortSharing is the WCF TCP port sharing service. It implements a centralized TCP listener so that multiple processes can listen on the same TCP port. This service is available even if IIS 7.0 is not installed.

NetTcpActivator is the WCF TCP Activation Service. It communicates TCP activation requests to WAS.

NetPipeActivator is the WCF-named pipe activation service which communicates named pipe activation requests to WAS.

NetMsmqActivator is the WCF MSMQ activation service; it communicates MSMQ activation requests to WAS.

Although these services all live in the same binary, they are separate Windows NT services and can be stopped and started individually to reduce both attack surface and overhead. They are all examples of listener services and all behave in a similar manner. For that reason, we will focus on TCP activation, which can also serve as an example for named pipes and MSMQ. The only difference is the particular type of network resource in use in each case.

Configuration and Multi-Protocol Addressing

In order for an application to be activated by WAS, it first needs to be configured. Each application hosted in WAS has a corresponding configuration entry located in %windir%\system32\inetsrv\config\applicationhost.config. This information includes items like the application pool that will host the application and a URL fragment that uniquely identifies it. We'll present an example of this later. Much like IIS 6.0, each application is associated with both an application pool and a site. Each site has a set of address bindings for the network protocols it supports. For example, an administrator might configure the default site to bind to HTTP on port 80 and TCP on port 7777. This site might host two applications (listening on /Foo and /Bar respectively) that live in separate application pools. In this configuration, the /Foo application would listen on http://myserver.com/Foo as well as net.tcp://myserver.com:2323/Foo, while /Bar would listen on http://myserver.com/Bar and net.tcp://myserver.com:2323/Bar. Regardless of how they were received—since each application pool gets its own worker process—all requests for /Foo would go to worker process 1, while everything destined for /Bar would go to worker process 2. In general, an application's site governs the set of network addresses associated with the application while its application pool governs the worker process instance that will host it. Note that the one worker process per application pool rule only holds true if you're not using Web gardens. In the Web garden case, each application pool maps to its own set of worker processes, up to the maximum number of worker processes specified in the application pool's configuration.

How Listeners Know to Listen

A listener needs to receive messages. For this, it needs to open a socket (or a pipe handle, or start an MSMQ read, and so on). However, in order to receive the proper messages, it needs to obtain the necessary addressing information from WAS. This is accomplished during listener startup. The protocol's listener adapter calls a function on the WAS listener adapter interface and essentially says, "I am now listening on the net.tcp protocol; please use this set of callback functions I'm handing you to tell me what I need to know." In response, WAS will call back with any configuration it has for applications that are set up to accept messages over the protocol in question. For the example above, the TCP listener would be informed that there were two applications (*:7777/Foo and *:7777/Bar) configured to use TCP. WAS also assigns to each application a unique listener channel ID used for associating requests with their destination applications.

The listener process uses the configuration information provided by WAS to build up a routing table, which it will use to map incoming requests to listener channel IDs as they arrive. The mechanics of this mapping is an implementation detail of the underlying protocol the listener is supporting. The important thing is that each listener service must be able to look at an incoming message, say, "Ah—this is destined for listener channel x," and dispatch the request to WAS accordingly. It happens that WCF uses URIs to indicate the destination of messages that arrive over TCP/MSMQ/named pipes, but other protocols could conceivably implement this mapping in whatever way is appropriate for them.

Once the listener service has connected to WAS and received configuration information, it can open its network resource and begin listening for messages. For TCP, this causes NetTcpActivator to trigger a socket to open and an asynchronous call to Socket.Accept to be made, at which point the listener essentially goes to sleep until a message arrives.

Message-Based Activation over Non-HTTP Protocols

The listener process wakes up and starts reading bytes when they arrive at the waiting socket. At this point, the goal is to read just enough information to determine the eventual destination of the forthcoming message and then pause while the listener calls back to WAS and spawns the worker process. For TCP, the destination address is determined by reading information out of the framing protocol that surrounds the SOAP message. Once the listener has the destination URI, it uses that URI as an index into its internal routing table and resolves the URI into the corresponding listener channel ID assigned to that address by WAS.

The listener service then asks WAS to activate a worker process to handle the pending request by calling the unmanaged WebhostOpenListenerChannelInstance method on the listener adapter interface. In addition to other parameters, WebhostOpenListenerChannelInstance takes a listener channel ID as well as a blob of data (represented as a byte array) that will be used to initialize the receiving side of the plumbing in the activated process. This blob is of no interest to WAS as it's merely a convenience that allows each protocol listener implementation to initialize its newly activated components.

When WAS receives the call to WebhostOpenListenerChannelInstance, the process manager part of WAS spawns a new instance of the worker process. The worker process then needs to do some initialization itself before it can process messages received by the listener. During this time the listener may continue to receive data from the network; however it cannot dispatch the message to the worker until it receives notification that worker process initialization is complete.

Worker Process Initialization

There are several important components that are active inside the worker process, as you can see in Figure 3. The architecture of a worker process is shown in Figure 4.

Figure 3 Important Components in the Worker Process (Close Figure 3)

Component | Description
Process Host | Loads the CLR into the worker process and initializes the default application domain.
Application Manager | Creates unique application domains in response to application-level activation requests and manages their lifetime.
Process Protocol Handlers | Implement protocol-specific process initialization logic.
Application Domain Protocol Handlers | Reside in the activated application domain and perform protocol-specific application domain initialization.
Figure 4 Worker Process Initialization Architecture
Figure 4 Worker Process Initialization Architecture(Click the image for a smaller view)
Figure 4 Worker Process Initialization Architecture
Figure 4 Worker Process Initialization Architecture (Click the image for a larger view)

Since IIS does not implement any managed code components, the job of loading the CLR into the worker process falls to the ASP.NET process host. WAS creates a process host inside the worker process by calling an API exposed by aspnet_isapi.dll. WAS passes this function a callback interface that the managed process host can use for subsequent communication with WAS. The newly created process host is then returned to WAS, allowing for two-way communication between WAS and the worker process.

WAS uses the instance of the process host it obtained from the worker process to start the protocol-specific initialization routines. The activation system StartProcessProtocolListenerChannel on the process host, passing it a protocol key (such as "net.tcp") and a callback interface that the protocol-specific handlers can use to talk back to WAS.

When the process host receives a request to start a new process protocol listener channel inside the worker process, the first thing it does is look up the protocol key in the configuration. Users of WAS are required to register this prior to starting their listeners. For example, WCF registers both a process protocol handler and an application domain protocol handler for net.tcp. The configuration data shown in Figure 5 can be found in the master web.config file in %windir%\Microsoft.NET\Framework\v2.0.50727\CONFIG.

Figure 5 Configuration for net.tcp Protocol Handlers (Close Figure 5)
<system.web>
  <protocols>
    <add name="net.tcp"
     processHandlerType=
      "System.ServiceModel.WasHosting.TcpProcessProtocolHandler,
       System.ServiceModel.WasHosting, Version=3.0.0.0, Culture=neutral,
       PublicKeyToken=b77a5c561934e089"
     appDomainHandlerType=
      "System.ServiceModel.WasHosting.TcpAppDomainProtocolHandler,
       System.ServiceModel.WasHosting, Version=3.0.0.0, Culture=neutral,
       PublicKeyToken=b77a5c561934e089" validate="false" />
  </protocols>
</system.web>

Once the process host has resolved the protocol key in configuration, it creates a new instance of the configured process protocol handler type in the default application domain and calls StartListenerChannel on it. This method takes two separate callback interfaces. One allows communication with WAS (it is, in fact, the same WAS callback object as passed to the process host). The other one allows communication with the process host directly (see Figure 6).

Figure 6 Registering Protocol Handlers
Figure 6 Registering Protocol Handlers(Click the image for a smaller view)
Figure 6 Registering Protocol Handlers
Figure 6 Registering Protocol Handlers (Click the image for a larger view)

A process protocol handler (PPH) has a large degree of control over the hosting model for a particular protocol. For example, the PPH could simply stop at this point in the initialization routine and handle all subsequent requests in the default application domain. However, WCF has chosen to retain the IIS 6.0 application pool model, which activates each application in its own application domain so that each one can be independently monitored and recycled. As such, the WCF PPHs need to collaborate with the ASP.NET application manager, which governs applications and application domain lifetime.

The process host provides the PPH with access to the application manager's functionality through the IAdphManager interface it passes to StartListenerChannel. In order to determine the destination application for the activation request, the PPH calls back to WAS and asks for the data blob that the listener provided when it originally called WebhostOpenListenerChannelInstance. After doing some twiddling on the returned byte array, the WCF PPH has access to the application key that uniquely identifies the destination application (for example, /w3svc/1/Foo) because, by convention, the WCF listener includes the application key in the data blob it sends to the listener channel. WAS places no restrictions on the data passed in the blob—from the WAS perspective, it's merely an opaque byte array. It can then pass the application key, the protocol key, and the listener callback interface to IAdphManager.StartAppDomainProtocolListenerChannel to activate the application's application domain.

In general, the application manager uses the application key as an index into a table of all application domains. When it receives a call to StartAppDomainProtocolListener, it first consults this table to see if the application has already been activated by a previous request that arrived over a different protocol. Finding none, it uses the protocol key to look up the protocol-specific application domain protocol handler type in much the same way as the process host used this key to determine the process protocol handler type. Once the application domain protocol handler type is resolved, the application manager creates a new instance of it in the target application domain. Finally, the application manager calls AppDomainProtocolHandler.StartListenerChannel, again passing the WAS callback interface so the application domain can communicate with WAS if necessary.

Once the application domain protocol handler is instantiated, the internals of the worker process look like Figure 7. At this point, WAS considers the target application to be fully activated and has no further involvement in the process until the application shuts down.

Figure 7 Application Domain Protocol Handler Initialization
Figure 7 Application Domain Protocol Handler Initialization(Click the image for a smaller view)
Figure 7 Application Domain Protocol Handler Initialization
Figure 7 Application Domain Protocol Handler Initialization (Click the image for a larger view)

Getting Data from the Listener to the Worker

Activating worker processes and application domains is the first step towards a larger goal: actually delivering the message to the application. WAS, since it is an activation service and not a message delivery service, does not participate in this activity. Instead, each protocol implementation is free to implement communication between the listener and the worker in whatever way it sees fit. This architecture allows for various protocol-specific optimizations and for each implementation to exploit the nature of the underlying protocol. For example, the WCF MSMQ activation service simply passes the name of the MSMQ queue in the data blob, allowing the activated application to read directly from the data source and bypass the listener completely.

Hosting a WCF Service in WAS

Now let's get our hands dirty and actually host a WCF service in WAS. First we will set up an HTTP-based service, which is not much different from what we'd do in IIS 6.0. Later, we will add support for TCP, named pipes, and MSMQ and take a closer look at the available configuration options (see Figure 8 for the source code of the simple service).

Figure 8 Sample WCF Service (Close Figure 8)
[ServiceContract(Name = "WasDemoServiceContract",
                Namespace = "urn:msdnmag")]
public interface IDemoService
{
  [OperationContract(Name="Ping", Action="Ping",
                     ReplyAction="PingReply")]
  string Ping(string data);
}

[ServiceBehavior(
  Name="WasDemoService", Namespace="urn:msdnmag",
  InstanceContextMode=InstanceContextMode.PerCall,
  ConcurrencyMode=ConcurrencyMode.Single)]
public class DemoService : IDemoService
{
  public string Ping(string data)
  {
    return "pinged with: " + data;
  }
}

The first step is to create a new application in WAS—you can do that either using the IIS 7.0 management tool or with Visual Studio®. The virtual directory has to contain three things: a service (either compiled in the /bin directory or as source code in /App_Code folder), a .svc service endpoint, and a configuration file.

The .svc file makes the connection between a URI and a service implementation (very similar to .asmx files). The @ServiceHost directive inside the .svc file tells the WCF plumbing to create a ServiceHost instance for the type specified in the Service attribute:

<% @ServiceHost Service="DemoService" %>

Configuration information for WAS is stored in a web.config file. The entire WCF configuration takes place in the system.serviceModel section. One thing to note here is that you do not specify a base address when hosting in WAS since that address is determined by the Web site and virtual directory system (see Figure 9 for the basic configuration file). The configured service behavior enables metadata publishing and you can directly point tools like svcutil.exe or the "Add Service Reference" dialog in Visual Studio to the .svc file to generate a client-side proxy.

Figure 9 Basic Configuration for the Sample Service (Close Figure 9)
<configuration>
  <system.serviceModel>
    <services>
      <service name="DemoService" behaviorConfiguration="Behavior">
        <endpoint address="" binding="wsHttpBinding"
         bindingNamespace="urn:msdnmag" contract="IDemoService" />
      </service>
    </services>
    ...
    <behaviors>
      <serviceBehaviors>
        <behavior name="Behavior">
          <serviceMetadata httpGetEnabled="true" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
  </system.serviceModel>
</configuration>

WCF endpoints that use the HTTP transport traverse the common IIS processing pipeline. If you've worked with ASP.NET Web services in the past this may look very familiar, but internally it works a little bit differently. There are actually two modes of operation for HTTP-based endpoints: ASP.NET compatibility and no compatibility. By default ASP.NET compatibility is turned off, which means that HttpContext.Current is null and that the actual activation of the endpoint does not happen in an HttpHandler but in an HttpModule that hands off the request to the ServiceHost in the ASP.NET PostAuthenticateRequest pipeline event. This has some implications; first of all, if you are used to the ASP.NET extensibility model, you will only receive pipeline notifications up to PostAuthenticateRequest. From there on, processing will jump directly to the EndRequest event. Also, WCF services will not be able to access any ASP.NET host features such as session state. This is intentional. Since the WCF service host has transport-independent equivalents of all these features (sessions, authentication, and so forth), you should not attempt to tie your service to a specific hosting environment.

If you absolutely need ASP.NET compatibility (to share session state between the service and an ASP.NET application hosted in the same application domain, for example), you can turn it on using a configuration switch:

<serviceHostingEnvironment aspNetCompatibilityEnabled="true" />

Furthermore, you have to specifically enable your service for ASP.NET compatibility:

[AspNetCompatibilityRequirements(
    RequirementsMode=AspNetCompatibilityRequirementsMode.Required)]
class DemoService : IDemoService { ... }

These two switches in combination change the way the internal processing of service requests work. Once they are used, the requests will get dispatched to the WCF runtime using an HttpHandler and the whole processing pipeline behaves very much like ASP.NET Web services. Furthermore the current HttpContext is available and can be used as usual. WCF services running in ASP.NET compatibility mode support HTTP endpoints only.

Now that you've seen how to host an HTTP-based service in WAS, let's add non-HTTP endpoints to the service. As explained earlier, the protocol listeners are in charge of opening the actual transport and dispatching the connection to the application domain running the service. These protocol listeners are Windows NT services. When you open services.msc you will find a Net.tcp Listener Adapter as well as listeners for named pipes and MSMQ. These, along with the port sharing service for TCP, have to be up and running for successful non-HTTP activation.

The host name and port configuration for the listener adapters can be found in the WAS configuration file applicationHost.config. You can either edit the settings manually, use the appcmd.exe command-line tool, or use the IIS 7.0 configuration GUI. These settings, called bindings, are done at the site level (see Figure 10). (Note that there are some differences between the versions of WAS that ship with Windows Vista and Windows Server 2008. In Windows Vista, the bindings are not pre-configured and there is currently no GUI to add them graphically. Later you'll see the necessary appcmd.exe commands. On Windows Server 2008 the bindings are configured by default.)

Figure 10 IIS 6.0 Basic Architecture
Figure 10 IIS 6.0 Basic Architecture(Click the image for a smaller view)
Figure 10 IIS 6.0 Basic Architecture
Figure 10 IIS 6.0 Basic Architecture (Click the image for a larger view)
Figure 10 IIS 6.0 Basic Architecture
Figure 10 IIS 6.0 Basic Architecture

<bindings>
  <binding protocol="https" bindingInformation="*:443:" />
  <binding protocol="http" bindingInformation="*:80:" />
  <binding protocol="net.tcp" bindingInformation="808:*" />
  <binding protocol="net.pipe" bindingInformation="*" />
  <binding protocol="net.msmq" bindingInformation="localhost" />
  <binding protocol="msmq.formatname" bindingInformation="localhost" />
</bindings>

Each protocol has associated protocol-specific configuration settings in the bindingInformation attribute. For example, bindingInformation for the TCP binding specifies the port as well as on which host name the listener should accept connections (the * specifies that all host names are valid).

The last step is to enable the protocols of choice in the application that hosts the service. As of Windows Server 2008 Beta 3, there is no GUI for this; you have to manually edit applicationHost.config. Later we'll look at how to do this either using a tool or programmatically. Search for the application element in the configuration file and add an enabledProtocols attribute, like so:

<application path="/WasDemo"
    enabledProtocols="http,net.tcp,net.pipe,net.msmq">
  <virtualDirectory path="/" physicalPath="C:\MSDN\WasDemo\web" />

After that you simply have to add additional endpoints to your WCF service configuration to make them reachable over the above protocols:

<endpoint address="" binding="netTcpBinding"
  bindingNamespace="urn:msdnmag" contract="IDemoService" />
<endpoint address="" binding="netNamedPipeBinding"
  bindingNamespace="urn:msdnmag" contract="IDemoService" />

When you generate the client proxy from metadata, the configuration file will include a client element for each endpoint. In the client code you can specify which transport to use by passing the endpoint name to the proxy constructor, like this:

WasDemoServiceContractClient proxyTcp =
  new WasDemoServiceContractClient(
    "NetTcpBinding_WasDemoServiceContract");
Console.WriteLine(proxyTcp.Ping("Hello"));

Adding an MSMQ endpoint to the service requires some additional steps. First you have to create a queue using the Message Queuing MMC snap-in (or by using any of the available programmatic approaches). Furthermore, you have to set access control lists (ACLs) on the queue to allow the worker process (NETWORK SERVICE, by default) to read and peek from the queue.

It's important to note that the activation of MSMQ endpoints only works correctly if the queue has the same name as the .svc file (minus the machine name). That means that if your service endpoint is /server/app/service.svc, the queue name must be app/service.svc.

In the service configuration you have to specify the name of the queue the service should listen on:

<endpoint address="net.msmq://localhost/private/wasdemo/service.svc"
  binding="netMsmqBinding" bindingNamespace="urn:msdnmag"
  contract="IServiceMsmq" />

Non-HTTP endpoints do not pass the IIS processing pipeline and will get routed directly to the WCF runtime. This means that you can't use an HttpModule to pre- or post-process requests. In addition, the Application_Start and Application_End of the HttpApplication class (global.asax) don't fire. So if you want to run startup or cleanup code for such services, you have to use the events of the ServiceHost class. We'll take a closer look at how that works in the next section.

Lifetime Management of WAS-Hosted Services

When you self-host a service (in a Windows NT service, for example) the lifetime of that service is deterministic. It is running after calling Open and it will shut down after calling Close (ignoring catastrophic failures for now). Additionally, Windows NT services usually start at boot time. This is different when hosting in WAS.

First of all, WAS does demand activation (as you know, the "A" in "WAS" stands for activation). This means that the application domain hosting a service only gets created when a request message comes in. Application domains shut down again after a configurable idle period. There are also several reasons why WAS or the ASP.NET runtime may decide to recycle the application domain or even the whole worker process.

First of all, the application pool settings in WAS specify regular recycles of the worker process—by default it is every 29 hours (please note that this prime number was chosen intentionally so that the chance that recycling will actually happen at the same time of day will be minimized).

There are several other reasons why an application domain may recycle. Configuration settings in machine.config, web.config, or applicationHost.config affecting this application domain may have changed. Also the /bin or /App_Code directory or its contents might have been modified, the number of re-compilations (for .aspx, .asmx, .svc, and so on) exceeds the limit specified by the <compilation numRecompilesBeforeAppRestart /> setting in machine.config or web.config (by default this is set to 15), or the physical path of the virtual directory is modified. Finally, the code access security (CAS) policy might have been modified or the application sub-directories might have been deleted. When this happens, you lose all in-memory state including sessions or instance and static variables. That means in essence that out-of-the-box WAS hosting is not something that is really suited for sessionful or singleton services. It is more suitable for stateless per-call services.

The WCF ServiceHost type features two events called Opening and Closing. They are the only proper ways to execute code at service startup and shutdown. The problem is that you have to wire up the handlers for these events between creating a new instance of the ServiceHost and calling Open on it. This is not possible when using the @ServiceHost directive in .svc files as described above. A viable option in this case is to host a custom service host factory. This gives you more control and allows you to handle the aforementioned events.

To do this you have to derive from a class named ServiceHostFactoryBase and implement the CreateServiceHost method. This method receives the service type name and the base addresses from the hosting environment and returns an instance of ServiceHostBase. Now it is your responsibility to create the appropriate ServiceHost, configure it according to your needs, and return it to the WCF runtime. Doing this then allows you to programmatically access the extensibility model of WCF and wire up event handlers (see Figure 11).

Figure 11 Using CreateServiceHost (Close Figure 11)
public class HostFactory : ServiceHostFactoryBase
{
  public override ServiceHostBase CreateServiceHost(
    string constructorString, Uri[] baseAddresses)
  {
    Type service = Type.GetType(constructorString);
    ServiceHost host = new ServiceHost(service, baseAddresses);
    // hook up event handlers
    host.Opening += OnOpening;
    host.Closing += OnClosing;
    return host;
  }
}

To make the connection between the already-existing .svc endpoint and the custom factory, you have to add the Factory attribute to the @ServiceHost directive:

<%@ServiceHost Service="Service" Factory="HostFactory" %>

Automating Setup of WAS-hosted Services

As we've discussed, there are several steps necessary to set up a WAS-hosted WCF service:

  1. Setting up the physical directory (and ACLs if necessary).
  2. Setting up the application and virtual directory in WAS.
  3. Configuring the enabled protocols.
  4. Creating the queue and setting ACLs (if MSMQ is used).

WAS offers various interfaces for configuration including Windows Management Instrumentation (WMI), a command-line tool (appcmd.exe), and a brand-new managed API. While WMI and appcmd.exe are ideal for scripting and batch files, the new managed API gives you easy access to configuration for more complex deployment scenarios like MSI packages. See Figure 12 for some appcmd.exe examples.

Figure 12 Using appcmd.exe to Set Up a WAS Application (Close Figure 12)
REM adding bindings to the default web site
appcmd.exe set site "Default Web Site" –
    +bindings.[protocol='net.tcp',bindingInformation='808:*']
appcmd.exe set site "Default Web Site" –
    +bindings.[protocol='net.pipe',bindingInformation='*']
appcmd.exe set site "Default Web Site" –
    +bindings.[protocol='net.msmq',bindingInformation='localhost']

REM add a new application
appcmd add app /site.name:"Default Web Site" /path:/WasDemo
    /physicalpath:c:\etc\WasDemo

REM enabled protocols for application
appcmd.exe set app "Default Web Site/WasDemo"
    /enabledProtocols:http,net.pipe,net.tcp,net.msmq

The managed configuration API can be found in the new Microsoft.Web.Administration.dll assembly and is centered around the ServerManager class. From there you will find a complete object model covering application pools, worker processes, sites, applications and configuration files. Setting up an application using that API is a matter of a few lines of code, as you can see in Figure 13. Also take a look at Figure 14 for a little helper method that sets up a queue and the minimum ACLs for NETWORK SERVICE account.

Figure 14 Creating Queues and Setting ACLs (Close Figure 14)
private static void SetupMsmq(string queuename)
{
  // create queue – in this case a transactional queue is created
  MessageQueue queue = MessageQueue.Create(queuename, true);
  queue.Label = queuename;

  // set ACL
  queue.SetPermissions(GetAccountName(
    WellKnownSidType.NetworkServiceSid),
    MessageQueueAccessRights.ReceiveMessage);
}

private static string GetAccountName(WellKnownSidType wellKnownSid)
{
  NTAccount account = (NTAccount) new SecurityIdentifier(wellKnownSid,
    null).Translate(typeof(NTAccount));
  return account.Value;
}
Figure 13 Using the Managed Configuration API (Close Figure 13)
private static void SetupWasApp(string virtualPath, string physicalPath,
  string enabledProtocols)
{
  ServerManager manager = new ServerManager();

  // creating the application
  manager.Sites[0].Applications.Add(virtualPath, physicalPath);
  manager.CommitChanges();

  // setting up the enabled protocols
  Application wasApp = manager.Sites[0].Applications[virtualPath];
  wasApp.EnabledProtocols = enabledProtocols;
  manager.CommitChanges();
}

Extending WAS

WAS is extensible, meaning you can write your own support for custom WCF transport channels. The Windows SDK contains a sample for a User Datagram Protocol (UDP)-based WAS activation extension, but extending WAS to add your own features is no simple task. It involves implementing custom process and application domain protocol handlers as well as creating code for listening to receive messages on the respective and targeted transport mechanism. An in-depth article on that topic alone would be needed to explain it sufficiently.

There is one very essential point we must mention, however. The maximum request execution limit in WAS has different values depending on the OS in question. For Windows Server 2008 it is unlimited, but for Windows Vista Ultimate and Business it is a 10-connections limit. The home-targeted versions of Windows Vista, namely Windows Vista Home Basic and Home Premium, as well as Windows Vista Starter, have a request execution limit of 3. Keep this in mind as you plan your implementation.

Dominick Baier is the security consultant at thinktecture (thinktecture.com) in Germany. In addition he is the security and WCF curriculum lead at DevelopMentor (develop.com), developer security MVP, and the author of Developing More Secure Microsoft ASP.NET 2.0 Applications (Microsoft Press, 2006). You can find his blog at leastprivilege.com.

Christian Weyer is co-founder and principal architect of thinktecture and has been modeling and implementing distributed applications with Java, COM, DCOM, COM+, Web Services, WCF, and other technologies. Get in touch with him at thinktecture.com/staff/christian.

Steve Maine is a Senior Program Manager on the Connected Framework team at Microsoft. He's currently working on many Web-focused programming model features for the upcoming release of WCF in the .NET Framework 3.5. He blogs at hyperthink.net/blog.


Unit Testing: Exploring The Continuum Of Test Doubles

Unit Testing

Exploring The Continuum Of Test Doubles

Mark Seemann
Code download available at: Testing2007_09.exe (271KB)
This article discusses:
  • Unit testing with test doubles
  • Dummies, stubs, spies, fakes, and mocks
  • Manual and dynamic mocks
  • Knowing when and how to use each
This article uses the following technologies:
Unit Testing
Contents
Unit Testing for Dummies
Knowing When to Stub
Adding Spies to Your Tests
Making a Mockery of Unit Tests
Returning Values
Faking It
Manual Mocks
Mock Reality
Comparing Types

I
n the past couple of years, unit testing has gained tremendously in popularity; but while most developers understand the overall concept, certain aspects have been more elusive. Among these is how to effectively replace component servers for testing purposes. Most people call these replacements stubs or mocks, but as I will show in this article, these are only two types in a larger continuum of replacements.

Gerard Meszaros (xunitpatterns.com) has suggested the term "test double" (similar to the term "stunt double") as a general name for objects used to replace real component servers for testing purposes. While test double is in itself a generic term, specific names are used to describe the specific implementations, as explained in Figure 1.

Figure 1 Test Doubles Defined (Close Figure 1)

Test Double Type | Description
Dummy | The simplest, most primitive type of test double. Dummies contain no implementation and are mostly used when required as parameter values, but not otherwise utilized. Nulls can be considered dummies, but real dummies are derivations of interfaces or base classes without any implementation at all.
Stub | A step up from dummies, stubs are minimal implementations of interfaces or base classes. Methods returning void will typically contain no implementation at all, while methods returning values will typically return hard-coded values.
Spy | A test spy is similar to a stub, but besides giving clients an instance on which to invoke members, a spy will also record which members were invoked so that unit tests can verify that members were invoked as expected.
Fake | A fake contains more complex implementations, typically handling interactions between different members of the type it's inheriting. While not a complete production implementation, a fake may resemble a production implementation, albeit with some shortcuts.
Mock | A mock is dynamically created by a mock library (the others are typically produced by a test developer using code). The test developer never sees the actual code implementing the interface or base class, but can configure the mock to provide return values, expect particular members to be invoked, and so on. Depending on its configuration, a mock can behave like a dummy, a stub, or a spy.

Although these types seem distinct in theory, the differences become more blurred in practice. For that reason, I think it makes sense to think of test doubles as inhabiting a continuum, as illustrated in Figure 2. At the one extreme you'll find dummies with absolutely no implementation, and at the other end are full production implementations. Dummies and production implementations are both well-defined, but stubs, spies, and fakes are more difficult to pin down: when does a test spy become a fake? In addition, mocks inhabit a rather large interval in the continuum, since they can be quite complex in some instances but very simple in others.

Figure 2 Spectrum of Test Doubles
Figure 2 Spectrum of Test Doubles(Click the image for a smaller view)
Figure 2 Spectrum of Test Doubles
Figure 2 Spectrum of Test Doubles (Click the image for a larger view)

If this sounds too philosophical, it's really not, and I'll use a lot of examples throughout the rest of the article to illustrate the concepts. Due to space considerations, I have elected not to show a few classes here and there, but only for trivial implementations. If you feel that you are missing out on anything, the complete code is contained in the download for the article, available from the MSDN® Magazine Web site.

Unit Testing for Dummies

While mocks can be said to inhabit a slightly more extreme place on the test double continuum than dummies, the latter are far easier to understand, so I'll start by giving an example involving a dummy.

In a particularly unimaginative moment, I decided to use the tried and true online shop order-handling scenario for my example. Consider the simple Order class reproduced in Figure 3. The most important thing to notice is that the constructor takes an instance of IShopDataAccess, which is defined as:

Figure 3 Order Class (Close Figure 3)
public class Order
{
    private int orderId_;
    private IShopDataAccess dataAccess_;
    private OrderLineCollection orderLines_;

    public Order(int orderId, IShopDataAccess dataAccess)
    {
        if (dataAccess == null)
        {
        throw new ArgumentNullException("dataAccess");
        }
        this.orderId_ = orderId;
        this.dataAccess_ = dataAccess;
        this.orderLines_ = new OrderLineCollection(this);
    }

    public OrderLineCollection Lines
    {
        get { return this.orderLines_; }
    }

    public void Save()
    {
        this.dataAccess_.Save(this.orderId_, this);
    }

    internal IShopDataAccess DataAccess
    {
        get { return this.dataAccess_; }
    }
}

public interface IShopDataAccess
{
    decimal GetProductPrice(int productId);
    void Save(int orderId, Order o);
}

To unit test the Order class, I'll need to provide a test double for IShopDataAccess, since the constructor will throw an exception if I pass in null. The simplest test double I can create is a dummy, and Visual Studio® 2005 makes this very easy: I just create a new class in the unit test project, call it DummyShopDataAccess, and have it implement IShopDataAccess. In Visual Studio, I can then click on the smart tag for the interface name and select "Implement interface IShopDataAccess," which causes Visual Studio to create all the members of the interface. All member implementations are identical except for their signatures, so I'll just show the Save method here:

public void Save(int orderId, Order o)
{
    throw new Exception("The method or operation is not implemented.");
}

Creating a dummy with Visual Studio takes less than 10 seconds, and you only have to write a single line of code regardless of the number of members of the class: the class declaration containing the interface declaration.

Now that I have the DummyShop DataAccess class, it's easy to write a simple unit test of the Order class:

[TestMethod]
public void CreateOrder()
{
    DummyShopDataAccess dataAccess = new DummyShopDataAccess();

    Order o = new Order(2, dataAccess);
    o.Lines.Add(1234, 1);
    o.Lines.Add(4321, 3);

    Assert.AreEqual<int>(2, o.Lines.Count);
    // More asserts could go here...
}

At this point, the dummy suffices since the IShopDataAccess interface isn't being used in any of the methods exercised on the test target, so I only need it to get past that insistent input validation in the constructor. Obviously, as soon as I attempt to invoke a member that actually uses the IShopDataAccess instance (such as the Save method), my tests would come crashing down if I were to use DummyShopDataAccess, since it would throw an exception if invoked. For these tests I can use one of the other test double types, as I'll proceed to demonstrate.

Knowing When to Stub

As soon as your unit test invokes a member on the test target that in turn invokes a member on a test double, you need something that, at a minimum, doesn't throw an exception. The simplest type of test double that meets this criterion is a stub.

In my next example, I'll write a unit test that invokes the Save method on the Order class. This will in turn invoke the Save method on the IShopDataAccess interface, so I'll need to implement the Save method in such a way that it doesn't throw an exception. Since this method returns void, the implementation is very simple:

public void Save(int orderId, Order o) { }

If you are interested in semantics, you may start to wonder whether this implementation is actually a dummy or a stub. It would be a stretch to claim that there's much implementation in this line of code, and if I leave the other members at throwing exceptions, you could very well argue that this test double is, in fact, a dummy. On the other hand, dummies are supposed to be used only to fill in required parameters, but this implementation can be used for something else since a client can invoke its save method without crashing; thus you could argue that this implementation is a stub.

While it would probably be possible to propose a stronger definition of either dummy or stub, I find that this ambiguity has few practical consequences aside from naming. It does, however, nicely illustrate why I think that test doubles inhabit a continuum instead of being discreet types. In this particular case, I'll choose to call my new class StubShopDataAccess, since I may want to add more implementation to some of the other members later.

With StubShopDataAccess in place, I can now write the following unit test:

[TestMethod]
public void SaveOrder()
{
    StubShopDataAccess dataAccess = new StubShopDataAccess();

    Order o = new Order(3, dataAccess);
    o.Lines.Add(1234, 1);
    o.Lines.Add(4321, 3);

    o.Save();
}

Calling Save on the Order class will invoke the Save method on StubShopDataAccess, which does absolutely nothing. Accordingly, the test succeeds simply due to the fact that it doesn't fail, but it doesn't really verify the test target.

Adding Spies to Your Tests

Although investigating test doubles is the subject of this article, keep in mind that the purpose of a unit test is to demonstrate that the test target (in this case, the Order class) works as intended. For the Save method, the intention is that it should invoke the Save method on the IShopDataAccess interface, but the test defined earlier doesn't verify this, since it would also succeed if Order.Save had a blank implementation—in other words, if it didn't invoke IShopDataAccess.Save.

In order to verify that the Save method was called, the test double should record whether it was invoked or not. Since recording member invocations for later verification is the primary distinguishing trait of a test spy, I'll create a new class and call it SpyShopDataAccess, as illustrated in Figure 4.

Figure 4 Spy Class (Close Figure 4)
internal class SpyShopDataAccess : IShopDataAccess
{
    private bool saveWasInvoked_;

    #region IShopDataAccess Members

    // Other IShopDataAccess members ommitted for brevity

    public void Save(int orderId, Order o)
    {
        this.saveWasInvoked_ = true;
    }

    #endregion

    internal bool SaveWasInvoked
    {
        get { return this.saveWasInvoked_; }
    }
}

Once again, I want to point out that the test spy type doesn't occupy a discreet space in the test double continuum. Given this, what if the other method of the IShopDataAccess interface still throws an exception? Is it half dummy and half spy, then? Or would it mostly resemble a stub if the other method provided a hardcoded return value but didn't record that it was invoked?

Even when recording invocations in all members, there are different ways to do it. In Figure 4, I used the simplest approach by just setting a flag, but I could also have recorded the number of invocations, like this:

private int saveInvocationCount_;

public void Save(int orderId, Order o)
{
    this.saveInvocationCount_++;
}

internal int SaveInvocationCount
{
    get { return this.saveInvocationCount_; }
}

This implementation clearly captures more information than the first one, but does that mean that it's more spy than the other spy? It's possible to create even more advanced spies by also recording the input parameters for every call, but that requires more code to implement and would thus place such a spy further to the right on the test double continuum. Once again, the point is that the boundary between stubs and spies is fuzzy and it makes sense to view test doubles as inhabiting a continuum.

With the SpyShopDataAccess class, I can now write a unit test that verifies that the Order is, in fact, being saved:

[TestMethod]
public void SaveOrderWithDataAccessVerification()
{
    SpyShopDataAccess dataAccess = new SpyShopDataAccess();

    Order o = new Order(4, dataAccess);
    o.Lines.Add(1234, 1);
    o.Lines.Add(4321, 3);
    o.Save();

    Assert.IsTrue(dataAccess.SaveWasInvoked);
}

This simple test only verifies that the Save method was invoked, but not how many times or which parameters were used. It will often be a good idea to verify this type of data as well, but then you will have to either write a more advanced test spy or resort to mocks, which typically record all calls by default.

Making a Mockery of Unit Tests

Mocks are very different beasts than dummies, stubs, spies, and fakes, which are all created by the test developer. A mock is created by calling methods on a dynamic mock object that is being created at run time.

In my October 2004 MSDN Magazine article, "Unit Testing: Mock Objects to the Rescue! Test Your .NET Code with NMock" (msdn.microsoft.com/msdnmag/issues/04/10/NMock), I described the basics of dynamic mocks and how to use a particular mock library called NMock. Here, I will use another free mock library called Rhino Mocks (available at ayende.com/projects/rhino-mocks.aspx).

Instead of creating a new code file that implements the IShopDataAccess interface, I'll instruct Rhino Mocks to create an object implementing the interface at run time, as shown in Figure 5. To work as intended, a mock needs to be configured to expect which members will be invoked on the interface. Mock libraries do this in different ways, but Rhino Mocks works by allowing you to start out by recording expectations. Essentially, the mock has two modes: a recording mode where the unit test defines expectations, and a replay mode where clients can invoke members on the mock.

Figure 5 Dynamic Mock (Close Figure 5)
[TestMethod]
public void SaveOrderAndVerifyExpectations()
{
    MockRepository mocks = new MockRepository();
    IShopDataAccess dataAccess = mocks.CreateMock<IShopDataAccess>();

    Order o = new Order(6, dataAccess);
    o.Lines.Add(1234, 1);
    o.Lines.Add(4321, 3);

    // Record expectations
    dataAccess.Save(6, o);

    // Start replay of recorded expectations
    mocks.ReplayAll();

    o.Save();

    mocks.VerifyAll();
}

In the example in Figure 5, it's simple to set up the expectations, since the only thing I expect is a single call to the Save method. After all expectations have been recorded, I switch the mock to replay mode by calling ReplayAll.

When I call Save on the Order object, it will subsequently invoke the Save method on the mock. And since the mock is in replay mode, it will record that the method was called with the expected parameters. If the call to Save is made, but with different parameters, the mock will throw an exception. Calling VerifyAll at the end ensures that the Save method was invoked on the mock at all—if not, an exception is thrown.

Since the mock verifies that the expected calls were made, it works much like a test spy. However, it only exists at run time, so it's difficult to pin down exactly where on the test double continuum a mock exists. That's why I prefer to think of mocks as inhabiting quite a large part of the continuum.

Returning Values

Until now, I've only looked at mocking a single method (Save) that doesn't return any value. As I've shown you already, implementing a stub for a method that returns void can be very simple. For methods that return values, things get a tiny bit more complicated. To show you how, I'll expand on the Order example.

The Order class contains a collection of OrderLine objects. Each OrderLine object contains a productId, a quantity, and a reference to the Order that owns it. To calculate each line total, the OrderLine class contains the Total property:

private decimal? total_;

public decimal Total
{
    get
    {
        if (!this.total_.HasValue)
        {
            decimal unitPrice =
                this.owner_.DataAccess.GetProductPrice(
                    this.productId_);
            this.total_ = unitPrice * this.quantity_;
        }
        return this.total_.Value;
    }
}

Notice that this implementation calls the GetProductPrice on IShopDataAccess. The implication of this is that to test the Total property, any test double used must return a value from GetProductPrice. The simplest way to do this is just to return a hard-code value:

public decimal GetProductPrice(int productId)
{
    return 25;
}

With this implementation in StubShopDataAccess, I can now write a unit test:

[TestMethod]
public void CalculateSingleLineTotal()
{
    StubShopDataAccess dataAccess = new StubShopDataAccess();

    Order o = new Order(7, dataAccess);
    o.Lines.Add(1234, 2);

    decimal lineTotal = o.Lines[0].Total;
    Assert.AreEqual<decimal>(50, lineTotal);
}

Since the first order line has a quantity of 2 of the product 1234, the line total becomes 50, since the unit price returned by StubShopDataAccess is 25.

Although this implementation is very simple, it's not particularly flexible since it's not possible to test an Order with several order lines where each product has a different unit price. One possible solution is obviously to change the implementation of GetProductPrice, like this:

public decimal GetProductPrice(int productId)
{
    switch (productId)
    {
        case 1234:
            return 25;
        case 2345:
            return 10;
        default:
            throw new ArgumentException("Unexpected productId");
    }
}

Notice how conditional logic has crept into the test double, making the implementation a bit more complex. At this point you might begin to wonder whether it's still a stub: it's returning hardcoded values, but it contains conditional logic.

In principle, you can add an arbitrary number of case statements, but as you may have noticed from the previous unit test example, it's not very clear just from looking at the test code what the stub is going to return.

Faking It

Creating a more configurable test double may be the way to go in some instances. In the case of IShopDataAccess, this interface is really an abstraction of a database, so one approach would be to create a primitive, in-memory database. The FakeShopDataAccess class shown in Figure 6 is about as primitive as it gets since it has no referential integrity or any other features normally desirable in a database. It contains a collection object that basically acts as a stand-in for a database table. Since I need to be able to index products on product ID, I've created a ProductCollection class (not shown) that simply derives from KeyedCollection<int, Product>; the Product class is basically a property bag of product data and is another test-only class created solely for this purpose.

Figure 6 Primitive Test Database (Close Figure 6)
internal class FakeShopDataAccess : IShopDataAccess
{
    private ProductCollection products_;

    internal FakeShopDataAccess()
    {
        this.products_ = new ProductCollection();
    }

    #region IShopDataAccess Members

    public decimal GetProductPrice(int productId)
    {
        if (this.products_.Contains(productId))
        {
            return this.products_[productId].UnitPrice;
        }
        throw new ArgumentOutOfRangeException("productId");
    }

    public void Save(int orderId, Order o) { }

    #endregion

    internal IList<Product> Products
    {
        get { return this.products_; }
    }
}

You might notice that FakeShopDataAccess doesn't save any order passed to its Save method. This was a shortcut I chose to take since the IShopDataAccess interface doesn't have any members that return saved orders. However, if I had wanted to use this class for verification of the Save method, I could have added the Order object to an internal dictionary and later verified the contents of that dictionary. This would have added a test spy trait to the fake, once again illustrating that the boundaries between the different test double types are blurred.

With the FakeShopDataAccess class, it now becomes more apparent what's going on in the data access layer, as you can see in Figure 7. Before creating the test target (the Order object), I create and populate the list of products and their unit prices. When the test executes, GetProductPrice returns the unit price for the requested product in the list.

Figure 7 Testing with the Fake Database (Close Figure 7)
[TestMethod]
public void CalculateLineTotalsUsingFake()
{
    FakeShopDataAccess dataAccess = new FakeShopDataAccess();
    dataAccess.Products.Add(new Product(1234, 45));
    dataAccess.Products.Add(new Product(2345, 15));

    Order o = new Order(9, dataAccess);
    o.Lines.Add(1234, 3);
    o.Lines.Add(2345, 2);

    Assert.AreEqual<decimal>(135, o.Lines[0].Total);
    Assert.AreEqual<decimal>(30, o.Lines[1].Total);
}

Notice how this test is more readable than a similar test utilizing a stub since you don't have to flip to the stub code to see which values it will return. The disadvantage of creating a fake is that it may turn out to be a bit of work in itself. In the case of FakeShopDataAccess, I ended up creating three classes: FakeShopDataAccess itself, the ProductCollection class, and the Product class. The IShopDataAccess interface is even pretty simple, so for more complex interfaces, this disadvantage is only more pronounced.

Manual Mocks

One of the golden rules of unit testing is that you are supposed to write some fairly simple code to test your complicated code. Creating complex fakes would somewhat defy that purpose. If you ever begin thinking about unit testing the fake itself, you have probably overdone it.

Dynamic mocks are the ultimate answer to this conundrum, but you may not always want to use a full mock library. A mock library is an extra assembly that you need to reference in your test project, and if you are working on a team-based project, you need to address the issue of standardizing on a common mock library, installing it on all developer machines (or putting the binary under source control), and all the other details that go along with application deployment. That's why I sometimes like to throw together something I call a manual mock.

Let's say that I want to rewrite the test in Figure 7 without using a fake, but still keeping my test double translucent. The only member of IShopDataAccess I care about for this test is the GetProductPrice method, so I create the manual mock class shown in Figure 8 to handle this particular method. Notice that the constructor takes a Converter<int, decimal> that it saves and uses to implement the GetProductPrice method.

Figure 8 Manual Mock Class (Close Figure 8)
internal class ProductPriceMockShopDataAccess : IShopDataAccess
{
    private Converter<int, decimal> implement_;

    internal ProductPriceMockShopDataAccess(
        Converter<int, decimal> productPriceCallback)
    {
        this.implement_ = productPriceCallback;
    }

    #region IShopDataAccess Members

    public decimal GetProductPrice(int productId)
    {
        return this.implement_(productId);
    }

    public void Save(int orderId, Order o)
    {
        throw new NotImplementedException();
    }

    #endregion
}

Converter<int, decimal> is a delegate that takes an int as input and returns a decimal. If you refer to the definition of IShopDataAccess, you will notice that this is the same signature as GetProductPrice. This means that I can use Converter<int, decimal> to implement GetProductPrice, and that's exactly what I'm doing in Figure 8.

This manual mock contains no implementation in itself, but is just a scaffolding to be used by the unit test, as shown in Figure 9. In this unit test, I initialize the mock with an anonymous method that will get invoked when the GetProductPrice method is called. This is essentially similar to how dynamic mocks are configured, where you first define expectations for the mock, and then subsequently create and exercise the test target.

Figure 9 Using the Manual Mock (Close Figure 9)
[TestMethod]
public void CalculateLineTotalsUsingDelegate()
{
    ProductPriceMockShopDataAccess dataAccess =
        new ProductPriceMockShopDataAccess(delegate(int productId)
    {
        switch (productId)
        {
            case 1234:
                return 45;
            case 2345:
                return 15;
            default:
                throw new ArgumentOutOfRangeException("productId");
        }
    });

    Order o = new Order(10, dataAccess);
    o.Lines.Add(1234, 3);
    o.Lines.Add(2345, 2);

    Assert.AreEqual<decimal>(135, o.Lines[0].Total);
    Assert.AreEqual<decimal>(0, o.Lines[1].Total);
}

While the unit test in Figure 9 may look more complicated than the test using a fake shown in Figure 7, looks can be deceiving. Although the test itself is more verbose, the total size of the code supporting the test is smaller, since the fake requires three supporting classes (including the fake itself), while the manual mock only requires the mock itself.

The nice thing about using delegates for mocking is that the test developer writes a new implementation for each unit test. This not only enables you to change return values and behavior, but also to decide on the level of spying you want to perform. If you only want to return values, you can get by with code similar to that shown in Figure 9, but if you also want to record method calls for later verification, you can easily do this as well since anonymous methods allow access to outer variables.

The main drawback of the ProductPriceMockShopDataAccess class shown in Figure 8 is that it only mocks the GetProductPrice method, but throws a NotImplementedException in the other member. Obviously, you can take the concept of manual mocks a couple of steps further by generalizing the approach. One thing you could do in a general-purpose manual mock for IShopDataAccess is to define a Converter<TInput, TOutput> for every member of the interface, but that can soon become unwieldy if your interface has many members.

A better approach would be to define a delegate that can be used to implement any member, like this one:

public delegate void ImplementationCallback(MemberData member);

This delegate simply defines a method that returns void and takes a MemberData object as input. The MemberData class is rather trivial, so I'll not use space to show it here, but it is included in this article's code download. It is basically a property bag that contains the name of the invoked member, a list of parameter values, and a property that can be used to set any return value an implementation may need to provide. The return value is optional since not all methods need a return value (for instance, the Save method returns void).

With the ImplementationCallback delegate, I can create the more general manual mock shown in Figure 10. As before, the constructor is used to populate the mock with a delegate that will contain the implementation. Each member implementation follows a common pattern: a MemberData object is constructed to contain the name of the invoked method as well as the name and value of any parameters. The member variable is then used as a parameter when invoking the delegate. If the method should return any value, the value is extracted from the member variable and returned. It's the responsibility of the implementing delegate to set the ReturnValue property.

Figure 10 Generalized Manual Mock (Close Figure 10)
internal class MockShopDataAccess : IShopDataAccess
{
    private ImplementationCallback implement_;

    internal MockShopDataAccess(ImplementationCallback callback)
    {
        this.implement_ = callback;
    }

    #region IShopDataAccess Members

    public decimal GetProductPrice(int productId)
    {
        MemberData member = new MemberData("GetProductPrice");
        member.Parameters.Add(new ParameterData("productId", productId));

        this.implement_(member);

        return (decimal)member.ReturnValue;
    }

    public void Save(int orderId, Order o)
    {
        MemberData member = new MemberData("Save");
        member.Parameters.Add(new ParameterData("orderId", orderId));
        member.Parameters.Add(new ParameterData("o", o));

        this.implement_(member);
    }

    #endregion
}

A test using this approach looks similar to the previous test, as shown in Figure 11. The MemberData contains information about the member that was invoked, so it's sensible to perform an initial check whether the expected method is being called and throw an exception if this is not the case. If the method invoked was the GetProductPrice method, the test code then sets the relevant return value on the MemberData input parameter. This is the value that MockShopDataAccess will then return to its caller.

Figure 11 Using the Generalized Mock (Close Figure 11)
[TestMethod]
public void CalculateLineTotalsUsingManualMock()
{
    MockShopDataAccess dataAccess =
        new MockShopDataAccess(delegate(MemberData member)
    {
        if (member.Name == "GetProductPrice")
        {
            int productId = (int)member.Parameters["productId"].Value;
            switch (productId)
            {
                case 1234:
                    member.ReturnValue = 45m;
                    break;
                case 2345:
                    member.ReturnValue = 15m;
                    break;
                default:
                    throw new ArgumentOutOfRangeException(
                        "productId");
            }
        }
        else
        {
            throw new InvalidOperationException("Unexpected member");
        }
    });

    Order o = new Order(11, dataAccess);
    o.Lines.Add(1234, 3);
    o.Lines.Add(2345, 2);

    Assert.AreEqual<decimal>(135, o.Lines[0].Total);
    Assert.AreEqual<decimal>(30, o.Lines[1].Total);
}

You may have noticed that creating such a general-purpose manual mock follows a repeatable pattern: first, create the type and its members; then, in each member, create a MemberData object, invoke the delegate and return the ReturnValue property, if applicable. This implementation pattern can be automated, so it's possible to use a bit of Reflection and CodeDOM logic to create and emit a run-time object that works exactly like that. While that is beyond the scope of this article, I've provided a primitive proof of concept demonstrating this approach in the code download. Here, the DelegateMock class can create delegate-based mocks on the fly. The unit test looks almost identical to the test in Figure 11, except for the line where the mock is created, which now appears like this:

IShopDataAccess dataAccess =
    DelegateMock.Create<IShopDataAccess>(
        delegate(MemberData member){...});

Just like a regular mock, this manual mock is dynamically created and exists only at run time. In fact, the only difference between DelegateMock and dynamic mocks lies in how they implement the interface or base class they are mocking.

The advantage of delegate-based mocks lies in the high degree of freedom they offer and the relatively low barrier of entry. While many developers find dynamic mocks difficult to understand and learn, more developers understand delegates and know how to write code. Since all expectations, return values, recording of invocations, and so on have to be written explicitly in code, most developers already know how to accomplish those tasks without having to learn a new object model.

Mock Reality

While there are advantages to manual mocks, there are certainly also disadvantages, one of which is that providing delegate-based implementations in every unit test quickly becomes rather verbose—particularly when you also need to record invocations like a test spy. Perhaps the biggest disadvantage, however, is that writing delegates as implementations can rapidly become rather tedious and error-prone work.

Although I find manual mocks handy as quick-and-dirty mocks, it is no secret that I personally prefer dynamic mocks. A good mock library will create mocks that are easy to configure and that act as test spies at the same time.

The line total example that I've shown variations of in this article is very easy to write using Rhino Mocks. Creating the mock itself is accomplished in two lines of code and no supporting classes:

MockRepository mocks = new MockRepository();
IShopDataAccess dataAccess = mocks.CreateMock<IShopDataAccess>();

The dataAccess variable can then be used to create a new Order instance, just like the previous examples. The mock is created in recording mode, so defining expectations and return values can be done in only two lines of code:

Expect.Call(dataAccess.GetProductPrice(1234)).Return(45m);
Expect.Call(dataAccess.GetProductPrice(2345)).Return(15m);

The first line simply states that the mock should expect a call to the GetProducPrice method with the value 1234, and, when that happens, it should return 45. The second line obviously sets up a similar expectation for the product ID 2345.

By default, the mock acts as a test spy, so without further work, you can just call mocks.VerifyAll at the end of the test to verify that all expectations were met.

Comparing Types

In Figure 12, I've listed the different types of test doubles and their advantages and disadvantages. While such a table seems to indicate that the types are discreet and different, I hope that this article has demonstrated that the boundaries are, in fact, vague and that all test doubles are inhabitants of a continuum. It may seem that this distinction is mostly of semantic value, but it is my firm belief that the words we use shape how we think about abstract concepts, so it's important to be as precise as possible when describing such phenomena.

Figure 12 Advantages and Disadvantages of Test Doubles (Close Figure 12)

Test Double Type | Advantages | Disadvantages
Dummy | Very easy to create. | Not very useful.
Stub | Easy to create. | Limited flexibility. Opaque when observed from unit tests. No ability to verify that members were invoked correctly.
Spy | Can verify that members are invoked correctly. | Limited flexibility. Opaque when observed from unit tests.
Fake | Offers a semi-complete implementation that can be used in many different scenarios. | Harder to create. May be so complex that it requires unit testing in itself.
Mock | Efficient creation of test doubles. Can verify that members are invoked correctly. Transparent when observed from unit tests. | Steeper learning curve.

In my opinion, a good dynamic mock library can replace dummies, stubs, and spies, at least for the majority of situations. If your circumstances require creating a test double for a very complex interface or base class, there may be times where fakes offer more productivity than mocks.

Mocks need to have all expectations explicitly defined in each and every test. As the interaction between the test target and the test double becomes more intense, the work involved in defining a myriad of expectations may become too tedious and labor-intensive compared to the effort required to write a workable fake. Identifying the point at which this happens is, unfortunately, more an art form than an exact science.

Mark Seemann works for Microsoft Services in Copenhagen, Denmark, where he helps Microsoft customers and partners architect, design, and develop enterprise-level applications. Mark can be reached via his blog at blogs.msdn.com/ploeh.


Editor's Note: Summer in New York

Editor's Note
Summer in New York
Joshua Trupin

Even though it's still June, we're finishing up the September 2007 issue right now, and it's about a billion degrees outside. Seriously, a billion. New York is undergoing a "Vaporization Alert" week—people who have ventured out onto the streets are sinking ankle-deep into asphalt. Flaming, overturned taxis light up the ozone-filled sky. It's a typical summer in New York. The great outdoors is no place for people to go on days like this. So stay inside, crank up the air conditioning to high (global warming is at least another three years off), and enjoy this month's MSDN Magazine.

We've put our enforced indoor time to good use by gathering up some core info about Web 2.0 technologies for this issue of the magazine. Have you ever overheard someone sitting next to you on the bus discussing AJAX technologies? You probably want to know more about it! This month, we'll take an in-depth look at ScriptManager, which is the core of ASP.NET AJAX. As the name implies, it is the component that manages your scripts. If you really want to understand what's going on under the hood, you should start here.

Taking a step back from the guts of the app, Dino Esposito begins a two-part series looking at AJAX application architecture in his Cutting Edge column. Since AJAX is the core of Web 2.0 applications, this series will certainly be important reading if you're just diving in.

If you've been building apps on the ASP.NET platform for a while, you'll be interested in learning about extending your site with search. Marco Bellinaso explains how you can easily implement advanced searching capabilities by tying your ASP.NET code into search engines like those provided by Live Search and SharePoint.

In addition to all this Webby goodness, we've got an article about the networking namespace in the Microsoft .NET Framework 3.5. Now, we don't know exactly who's responsible for naming conventions, but the namespace is called System.Net. Calling something in the .NET Framework something.Net is sort of, well, confusing. Not that we're particular experts on these things, but wouldn't System.Networking be clearer? Or maybe the oppressive heat is just getting to us.

We've also been thinking that the name "Web 2.0" is really annoying. Does anyone have a better suggestion? Web 1.1, perhaps? On second thought, the boat may have sailed on this one already.

As annoying as "Web 2.0" sounds to our untrained ears, we're actually making good use of enhanced Web features on the MSDN Magazine site (msdn.microsoft.com/msdnmag). You might not be aware of our innovative new Code Explorer feature, but all of our new code samples now feature it. Instead of downloading the entire sample, you can navigate through the ZIP file right on the site, find the individual file you want, and then click to display it.

(Click the image for a smaller view)

(Click the image for a larger view)

Why did we go to this effort? We want to make MSDN Magazine more useful than ever. It was also something we could do inside. Who wants to go outside when it's a billion degrees out? —J.T.

Thanks to Paul Andrew, Mason Bendixen, Naren Datha, Shawn Farkas, Kit George, Matt Gibbs, Ladi Prosek, Cosmin Radu, Karthik Raman, Varun Sekhri, Richard Turner, and Danny van Velzen.


Toolbox: ASP.NET Controls, FTP, and More

Toolbox
ASP.NET Controls, FTP, and More
Scott Mitchell

Contents
Make Your ASP.NET App More Appealing
Comprehensive FTP Library Is User Friendly
Rich Imaging Support
The Bookshelf

Make Your ASP.NET App More Appealing

Since most ASP.NET applications are merely data entry forms accessible over the Internet, developers usually have their time monopolized by important yet bland issues such as reporting, data-entry form design, and input validation. This focus is evident in third-party control suites, which typically have a grid control as their flagship product.

While an application's substance is paramount, the importance of an intuitive and aesthetic user interface should not be discounted. You won't find any grid components in the ASPxperience Suite 1.1 by Developer Express, but you will find 18 Web controls that are designed to enhance your Web application's aesthetics and usability by presenting information in interesting and attractive ways.

The 18 controls that make up the suite specialize in navigation, layout, and application flow control. For example, the ASPxDataView control displays tabular data in columns and rows but, unlike ASP.NET's blocky GridView control, the ASPxDataView control can span data across multiple columns or rows, allows for a fine detail of customization, and offers an improved paging interface. The ASPxSiteMapControl offers a variety of eye-pleasing ways to display the Web site's navigational structure, which can be specified programmatically or from a hierarchical data source, such as an XmlDataSource or SiteMapDataSource.

The controls that impressed me the most in the ASPxperience Suite were the ASPxCloudControl and ASPxTimer controls. The ASPxCloudControl renders a tag cloud, which is a visual metaphor that succinctly depicts a weighted list. Each item in a tag cloud is associated with some value, and the greater that value is with respect to other items in the cloud, the larger that item's text appears. This interface enables a visitor to quickly ascertain how the value of one item compares to the others. The ASPxTimer control creates a client-side timer component without requiring you to write any timer-related JavaScript code. Moreover, the rendered timer component includes a client-side API for starting and stopping the timer as well as responding to tick events.

The controls' appearances are very customizable and can be configured programmatically or declaratively and with the aid of formatting wizards. Furthermore, many of the ASPxperience Suite controls use or can be configured to use AJAX techniques to enhance the user experience. All major browsers are supported, including Internet Explorer®, Firefox, Netscape, Opera, and Safari.

Price: $249 per developer license; $299 for a license plus source code.

www.devexpress.com

The Intuitive ASPxCloudControl
The Intuitive ASPxCloudControl (Click the image for a smaller view)
The Intuitive ASPxCloudControl
The Intuitive ASPxCloudControl (Click the image for a larger view)

Comprehensive FTP Library Is User Friendly

The File Transfer Protocol (FTP) is an open and well-known protocol for exchanging files between a server and a client over a network. There are many server and client implementations available. Microsoft® Internet Information Services (IIS), for example, includes an FTP service, and there are plenty of both free and commercial standalone FTP client applications. Even your Web browser can double as an FTP client.

While the abundance of FTP client applications is beneficial for end users, as developers we are more interested in communicating with FTP servers through programmatic interfaces. The Microsoft .NET 2.0 Framework includes classes for sending and retrieving files from an FTP server (see the FtpWebRequest and FtpWebResponse classes in the System.Net namespace). There are also third-party FTP libraries that add features not found in the .NET Framework. One such library is FTP for .NET 2.0 by Rebex, which provides a comprehensive API for connecting, uploading, and downloading files to and from an FTP server, with support for both .NET and .NET Compact Framework applications. FTP for .NET allows for synchronous and asynchronous operations, binary and ASCII file transfers, and non-transfer related FTP functionality, such as creating and deleting files and folders on the FTP server.

What I liked best about FTP for .NET was its concise, easy-to-use API. The Ftp class makes up the core of the API. To start communicating with an FTP server, simply create an instance of this class and call its Connect method, passing in the FTP server name and port number. If you are working with an FTP server that does not allow anonymous visitors, use the Login method to provide your credentials. Uploading and downloading files is as simple as calling the PutFile and GetFile methods. The same straightforward API extends to FTP for .NET's other functions. In short, this product makes it possible to tackle common and tricky FTP tasks alike in a few lines of terse, readable code. And to help you get started, FTP for .NET ships with several helpful examples that showcase using the library in both command-line and Windows® Forms applications.

Keep in mind that FTP is inherently insecure since all communications occur in plaintext. There are two alternative FTP specifications that add secure communications: FTP/SSL, also called FTP over SSL; and SSH File Transfer Protocol (SFTP). Rebex offers a separate product line that adds support for the FTP/SSL and SFTP standards.

Price: $249 per developer license; source code available for $999 per license.

www.rebex.net/ftp.net

(Click the image for a smaller view)

(Click the image for a larger view)

Rich Imaging Support

If you have ever created or worked on an application that needed to manipulate or allow users to edit images, there's a strong chance you have spent time poring through the classes in the System.Drawing namespace. Sadly, .NET's built-in imaging classes provide such a low-level interface that even the simplest tasks can require a large amount of time, effort, and code. Moreover, the framework includes only limited image-related user interface controls. The good news is that there are many third-party products that abstract the complexities of GDI+ in powerful and easy-to-use imaging APIs. One such product is DotImage Photo Pro 5.0 by Atalasoft, which includes a robust imaging API as well as imaging controls for Windows Forms and ASP.NET applications.

To start working with an image, simply create a new Workspace object and use its Open method, passing in the path or URL to the image. Once a Workspace has been loaded, use its ApplyCommand method to perform any number of image manipulations. These include effects such as adding a drop shadow or applying a grayscale, cropping, adjusting the transparency, changing the palette, resizing, and performing transformations (image skewing, rotating, flipping, and so on). This succinct and uncomplicated API results in tight, readable code. For example, opening an image and saving a thumbnail requires just three lines of code.

DotImage Photo Pro also includes UI imaging controls for both Web and Windows applications. To allow users to interact with images through a Windows application, add a WorkspaceViewer control to the form. As the user interacts with the image—moving the mouse, clicking on the image, defining a region on the image, and so forth—the control raises corresponding events. Images can be displayed in a Web application using either ASP.NET's built-in Image control or DotImage's WebImageViewer control. In addition to displaying images, the WebImageViewer can also create a rich image editing interface that utilizes AJAX techniques to efficiently communicate with the Web server when opening, saving, and manipulating images.

DotImage Photo Pro works with a variety of popular image codecs (JPEG, PNG, BMP, TIFF, GIF, PCX, and PSD, to name a few). The API was designed with extensibility in mind and allows custom encoders and decoders to be created and plugged in at runtime. There are also add-ons that can be purchased from Atalasoft's Web site that add features such as barcode recognition, optical character recognition, and advanced photo effects.

Price: $999 per developer license.

www.atalasoft.com

DotImage Poto Pro Offers a Robust Imaging API
DotImage Poto Pro Offers a Robust Imaging API (Click the image for a smaller view)
DotImage Poto Pro Offers a Robust Imaging API
DotImage Poto Pro Offers a Robust Imaging API (Click the image for a larger view)

The Bookshelf

In my job as a consultant and trainer I frequently meet with companies whose existing line-of-business application is unable to meet their current needs. In some cases it's because the application is not scaling; in others, the company is extending its products or services and needs to update the application to address these new offerings. These meetings usually leave me amazed at just how easy it is to create databases and data-driven applications with Microsoft's developer tools.

The scalability, extensibility, and maintainability of a database are directly related to the quality of its design. As such, it is important that the developers creating the data model for a data-driven application have a solid understanding of the principles of good design. Pro SQL Server 2005 Database Design and Optimization, by Louis Davidson, Kevin Kline, and Kurt Windisch (Apress, 2006), introduces readers to these principles and illustrates how to implement them in SQL Server™ 2005. The first half of the book explores database design concepts such as entities, relations, data integrity, and constraints.

The book's second half looks at database design outside of the data model. Chapter seven, for example, examines security topics such as authentication, permissions, encryption, and auditing. Other chapters focus on query optimization and index creation and usage. I found the chapter on code-level architectural decisions to be the most interesting, as it included discussions on ad hoc SQL queries versus stored procedures and choosing between T-SQL and CLR database objects.

Database design is vital to creating a data-driven application. Regardless of your expertise, I am confident that Pro SQL Server 2005 Database Design and Optimization will help improve your database design skills and applications.

Price: $59.99.

www.apress.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.

All prices were confirmed at press time and are subject to change. The opinions expressed in this column are solely those of the author and do not necessarily reflect the opinions of Microsoft.

MSDN Magazine does not make any representation or warranty, express or implied with respect to any code or other information herein. MSDN Magazine disclaims any liability whatsoever for any use of such code or other information.


CLR Inside Out: COM Connection Points

CLR Inside Out
COM Connection Points
Thottam R. Sriram

Code download available at: CLRInsideOut2007_09.exe (252KB)
Contents
The Sample Scenario
Creating a Connection Point on a COM Server
The Client
Get the Add and ConnectionPoint Interface
Managed Client
Wrap-Up

A
typical scenario in COM has client objects instantiating server objects and then making calls to those objects. Without a special mechanism, however, it would be very difficult for those server objects to turn around and make calls back to the client objects. COM connection points provide this special mechanism, enabling two-way communication between the server and client. Using connection points, the server can call the client when certain events happen on the server.

With connection points, the server specifies the events that it is capable of raising by defining an interface. Clients that have actions to be taken when these events are raised on the server register themselves with the server. The clients subsequently provide the implementation for the interface defined by the server.

There are standard mechanisms through which clients can register themselves with the server. COM provides IConnectionPointContainer and IConnectionPoint interfaces for this.

The clients for a COM connection point server can be written in C++ and in C# managed code. The C++ client registers an instance of a class that provides the implementation for the sink interface. The managed client registers delegates for individual events, thus creating a single sink per event notification method. In the managed world, there are two ways by which the client can register itself—I detail both of these methods later in this column.

There are very few working samples for eventing and interop on the Web. In this column, I focus on creating an Active Template Library (ATL) connection point server. This involves exposing a COM method, defining an event interface that will be implemented by the client, and implementing the code that raises the events from the server. I also show you a sample C++ client that provides an implementation of the sink plus a sample C# client and the two ways you can register and listen to events from the server. Finally, I talk about the recommended way to implement a managed event sink.

The Sample Scenario

In my scenario, the server exposes a COM method:

HRESULT Add(int nFirst, int nSecond)

The server also defines ConnectionPointContainer and connection points so that clients can register with it. In addition, the server defines an interface, _IAddEvents, which has two methods in it:

HRESULT AdditionStarted()
HRESULT AdditionCompleted(int nResult)

The client provides the implementation for _IAddEvents and invokes the Add method on the server. The server fires the AdditionStarted and AdditionCompleted methods on the client to notify it appropriately. Then the client performs the appropriate actions associated with these events.

Creating a Connection Point on a COM Server

In the January 2007 installment of CLR Inside Out, I described in detail how to create a simple ATL COM server (see msdn.microsoft.com/msdnmag/issues/07/01/CLRInsideOut). Today's column assumes you've already gone through the process of creating an ATL COM server called ATLConnectionPointServer. So if you haven't done so, you may want to read the earlier column before proceeding.

So now you need to define a COM interface implemented by the server and make it a connection point. The process of creating a connection point on top of this COM server is straightforward. To do this, open the Class View in Visual Studio® and create a simple ATL object. Just right-click on ATLConnectionPointServer and add a class, select a simple ATL object, and provide the name of the class as Add. Be sure to select Supports: Connection Points when you walk through the wizard.

Now you have a server interface IAdd that can be called from the client. If you build the server, you will notice that there are two interfaces defined here. One is IAdd, which implements IDispatch, and the other is the dispinterface _IAddEvents.

Next add a new method, called Add, to the interface IAdd. This takes in two integers and returns an HRESULT. To do this, right-click on IAdd and select Add Methods. The signature of the methods will be:

HRESULT Add([in] int nFirst, [in] int nSecond)

Now open ATLConnectionPointServer.idl and add methods AdditionStarted and AdditionCompleted to the _IAddEvents interface as shown in Figure 1.

Figure 1 Adding AdditionStarted and AdditionCompleted (Close Figure 1)
library ATLConnectionPointServerLib
{
    importlib("stdole2.tlb");
    [
        uuid(7F45FEA6-4D7C-489C-A852-19BA8B29D8AB),
        helpstring("_IAddEvents Interface")
    ]
    dispinterface _IAddEvents
    {
        properties:
        methods:
        [id(1), helpstring("AdditionStarted")]HRESULT AdditionStarted();
        [id(2), helpstring("AdditionStarted")]
            HRESULT AdditionCompleted(int nResult);
    };
    [
        uuid(15B6C26A-0416-4C8F-9533-89F318355E31),
        helpstring("Add Class")
    ]
    coclass Add
    {
        [default] interface IAdd;
        [default, source] dispinterface _IAddEvents;
    };
};

If you compile the project at this point, you'll notice an autogenerated file called _IAddEvents_CP.h. This file, which is generated by ATL, contains an empty CProxy_IAddEvents class. This is the class that provides the firing of the events once the connection point is complete and hooked up.

Go to Class View and right-click on CAdd then select Add | Add Connection Point. In the following wizard, select _IAddEvents. If you open the _IAddEvents_CP.h file now, it will contain auto-generated code for two methods, namely Fire_AdditionStarted and Fire_AdditionCompleted. This is the code that calls back to the client sink objects when they register with the server.

You are now close to completing the implementation of the server. All you have remaining is the implementation of the Add method on the server and the trigger points for firing events from the server.

Open Add.cpp and provide an implementation for the Add method that you added. The implementation looks like this:

STDMETHODIMP CAdd::Add(int nFirst, int nSecond)
{
    // Fire AdditionStarted event
    Fire_AdditionStarted();

    int nResult = nFirst + nSecond;
    Sleep(1000); // simulate the addition taking a long time

    // Fire AdditionCompleted event
    Fire_AdditionCompleted(nResult);
    return S_OK;
}

Now just compile the solution and your server is ready.

The Client

Now you can move onto the client. I will start by looking at a C++ client and then I'll move on to a managed client.

The client is responsible for five main tasks:

  • It must provide you with the implementation for the _IAddEvents interface.
  • It must provide an interface pointer to the Add interface on the server.
  • It must get the ConnectionPoint of ConnectionPoinContainer of the Add interface and add the sink interface.
  • It must call the Add method and waits for the events to be fired from the server.
  • It must close cleanly and exit.

To implement the client, open a new C++ project called ConnectionPointClient and add a new C++ source file to the project. Add the ATLConnectionPointServer.h and ATLBase.h file to the project. The sink implements the _IAddEvents that the server has defined. There are two methods in this interface: AdditionStarted and AdditionCompleted. An implementation for these two methods is shown in Figure 2.

Figure 2 AdditionStarted and AdditionCompleted (Close Figure 2)
class CSink : _IAddEvents
{
    private:
    DWORD       m_dwRefCount;
    public:

    CSink::CSink() {m_dwRefCount = 0;}
    CSink::~CSink()    {}

    HRESULT STDMETHODCALLTYPE AdditionStarted()
    {
        printf("C++ SINK: Addition started event fired ... \n");
        return S_OK;
    };

    HRESULT STDMETHODCALLTYPE AdditionCompleted(int nResult)
    {
        printf("C++ SINK: Addition completed event fired ... \n");
        printf("C++ SINK: Addition result: %d \n",nResult);
        return S_OK;
    };
...

For simplicity, I have implemented the dispinterface on the client; the sample code provides an ATL client that does this automatically. The sink in this implementation merely prints the fact that it has been called and the result when the addition is completed. Your sink is now implemented and ready to go.

Get the Add and ConnectionPoint Interface

Now that the sink has been implemented, let's work on the client that will register this sink with the server. The client handles three main tasks.

  • It gets the interface pointer to the server Add interface.
  • It gets the ConnectionPoint of ConnectionPointContainer from the Add interface.
  • It registers the Sink interface with the server.

First, you get the interface IAdd to the server as follows:

CoInitialize(NULL);

hr = CoCreateInstance(
    CLSID_Add, NULL, CLSCTX_ALL, IID_IAdd, (void **)&pAdd);
if(hr != S_OK) { return; }

Next, you must get the connection point on the server so you can register the sink implementation with it. To do this, get the ConnectionpointContainer from the IAdd interface as follows:

// Using the interface for add,
// query for IConnectionPointContainer interface
hr = pAdd->QueryInterface(
    IID_IConnectionPointContainer,(void **)&pCPC);
if ( !SUCCEEDED(hr) ) { return; }

Now, you can get to the ConnectionPoint:

// Using the IConnectionPointContainer,
// get the IConnectionPoint interface
hr = pCPC->FindConnectionPoint(DIID__IAddEvents,&pCP);
if ( !SUCCEEDED(hr) ) { return; }

The client now has to create an instance of its sink implementation and register it with the server. To do this, the client creates an instance of the sink and gets its IUnknown interface pointer as follows:

// Create an instance of the sink object to pass
// to the server
pSink = new CSink();
if ( NULL == pSink ) { return; }

// Get the interface pointer to CSink's IUnknown pointer, which you
// will pass to the server
hr = pSink->QueryInterface (IID_IDispatch,(void **)&pSinkUnk);
if(!SUCCEEDED(hr)) { return; }

You are close to completing the client. All that remains is to register the sink with the server, call the server, and clean up. The client registers the instance of the sink with the server:

// Pass the sink interface to the server through the Advise
hr = pCP->Advise(pSinkUnk,&dwAdvise);
if(!SUCCEEDED(hr)) { return; }

You have now registered the client sink interface with the server.

The client calls the Add method on the server and passes it two arguments as required by the method. The result of the addition is returned through the AdditionCompleted event, not directly from the Add call. Now call the Add method on the IAdd interface pointer you obtained:

pAdd->Add(1, 5);

This call should trigger the firing of the events which in turn should call into the client. At this point, you can clean up the client by releasing all the interfaces you've obtained (see Figure 3).

Figure 3 Clean Up the Client (Close Figure 3)
// Release the IConnectionPointContainer interface.
if(pCPC != NULL) pCPC->Release();

// Unadvise the event call back we registered.
if(pCP != NULL) { pCP->Unadvise(dwAdvise); }

if(pSinkUnk != NULL) { pSinkUnk->Release(); }

// Disconnect from the server.
if(pCP != NULL) { pCP->Release(); }

// Release interfaces.
if(pAdd != NULL) { pAdd->Release(); }

CoUninitialize();
return;

You're finally done with your client. You can now compile the client and execute it:

cl COMConnectionPointClient.cpp

On execution, you should see the following output:

C++ SINK: Addition started event fired ...
C++ SINK: Addition completed event fired ...
C++ SINK: Addition result: 6

Managed Client

Now I want to discuss using the same ConnectionPointServer from managed code. The managed client is much simpler than the COM client. There are two ways you can implement the client. First, I'll focus on the recommended approach.

To begin, import the server DLL to managed code to obtain ATLConnectionPointServerLib.dll by using the Microsoft® .NET Framework Type Library to Assembly Converter tool, tlbimp.exe, running the following command:

tlbimp ATLConnectionPointServer.dll

You need to reference the resulting assembly in your managed project and then provide an implementation for the sink interface on the client, such as the one shown in Figure 4. The ManagedSink class implements two methods, AdditionStarted and AdditionCompleted, as defined in the _IAddEvents interface. With that, your sink event handler is now complete and ready. (Seems almost too simple compared to the COM client, doesn't it?)

Figure 4 Provide Implementation for Sink Interface (Close Figure 4)
public class ManagedSink :_IAddEvents
{
    public void AdditionStarted()
    {
        Console.WriteLine("C# SINK: Addition started event fired ...");
    }

    public void AdditionCompleted(int nResult)
    {
        Console.WriteLine("C# SINK: Addition completed event fired ...");
        Console.WriteLine("C# SINK: Addition result: {0}", nResult);
        return;
    }
};

As with the COM client, you have to register the sink with the server so the server can invoke the sink when firing the events. There are, however, some differences in the way the managed client registers itself with the server.

The COM client registered the instance of its sink object that implemented the interface _IAddEvents with the server. As a reminder, the following call registered the COM client:

// Pass the sink interface to the server through the Advise
hr = pCP->Advise(pSinkUnk,&dwAdvise);
if(!SUCCEEDED(hr)) { return;}

With the managed client, you register individual methods as delegates with the server. To accomplish this, you create an instance of the sink object:

ManagedSink ms = new ManagedSink();

Create an instance of the server object and add the AdditionStarted and AdditionCompleted event handlers separately, like this:

AddClass a = new AddClass();
a.AdditionStarted += ms.AdditionStarted;
a.AdditionCompleted += ms.AdditionCompleted;

The client registers two different interfaces for each of the event handlers with the server. The previous calls to add the delegates on the client add a ref count on the Runtime Callable Wrapper (RCW) on the client. The ref count has to be released by removing the event handler once the call is completed, like so:

a.Add(1, 5);
a.AdditionStarted -= ms.AdditionStarted;
a.AdditionCompleted -= ms.AdditionCompleted;

Finally, compile ManagedClient.cs:

csc /r:ATLConnectionPointServerLib.dll ManagedClient.cs

And run the executable. You should see the following output:

C# SINK: Addition started event fired ...
C# SINK: Addition completed event fired ...
C# SINK: Addition result: 6

Wrap-Up

Writing a client implementation for dispinterface in ATL can be slightly complicated. The sample I've discussed here deliberately works around the complexities by having its own Invoke implementation.

I want to thank Cosmin Radu, Ladi Prosek, Mason Bendixen, Varun Sekhri, and Claudio Caldato for all their help and suggestions in making this column happen.

Send your questions and comments to clrinout@microsoft.com.

Thottam R. Sriram has a master's degree in computer science from Concordia University, Montreal, Canada. He is currently a Program Manager on the CLR team working on COM interop. Before joining the CLR team, Thottam worked with the Windows team. Visit his blog at blogs.msdn.com/thottams.


Basic Instincts: Lambda Expressions

Basic Instincts
Lambda Expressions
Timothy Ng

Contents
What are Lambda Expressions?
Lambda Expressions as Callbacks
Why Add Lambda Expressions?
Type Inference
Code Generation under the Hood
Lambda Expressions and Variable Lifting
Make the Most of Lambda Expressions

L
ambda expressions, new in Visual Basic® 2008, are a handy addition to any programmer's toolbox. They are callable entities that are defined within a function, and they're first-class citizens; you can return a lambda expression from a function and you can pass lambda expressions to other functions. Lambda expressions were added to Visual Basic 2008, formerly code-named "Orcas," in order to support Language Integrated Queries (LINQ), which adds data programmability to Visual Basic (more on that later). As you use lambda expressions, you will begin to see the power and flexibility they promote. I invite you to sample the basic concepts of lambda expressions, explore their benefits, and witness how to use them to write more expressive programs.

What are Lambda Expressions?

The following is an example of a basic lambda expression definition. It defines doubleIt as a lambda expression that takes an integer and returns an integer. The lambda expression effectively takes the input, multiplies it by 2, and then returns the result.

Dim doubleIt As Func(Of Integer, Integer) = _
    Function(x As Integer) x * 2

The Func type is also new with Visual Basic 2008; it is essentially a delegate that has the return type specified as the last generic parameter and allows up to four arguments to be supplied as the leading generic parameters (there are actually several Func delegates, each of which accepts a specific number of parameters). The Func delegate type is defined in the assembly System.Core.dll in the System namespace. All new projects created with Visual Basic will automatically have a reference to System.Core.dll, so you can take advantage of the Func type immediately.

The following code shows the various overloads of Func:

Dim f0 As Func(Of Boolean)
Dim f1 As Func(Of Integer, Boolean)
Dim f4 As Func(Of Integer, Integer, Integer, Integer, Boolean)

Here, f0 is a delegate that returns a Boolean, f1 is a delegate that takes an integer and returns a Boolean, and f4 is a delegate that takes four integer arguments and returns a Boolean. The key point to note is that a lambda expression is typed as a delegate; that is, it is a callable entity, just like the delegates in Visual Basic 2005.

On the right-hand side of the assignment in the first code snippet, you can see the new lambda expression syntax. It starts with the keyword Function, which is then followed by an argument list and a single expression.

In that earlier example, the lambda expression takes one argument, x, which is an integer. Notice, however, that there is no return statement. That's because the Visual Basic compiler already knows the type based on the expression, and so a return statement is superfluous. In this case, since x is an integer, 2 * x is also an integer, so the result of the lambda expression is an integer.

The magic of a lambda expression is that you can simply invoke it like a normal delegate, as you see here:

Dim doubleIt As Func(Of Integer, Integer) = _
    Function(x As Integer) x * 2
Dim z = doubleIt(20)

If you run this code, you will see that the value stored in z is 40. Essentially, you have created a lambda expression that doubles any integer value that you throw at it.

Let's consider a more complex example—a lambda expression factory:

Dim mult As Func(Of Integer, Func(Of Integer, Integer)) = _
    Function(x As Integer) Function(y As Integer) x * y

Mult is a fairly complex lambda expression; it takes as input one integer argument and returns another lambda expression that takes as input one integer argument and returns an integer. The syntax is a little tricky, but using the following line continuations and formatting can help indicate the exact nesting structure:

Dim mult As Func(Of Integer, Func(Of Integer, Integer)) = _
    Function(x As Integer) _
        Function(y As Integer) x * y

That should be a little clearer; the outer lambda expression contains another lambda expression, which the compiler uses as the return statement. The inner lambda expression's signature matches the Func(Of Integer, Integer) delegate signature in the return argument of the outer lambda expression, so the compiler compiles the statement with no errors.

How would you use such a lambda expression?

Dim mult_10 = mult(10)
Dim r = mult_10(4)

The first line here defines mult_10 to be mult(10). Since Mult(10) returns a lambda that takes an argument and multiplies it by 10, the type of mult_10 is Func(Of Integer, Integer). The second line calls mult_10 with the value 4, so the result of r is 40 and the type of r is Integer.

Essentially, mult is a lambda expression factory. It returns lambda expressions that are customized by the first argument. You may have noticed that the inner lambda expression references the parameter of the outer lambda expression, but the lifetime of the inner lambda expression exceeds the lifetime of the outer one. I will discuss this variable lifting later.

Lambda Expressions as Callbacks

Since lambda expressions are simply delegates, you can use them anywhere you use delegates. Consider the following method, which takes a delegate and calls the delegate for each element in a list:

Delegate Function ShouldProcess(Of T)(element As T) As Boolean

Sub ProcessList(Of T)( _
        elements As List(Of T), shouldProcess As ShouldProcess(Of T))
    For Each elem in elements
        If shouldProcess(elem) Then
            ' Do some processing here
        End If
    Next
End Sub

This is a fairly standard application of delegates; the method ProcessList will loop through each of the elements of the list, check whether it should process the item, and then do some standard processing.

To use this in Visual Basic 2005, you would have to define a function in your class or module that has the same signature as the delegate, and then pass the address of the function to the ProcessList procedure, like so (note the code in red):

Class Person
    Public age As Integer
End Class

Function _PrivateShouldProcess(person As Person) As Boolean
    Return person.age > 50
End Function

Sub DoIt()
    Dim list As New List(Of Person)
    ' Obtain list of Person from a database, for example
    ProcessList(list, AddressOf _PrivateShouldProcess)
End Sub

This is troublesome at best; you often have to dig through the code documentation to figure out what signature the delegate imposes, and then you have to match it exactly. Furthermore, if you need to call ProcessList with many different shouldProcess functions, you pollute your code with lots of little private functions.

Let's take a look at how you can call this function with lambda expressions:

Class Person
    Public age As Integer
End Class

Sub DoIt()
    Dim list As New List(Of Person)
    ' Obtain list of Person from a database, for example
    ProcessList(list, Function(person As Person) person.age > 50)
End Sub

I love the elegance and simplicity of lambda expressions. There's no need to create your own function to perform the processing logic. The delegate is defined at the point where it is used, which is much better than having it defined in a private method somewhere and have it lose locality with the method using the private method.

I'm sure you can see how lambda expressions are powerful, convenient, and can make your code easier to read and maintain. More advanced features such as type inference add even more power.

One limitation to note is that a lambda expression is exactly that—a single expression. In Visual Basic 2008, you can only have a single expression in a lambda expression. Further on in this column, I will show you a new ternary operator introduced in Visual Basic 2008 that will allow you to construct conditional expressions, but the current feature will not support arbitrary statements in a lambda expression.

But before exploring the deeper concepts behind lambda expressions, I'll discuss why lambda expressions were introduced in the first place.

Why Add Lambda Expressions?

To support LINQ queries, a few features needed to be added; among them were Visual Basic and lambda expressions. Assume you have the following query statement in Visual Basic:

Dim q = From p In Process.GetProcesses() _
        Where p.PriorityClass = ProcessPriorityClass.High _
        Select p

A lot of work goes on under the hood to get this query statement to compile. At a high level, the compiler will iterate through the Process.GetProcesses collection, apply the Where filter to it, and return a list of processes that match the filter in the Where clause.

Notice that there is a Visual Basic expression inside the Where clause: p.PriorityClass = ProcessPriorityClass.High. To execute this filter, the compiler creates a lambda expression for the Where filter and applies it to each element in the process list:

Dim q = Process.GetProcesses().Where( _
            Function(p) p.PriorityClass = ProcessPriorityClass.High)

Essentially, the lambda expression provides a shorthand for the compiler to emit methods and assign them to delegates; this is all done for you. The benefit you get with a lambda expression that you don't get from a delegate/function combination is that the compiler performs automatic type inference on the lambda arguments. In the example above, the argument p's type is inferred by the usage; in this case, the Where argument defines the lambda expression type, and the type for the argument lambda expression is inferred by the compiler. The type inference features supported by the compiler are a powerful addition to Visual Basic. Let's see what they can do for you.

Type Inference

The introduction of powerful type inference mechanisms means that you don't need to worry about figuring out the type of each variable. Furthermore, type inference enables scenarios that are otherwise impossible. Let's look at three different ways types are inferred when using lambda expressions.

Inferring the Lambda Expression Argument Types This scenario is really handy if you have a delegate type you want to assign a lambda to and you don't want to fully specify the arguments:

Dim lambda As Func(Of Integer, Integer) = Function(x) x * x

In the example, the lambda variable is typed as a Func(Of Integer, Integer). This is a delegate that takes one integer argument and returns an integer argument. As a result, the compiler automatically infers the lambda argument x to be an integer, and the return value of the lambda to be an integer.

You also benefit from type inference of lambda expression arguments when you are calling a method that takes a delegate. Consider a modification of an earlier example:

Delegate Function ShouldProcess(Of T)(element As T) As Boolean

Sub ProcessList(Of T)( _
        elements As List(Of T), shouldProcess As ShouldProcess(Of T))
    ' Method body removed for brevity
End Sub

In this case, the ProcessList function takes a lambda expression.

You can call the procedure like this:

Sub DoIt()
    Dim list As New List(Of A)
    ' fill or obtain elements in list
    ProcessList(list, Function(a) a.x > 50)
End Sub

Notice that I didn't specify the type of the lambda argument as I did earlier, yet the compiler infers it to be Person. How does something like this happen? Well, there are actually several levels of type inference in this example.

First the compiler sees that ProcessList is a generic procedure that takes as input List(Of T) and ShouldProcess(Of T). In the call to ProcessList, the compiler sees that list is the first argument, and that list is a List(Of Person). Since the second argument offers no hints as to what the type T is, the compiler decides that T is of type Person. Next, it infers that the generic argument for ShouldProcess(Of T) is Person and so it infers the second argument to be ShouldProcess(Of Person). Finally, since the lambda expression did not supply a type for its argument, the compiler knows the argument type based on the delegate signature of ShouldProcess(Of Person), and it infers the type of the parameter (a) to be Person. This is an extremely powerful model for type inference; you do not need to know the type of the delegate arguments when you construct the lambda and, in fact, I recommend that you let the compiler do the hard work for you. Fortunately, however, you do get IntelliSense® and tooltips that help indicate what the inferred type is, so you can be just as productive, if not even more so.

Inferring the Result Type This scenario is really handy if you do not have a delegate type and want the compiler to synthesize one for you. It's a feature available only in Visual Basic:

Dim lambda = Function(x As Integer) x * x

In the example, the lambda expression is fully typed (the lambda argument x is of type integer, and the compiler infers that the return value is an integer, since integer * integer = integer). However, the lambda variable has no type. Therefore, the compiler will synthesize an anonymous delegate that matches the lambda expression shape and assign that delegate type to the lambda.

This is a great feature because it means you can create lambda expressions on the fly without having to statically construct their delegate types. For example, how often have you been in the situation in which you have a condition that you need to apply to a set of variables, and you need to do it in several places as in Figure 1? There have been quite a few times where I've been coding and I ran into a situation like this. Normally I would factor that out so that I could do the condition check in one place rather than scattered throughout the function.

Figure 1 Repeated Conditional Checks (Close Figure 1)
Class Motorcycle
    Public color As String
    Public CC As Integer
    Public weight As Integer
End Class

Sub PrintReport(motorcycle As New Motorcycle)
    If motorcycle.color = "Red" And motorcycle.CC = 600 And _
       motorcycle.weight > 300 And motorcycle.weight < 400 Then
       ' do something here
    End If

    ' do something here

    If motorcycle m.color = "Red" And motorcycle.CC = 600 And _
       motorcycle.weight > 300 And motorcycle.weight < 400 Then
       ' do something here
    End If
End Sub

But there are times when the check is used only in this function and nowhere else; I don't like polluting my class with random helper functions used only to support this function. Doing so negatively affects maintainability—what if someone else calls this function and I need to make a change? It also may cause name pollution—I find classes with lots of private methods really hard to follow. And it could make IntelliSense less useful since there are more and more entries in the IntelliSense list. In addition, locality of logic is impacted. If I make a separate private method, I'd like it to be physically close to the method that uses it. With many people working in the same code base, it can be hard to maintain this locality over the long run. Using lambda expressions and having the compiler automatically create delegate classes addresses these issues, as in Figure 2.

Figure 2 Lambda Expressions and Delegates (Close Figure 2)
Sub PrintReport(motorcycle As New Motorcycle)
    Dim check = Function(m As Motorcycle) m.color = "Red" And _
                                          m.CC = 600 And _
                                          m.weight > 300 And _
                                          m.weight < 400
    If check(motorcycle) Then
        ' do something here
    End If

    ' do something here

    If check(motorcycle) Then
        ' do something here
    End If
End Sub

I have factored out the logic to check some conditions on the Motorcycle class—not into a private method where there are disadvantages, but into a lambda expression where the compiler will automatically create a delegate type—that is hidden—and hook up all the necessary work so that I can call the lambda expression as though it were a method.

I love this approach because it puts the logic close to the implementation (within the method body), it's factored (only one copy), and the compiler takes care of much of the maintenance. This works well because you can build an arbitrarily complex expression as the body for the lambda expression.

Late Binding—Inferring Object In this scenario, neither the lambda variable nor the lambda expression is typed:

Dim lambda = Function(x) x * x

Here, an anonymous delegate is also generated for you by the compiler, but the lambda's types are System.Object. This means that late binding is enabled in this scenario when Option Strict is off.

This is a really nice scenario for those that rely on late binding. Lambda expressions fully support late-bound operations so, in the above example, as long as the * operator is defined on the types you give to the lambda, it will work:

Dim a = lambda(10)
Dim b = lambda(CDec(10))
Dim c = lambda("This will throw an exception because " & _
               "strings don't support the * operator")

As you can see in the examples above, as long as the runtime type has a * operator, everything works out well. Lambda expressions of this nature fit very nicely with the late binding model in Visual Basic.

Code Generation under the Hood

Now that I've explored lambda expressions, let's look at the kind of code the compiler generates. Consider the earlier example:

Sub TestLambda()
    Dim doubleIt As Func(Of Integer, Integer) = _
        Function(x As Integer) x * 2
    Console.WriteLine(doubleIt(10))
End Sub

You know that Func is a delegate and delegates are just pointers to functions, so how does the compiler get this magic to work? In this case, the compiler emits a new function for you and sets up the delegate so that it points to the new function:

Function $GeneratedFunction$(x As Integer) As Integer
    Return x * 2
End Sub

Sub TestLambda()
    Dim doubleIt As Func(Of Integer, Integer) = _
        AddressOf $GeneratedFunction$
    Console.WriteLine(doubleIt(10))
End Sub

The compiler essentially takes the lambda expression and creates a new function with its contents, and changes the assignment statement so the lambda expression will take the address of the generated function. In this case the function is generated in the same parent that contains the method that uses the lambda expression. If TestLambda is defined on a class C, then the generated function will be defined on C. Note that the generated function is not callable and is marked private.

Lambda Expressions and Variable Lifting

In the previous examples, the lambda expression bodies referred to variables that were passed into the lambda. However, the power of lambda expressions comes to fruition with variable lifting. As I hinted earlier, the compiler "performs some magic" in certain scenarios. Before exploring these scenarios, though, let's get some basic concepts down from the branch of mathematics called lambda calculus—lambda expressions are based on a similar concept.

The fundamental concept in lambda calculus is that a function can have free variables or bound variables. Free variables are those that are defined in the containing method's variables (locals and parameters). Bound variables are those that are defined in the lambda signature or are members of the class containing the lambda, including base classes.

It is very important to make the distinction between bound and free variables in your lambda expressions because they affect the semantics of the lambda expression, the code that is generated, and, ultimately, the correctness of your program. Here is an example of a lambda expression that contains bound and free variables:

Dim y As Integer = 10
Dim addTen As Func(Of Integer, Integer) = Function(ByVal x) x + y

Here, x is considered a bound variable inside the lambda since it is a formal parameter to the lambda expression, and y is considered a free variable since it is a variable that belongs to the containing method of the lambda expression.

Remember that after you define a lambda expression, it's treated like a delegate type (you can return the lambda expression from a method, for example). Consider this:

Function MakeLambda() As Func(Of Integer, Integer)
    Dim y As Integer = 10
    Dim addTen As Func(Of Integer, Integer) = Function(ByVal x) x + y
    Return addTen
End Function

Sub UseLambda()
    Dim addTen = MakeLambda()
    Console.WriteLine(addTen(5))
End Sub

This code will print "15" to the console when UseLambda is called. But, you might ask yourself, how does it work? The function MakeLambda defines y as a local variable, and the lambda is using y, but the lambda returns out of the function MakeLambda. The function UseLambda gets the lambda from MakeLambda and executes the lambda, and it seems like somehow the variable y is "remembered" in the lambda.

Is this some sort of behind-the-scenes trick? The lifetime of y is the method of MakeLambda. When we execute the lambda returned from MakeLambda, MakeLambda will be out of scope and its stack space should be removed. But y was defined on the stack and, somehow, it got "stuck" with the lambda.

This stickiness is the magic commonly referred to as variable lifting. In this case, the variable y is called a lifted variable. And, as you can see, lifted variables are powerful: the compiler does a lot of leg work for you to capture the state of variables and preserve them outside of their normal lifetimes.

More formally, when the compiler encounters a lambda expression that has free variables, it will lift the free variables into a class called a closure. The closure's lifetime exists beyond the lifetime of the free variables hoisted in it. The compiler rewrites the variable access in the method to access the one within the closure instance.

First, let's walk through the MakeLambda example again:

Function MakeLambda() As Func(Of Integer, Integer)
    Dim y As Integer = 10
    Dim addTen As Func(Of Integer, Integer) = Function(ByVal x) x + y
    Return Lambda
End Function

As we analyzed before, x is bound to the parameter of the lambda but y is a free variable. The compiler detects this and proceeds to create a closure class that captures the free variables, as well as the definition of the lambda expression:

Public Class _Closure$__1
    Public y As Integer
    Public Function _Lambda$__1(ByVal x As Integer) As Integer
        Return x + Me.y
    End Function
End Class

You can see that the closure variable captures the variable y and stores it in the closure class. The free variable is then converted to a bound variable on the closure class.

The compiler also rewrites the method that contains the lambda expression to look like this:

Function MakeLambda() As Func(Of Integer, Integer)
    Dim Closure As New _Closure$__1
    Closure.y = 10
    Return AddressOf Closure._Lambda$__1
End Function

Now you can see how the compiler creates the closure variable, rewrites the local variable y that is lifted into the closure variable, initializes the variable, and simply returns the address of the lambda expression that is written into the closure class.

It's important to note that the compiler only lifts free variables in the lambda expression. The state of the variable is captured in the closure, which exists as long as the lambda expression exists.

Let's look at another example:

Sub Test()
    Dim y As Integer = 10
    Dim Lambda As Func(Of Integer, Integer) = Function(ByVal x) x + y
    y = 20
    Console.WriteLine(Lambda(5))
End Sub

What value is displayed when you execute this function? If you said 25, you were spot on. Why 25? Well, the compiler captures and rewrites all of the free variables y to the closure's copy, like so:

Sub Test()
    Dim Closure As New $CLOSURE_Compiler_Generated_Name$
    Closure.y = 10
    Dim Lambda = AddressOf Closure.Lambda_1
    Closure.y = 20
    Console.WriteLine(Lambda(5))
End Function

As you can see, by the time the lambda expression gets executed, the value of y has been changed to 20 and thus, when the lambda expression gets executed, it returns 5 + 20.

This is really important when it comes to loops. Because the free variables are captured into a single closure, you may see unexpected behavior if, for example, you spawn a thread that uses the lambda expression with a captured variable that changes:

Sub Test()
    For I = 1 To 5
        StartThread(Function() I + 10)
    Next
End Function

In this example, suppose StartThread creates a new thread and prints the result of the lambda expression to the console. Since I is captured into the closure, it's possible that by the time the thread executes the lambda expression, the for loop has modified the value of I. In that case, the program may not print 11, 12, 13, 14, and 15 as expected. Instead, you have to scope the captured variable inside the for loop:

Sub Test()
    For I = 1 To 5
        Dim x = I
        StartThread(Function() x + 10)
    Next
End Function

This code will now capture the value of x in the closure and the program will print 11, 12, 13, 14, and 15 as expected. It is extremely important to know which variables are lifted, when the lambda expressions will be executed, and when the lifted variables may change, so that you can be sure your program executes correctly in all circumstances.

Make the Most of Lambda Expressions

In Visual Basic 2008, you can only supply one expression as the body of the lambda, but a new ternary keyword is also introduced that allows you to do simple, fully typed conditional expressions:

Dim x = If(condition, 10, 20)

The If keyword is similar to the IIF function call, except that the If keyword is fully type safe. This means that in the above example, the compiler deduced that both branches of the If keyword return an integer, and so it applies type inference rules and decides that the type of x is Integer. Using IIF, x will be of type object.

You can use the If keyword in a lambda expression:

Dim x = Function(c As Customer) _
    If(c.Age >= 18, c.Address, c.Parent.Address)

In this example, assume there is a Customer class whose definition includes an Address property that represents, as you'd expect, the current address of the customer. The lambda expression makes use of the ternary expression to apply a condition on the input argument; if the customer is 18 years or older, it returns his address. Otherwise, it returns the parent's address.

Now type inference kicks in, and the compiler determines the return type of the lambda expression to be Address. It then creates a delegate type for x (as discussed previously), where the delegate type takes as input a Customer and returns an Address.

If you're interested in learning more about lambda expressions, you can subscribe to my blog (blogs.msdn.com/timng) where I will be discussing lambda expressions (and other Visual Basic 2008 language features) in more detail.

Send your questions and comments to instinct@microsoft.com.

Timothy Ng is a Software Development Engineer on the Visual Basic compiler team at Microsoft. For the upcoming release of Visual Studio 2008 he has worked on several features, including type inference, friend assemblies, expression trees, and anonymous types. You can contact Tim at timng@microsoft.com.


Cutting Edge: AJAX Application Architecture, Part 1

Cutting Edge
AJAX Application Architecture, Part 1
Dino Esposito

Code download available at: CuttingEdge2007_09.exe (333KB)
Contents
AJAX to the Rescue
The AJAX Architecture
What's an AJAX Framework?
ASP.NET AJAX Extensions
A Look at Partial Rendering
Anatomy of an AJAX Postback
Optimization Techniques for Partial Rendering
Weighing Partial Rendering

W
hether you're a system administrator, a designer, or a developer, your job will be significantly affected by AJAX. Administrators have to ensure that the security bar is still high enough to face new types of possible attacks. Intranet administrators have to guarantee that JavaScript is not disabled on any browsers. Web designers have new challenges to pursue due to features attainable with AJAX that were once impossible or impractical. And developers have a new API and a new overall approach to programming to become familiar with. All that said, what does AJAX mean for architects?

AJAX applications are challenging because they introduce brand-new concepts and a new foundation. The role of the architect is essential because the AJAX paradigm straddles both the client and the server environments. A clear architectural vision is critical for determining what logic and processing happens on the client and what remains on the server, as well as what data objects the client and the server are able to exchange.

AJAX to the Rescue

The Web was originally used to share static pages of information. Over just a few years, it evolved into a far more dynamic medium. Today, the Web is made up of extremely interactive pages; however, it still relies heavily on a paradigm that is based on full-page transitions. The more the user interacts with a page, the more frequently pages must be created and served back to the user's browser. There are many negative consequences that result from serving interactive pages in this manner, such as slow page refreshes and flickering. It's an inelegant solution that frequently leads to a poor user experience. AJAX brings about a much-desired shift in this paradigm that allows for cleaner Web applications and it lays the groundwork for a new generation of applications. AJAX is really about the end user. With AJAX features, a Web application is much more interactive, more responsive. It's faster and friendlier. The user is encouraged to do more with the page as interaction and responses (and feedback in case of slow operations) are all handled more elegantly. Overall, the user has a much better experience.

With AJAX, more code is executed on the browser and this requires more script being given to the client pages. But this leaves some big questions to be answered. What kind of script? Who will write this script? And what architectural principles and patterns will this script adhere to?

In this column, I'll explain AJAX from an architectural standpoint and help you, the developers and architects, make thoughtful choices. At the same time, I'll give designers and administrators enough context information to help make their jobs easier. This is a pretty big topic, so I'm dividing it into two parts. Don't miss next month's column when I'll continue this discussion.

The AJAX Architecture

If you're considering AJAX, take a look at Figure 1, which illustrates the architectural implications of such a move. A traditional Web application does everything on the server. Only occasionally does the application emit script code to run a task on the client—for example, to handle data validation.

Figure 1 Classic Web Paradigm versus AJAX Paradigm
Figure 1 Classic Web Paradigm versus AJAX Paradigm(Click the image for a smaller view)
Figure 1 Classic Web Paradigm versus AJAX Paradigm
Figure 1 Classic Web Paradigm versus AJAX Paradigm (Click the image for a larger view)

An AJAX application uses a client-side framework that takes care of issuing calls to the Web server. An AJAX server-side framework then takes care of the request and returns a data feed to the client. This will often be a JavaScript Object Notation (JSON) data stream, but other formats—such as XML, RSS, and CSV—can be used.

The client receives data feeds and updates the UI using JavaScript. The server responds to requests by returning raw data, encoded in a given format. Bandwidth consumption is minimized, the speed of the app increases since requests take less time to complete, and UI updates are able to take effect with no visible postback. But while this solves many problems, the increase in action on the client also brings about new issues, such as new coding practices, new security hazards, accessibility concerns, and so on.

What's an AJAX Framework?

To keep AJAX alive in a Web page, a few conditions must be met. First, you need JavaScript support from the browser. A full-time Internet connection is also required—an AJAX application can't work offline. When all the requests are made at the page level, as they are with non-AJAX applications, the browser can offer to navigate through a cached set of these pages, and thus can proceed offline. This same behavior would have to be coded explicitly in the script-based framework that partners with the browser in an AJAX application. There's a lot going on in this area and some new tools to help are being developed. The ability to take AJAX applications offline is a challenge for many software vendors.

A special framework is required by any serious AJAX application. Usually the framework of choice is articulated in separate client and server parts.

There are two main types of script code in an AJAX page: system-level code from the framework of choice and user-level code that implements the page's expected behavior. The system-level code provides the engine used to send asynchronous requests to the Web server and process any responses. The user-level code updates the UI of the page, essentially using Document Object Model (DOM) scripting. So here's one of those questions I mentioned earlier: Who writes which? To discuss this, I'll focus on a particular AJAX framework—ASP.NET AJAX Extensions.

ASP.NET AJAX Extensions

An extension to ASP.NET 2.0, ASP.NET AJAX Extensions provides AJAX capabilities to new and existing Web sites. There are two programming models for ASP.NET AJAX: partial rendering and remote services. They assume two radically different architectures in the application and consequently have different pros and cons. (You've probably already figured out that most features in AJAX involve trade-offs.)

In brief, partial rendering allows you to maintain an architecture similar to a classic ASP.NET 2.0 application. It just provides you with a set of new server-side tools you can use to implement flicker-free page updates.

Remote services, on the other hand, involve a service-oriented back end invoked by a relatively script-heavy AJAX front end. Nearly all fundamental application processes—including authentication, data paging, and sorting—must be redesigned. Server-side code must be factored out to application-specific services and a format (such as JSON) must be selected for data exchange. Finally, a front end must be arranged, paying due attention to limit your coding to UI-level tasks and keeping most business logic off the client.

The remote services approach provides a more complete AJAX experience, while partial rendering offers a smoother transition with a progressive enhancement of the features already in your existing application.

The two approaches can be mixed to some extent. You can, for instance, keep the same traditional Web architecture, add some flicker-free updates here and there, and then take time to refactor some key features in a service-oriented manner. ASP.NET AJAX Extensions provides several key ingredients, but significant pieces of the work for creating great AJAX applications is still up to you. Figure 2 shows the client and server components of the ASP.NET AJAX Extensions framework.

Figure 2 Components of the ASP.NET AJAX Extensions Framework (Close Figure 2)

File | Location | Description
MicrosoftAjax.js | Client | Contains functions to extend JavaScript with object-oriented constructs. It extends the prototype of built-in JavaScript objects and defines new helper classes, such as StringBuilder.
MicrosoftAjaxWebForms.js | Client | Contains the JavaScript classes that form the network stack and all classes that implement the client-side partial rendering engine.
MicrosoftAjaxTimer.js | Client | Represents the client-side object model for the new AJAX System.Web.UI.Timer server control. This is basically a server wrapper for a browser's timer object.
System.Web.UI.ScriptManager | System.Web.Extensions | Orchestrates the download of proper JavaScript files and client-side data, including the AJAX library, proxy classes for remote services, localized version of script files, and globalization data.
System.Web.UI.UpdatePanel | System.Web.Extensions | Defines a region of a page that can be refreshed through an AJAX postback in a flicker-free way.
System.Web.UI.UpdateProgress | System.Web.Extensions | Defines a server-side template that can be used to provide feedback to the user while a long partial rendering operation is taking place.
System.Web.UI.Timer | System.Web.Extensions | Helper server control that posts back the AJAX way when a child client timer times out.

A Look at Partial Rendering

Jeff Prosise's June 2007 Wicked Code column (msdn.microsoft.com/msdnmag/issues/07/06/WickedCode) offers an excellent discussion about partial rendering with ASP.NET AJAX. From an architectural viewpoint, partial rendering doesn't add anything new. It's just a cool way of enhancing legacy applications with some AJAX capabilities—the most important of which is flicker-free page updates.

Partial rendering doesn't require learning a lot of new skills, and it has a very limited impact on existing code. It also offers a straightforward fallback mechanism for aspects of an application that may suffer from the adoption of AJAX—security and accessibility, for example. Basically, if in the upgrade process you get into trouble when turning certain parts of existing code into AJAX, you can just leave those code blocks as they were and move on.

A partial rendering request is often referred to as an AJAX postback. The term is quite appropriate, perfectly representing what really happens. An AJAX postback is like a classic ASP.NET postback, except that it is carried out by a piece of script code defined in the client-side ASP.NET AJAX library. Although dressed like a pure AJAX remote call, an AJAX postback looks like a regular postback request to the ASP.NET runtime components. Once on the server, the request goes through the typical lifecycle of postback requests and raises such events as Init, Load, and PreRender. On the server, an AJAX postback differs from a classic ASP.NET postback only in the algorithm it uses to render the final markup. The different algorithm is key to the improved performance and absence of page flickering. However, the application model remains the same as in ASP.NET. Figure 3 shows the modified lifecycle of an AJAX postback request.

Figure 3 Lifecycle of an AJAX Postback
Figure 3 Lifecycle of an AJAX Postback

Everything is the same in the AJAX and ASP.NET postbacks, except the implementation of the rendering stage. An AJAX postback still fires notification events, such as Load and PreRender, and it still processes viewstate information and raises state changed and postback events, such as TextChanged and Click.

An ASP.NET AJAX page must include one instance of the ScriptManager control. This control is the real nerve center of ASP.NET AJAX pages. It takes care of linking the page to any required framework script files and orchestrates partial rendering if it detects that an AJAX postback is occurring. The ScriptManager control checks an HTTP header in the request to determine whether the request is an AJAX postback. The following is an excerpt from the MicrosoftAjaxWebForms.js file that sets the HTTP header before triggering the AJAX request:

   request.get_headers()['X-MicrosoftAjax'] =
       'Delta=true';

To see what makes the rendering stage different for an AJAX postback, take a close look at the following excerpt from the System.Web.Extensions assembly:

   protected override void OnPreRender(EventArgs e) {
       base.OnPreRender(e);
       if (IsInAsyncPostBack)
           PageRequestManager.OnPreRender();
   }

In particular, the code snippet shown above comes from the ScriptManager control. As you can see, the script manager registers a pre-rendering event handler. When fired, the handler detects whether it is engaged in an AJAX postback and, if so, invokes a method from the PageRequestManager internal class. The OnPreRender method on the PageRequestManager class does the following:

   internal void OnPreRender()
   {
       _owner.SetRenderMethodDelegate(
          new RenderMethod(this.RenderPageCallback));
   }

Quite simply, the method sets a specific subroutine to render the markup for the current request. SetRenderMethodDelegate is a method defined on the System.Web.UI.Control class. Not intended for public use, this method basically assigns an event handler that the page or the control will use to render its contents. In the preceding code snippet, the method is invoked on the current page and ends up overriding the whole rendering process for the current request. The actual code responsible for the markup of an AJAX postback is therefore defined in the RenderPageCallback method—an internal method in the PageRequestManager class. (For more information on the ScriptManager control, see the article by Ben Rush in this issue of MSDN® Magazine.)

Basically the modified rendering process for an AJAX postback loops through all updatable panels in the page that are involved with the postback and accumulates the markup of each. All markup chunks, along with hidden fields and any error information, are packed into a response stream and passed to the client. Now let's dissect a typical partial rendering call in the context of a sample page.

Anatomy of an AJAX Postback

To make an ASP.NET page a partially rendered page, you must first add a script manager to the page and then define independently updatable regions by wrapping them with an UpdatePanel control. Here's an example:

<asp:ScriptManager runat="server" />
<asp:UpdatePanel runat="server" ID="UpdatePanel1">
   <ContentTemplate>
      <%-- Markup of the region goes here --%>
   </ContentTemplate>
</asp:UpdatePanel>

The UpdatePanel control doesn't alter in any way the visible markup generated for the region. It simply adds a surrounding <div> tag to the original markup:

<div id="UpdatePanel1">
      <%-- Markup of the region goes here --%>
</div>

What triggers an AJAX postback? How is it managed? And by whom? Whenever the script manager detects one or more UpdatePanel controls in the page, it emits the block of script as in the following:

<script type="text/javascript">
Sys.WebForms.PageRequestManager._initialize('ScriptManager1',
        document.getElementById('form1'));
Sys.WebForms.PageRequestManager.getInstance()._updateControls(
    ['tUpdatePanel1','tUpdatePanel2'], [], [], 90);
</script>

The _initialize method is a static method on the client-side PageRequestManager object (see MicrosoftAjaxWebForms.js). It creates a global instance of the PageRequestManager class and initializes it. The class acts as a singleton and the only available instance can be retrieved later through the getInstance method. The second statement of the preceding snippet registers an array of UpdatePanel controls with the client framework. Each server-side UpdatePanel control is referenced through its ID.

The key action happening here is inside the _initialize method. As I said, after creating the singleton instance of the class, the code initializes it. At this time, among other things, a handler is registered for the submit event of the DOM form object. This means that whenever the page submits a form, the AJAX script kicks in and places the request using XMLHttpRequest instead of letting the request go through a normal browser postback. The original set of form fields is maintained and some extra information is appended for the convenience of the server-side script manager. Hence, an AJAX postback uploads a bit more information than a regular ASP.NET postback.

The viewstate, as well as any other hidden fields, are carried out and uploaded to the server with the request. On the way back, an updated viewstate is downloaded along with new hidden fields, if any, and a likely shorter markup. In particular, the response includes only the markup for the updatable regions that have been modified during the postback. The list includes the UpdatePanel control that triggered the postback (and any nested panels), any other UpdatePanel controls in the page that have the UpdateMode property set to Always, and any UpdatePanel control that is programmatically refreshed. The following code is an example of how to programmatically refresh a panel based on run-time conditions:

   UpdatePanel1.Update()

Consider the sample page shown in Figure 4 and the response returned. You can use a variety of tools to monitor inbound and outbound HTTP packets. For this column, I used Nikhil Kothari's Web Development Helper tool, which you can download for free at projects.nikhilk.net/Projects/WebDevHelper.aspx.

Figure 4 Sample Page (Close Figure 4)
<%@ Page Language="VB" CodeFile="Test.aspx.vb" Inherits="Test" %>

<html xmlns="http://www.w3.org/1999/xhtml" >
<head runat="server">
    <title>Test :: Partial Rendering</title>
</head>
<body>
    <form id="form1" runat="server">
    <div>
        <asp:ScriptManager runat="server" ID="ScriptManager1" />

    <asp:UpdatePanel runat="server" ID="UpdatePanel1">
        <ContentTemplate>
            <asp:TextBox runat="server" ID="TextBox1" />
            <asp:Button runat="server" ID="Button1" Text="Update" />
            <hr />
            <asp:Label runat="server" ID="Label1" />
        </ContentTemplate>
    </asp:UpdatePanel>
    </div>
    </form>
</body>
</html>

The response of an AJAX postback is a text stream that can be seen as a table of records with Size, Type, ID, and Content columns. Each record is referred to as a delta node. Figure 5 shows the table you obtain for my sample page.

Figure 5 Delta Node Table for the Sample Page (Close Figure 5)

Size | Type | ID | Content
249 | updatePanel | UpdatePanel1 | <input name="TextBox1" type="text" value="Test" id="TextBox1" /> <input type="submit" name="Button1" value="Update" id="Button1" /> <hr /> <span id="Label1"></span>
52 | hiddenField | __VIEWSTATE |
56 | hiddenField | __EVENTVALIDATION |
0 | asyncPostBackControlIDs |  |
0 | postBackControlIDs |  |
13 | updatePanelIDs |  | tUpdatePanel1
0 | childUpdatePanelIDs |  |
12 | panelsToRefreshIDs | UpdatePanel1 |
2 | asyncPostBackTimeout | 90 |
9 | formAction |  | Test.aspx
25 | pageTitle |  | Test :: Partial Rendering

A delta node identifies a possible change that occurred during the AJAX postback. Not all responses feature the same set of delta nodes; it depends on what happens during the server lifecycle of the request and the structure of the original markup.

Figure 6 lists all currently supported delta nodes. The JavaScript code that is responsible for processing the delta table can be found in the MicrosoftAjaxWebForms.js file.

Optimization Techniques for Partial Rendering

After examining Figure 6, you'll note that the response is made up of two big blocks: markup and viewstate. Event validation data (a security-related feature of ASP.NET 2.0), custom hidden fields, scripts, and any other types of nodes usually take just a few dozen bytes all together. The size of the viewstate is an old problem for ASP.NET pages. Partial rendering with ASP.NET AJAX, unfortunately, does nothing to correct the problem. You still need to keep your viewstate lean and mean to improve download time. So what can you do to reduce the markup for a page? Essentially, you can have smaller panels that bring back just the minimum amount of markup you need for that particular click.

Figure 6 Partial Rendering Delta Node Type (Close Figure 6)

Delta Node Type | Description
updatePanel | Stores the updated markup for a partial rendering region.
hiddenField | Stores the content of a hidden field.
arrayDeclaration | Stores an array declaration added to the response using the appropriate RegisterXXX method on the script manager.
scriptBlock | Stores a script block added to the response using the appropriate RegisterXXX method on the script manager. The specified script is executed or inserted in the page based on which RegisterXXX method you used.
expando | Stores an expando property added to the response using the appropriate RegisterXXX method on the script manager. The specified value is assigned to the specified property ID when the response is processed.
onSubmit | Stores an onsubmit script block added to the response using the appropriate RegisterXXX method on the script manager.
asyncPostBackControlIDs | Stores the ID of controls registered as asynchronous triggers for the update panels in the call.
postBackControlIDs | Stores the ID of controls registered as postback triggers for the update panels in the call.
updatePanelIDs | Stores the ID of the update panels involved with the call.
asyncPostBackTimeout | Stores the timeout of the request in seconds.
childUpdatePanelIDs | Stores the ID of any nested update panel refreshed in the call.
panelsToRefreshIDs | Stores the ID of any update panels refreshed during the call, including panels refreshed programmatically.
formAction | Stores the URL of the action form.
dataItem | Stores any extra information generated on the server to be consumed by client components.
dataItemJson | Stores any extra JSON serialized information generated on the server to be consumed by client components.
scriptDispose | Stores a dispose script added to the response using the RegisterDispose method on the script manager. The specified script is executed on the client for the specified DOM element.
pageRedirect | Stores the new URL to reach in case of a redirection.
error | Stores error information in case an exception is raised during the postback.
pageTitle | Stores the new title of the page.
focus | Stores the ID of the new control holding the input focus.

Look again at the code in Figure 4. The logic in the page is fairly simple, but it illustrates the point. Basically, when the user clicks the button, any content in the TextBox is used to update the label.

The page contains only one UpdatePanel control that wraps everything: textbox, button, separator, and label. Logically speaking, this may be acceptable. If you have to identify a region in the page to refresh autonomously, this is a valid option. Defined in this way, though, the region contains a significant portion of code that isn't updated across postbacks. The TextBox and Button controls, for instance, are not updated in this page and the <hr> separator is plain static text.

A better option would be to reduce the UpdatePanel to comprehend the sole Label control—the unique server control that gets updated in any postback triggered by the button. In general, you should carefully identify postback relationships between controls—basically examining which updates which—and design updatable panels to include the minimum number of controls for a given user action. In the context of the sample shown in Figure 4, here's a better schema:

<asp:TextBox runat="server" ID="TextBox1" />
<asp:Button runat="server" ID="Button1" Text="Update"
     OnClick="Button1_Click" />
<hr />
<asp:UpdatePanel runat="server" ID="UpdatePanel1">
   <ContentTemplate>
       <asp:Label runat="server" ID="Label1" />
   </ContentTemplate>
   <Triggers>
      <asp:AsyncPostBackTrigger ControlID="Button1" />
   </Triggers>
</asp:UpdatePanel>

The UpdatePanel control now contains only the Label control. This means that no (unchanged) markup will be returned for the textbox and button controls. By default, an UpdatePanel control is refreshed when either one of its child controls causes a postback or another UpdatePanel in the page is refreshed. This behavior can be altered with some public properties—namely UpdateMode, ChildrenAsTriggers, and Triggers.

In particular, UpdateMode determines whether the UpdatePanel is set to refresh unconditionally (the default) or only under certain conditions. If you want a conditional refresh—a key setting for optimizing updatable panels—start by setting UpdateMode to Conditional:

<asp:UpdatePanel runat="server" ID="UpdatePanel1"
     UpdateMode="Conditional">

Note, though, that conditional updates are relevant only if you have multiple panels in a page.

There might be situations where you find it difficult to split a page into a set of normalized panels. A technique that might help you in these cases is preventing children from triggering postbacks. The ChildrenAsTriggers property is a Boolean property (true by default) that determines whether child controls of a panel act as triggers of AJAX postback events. In the following snippet, the Button1 control doesn't cause a refresh of the panel:

<asp:UpdatePanel runat="server" ID="UpdatePanel1"
     ChildrenAsTriggers="false">
   <ContentTemplate>
       <asp:Button runat="server" ID="Button1" Text="Update"
          OnClick="Button1_Click" />
       <asp:Label runat="server" ID="Label1" />
   </ContentTemplate>
</asp:UpdatePanel>

When a child button is clicked in a panel configured like this, an AJAX postback still occurs to execute any postback code attached to the button—for instance, the Button1_Click method—but no markup is returned to refresh the panel.

The third property involved with conditional refreshing is Triggers. This is a collection property and lists control events that will command a refresh of the panel, like so:

<asp:UpdatePanel runat="server" ID="UpdatePanel1">
   <ContentTemplate>
       <asp:Label runat="server" ID="Label1" />
   </ContentTemplate>
   <Triggers>
      <asp:AsyncPostBackTrigger ControlID="Button1"
           EventName="Click" />
   </Triggers>
</asp:UpdatePanel>

In this case, the Click event on Button1 will refresh the panel. The Button1 control may be placed anywhere in the page. Note that I'm talking about server events. In other words, if you want to refresh a panel when the user changes the selection on a dropdown list, you must first add the AutoPostBack attribute to the dropdown list.

You can still configure a panel to refresh when a state-changed event occurs on the server during an AJAX postback. Consider the following code:

<asp:TextBox runat="server" ID="TextBox1" />
<asp:UpdatePanel runat="server" ID="UpdatePanel1">
   <ContentTemplate>
       <asp:Label runat="server" ID="Label1" />
   </ContentTemplate>
   <Triggers>
      <asp:AsyncPostBackTrigger
           ControlID="TextBox1"
           EventName="TextChanged" />
   </Triggers>
</asp:UpdatePanel>

The label is refreshed whenever the content of TextBox1 changes:

Sub TextBox1_TextChanged(ByVal sender As Object, ByVal e As System.EventArgs) _
Handles TextBox1.TextChanged
   Label1.Text = TextBox1.Text
End Sub

The TextChanged event, as well as other state-changed events, are server-side events that fire as a consequence of a postback. The page still needs a button, or an auto-postback control, to trigger the postback in order to update the label.

The chart in Figure 7 summarizes the potential benefits of applying optimization techniques to updatable panels. With partial rendering, the size of the request packet increases a bit, but even in the least optimized case the response you get back is smaller than in a classic ASP.NET scenario. And, of course, users won't experience flickering.

Figure 7 Benefits of Optimizing Updatable Panels
Figure 7 Benefits of Optimizing Updatable Panels(Click the image for a smaller view)
Figure 7 Benefits of Optimizing Updatable Panels
Figure 7 Benefits of Optimizing Updatable Panels (Click the image for a larger view)

I should warn you, though, that pages with a huge viewstate may not benefit much from partial rendering. You might save 5KB of markup, but that's faint relief if you have to carry 30KB of viewstate.

But sometimes, viewstate is only an innocent scapegoat for an issue that originated elsewhere. Imagine you have a calendar control in an ASP.NET page and you use updatable panels to keep the refresh smooth:

<asp:UpdatePanel runat="server" ID="UpdatePanel1">
   <ContentTemplate>
       <asp:Calendar runat="server" ID="Calendar1" />
   </ContentTemplate>
</asp:UpdatePanel>

The markup for this fragment is no less than 7KB and can grow up to 10KB if you add some styles. Now, if you disable the viewstate on the calendar, you save approximately 1KB of data, which is only about 10 percent of the total size. This is because the Calendar is a heavy control in terms of markup. The viewstate has little to do with it. An ASP.NET AJAX page shouldn't use a Calendar control for date-picking functions. Instead, the AJAX Control Toolkit provides an extender—the CalendarExtender control—that, when added to a plain TextBox, lets you pick dates, effectively saving 10KB of markup.

This code pops up a calendar whenever the textbox gets the input focus:

<asp:textbox runat="server" ID="TextBox1" />
<act:CalendarExtender ID="CalendarExtender1" runat="server"
    TargetControlID="TextBox1"
    OnClientDateSelectionChanged="updateLabel"
    Format="dd/MM/yyyy" />

<asp:UpdatePanel runat="server" ID="UpdatePanel1"
     UpdateMode="Conditional">
    <ContentTemplate>
        <asp:label runat="server" ID="Label1" />
    </ContentTemplate>
</asp:UpdatePanel>

The calendar is created entirely on the client and only requires a couple of small icons to be downloaded. The extender automatically writes the selected date to the companion TextBox. However, with a bit of JavaScript code, you can handle the client date selection event and do whatever you need.

Weighing Partial Rendering

Partial rendering is the quickest way of spicing up an ASP.NET Web site with AJAX capabilities. It requires neither new skills nor a new architecture. Instead, it lends itself to progressively enhancing an existing Web site. But compared to a pure AJAX approach—direct client calls to remote services—partial rendering suffers from significant performance penalties.

As a rule of thumb, I recommend partial rendering as the best approach for building a rock-solid first AJAX version of a site. You can then progressively move away from the classic ASP.NET architecture and start building a set of back-end services and a thick presentation layer. In the next installment of Cutting Edge, I'll dive into the remote services approach to AJAX. It's neat, architecturally speaking, but it may require a complete redesign of your application and it raises a few new issues.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is a mentor at Solid Quality Learning and the author of Programming Microsoft ASP.NET 2.0 (Microsoft Press, 2005). Based in Italy, Dino is a frequent speaker at industry events worldwide. Get in touch with Dino at cutting@microsoft.com or join the blog at weblogs.asp.net/despos.


Team System: Work Items and Undo Support

Team System
Work Items and Undo Support
Brian A. Randell

Code download available at: TeamSystem2007_09.exe (197KB)
Contents
Changes
Work Item Support
Returning Work Item Queries
Updating the Check-In Dialog
Final Details

I
n the January 2007 installment of this column (msdn.microsoft.com/msdnmag/issues/07/01/TeamSystem), I described how to build a Microsoft® Word 2003 add-in to work with the Team Foundation Server version control subsystem. In the April 2007 column (msdn.microsoft.com/msdnmag/issues/07/04/TeamSystem), I drilled down into the work item tracking subsystem. In this month's column, I'll describe how you can add support for work items to the add-in. In addition, you'll learn how to add a feature that should have been in the first version of the add-in—undo support.

Changes

In the first column describing the add-in, I used the beta release of Microsoft Visual Studio® 2005 Tools for the 2007 Microsoft Office system (Visual Studio 2005 Tools for Office Second Edition, or VSTO 2005 SE for short). Since that time, Microsoft has released the final version, which supports building application add-ins for both Office 2003 and the 2007 Office system. Thus, if you're working along with the article, you need to upgrade the first "release" of the add-in to the RTM version. To do this, simply open the solution and recompile on a machine with VSTO 2005 SE installed. Once you've verified the new version works, the next step is to add undo pending changes support. This feature requires you modify the tfsvcUtil class and the ThisAddin class.

In tfsvcUtil, add a new shared method UndoPendingChanges that accepts the full path to the document currently being modified, and have it return a Boolean. It turns out the core functionality of this new method mimics the existing CheckInDocument method. The method checks to ensure a valid connection has been made to the Team Foundation Server and that a valid workspace is loaded. Once this is accomplished, it gets an array of PendingChanges objects for the passed-in document. Assuming a pending change is returned, the Undo method of the user's workspace is called, passing in the PendingChanges array. This method should return an integer value of 1. If it does, the method returns true, otherwise a tfsUtilException is thrown using the newly defined MSG_UNDO_NOT_ZERO constant. Figure 1 provides the full method listing.

Figure 1 UndoPendingChanges Method in tfsvcUtil (Close Figure 1)
Public Shared Function UndoPendingChanges( _
  ByVal docPath As String) As Boolean
  If serverValidated Then
    If m_userWorkspace Is Nothing Then
      Dim userWorkstation As Workstation = Workstation.Current
      Dim docWSInfo As WorkspaceInfo = _
        userWorkstation.GetLocalWorkspaceInfo(docPath)
      m_userWorkspace = m_tfsVCS.GetWorkspace(docWSInfo)
    End If

    Dim pc As PendingChange() = _
      m_userWorkspace.GetPendingChanges(docPath)

    If pc IsNot Nothing AndAlso pc.Length > 0 Then
      Dim retval As Integer = m_userWorkspace.Undo(pc)
      ' Retval should equal the number of items 'undone'
      If retval = 1 Then
        Return True
      Else
        Throw New tfsUtilException(String.Format( _
          MSG_UNDO_NOT_ZERO, docPath, retval))
      End If
    Else
      Return False
    End If
  Else
    Throw New tfsUtilException(MSG_SERVER_NOT_VALIDATED)
  End If
End Function

The changes you need to make to the ThisAddIn class are pretty straightforward. As before, due to space restrictions, I won't go into any great detail on the Word add-in specific code (you can compare this column's download to the previous version to see the changes). Needless to say, you will have to modify the class to add a new Undo Pending Changes button to the toolbar, add a click handler for the new button, and add additional code to handle state changes to control whether the button is enabled. Basically, the Undo Pending Changes button's enabled state should mirror that of the existing Check-in button.

Work Item Support

Adding support for associating work items with a check-in requires three major sets of changes to take place. First, you need to modify the tfsvcUtil class to support connecting to the work item store. Second, you need to modify the CheckInDocument method to do the actual association of work items. Finally, you need to modify the existing frmCheckIn dialog to support listing work items for the user to select as part of check-in.

To program work items, you add a reference to the Microsoft.TeamFoundation.WorkItemTracking.Client.dll assembly from the TFSUtil project. As mentioned in earlier columns, Team Explorer installs its supporting assemblies in the Global Assembly Cache (GAC) by default. However, the installer does not register them to show up in the Visual Studio 2005 Add References dialog. You'll need to either modify your registry so the Add References dialog sees the assemblies or manually browse for the assemblies. You'll find the assemblies at %Program Files%\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies\.

Once you've added the reference, you need to modify the tfsvcUtil class and add an imports statement at the top of the source file as follows:

Imports Microsoft.TeamFoundation.WorkItemTracking.Client

Currently when the add-in connects to the Team Foundation Server box, it only connects to the version control service. You could modify the existing connect method to also connect to the work item store. However, there might be times when the user of the add-in is not going to associate work items, and thus this would add unnecessary overhead to the connect method. Instead, you'll add a new method that is called separately to connect to the work item store. You'll make this method public; however, most of the time existing methods will call the method before doing any work item-related operations. If the connection to the work item store is already in place, then it effectively becomes a no-op.

You need to add a new shared method ConnectWIS to the tfsvcUtil class. This method accepts no parameters and returns nothing. The method body is simple. First, it checks to make sure a valid Team Foundation Server instance exists. If not, it throws an exception since the user needs to initiate the connection before this method is called. That said, you could modify the architecture of the add-in to support attempting to login to the user's default Team Foundation Server installation. If the server is available, a connection is made to the work item store using the GetServiceMethod of the m_tfs instance, typed as a TeamFoundationServer object:

Public Shared Sub ConnectWIS()
  If Not m_tfs Is Nothing Then
    If tfsvcUtil.m_tfsWIS Is Nothing Then
      tfsvcUtil.m_tfsWIS = CType( _
        m_tfs.GetService(GetType(WorkItemStore)), WorkItemStore)
    End If
  Else
    Throw New tfsUtilException(MSG_SERVER_NOT_VALIDATED)
  End If
End Sub

You'll note the work item store reference is cached in a class-level variable m_tfsWIS, typed as a WorkItemStore object. You'll need to add this variable to the class like this:

Private Shared m_tfsWIS As WorkItemStore = Nothing

Once you've done this, you need to create an overloaded version of the CheckInDocument method so that it matches the existing version with an additional input parameter that accepts an array of WorkItemCheckinInfo instances. Next, cut the body from the existing method and paste it into your new method. Modify the original method so that it calls the new version, passing Nothing for the WorkItemCheckinInfo array. The modified version is now one line of code:

Return tfsvcUtil.CheckInDocument(docPath, comment, Nothing)

Now you need to modify the code in CheckInDocument to see if any WorkItemCheckinInfo objects were passed into the method. If this is the case, the code calls the ConnectWIS method to ensure a connection to the work item store has been made. Once you've done that, you call the overloaded version of CheckIn that accepts an array of WorkItemCheckinInfo instances, passing Nothing for check-in notes and policy overrides. Figure 2 provides the complete listing.

Figure 2 Check-In with Support for Work Items (Close Figure 2)
Public Shared Function CheckInDocument( _
  ByVal docPath As String, ByVal comment As String, _
  ByVal SelectedWorkItems() As WorkItemCheckinInfo) _
  As Integer

  If serverValidated Then
    If m_userWorkspace Is Nothing Then
      Dim userWorkstation As Workstation = Workstation.Current
      Dim docWSInfo As WorkspaceInfo = _
        userWorkstation.GetLocalWorkspaceInfo(docPath)
      m_userWorkspace = m_tfsVCS.GetWorkspace(docWSInfo)
    End If

    Dim pc As PendingChange() = _
      m_userWorkspace.GetPendingChanges(docPath)

    If pc IsNot Nothing AndAlso pc.Length > 0 Then
      If SelectedWorkItems IsNot Nothing _
        AndAlso SelectedWorkItems.Length > 0 Then

        tfsvcUtil.ConnectWIS()

        Return m_userWorkspace.CheckIn( _
           pc, comment, Nothing, SelectedWorkItems, Nothing)
       Else
         Return m_userWorkspace.CheckIn(pc, comment)
      End If
    Else
      Return -1
    End If
  Else
    Throw New tfsUtilException(MSG_SERVER_NOT_VALIDATED)
  End If
End Function

Returning Work Item Queries

At this point, you've added work item association support to the check-in process. However, you need to enhance the tfsvcUtil class to support returning a list of work item queries for the current Team Project. In addition, you'll need a method that runs a query and returns the work items to the caller so that they can be displayed to the user as part of the integrated check-in experience.

To do this, add two methods. The first method, GetWIQ, returns a StoredQueriesCollection. The second method, RunWIQ, accepts a query name as string parameter and returns a WorkItemsCollection object. I detailed how to do this in the April 2007 edition of this column (msdn.microsoft.com/msdnmag/issues/07/04/TeamSystem). Figure 3 provides the code you need.

Figure 3 GetStoredQueries and RunWIQ Methods (Close Figure 3)
Public Shared Function GetStoredQueries(ByVal docPath As String) _
  As StoredQueryCollection

  If IsServerReady Then
    tfsvcUtil.ConnectWIS()

    If m_userWorkspace Is Nothing Then
      Dim userWorkstation As Workstation = Workstation.Current
      Dim docWSInfo As WorkspaceInfo = _
        userWorkstation.GetLocalWorkspaceInfo(docPath)
      m_userWorkspace = m_tfsVCS.GetWorkspace(docWSInfo)
    End If

    Dim vcTeamProject As TeamProject = _
      m_userWorkspace.GetTeamProjectForLocalPath(docPath)

    m_wisProject = m_tfsWIS.Projects(vcTeamProject.Name)

    Return m_wisProject.StoredQueries()
  Else
    Throw New tfsUtilException(MSG_SERVER_NOT_VALIDATED)
  End If
End Function

Public Shared Function RunWIQ(ByVal docPath As String, _
  ByVal WIQ As StoredQuery) As WorkItemCollection

  Dim wiqlToExecute As String = WIQ.QueryText.ToLower()
  Dim params As Hashtable = Nothing
  Dim wic As WorkItemCollection = Nothing

  If wiqlToExecute.Contains(MACRO_PROJECT) Then
    If params Is Nothing Then
      params = New Hashtable
    End If
    params.Add(MACRO_PROJECT.Substring(1), WIQ.Project.Name)
  End If
  If wiqlToExecute.Contains(MACRO_ME) Then
    If params Is Nothing Then params = New Hashtable
    params.Add(MACRO_ME.Substring(1), _
      m_tfsVCS.AuthenticatedUser.Substring( _
      m_tfsVCS.AuthenticatedUser.IndexOf("\"c) + 1))
  End If

  If params IsNot Nothing Then
    wic = m_tfsWIS.Query(wiqlToExecute, params)
  Else
    wic = m_tfsWIS.Query(wiqlToExecute)
  End If

  If wic.Count = 0 Then
    Return Nothing
  Else
    Return wic
  End If
End Function

Updating the Check-In Dialog

Now that you've got the main code written to handle associating work items at check-in, you need to modify the existing check-in dialog to support work item association. Start by making the existing form larger, adjusting the size property to something like 640×480 to give you room to adjust the layout. Next, add a panel control to the form and name it pnlComments. Then cut the existing textbox txtComment from the form and paste it into the new panel, setting the Dock property to Fill. You'll need to re-add a handles clause to the txtComment_TextChanged event handler because, when you cut the textbox, the existing clause is removed.

You'll need a GroupBox control called grpOptions added to the form on the left. Add two RadioButton controls into the group box and name them rdoComments and rdoWorkItems. Change their AutoSize property to False and their Appearance property to Button. Set the rdoComments radio button's Checked property to True. Finally, rearrange the form so that it resembles Figure 4.

Figure 4 Modified Check-In Dialog with Comments Page
Figure 4 Modified Check-In Dialog with Comments Page(Click the image for a smaller view)
Figure 4 Modified Check-In Dialog with Comments Page
Figure 4 Modified Check-In Dialog with Comments Page (Click the image for a larger view)

With the basic UI changes in place, you need to add support for showing work items in a grid when the Work Items toggle button is clicked. Add a panel control to the form called pnlWorkItems. Inside that panel, add two panels: pnlSQCombo and pnlGridHolder. Set pnlSQCombo's Dock property to Top and pnlGridHolder's to Fill. You then need to add a label and a combobox control to pnlSQCombo and a DataGridView control to pnlGridHolder. Name the combobox cboStoredQueries and the grid dgvWorkItems. Dock the grid within the panel control using the smart tag. Adjust the size and location of pnlWorkItems to match pnlComments and set its Visible property to False.

When the user clicks the Work Items toggle button, the click event handler will perform the work necessary to make the work items available. There's no point in adding the extra overhead to the form load time unless you know you'll always want work items associated with a document check-in. There's quite a bit of code that makes things happen, much of which is generic Windows® Forms code necessary to set the UI correctly.

The order of operations when the Work Items toggle button is clicked is as follows: load the check-in action combobox data sources, define the grid's columns, load the stored queries, and then run the My Work Items query, binding the results of the query to the grid control. Figure 5 lists the code that gets called when the user clicks the Work Items toggle button.

Figure 5 rdoWorkItems_Click Event Handler (Close Figure 5)
Private Sub rdoWorkItems_Click(ByVal sender As Object, _
  ByVal e As System.EventArgs) Handles rdoWorkItems.Click
  ' Load ComboBox Data Sources
  LoadCheckInActionComboBox()

  ' Define Grid Layout
  DefineWorkItemGrid()

  ' Load WIQs
  LoadStoredQueries()
  For Each sq As StoredQuery In msq
    If sq.Name = "My Work Items" Then
      Me.cboStoredQueries.SelectedItem = sq
      Exit For
    End If
  Next

  ' Run WIQ
  LoadGridData()

  Me.pnlComments.Visible = False
  Me.pnlWorkItems.Visible = True
End Sub

The rdoWorkItems_Click event handler calls a number of supporting procedures to do its work. LoadCheckInActionComboBox initializes the data sources for the check-in action combobox that will appear in the grid. If you examine the Microsoft check-in dialog, you'll note that, depending upon the type of work item being selected, the check-in action combobox will either show only the word Associate or it will show Associate and Resolve. Mimicking this behavior requires a bit of behind-the-scenes work in the form.

In order to make this work, your first step is to define two class-level arrays:

Private listAssociate(0) As String
Private listAssociateResolve(1) As String

The next thing you'll do is swap the check-in action combobox's data source between these two arrays depending upon the type of work item selected. LoadCheckInActionComboBox loads the string values into the arrays.

The event handler then runs DefineWorkItemGrid to initialize the grid control. This method adds an unbound checkbox control, four data-bound text columns (Work Item Type, Work Item ID, Title, and State), and an unbound combobox control for the check-in action. Next, rdoWorkItems_Click calls LoadStoredQueries. This method calls the GetStoredQueries method you created earlier. It then data binds the results to the cboStoredQueries combobox control.

Back in the rdoWorkItems_Click method, the code walks the list of stored queries until it finds the My Work Items query. Once it finds this, the code sets that to be the currently selected query in the cboStoredQueries combobox control. Then the method executes LoadGridData, which retrieves the work items returned by the My Work Items query and binds the results to the grid. Finally, rdoWorkItems_Click hides the comments panel and shows the work items panel.

To make the grid work in a similar fashion to the Microsoft version, you need to write event handlers for two grid-related events. The first event is the CellFormatting. One of the columns displayed by the grid is the Type property from the WorkItem object. This property is typed as WorkItemType object. You want to display the Name property of this object in the grid. Unfortunately, when the ToString method is executed, the object returns its fully qualified type name, not its Name property. To display the Name property, you need to change the cell's value in the CellFormatting event handler. This code checks to see whether the column to format is the Work Item Type column:

Private Sub dgvWorkItems_CellFormatting(ByVal sender As Object, _
  ByVal e As DataGridViewCellFormattingEventArgs) _
  Handles dgvWorkItems.CellFormatting

  If e.ColumnIndex = typeColumnIndex Then
    If e.Value IsNot Nothing Then
      Dim wit As WorkItemType = CType(e.Value, WorkItemType)
      e.Value = wit.Name
      e.FormattingApplied = True
    End If
  End If
End Sub

If it is and the cell's value is not null, the code retrieves the WorkItemType instance from the cell and changes the value of the cell to the value of the Name property.

Next, you need to implement an event handler for the Current-CellDirtyStateChanged event. In this handler you're changing the data source of the check-in action combobox based upon the type of work item selected. Write the code so that it only executes when the checkbox column's state is changed.

To know what the data source should be, you need to access the WorkItem object that is bound to the current row. Once you have it, you call its GetNextState method passing in a string value of Microsoft.VSTS.Actions.Checkin. The code is checking to see if there's a valid state transition from a check-in. If there is, the code receives a valid string back that tells it that the combobox should display both Associate and Resolve. If a valid string is not retrieved, then only Associate should be available. In this way, the code sets the correct data source and then commits the current edit, changing the state of the checkbox.

The code then checks the value of the checkbox and, if it has been set to checked (True), the check-in action combobox is enabled and made the active control, and an edit is started. If the checkbox is returning to an unchecked state, the code resets the combobox. Figure 6 provides the details.

Figure 6 Complete CurrentCellDirtyStateChanged Event Handler (Close Figure 6)
Private Sub dgvWorkItems_CurrentCellDirtyStateChanged( _
  ByVal sender As Object, ByVal e As System.EventArgs) _
  Handles dgvWorkItems.CurrentCellDirtyStateChanged

  Dim c As DataGridViewCell = dgvWorkItems.CurrentCell

  If c.ColumnIndex = checkedColumnIndex Then
    Dim dcb As DataGridViewComboBoxCell = _
      CType(dgvWorkItems(comboBoxColumnIndex, _
      c.RowIndex), DataGridViewComboBoxCell)

    Dim currentRow As DataGridViewRow = dgvWorkItems.Rows(c.RowIndex)
    Dim wi As WorkItem = CType(currentRow.DataBoundItem, WorkItem)

    Dim nextState As String = _
      wi.GetNextState("Microsoft.VSTS.Actions.Checkin")

    If String.IsNullOrEmpty(nextState) Then
      dcb.DataSource = listAssociate
    Else
      dcb.DataSource = listAssociateResolve
    End If

    dgvWorkItems.CommitEdit( _
      DataGridViewDataErrorContexts.CurrentCellChange)

    If CBool(c.Value) Then
      dcb.ReadOnly = False
      dcb.Value = dcb.Items.Item(0)

      dgvWorkItems.CurrentCell = dcb
      dgvWorkItems.BeginEdit(True)
    Else
      dcb.Value = Nothing
      dcb.ReadOnly = True
    End If
  End If
End Sub

The last bit of code you need to add to process work items is developed by creating an array of WorkItemCheckinInfo objects in the FormClosing event (see Figure 7). You create this array by walking the rows of the grid, creating a WorkItemCheckinInfo instance if the checkbox column is checked. When you create a WorkItemCheckinInfo instance, you specify the check-in action. In addition to this method, you need to add a public property, WorkItems, to the form so that the array can be retrieved once the dialog is closed. You set this property at the end of the FormClosing method.

Figure 7 FormClosing Event Handler in the frmCheckIn Dialog (Close Figure 7)
Private Sub frmCheckIn_FormClosing( _
  ByVal sender As Object, ByVal e As FormClosingEventArgs) _
  Handles Me.FormClosing

  Dim chkCol As DataGridViewCell
  Dim cboCol As DataGridViewCell

  Dim wiciList As New List(Of WorkItemCheckinInfo)
  Dim cia As WorkItemCheckinAction = WorkItemCheckinAction.None

  For Each row As DataGridViewRow In Me.dgvWorkItems.Rows
    chkCol = row.Cells(checkedColumnIndex)
    If CBool(chkCol.Value) Then
      ' Item is selected
      cboCol = row.Cells(comboBoxColumnIndex)
      Select Case cboCol.Value.ToString()
        Case "Associate"
          cia = WorkItemCheckinAction.Associate
        Case "Resolve"
          cia = WorkItemCheckinAction.Resolve
        Case Else
          cia = WorkItemCheckinAction.None
      End Select

      wiciList.Add(New WorkItemCheckinInfo( _
        CType(row.DataBoundItem, WorkItem), cia))
    End If
  Next

  If wiciList.Count > 0 Then
    Me.WorkItems = wiciList.ToArray()
  Else
    Me.WorkItems = Nothing
  End If
End Sub

Final Details

You'll find some additional code in the form to handle the stored queries combobox's SelectedValueChanged event. The only thing left to do for the add-in to support work items is to modify the cbbCheckInDoc_Click method of the ThisAddIn class in the Word add-in. The modified version simply checks to see if the new WorkItems property of the dialog has data. If it does, the new version of the CheckInDocument method is called. Otherwise, the old version is executed. Figure 8 displays the completed check-in experience working with work items.

Figure 8 Completed Check-In Dialog with Work Item Support
Figure 8 Completed Check-In Dialog with Work Item Support(Click the image for a smaller view)
Figure 8 Completed Check-In Dialog with Work Item Support
Figure 8 Completed Check-In Dialog with Work Item Support (Click the image for a larger view)

At this point, if you run the add-in, you can add a document to source control, check it in with comments and associate work items, check it out, and even undo pending changes. By this time, a lot of work has been accomplished, but there's even more that can be done. In the next edition of this column, I'll look at adding check-in notes, policy support, and possibly other interesting features.

Send your questions and comments to mmvsts@microsoft.com.

Brian A. Randell is a senior consultant with MCW Technologies LLC. Brian spends his time speaking, teaching, and writing about Microsoft technologies. He is the author of Pluralsight's Applied Team System course and is a Microsoft MVP. Contact Brian via his blog at mcwtech.com/cs/blogs/brianr.


Foundations: Workflow Communications

Foundations
Workflow Communications
Matt Milner

Code download available at: Foundations2007_09.exe (165KB)
Contents
The Context for Communications
Communicating with the Host
Communicating with Workflow Instances
Two-Way Communications
A Note on Local Communications
Wrap-Up

W
orkflows are meant to coordinate systems and people, and that means communicating beyond the boundaries of the workflow instance itself. Whether sending e-mail or instant messages (IM) or interacting directly with the user interface in a host process such as ASP.NET or with Web services, communication is a key component of most workflows. This month I will introduce the Windows® Workflow Foundation (WF) communication architecture. Furthermore, I will cover sending data out of a workflow and submitting data to running workflow instances. As a bonus, I will also show you how to send data to a running workflow and get a response.

The Context for Communications

Some developers approaching Windows WF might wonder why a communication architecture even needs to exist. After all, aren't workflows just instances of Microsoft® .NET Framework classes that are being executed? Can't I just call methods on other .NET classes and invoke members on the workflow instance itself? Once you consider how Windows WF manages workflow instances, however, you begin to see that this traditional .NET approach will not work.

It is true that workflows are composed of .NET objects, but they are also managed by the WorkflowRuntime. Workflows model business logic or a business process and may therefore exist logically over the course of several days, being persisted and restored several times. Moreover, workflows are often executed on different threads from the host processing code, meaning that communication with the workflow must take threading into account. Because of these and other considerations, communications in workflows rely on a certain level of indirection that, in the end, provides a much richer framework for interaction.

Communications in Windows WF are often categorized as local (communication between the workflow and objects residing in the same host app) or remote (communication involving an application or service outside of the host process). The same architecture underpins both types of communication, and this article will focus on the core technologies. In a future edition of this column, I will look in more detail at remote communications using Windows WF with Windows Communication Foundation (WCF).

Communicating with the Host

When you have a workflow and you want it to interact with code running in the host application, you need to first get a reference to the object in the host. Windows WF uses the common .NET service provider model for managing communications from workflow instances to objects in the host. (See msdn2.microsoft.com/system.iserviceprovider for more information.) This model requires a service provider, a service, and a consumer, and is used in many different areas of the .NET Framework Base Class Library.

In Windows WF, the WorkflowRuntime is the main service provider and activities are the primary consumers of services. The services themselves are classes you write and load into the WorkflowRuntime. At run time, activities query the service provider for a service and invoke operations on that service as shown in Figure 1.

Figure 1 Communicating with Host Services
Figure 1 Communicating with Host Services

To see how these pieces fit together, consider a simple notification activity. The purpose of this activity is to alert the interactive user with a message of some sort by presenting it in the user interface. The first step is to build a service interface and a service implementation that can be used by the activity. The interface and class in Figure 2 show a very simple service that can be used by activities to notify the user.

Figure 2 Service to Notify Users (Close Figure 2)
public interface INotificationService
{
    void SendNotification(string notification);
}

public class ConsoleNotificationService : INotificationService
{
    public void SendNotification(string notification)
    {
        Console.WriteLine(notification);
    }
}

Once a service is defined, it needs to be added to the WorkflowRuntime, the service provider. This is usually done in the hosting application when the WorkflowRuntime is created and can be accomplished in code or configuration. Here's an example of adding the ConsoleNotificationService to the runtime in code:

ConsoleNotificationService notificationService =
    new ConsoleNotificationService();
workflowRuntime.AddService(notificationService);

At this point, the service is hosted within the WorkflowRuntime and is available to consumers that can access the service and interact with it. Most often, you will access these services from your custom activities. In the code download for this article, there is a custom activity named NotifyActivity that acts as the consumer for this service. The activity has one property named Message that identifies the text to use to notify the user. In the Execute method, the activity needs to get a reference to the notification service so that it can call operations on it. The IServiceProvider interface is implemented by all service providers and defines a single method called GetService. The GetService method takes a single parameter of System.Type that identifies the type of service to retrieve. Note that the type you pass in can be the actual type of the service class, or it can be an interface that is implemented by that service.

When writing activities, the key class that allows access to the services is the ActivityExecutionContext, which is passed as a parameter to all methods that the workflow runtime invokes on the activity. ActivityExecutionContext implements the IServiceProvider interface and also provides a generic helper method to remove the need for casting the result from an object to your specific service type. This is the mechanism used by the NotifyActivity as shown here:

INotificationService service =
    executionContext.GetService<INotificationService>();
service.SendNotification(Message);

The other point in an activity where a service provider is available is in the Initialize method of the activity. In this case, an IServiceProvider parameter is defined on the Initialize method and can be used to get access to services when the activity is first initialized. The reason for the difference is that activities are created and have their Initialize method called when you call CreateWorkflow, but there is no ActivityExecutionContext until the workflow begins executing. Passing in an IServiceProvider allows your activities to interact with services before the workflow begins executing.

Now you've seen the basics of using services to communicate with code in the host, but there are a few important considerations to bear in mind when using this type of communication. The first point to note is that since I've added only one instance of the service class to the runtime, this service acts as a singleton. But generally you'll be running multiple instances of workflows on different threads and they will all be accessing your service. In that case, thread safety quickly becomes an issue. If you are managing state in a service class, you will want to make sure to use appropriate synchronization semantics and potentially manage state for individual workflow instances. If you choose to do the latter, then you can use the System.Workflow.Runtime.WorkflowEnvironment.WorkflowInstanceId property to get the instance id of the current workflow instance that is calling into your code. You can then use that id as an index into a collection of state.

Another consideration related to threading is that code in the service class will run on the same thread the workflow instance is running on. If you are using the DefaultWorkflowScheduler (which you are if you haven't configured something different) then the code is running on a thread that is different from the host thread. The most obvious case where this causes problems is when hosting your workflows inside a Windows Forms application. When the workflow will interact with controls or your form, you must use a Control's Invoke method and delegates where appropriate to make changes to these elements. (For more information on thread-safe programming with Windows Forms, see msdn2.microsoft.com/system.windows.forms.control.invokerequired.)

Finally, notice that the activity code asks the service provider for the service based on an interface instead of the concrete class type. The primary benefit of using interface-based programming in your activities is that it allows the activity to be neutral in regard to the hosting application. This example uses a console-based notification, but suppose you want to reuse this activity in workflows that are hosted in Windows Forms or ASP.NET applications. Because the activity uses the interface, the host application can insert an appropriate implementation of the service for use in the host. In the Windows Forms example, a new class can be written that also implements the INotificationService interface, but instead of doing a Console.WriteLine, the class can show a message box. An ASP.NET implementation might redirect the user to a Web page that shows the message. The key is that the activity is not aware of these issues because they relate to the hosting code and the host is responsible for adding the service to the runtime.

But what if you just want to send a message from your activity, do you really need to use this service model? The short answer is no, you can write code directly in your activity to, for example, send an e-mail message. But flexibility is one key reason you would not put the code directly in your activity. Instead, allow the host to define the implementation details of sending that e-mail message. Perhaps in many environments SMTP would do the trick, but in others the consumer of your activity might want to handle that message differently. By using a service and coding to an interface, you make your activity more flexible. The other main reason you wouldn't use code directly in your activity is if you need to manage state or keep an object alive in memory. Consider a connection to a server or a Web service proxy that you use to communicate with a service. If you have multiple activities in a workflow, or multiple instances of your workflow, that are going to use that connection, you will want to manage it outside of the activity and outside of the workflow. Using a service hosted in the runtime means that the service instance will be around as long as the runtime; you cannot say the same for your workflow instance.

Communicating with Workflow Instances

Besides initiating communication, workflows must also be able to receive communication. The mechanism for receiving communication in a workflow instance is based on an internal queuing system: activities create queues, then register to get notified when data arrives. The host application or a service loaded into the WorkflowRuntime initiates communication with the workflow instance by placing data on the queue as shown in Figure 3.

Figure 3 Queuing Data to the Workflow Instance
Figure 3 Queuing Data to the Workflow Instance

In a previous article (see msdn.microsoft.com/msdnmag/issues/06/12/WindowsWorkflow), I discussed how activities can create queues and register to get notified when data arrives, so I won't dwell on the activity portion of the process here. Suffice it to say that it is the responsibility of the activity to create the queue and register for notifications. The queue can be created either when the activity is initialized or when it is subscribed/executed, depending on the needs of the activity.

From the host perspective, several issues emerge when submitting information to the workflow instance: knowing the instance id and queue name to submit information to, making sure the workflow instance is loaded, and optionally getting notified when the information has successfully been submitted to the workflow instance. In the case of the workflow instance id, it is the responsibility of the host application or the service loaded into the runtime to be able to determine this information. The id could be part of the data structure that is being exchanged, it could be managed in a context property such as an HTTP cookie or SOAP header, or it could be managed in a separate data store and looked up based on the data being submitted to the workflow. Regardless of the implementation, the host must be able to resolve the id of the workflow in order to get a handle to the instance.

Using the instance id, the host can get a reference to the workflow instance by using the GetWorkflow method on the WorkflowRuntime. This method returns the WorkflowInstance class which is the object used to represent the workflow and send data to it. When you call GetWorkflow, the runtime ensures that the workflow is loaded from the persistence store if necessary, saving you the responsibility of managing this important step.

The second piece of data that is required to communicate with a workflow is the name of the queue to which data is to be delivered. Since a workflow could have several activities waiting for data at one time, it is possible to have several different queues available on a single workflow instance. Rather than use simple string values for queue names, the IComparable interface is used, which allows activities to create complex and unique queue names based on their configuration and type.

The host application code needs to know the name of the queue it is submitting data to, but unlike with the instance id, the workflow itself can provide some help. The WorkflowInstance class provides a GetWorkflowQueueData method that returns a collection of WorkflowQueueInfo objects to provide information about the currently available queues on the workflow instance:

public class WorkflowQueueInfo
{
    public ICollection Items { get; }
    public IComparable QueueName { get; }
    public ReadOnlyCollection<string>
        SubscribedActivityNames { get; }
 }

As you can see, the information returned from this operation includes not only the name of the queue, but also the list of items already in the queue and the activities which are registered to get notified when data is submitted to the queue. Using this information, it is often possible to resolve the queue name for the queue you desire.

User interface interaction is a key scenario where GetWorkflowQueueData is used. Many applications present users with options as to how they can interact with a process, and need the ability to filter the options based on the current state of the business process. For example, if you are managing an order for a customer, you might only want to show the button used for returns after the item has shipped. Your workflow will likely model this type of behavior with an activity that waits for a request signaling a return. By querying the queue data on a workflow instance, your host application can tailor the user interface to the current state of the business process.

One final note about managing the instance id and the queue names—you can use a runtime service in conjunction with your activity to manage this information. When your activity creates a queue and registers for queue events, it can also register with your service in the runtime and provide the instance id, queue name, and other data that helps the service resolve the keys it needs, as shown in Figure 4. For example, if your activity is interested in data coming from a Web page, it might use the page name and user ID as the key. In your host (ASP.NET), you would call an operation on the runtime service passing in the page name, user ID, and the data to be submitted to the queue. The runtime service could then look up the instance id and queue name based on the page name and user ID, and pass the data to the queue.

Figure 4 Registering with a Runtime Service
Figure 4 Registering with a Runtime Service

I have mentioned several times now that the host or a service enqueues data to the workflow instance, but haven't yet discussed how this is accomplished. The WorkflowInstance class contains two methods for enqueuing data, as shown here:

public void EnqueueItem(IComparable queueName, object item,
    IPendingWork pendingWork, object workItem);

public void EnqueueItemOnIdle(IComparable queueName, object item,
    IPendingWork pendingWork, object workItem);

Both methods take the same parameters, including the queue name and the object to be enqueued. (I'll talk about the other two parameters shortly.) At this point, it becomes apparent why the instance id and the queue name are so important. In order to get the WorkflowInstance object, you need the instance id, and in order to enqueue data to that instance, you need the queue name. The difference between the two methods is in how they handle delivering the data to the queue.

The EnqueueItem method attempts to deliver the data to the queue immediately and assumes that the workflow has reached a point where the queue has been created and is awaiting data. In some scenarios, this is exactly what you would like to have happen. In other cases, this type of behavior can create a race condition where your workflow has not yet reached the point where the activity creates the queue you are interested in. You could check this by calling the GetWorkflowQueueData and continuing to check until the queue is there, but this is cumbersome and messy.

The EnqueueItemOnIdle method is intended to address those race conditions. When you use this method to enqueue data, the runtime does not attempt to deliver the data to the queue until the workflow has become idle (when all activities that are currently executing are waiting for input of some sort). This can provide a higher level of certainty to the host application that the workflow is ready for the data being submitted as it is more likely that the activity creating the queue has begun to execute and has in fact created that queue.

Even using EnqueueItemOnIdle, it is still possible that the workflow is waiting for another activity before executing the one you are interested in. For example, consider Figure 5, which shows a delay activity that occurs before a ReceiveData activity (available in the code download for this article). If data is enqueued before the delay, the idle event will occur on the delay, but the queue is not available yet. In this scenario, you might need to use an alternate method for enqueuing data. One possibility is to attempt to enqueue, but if there is a failure, wait for a short time and then attempt to enqueue the data again.

Figure 5 Multiple Idle Points
Figure 5 Multiple Idle Points

The next big question for a host is how to get notified as to whether an item sent to the queue actually made it to the workflow. You can use two different approaches here that are not mutually exclusive: exceptions and transactions. With the exception approach, you can catch a System.InvalidOperationException when calling either of the methods. If a queue by the name you are using is not found, for example, the exception will tell you just that. The Message property of the exception should provide some meaningful reason why the operation failed (meaningful to a developer, of course, not to your users). Exceptions should always be caught so that the host knows the data did not reach the workflow.

Using transactions allows you to get notified when the message has been successfully delivered to the queue and the workflow has committed the current transaction. This is where those two extra parameters to the EnqueueItem and EnqueueItemOnIdle come into play. The IPendingWork interface identifies a class that can be added to the workflow's work batch and participate in a transaction. The workItem parameter is state or data that the class implementing IPendingWork can use when committing or completing transactions. Figure 6 shows a simple class that implements the IPendingWork interface and can be used to get notified when the data has been delivered to the queue.

Figure 6 Getting Notified of Data Delivery (Close Figure 6)
public class EnqueuePendingWorkItem : IPendingWork
{
    private ManualResetEvent resetEvent;
    private bool result;

    public bool Success
    {
        get { return result; } set { result = value; }
    }

    public void WaitForCommit()
    {
        if (resetEvent != null) resetEvent.WaitOne();
        else throw new InvalidOperationException("Already completed");
    }

    public EnqueuePendingWorkItem()
    {
        resetEvent = new ManualResetEvent(false);
    }

    public void Commit(System.Transactions.Transaction transaction,
        System.Collections.ICollection items)
    {
       //option to do work here using the workflow's tx
       Console.WriteLine("Commit called on pending work item");
    }

    public void Complete(bool succeeded,
        System.Collections.ICollection items)
    {
        Success = succeeded;
        resetEvent.Set();
        resetEvent = null;
    }

    public bool MustCommit(System.Collections.ICollection items)
    {
        return true;
    }
}

The IPendingWork interface has three methods that must be implemented by inheritors: Commit, Completed, and MustCommit. The Commit method is where work can be done in the workflow's transaction. The Completed method is the callback to the class to indicate the outcome of the transaction. Finally, the MustCommit method allows the IPendingWork item to indicate if it must currently be committed. I will talk more about the impact of this last method shortly.

To use this class, I add it to my call to the EnqueueItem method and then use the WaitForCommit method to know that the transaction has completed so I can check the success or failure, like so:

EnqueuePendingWorkItem workItem = new EnqueuePendingWorkItem();
instance.EnqueueItemOnIdle(
    "ReceiveData:FirstInput", inputs, workItem, "");
workItem.WaitForCommit();

Because you can take actions within the same transaction as the workflow, you can take steps in your host that will be transactionally consistent with your workflow—especially when you use a persistence service. However, when you submit data to a queue using a transaction, it is important to understand when the transaction will commit and thus when you will know that your data was processed. The table in Figure 7 shows the interaction between using a persistence service, transaction scopes, and the MustCommit method on the IPendingWork item and when the transaction actually commits.

Figure 7 Transaction Committing with Enqueued Data (Close Figure 7)

TransactionScope | MustCommit | PersistenceService | Result
False | True | False | OnIdle
False | False | False | Hang
True | True/False | False | Error
True | True/False | True | EndOfTxScope

The key take away from this grid is that your work item should always return True from the MustCommit method in order to ensure that you don't get into a hang scenario. When you use a persistence service and a transactionscope, then you can force the transaction to commit sooner than when the workflow goes idle. For more information on transactions in workflows, see msdn.microsoft.com/msdnmag/issues/07/06/CuttingEdge.

Two-Way Communications

What you might have noticed in the discussion so far is that while you can call a service operation from your workflow and get a response value, there is no direct way to send data into a workflow and get a response back: enqueuing is a one-way operation. However, you often want to submit data to the workflow, let it process that data, and receive a response that the host application can use. For example, in an ASP.NET application, as part of processing a page, you could submit data to the workflow, wait for a response, and then use the data coming out of the workflow in your page or display logic.

In order to achieve this, you need a way to send an object to a custom activity in the workflow and a means to wait until the activity signals that it has updated the object. The simplest way to manage this is by submitting a message or data structure that provides input to the workflow and has properties for the information you need to get back out. This message should also have a way for your custom activity to signal to the host that it has updated the object with the results.

In the code download for this article, available from the MSDN® Magazine Web site, there is a RequestResponseData activity that shows how to use this type of pattern to submit data to the workflow and wait for a response. Figure 8 shows the TwoWayMessage class that is enqueued by the host, updated by the activity, and then used to signal to the host that the output is ready.

Figure 8 Request- Response Message Class (Close Figure 8)
[Serializable]
public class TwoWayMessage
{
    ManualResetEvent resetEvent;
    bool isResponse;

    public TwoWayMessage()
    {
        resetEvent = new ManualResetEvent(false);
        isResponse = false;
    }

    private Dictionary<string,string> input;
    public Dictionary<string,string> InputValues
    {
        get { return input; } set { input = value; }
    }

    private Dictionary<string,string> output;
    public Dictionary<string,string> OutputValues
    {
        get { return output; } set { output = value; }
    }

    public void SendResponse(Dictionary<string, string> returnValues)
    {
        OutputValues = returnValues;
        this.isResponse = true;
        resetEvent.Set();
    }

    public void WaitForResponse()
    {
        resetEvent.WaitOne();
        resetEvent = null;
    }
}

As you can see, this class allows for input and output values and uses a ManualResetEvent to allow the activity to signal when it has set the response. The caller/host uses the WaitForResponse method to block its thread until the output values are set, while the activity uses the SendResponse method to set the output values and signal the completion. The RequestResponseData activity in the sample code provides the activity implementation for this sample. It is a composite activity that receives data, executes all of its children in sequence, and then sends the response. Figure 9 shows this activity used in a workflow. The code activity contained within the activity represents the logic that sets the output values.

Figure 9 Request-Response
Figure 9 Request-Response

A Note on Local Communications

The observant reader may notice that I have not discussed what is frequently referred to as local communications in Windows WF. Local communications refers to a set of activities and a runtime service that abstract away much of what I have explained so far. The goal of these activities, along with the ExternalDataExchangeService, the runtime service they interact with, is to provide a simpler .NET development model on top of the communication architecture. As a developer, you use the CallExternalMethod and HandleExternalEvent activities to model the communication and .NET methods and events for communication to and from the host. In some cases, this may be the simplest option for you, but the goal of this article is to help you understand the core communication architecture, not a single abstraction that may prove limiting. There are several examples on the Web of using these activities and of the ExternalDataExchange service layered on the communication architecture discussed here.

Wrap-Up

The communication capabilities of Windows WF are critical to building real-world workflows that interact with systems and people. Understanding the communication architecture can help save you troubleshooting time and allow you to solve difficult problems that do not easily fit the common examples.

Send your questions and comments to mmnet30@microsoft.com.

Matt Milner is an independent software consultant specializing in Microsoft technologies including .NET, Web services, Windows WF, WCF, and BizTalk Server. As an instructor for Pluralsight, Matt teaches courses on Workflow, BizTalk Server, and WCF. Matt lives in Minnesota with his wife Kristen and his two sons.


Security Briefs: Exploring Claims-Based Identity

Security Briefs
Exploring Claims-Based Identity
Keith Brown

Code download available at: SecurityBriefs2007_09.exe (206KB)
Contents
Finding Common Ground
Traditional Representation of Client Identity
ClaimSets and Claims
Claims Transformation and IAuthorizationPolicy
Security Token Service
Trust and Federated Identity
Until Next Time

M
ost enterprise applications need some basic user security features. At a minimum, they need to authenticate their users, and many also need to authorize access to certain features so that only privileged users can get to them. Some apps must go further and audit what the user does. On Windows®, these features are built into the operating system and are usually quite easy to integrate into an application. By taking advantage of Windows integrated authentication, you don't have to invent your own authentication protocol or manage a user database. By using access control lists (ACLs), impersonation, and features such as groups, you can implement authorization with very little code. Indeed, this advice applies no matter which OS you are using. It's almost always a better idea to integrate closely with the security features in your OS rather than reinventing those features yourself.

But what happens when you want to extend reach to users who don't happen to have Windows accounts? What about users who aren't running Windows at all? More and more applications need this type of reach, which seems to fly in the face of traditional advice. This column will introduce you to the new identity model in the Microsoft® .NET Framework 3.0, which is designed to help address these and other problems.

Finding Common Ground

On Windows, the most common type of credential used to access an enterprise application is simply the user's domain account. An application that uses Windows integrated authentication receives a Kerberos ticket to represent a client. An application using SSL might instead receive an X.509 certificate for the client. A Kerberos ticket and an X.509 certificate are two very different beasts, and the code in the OS that parses, validates, and ultimately presents the data inside is very different. But if you take a step back and think about what they really represent, you'll see that these two credentials have a lot in common.

To make this discussion more concrete, and to help introduce what might be new terminology for some readers, let's imagine that Alice is a user who wants to access a purchasing service using her Windows domain account. Her domain controller authenticates her and creates a Kerberos ticket with a bunch of security identifiers (SIDs) in it. These SIDs represent Alice's user account as well as the domain groups in which she's a member, and they are embedded in the ticket along with a signature from the domain controller. In identity-speak, an issuer (the domain controller) has given a subject (Alice) a security token that she can use in order to prove her identity.

These same ideas apply when Alice uses a certificate instead. A certificate is just another type of security token. The issuer in this case is a certificate authority (CA) and the subject is Alice. Both the Kerberos ticket and the certificate are, in essence, signed statements by an issuer about a subject. These are just two different ways that a trusted authority can vouch for one of its subjects. Each signed statement can be thought of as a collection of claims. You should think of it this way: the domain controller is making claims about Alice's identity when it signs the list of SIDs in her ticket. Each SID becomes a claim. The certificate authority makes claims about Alice's identity when it signs her name and public key. The name and public key are examples of claims in the certificate.

If the terminology I'm using sounds a little academic, bear with me because these abstractions turn out to be incredibly useful in practice. In fact, the entire goal of this new identity model is to abstract identity in a way that reduces your dependency on specific types of credentials without compromising the security of your application. By coding to the identity model in the .NET Framework 3.0, you'll be able to process not only Kerberos tickets and certificates, but also Security Assertion Markup Language (SAML) tokens, which opens the door to some really interesting identity architectures, including federated identity.

Traditional Representation of Client Identity

That's enough abstract talk; let's see this stuff in action. I'll start with a simple Windows Communication Foundation (WCF) service that accepts either Windows credentials or X.509 certificates from clients. It exposes a single method that takes no arguments, and when called by a client, it prints out details about the client's identity. Figure 1 shows this method, called Hello. In this version, it uses the traditional IIdentity interface introduced in the .NET Framework 1.0 to inspect the client's identity.

Figure 1 Hello Method (Close Figure 1)
public void Hello() {
    ServiceSecurityContext sctx = ServiceSecurityContext.Current;
    if (null == sctx) {
        Console.WriteLine("Security context is null."+
                          "User must not be authenticated.");
        return;
    }
    IIdentity id = sctx.PrimaryIdentity;
    Console.WriteLine("Primary identity type: {0}", id.GetType().Name);
    Console.WriteLine("AuthenticationType: {0}", id.AuthenticationType);
    Console.WriteLine("IsAuthenticated: {0}", id.IsAuthenticated);
    Console.WriteLine("Name: {0}", id.Name);
}

The first step here is to grab the ServiceSecurityContext for the call, which is how WCF communicates details of the client's identity to a service. I'm careful to check for null, which would indicate an anonymous client. Then I grab the PrimaryIdentity property and dump out its contents. Here's what the output looks like when I configure the sample to use Windows credentials:

Primary identity type: WindowsIdentity
AuthenticationType: NTLM
IsAuthenticated: True
Name: GROMIT\Alice

The concrete type in this case is WindowsIdentity, which exposes a lot more information than IIdentity. By downcasting to WindowsIdentity, I'd be able to get a list of the groups of which Alice is a member, get her user SID and use that to look up her user record in Active Directory®, and I might even be able to impersonate her in some circumstances.

Now let's run this same example with a different WCF configuration. This time I'll have the client submit an X.509 certificate.

Primary identity type: X509Identity
AuthenticationType: X509
IsAuthenticated: True
Name: CN=SampleClient; 33BB8518E4B7...

The Name property in this case is a combination of the common name in the client certificate and its thumbprint. If you wanted further information, such as the issuer, expiration date, and so on, you might try downcasting to X509Identity, but you'd quickly find that this class is marked internal. But regardless of this limitation, even if you could downcast, you'd be dealing with an entirely different programming model from that in WindowsIdentity. This is one problem that the new identity model solves: it gives you a single programming model no matter what shape of client credential you get. In the next section, I've rewritten the Hello sample to be claims-aware. Read on and see what identity looks like through a claims-based lens.

ClaimSets and Claims

In Figure 2, you'll see a new version of Hello that's been rewritten using classes in System.IdentityModel, the new identity model introduced by the .NET Framework 3.0. This model steps back and treats identity using the abstractions of subject, issuer, and claim. Issuers make claims about their subjects and sign those claims to create security tokens. Each statement from any given issuer is modeled as a ClaimSet, which is a collection of Claim objects coupled with a property called Issuer, which is simply another ClaimSet.

Figure 2 Hello Using System.IdentityModel (Close Figure 2)
public void Hello() {
    ServiceSecurityContext sctx = ServiceSecurityContext.Current;
    if (null == sctx) {
        Console.WriteLine("Security context is null." +
                          "User must not be authenticated.");
        return;
    }
    AuthorizationContext actx = sctx.AuthorizationContext;

    Console.WriteLine();
    Console.WriteLine("HelloService.Hello sees {0} claim set(s):",
        actx.ClaimSets.Count);
    Console.WriteLine();

    foreach (ClaimSet cs in actx.ClaimSets) {
        Console.WriteLine("------------------------------");
        displayClaimSet(cs);
    }
}

const string CLAIM_FORMAT_STRING = "{0, -5} {1, -15} {2}";

private void displayClaimSet(ClaimSet cs) {
    Console.WriteLine();
    Console.WriteLine(CLAIM_FORMAT_STRING,
        "RIGHT", "CLAIM_TYPE", "RESOURCE");

    // display claims for subject of this claimset
    foreach (Claim c in cs) {
        displayClaim(c);
    }

    Console.WriteLine();
    Console.Write("ISSUED BY: ");

    // display claims for issuer of this claimset
    if (null == cs.Issuer) {
        Console.WriteLine("null");
    }
    else if (object.ReferenceEquals(cs, cs.Issuer)) {
        // self-asserted claims (issuer is the same as subject)
        Console.WriteLine("self");
    }
    else {
        displayClaimSet(cs.Issuer);
    }
}

// print a compact display of the supplied claim
private void displayClaim(Claim c) {
    string right;
    if (c.Right.Equals(Rights.Identity)) {
        right = "ID";
    }
    else if (c.Right.Equals(Rights.PossessProperty)) {
        right = "PP";
    }
    else right = c.Right;

    // make things a little more readable
    string claimType = c.ClaimType.Replace(
        "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/",
        ".../");
    string value = stringizeResource(c.Resource);

    Console.WriteLine(CLAIM_FORMAT_STRING, right, claimType, value);
}

private string stringizeResource(object resource) {
    // sometimes the claim's value will simply be a string
    string stringResource = resource as string;
    if (null != stringResource) return stringResource + " (string)";

    SecurityIdentifier sid = resource as SecurityIdentifier;
    if (null != sid) return getAccountName(sid) + " (SecurityIdentifier)";

    X500DistinguishedName dn = resource as X500DistinguishedName;
    if (null != dn) return dn.Name + " (X500DistinguishedName)";

    RSACryptoServiceProvider key = resource as RSACryptoServiceProvider;
    if (null != key) return string.Format(
        "{0} bit RSA public key (RSACryptoServiceProvider)",
        key.KeySize);

    // for anything else, just display the type name
    return string.Format("({0})", resource.GetType().Name);
}

private string getAccountName(SecurityIdentifier sid) {
    try {
        return sid.Translate(typeof(NTAccount)).ToString();
    }
    catch (IdentityNotMappedException) {
        return sid.Value;
    }
}

Before drilling straight into the code, I want you to see some output so you can get a feeling for what a ClaimSet represents. Have a look at Figure 3, which shows the output of the new Hello method when the client uses Windows credentials. The displayClaimSet helper method in Figure 2 pretty-prints the list of claims in the ClaimSet, and then recursively walks up the chain of issuers, printing their ClaimSets in turn. Not surprisingly, with Windows credentials, the ClaimSet mainly consists of SIDs. Looking a bit more closely at this output, we can see how System.IdentityModel represents claims.

Figure 3 Output of the New Hello Method (Close Figure 3)
HelloService.Hello sees 1 claim set(s):

------------------------------

RIGHT CLAIM_TYPE RESOURCE
ID    .../sid    GROMIT\Alice (SecurityIdentifier)
PP    .../sid    GROMIT\Alice (SecurityIdentifier)
PP    .../name   GROMIT\Alice (string)
PP    .../sid    GROMIT\None (SecurityIdentifier)
PP    .../sid    Everyone (SecurityIdentifier)
PP    .../sid    BUILTIN\Users (SecurityIdentifier)
PP    .../sid    NT AUTHORITY\INTERACTIVE (SecurityIdentifier)
PP    .../sid    NT AUTHORITY\Authenticated Users (SecurityIdentifier)
PP    .../sid    NT AUTHORITY\This Organization (SecurityIdentifier)
PP    .../sid    LOCAL (SecurityIdentifier)
PP    .../sid    NT AUTHORITY\NTLM Authentication (SecurityIdentifier)

ISSUED BY:
RIGHT CLAIM_TYPE RESOURCE
ID    .../sid    S-1-5 (SecurityIdentifier)
PP    .../sid    S-1-5 (SecurityIdentifier)

ISSUED BY: self

The Claim class has three key properties: ClaimType, Right, and Resource. In Figure 3, you can see two types of claims: a SID and a Name. The ClaimTypes for these are URIs, but I've abbreviated them in the displayClaim helper method by removing the first part of the URI, otherwise the output would be pretty tough to read.

While there is no centralized organization that defines all possible types of claims, we really don't need that. All we need is for the issuer to use claim types understood by the consumer (in this case, our Hello method). In practice, you'll see different claim types defined for use in different contexts. As an example, if you read the Information Card Profile, you'll see a dozen or so claim types defined for use with information cards.

Besides ClaimType, the Claim class also exposes a Right property that can often help you find a claim you need. Look once again at Figure 3 and note that of all the SIDs shown, only the first SID represents the subject's identity (Alice). The rest of them represent groups. The RIGHT column in the output shows the value of Claim.Right, which is technically a URI that is also abbreviated in the displayClaim routine. System.IdentityModel only defines two rights today: Identity and PossessProperty. You can see that WCF thinks that the user's SID in this example should be used as an identity claim. Of all the claims presented, it's the one you'd want to use to identify the user. All the other claims are PossessProperty claims.

Claim.Resource is the value of the claim. My claim dump code in Figure 2 has a helper method, stringizeResource, which pretty-prints these values along with their type names. You see, the Resource property is of type object, and can contain just about anything, depending on the type of the claim. For example, the SID claims you see in Figure 3 are represented by instances of the SecurityIdentifier class introduced in the .NET Framework 2.0.

In a nutshell, the Claim class tells you the type and value of the claim, and often gives you a hint as to whether the claim can be used to uniquely identify a subject or issuer. The ClaimSet is a collection of these Claim objects, coupled with a reference to the issuer's ClaimSet. In Figure 3, note that the issuer is represented with a SID of S-1-5. That SID represents the "NT Authority," which basically means the OS. Alice is using a local account in this example; otherwise the issuer would be identified by the SID of Alice's domain.

As you traverse up the chain of issuers, you'll eventually find one that is self-referential, that is, it points to its own ClaimSet. The displayClaimSet method in Figure 2 shows how you can check for this using Object.ReferenceEquals. This happens when you reach a root authority such as a domain controller or certificate authority. It's important to watch for this. Otherwise, if you are traversing up the issuer chain, you might just end up in an infinite loop when you reach the root issuer!

Now have a look at Figure 4, which shows the output of Hello when Alice sends a certificate. In this case, WCF uses Claim.Right to hint that the certificate's thumbprint can be used to identify the client. And since I'm cheating by using self-signed certificates for my sample code, the Issuer property points to the subject's claim set, indicating that it was self-issued. Note that different classes are used to represent the claims, but the overall structure is very similar to the previous example.

Figure 4 Hello Output when Alice Sends a Certificate (Close Figure 4)
HelloService.Hello sees 1 claim set(s):

------------------------------

RIGHT CLAIM_TYPE      RESOURCE
ID    .../thumbprint  (Byte[])
PP    .../thumbprint  (Byte[])
PP    .../x500distinguishedname CN=SampleClient (X500DistinguishedName)
PP    .../dns         SampleClient (string)
PP    .../name        SampleClient (string)
PP    .../rsa         2048 bit RSA public key (RSACryptoServiceProvider)

ISSUED BY: self

Finally, if you look at Figure 5, you'll see the output of Hello when the client uses an information card to authenticate with the service. In this case, the client's e-mail address and first name were sent as claims, and because a personal card was used, the issuer is identified with an RSA public key. If you want to know more about information cards and what these particular ClaimSets mean, read my two recent columns on Windows CardSpace™ in the October 2006 and May 2006 issues of MSDN® Magazine (msdn.microsoft.com/msdnmag/issues/06/10/SecurityBriefs and msdn.microsoft.com/msdnmag/issues/06/05/SecurityBriefs).

Figure 5 Hello Output from Information Card (Close Figure 5)
HelloService.Hello sees 2 claim sets:

------------------------------

RIGHT CLAIM_TYPE      RESOURCE
PP    .../hash        (Byte[])

ISSUED BY: self
------------------------------

RIGHT CLAIM_TYPE       RESOURCE
PP    .../emailaddress alice@contoso.com (string)
PP    .../givenname    Alice (string)

ISSUED BY:
RIGHT CLAIM_TYPE      RESOURCE
ID    .../rsa         2048 bit RSA public key (RSACryptoServiceProvider)
PP    .../rsa         2048 bit RSA public key (RSACryptoServiceProvider)

ISSUED BY:
RIGHT CLAIM_TYPE      RESOURCE

ISSUED BY: self

There are several advantages to building systems that use a claims-based lens to look at client identity. One obvious benefit is that you get a single programming model to inspect any kind of client identity. Just program against ClaimSet and you're all done, right? Well, in practice, if you're expecting a certificate, you're going to be looking for different types of claims than if you were expecting Windows credentials or a personal information card. In fact, there are classes that derive from ClaimSet, such as X509CertificateClaimSet and WindowsClaimSet, that can make things a little easier when you know what type of credential you are expecting. Having a single programming model is a good thing. But you'll discover the real power of claims once you realize that you can model just about any sort of authentication and authorization system using these concepts of subjects, issuers, and claims. And there's no better way to show this than through claims transformation.

Claims Transformation and IAuthorizationPolicy

Imagine that you need to build a WCF service that accepts lots of different types of credentials, and you want to use roles to authorize access to features. If the user is in the Manager role, she will have access to more features than if she is only in the Clerk role, for example. You can model roles using claims, and you can use claims transformation to deal with the multitude of different credential formats you might accept.

System.IdentityModel has an interface that allows you to perform arbitrary claims transformation. It's called IAuthorizationPolicy, and its one key method is called Evaluate, which takes an instance of the following class as an argument:

public abstract class EvaluationContext {
    // Methods
    protected EvaluationContext();
    public abstract void AddClaimSet(
        IAuthorizationPolicy policy, ClaimSet claimSet);
    public abstract void RecordExpirationTime(DateTime expirationTime);

    // Properties
    public abstract ReadOnlyCollection<ClaimSet> ClaimSets { get; }
    public abstract int Generation { get; }
    public abstract IDictionary<string, object> Properties { get; }
}

This context allows you to do a number of things, but what I care about here is that it allows you to examine the claim sets presented by a client, do some sort of mapping and then add your own claim set. Say, for example, the client submitted a certificate. Your authorization policy could map the subject name in the certificate onto a set of roles, and expose those roles as claims in a new ClaimSet that indicates your application as the issuer. If instead the user supplied a Windows credential, you could map her groups onto those same roles using a similar ClaimSet. Once this transformation occurs, your service simply needs to look for the ClaimSet issued locally, and then read the user's roles from there.

WCF has excellent support for this model. In your service's configuration file, you can wire up a ServiceAuthorization behavior that tells WCF to call your implementation of IAuthorizationPolicy before it calls your service's method (Hello, for example). This allows you to centralize the code that deals with the myriad of client credentials that your service accepts. Note that if you're planning to use role-based security, you can take this one step further and set the PrincipalPermissionMode attribute on the ServiceAuthorization behavior to Custom, and then supply an instance of IPrincipal that contains your roles via the evaluation context. This will allow you to use the PrincipalPermission attribute on your service's methods to grant access based on roles. Here's what this behavior looks like in config:

<serviceAuthorization principalPermissionMode="Custom">
  <authorizationPolicies>
    <add policyType="MyAuthorizationPolicy, MyAssembly"/>
  </authorizationPolicies>
</serviceAuthorization>

And here's how you communicate a custom principal to WCF using the evaluation context:

string[] roles = lookupRolesForUser(evaluationContext);
GenericIdentity id = new GenericIdentity(clientName);
evaluationContext.Properties["Principal"] =
    new GenericPrincipal(id, roles);

WCF will then set up Thread.CurrentPrincipal with the supplied principal before calling into the service's method. I only mention this custom principal trick because a lot of people like using PrincipalPermission and they will find this to be handy. But let's get back to claims again, because that's really where the power lies. So, how would you go about creating your own ClaimSet to represent a set of roles? While a full example can be found in the sample code available with this column on the MSDN Magazine Web site, here's an excerpt that shows how to do this:

string clientName = getClientNameFromClaims(EvaluationContext);
Claim c1 = new Claim(ClaimTypes.Name, clientName, Rights.Identity);
Claim c2 = new Claim("urn:role", "foo", Rights.PossessProperty);
Claim c2 = new Claim("urn:role", "bar", Rights.PossessProperty);
ClaimSet newClaimSet = new DefaultClaimSet(
    DefaultClaimSet.System, c1, c2, c3);

I used a well-known claim set called System to indicate that this is an internal claim set issued by my own code. The System claim set has nothing to do with the SYSTEM account in Windows. It's just a simple claim set that is easy to search for. Note that I add roles as PossessProperty claims, with a custom URI. You'll probably want to come up with something more elaborate than "urn:role" for the claim type URI, but since System.IdentityModel doesn't define a URI for a role claim, you can pick one that makes sense for your app.

Once you've created the new claim set, you can add it to the evaluation context via the AddClaimSet method. Now the code in your application simply needs to look for this ClaimSet you've issued, and it will get a list of roles for the user, regardless of whether she used a certificate or a Windows account to authenticate.

Security Token Service

Imagine for a moment that you've taken the time to build a great implementation of IAuthorizationPolicy that maps different credential types onto a simple set of role-based claims as I described above. You may have lots of different services that need this feature. One approach to reuse this code would be to drop the class into its own assembly and wire it into every service that needs it.

A more interesting approach would be to issue your own security tokens by using your authorization policy as the basis for implementing a Security Token Service (STS), which issues security tokens via a Web service protocol known as WS-Trust. An STS can accept a security token of one type and issue a token of another type, and can deal with whatever claims transforms are necessary, such as mapping a client certificate onto a SAML token that contains a set of roles that an application understands. In this scenario, the applications in your enterprise would look for a ClaimSet issued (and signed) by your own STS.

As I am writing this in June 2007, implementing an STS is possible, but not very easy. (Microsoft is working on making this easier.) Regardless of whether or not you happen to implement your own STS, building a claims-aware application will make it easier for you to leverage one should it ever become available to you.

Trust and Federated Identity

WCF and other communication frameworks use cryptography to ensure that the sender of a security token is indeed the subject and that the claims in the token were signed by the issuer named in the token. But all of this fancy plumbing doesn't have any idea how much you trust the issuer. If you don't trust him, you're not going to trust the claims he makes about his subjects! That's why the issuer is always identified when you receive a claim set, and it's the first thing you should look at when processing a claim set.

It's easy to write code that accepts tokens from a single trusted issuer. Just make sure the claim set you received was issued by the one authority you trust, and then you can use those claims to make security decisions. You've essentially delegated responsibility to the STS for doing the heavy lifting such as mapping users onto roles and dealing with different types of security tokens.

Now imagine you wanted to take this one step further. Instead of only accepting Windows credentials and X.509 certificates, what if your STS also accepted signed SAML tokens issued by an STS at a trusted partner? This leads to the realm of federated identity, which is very powerful. Instead of having to worry about managing user accounts for external users from partner companies, you can instead accept signed statements from those partners in the form of SAML tokens. I discuss the many benefits of automating these sorts of business-to-business trust relationships in my article on Active Directory Federation Services (msdn.microsoft.com/msdnmag/issues/06/11/SingleSignOn).

Federated identity ultimately boils down to claims transformation, if you think about it. The partner's STS makes the client's life easy by accepting as input whatever credential is most natural for her, given her operating system and platform. For example, if the client is running Windows, the STS could use Kerberos to automatically authenticate her and issue a SAML token. Another partner company might run a completely different OS that uses other strong authentication protocols. But the STS at that company would use those protocols to seamlessly authenticate the user and issue a SAML token. Meanwhile, the user enjoys the benefits of single-sign on, even when using applications like yours from federated partner companies.

The STS in your organization makes life easy for your applications. Your STS will ensure that only SAML tokens from trusted partners are accepted, and will then issue a new token for use with your application. Your application can live a long and happy life simply looking for ClaimSets issued by your company's STS. Figure 6 shows the flow of messages in this federated system. The client first goes to her company's account STS to obtain a SAML token. She sends this token to the resource STS to obtain a new SAML token that the service will consume.

Figure 6 Federated Authentication Message Flow
Figure 6 Federated Authentication Message Flow

Security token services are designed to bridge the gap between security or technology realms. By transforming one token type into another, and at the same time transforming the claims inside those tokens, they are a great way to centralize authentication and authorization policies.

I plan on drilling deeper into these federated identity models in the future. For now I hope you'll trust me (pun intended) when I say that federated identity solutions are becoming more and more common, and you'll be able to fit into this model if you're building claims-aware applications today. Expect further innovation in this space in the future.

Until Next Time

Building claims-aware applications will prepare you for the future of identity on the Windows platform. For example, you'll be able to accept not only traditional token formats such as Windows credentials or X.509 certificates, but you'll also be ready to accept information cards. You'll also be better prepared to implement federated identity when the time comes for that.

The first step to learning about claims is to spend some time exploring System.IdentityModel, which is the new claims-based programming model introduced in the .NET Framework 3.0. Use the sample code with this column to get started and see how you can apply the power of claims-based identity in your own applications today.

Send your questions and comments for Keith to briefs@microsoft.com.

Keith Brown is a co-founder of Pluralsight, a premier Microsoft .NET training provider. Keith is the author of Pluralsight's Applied .NET Security course as well as several books, including The .NET Developer's Guide to Windows Security, which is available both in print and on the Web. Learn more at www.pluralsight.com/keith


.NET Matters: Tales from the CryptoRandom

.NET Matters
Tales from the CryptoRandom
Stephen Toub and Shawn Farkas

Code download available at: NETMatters2007_09.exe (151KB)

Q I'm using the System.Random class to generate some random numbers in my application. A coworker reviewed my code and suggested that I use RNGCryptoServiceProvider instead. I'd like to follow her suggestion, but I'd also like not to have to modify all of my code that uses Random, and RNGCryptoServiceProvider looks nothing like Random in terms of the methods it exposes. Do you have any suggestions for making this easier?

A What you're looking for is something called an adapter. In design pattern terms, an adapter takes the functionality of one class and "adapts" it to fit the interface provided (or expected) by another. In this case, you want to adapt the functionality of RNGCryptoServiceProvider to the interface of Random. As luck would have it, Random is not sealed and its public methods are all virtual. This implies that the designers of the Microsoft® .NET Framework actually expected derived implementations to be created that provide their own random number generation algorithms while still maintaining the same interface. Thus, we can implement a type that derives from Random and overrides all of its public methods, providing new implementations that rely on RNGCryptoServiceProvider.

RNGCryptoServiceProvider, which derives from the abstract RandomNumberGenerator class, exposes two public methods: GetBytes and GetNonZeroBytes. The GetBytes method accepts a byte array and fills it with a cryptographically strong sequence of random values. GetNonZeroBytes does the exact same thing, except that it guarantees that none of the bytes are zero.

For our purposes, the former is the most useful, as using the latter actually decreases the possible values that can be generated, since it restricts each byte into the range [1,255] instead of [0,255]. In fact, GetNonZeroBytes should almost never be used, since one test of a truly random number is that each bit should have an equal probability of being either 0 or 1. That is not true if you can never have a sequence of 15 or more consecutive zero bits, which is the case if no byte can be 0 (you can still get up to 14 sequential zero bits from the two bytes 0x8001, but to get to 15, at least one byte must be 0). If we're using GetNonZeroBytes and keep a counter of consecutive 0 bits, we know for certain that once we've seen 14 zero bits, the next bit must be 1. Since that bit had a 100 percent chance of being 1 and a 0 percent chance of being 0, it automatically makes the sequence more predictable and thus not truly random. One of the few cases for which you actually need GetNonZeroBytes is when generating PKCS #1 padding for RSA encryption.

In any event, given this cryptographically strong random number generator, our task is to create a class that looks like Random but that implements Random's methods using the GetBytes method of an underlying RNGCryptoServiceProvider. Our implementation of this is shown in Figure 1.

Figure 1 A More "Random" Random (Close Figure 1)
public class CryptoRandom : Random
{
    private RNGCryptoServiceProvider _rng =
        new RNGCryptoServiceProvider();
    private byte[] _uint32Buffer = new byte[4];

    public CryptoRandom() { }
    public CryptoRandom(Int32 ignoredSeed) { }

    public override Int32 Next()
    {
        _rng.GetBytes(_uint32Buffer);
        return BitConverter.ToInt32(_uint32Buffer, 0) & 0x7FFFFFFF;
    }

    public override Int32 Next(Int32 maxValue)
    {
        if (maxValue < 0)
            throw new ArgumentOutOfRangeException("maxValue");
        return Next(0, maxValue);
    }

    public override Int32 Next(Int32 minValue, Int32 maxValue)
    {
        if (minValue > maxValue)
            throw new ArgumentOutOfRangeException("minValue");
        if (minValue == maxValue) return minValue;
        Int64 diff = maxValue - minValue;
        while (true)
        {
            _rng.GetBytes(_uint32Buffer);
            UInt32 rand = BitConverter.ToUInt32(_uint32Buffer, 0);

            Int64 max = (1 + (Int64)UInt32.MaxValue);
            Int64 remainder = max % diff;
            if (rand < max - remainder)
            {
                return (Int32)(minValue + (rand % diff));
            }
        }
    }

    public override double NextDouble()
    {
        _rng.GetBytes(_uint32Buffer);
        UInt32 rand = BitConverter.ToUInt32(_uint32Buffer, 0);
        return rand / (1.0 + UInt32.MaxValue);
    }

    public override void NextBytes(byte[] buffer)
    {
        if (buffer == null) throw new ArgumentNullException("buffer");
        _rng.GetBytes(buffer);
    }
}

Our CryptoRandom class derives from Random, which provides two constructors: one that's parameterless and one that accepts an Int32 "seed" value. The former simply delegates to the latter with Environment.TickCount as the seed value, and the latter uses the seed value to calculate a starting value for the pseudo-random number sequence. Note the use of "sequence" here. Provided with the same seed value, Random always produces the same sequence of "random" values. This is great for debuggability (as it allows you to specify a debugging seed value that will guarantee you reproducible "random" results), but it's lousy for randomness.

There are only 2^32 possible Int32 values you can use as a seed, which means Random can only produce at most 2^32 unique sequences (and in reality, based on its current implementation, it's actually significantly fewer than that). More important, the lack of cryptographically strong sequences means that an attacker can analyze the history of numbers generated in a sequence to anticipate future "random" values.

RNGCryptoServiceProvider does not accept a starting seed value from the developer, though internally a seed is still used (for information on what contributes to the seed, see the documentation for the CryptGenRandom function at msdn2.microsoft.com/aa379942). In fact, while RNGCryptoServiceProvider provides several constructors, including one that accepts a string and one that accepts a byte array, the only one that actually does anything with the parameter is the one that accepts cryptographic service provider (CSP) parameters, and we'll ignore that one for our purposes. To maintain the exact same API as Random, our CryptoRandom class exposes both constructors that Random does, but the seed is ignored (nothing about the seed is passed along to the underlying RNGCryptoServiceProvider). By exposing the same API, a find and replace substitution in your existing code becomes very easy.

Now, the whole point of CryptoRandom is to adapt the RNGCryptoServiceProvider class to the interface exposed by Random, so CryptoRandom contains a private RNGCryptoServiceProvider member; the instance stored here is used to serve all requests for random values. In addition, CryptoRandom contains another private field, a byte array, that's used with the GetBytes method of RNGCryptoServiceProvider. Our implementation only ever requires one size byte array for all calls to GetBytes, and as the Random class is not thread-safe, CryptoRandom does not need to be either. As such, it makes sense to create this byte buffer once and reuse it on every call. (Note that, as currently implemented in the .NET Framework 2.0, the parameterless constructor of RNGCryptoServiceProvider creates thread-safe instances. As such, we could have created our private member to instead have been a private static member, and in doing so not have to create a new RNGCryptoServiceProvider instance for each instance of CryptoRandom. However, this thread-safety is not currently documented nor is it in any way baked into the class's contract or interface. Given that, we haven't relied on it for our implementation.)

With the structure taken care of, we can implement the public methods, of which there are five. The easiest to implement is NextBytes, which accepts a byte array and fills it with random values. This is the same signature and behavior as RNGCryptoServiceProvider's GetBytes method, which makes NextBytes painless to implement:

_rng.GetBytes(buffer);

The next method to implement is NextDouble, which returns a random real value in the range [0.0, 1.0). One standard solution for generating such a value is to generate a non-negative integer and then divide that value by one more than the maximum possible integral value. This translates directly into an implementation:

_rng.GetBytes(_uint32Buffer);
UInt32 rand = BitConverter.ToUInt32(_uint32Buffer, 0);
return rand / (1.0 + UInt32.MaxValue);

This isn't a perfect solution, however. In order to generate a random double, we have to define what we mean by a random double in the first place. Do we want it so that each bit pattern has equal probability of being generated, or do we want it such that if we were to generate many random doubles they would be evenly spread over the given interval? Given the nature of the real number implementation in the .NET Framework, those are not the same thing. Most likely we mean the second definition, which means that we don't want to be generating completely random bits (which would skew the overall range of numbers toward the lower end of the range). Instead, we do want to use the division method shown in the previous code snippet, which will not cause all bit patterns to appear with equal probability, but will cause us to generate numbers that are evenly spread out across the desired range. Somewhat paradoxically, this approach will cause us to start failing tests for randomness based on the first definition mentioned earlier, but it's very unlikely that consumers of the NextDouble method really want each valid bit pattern to appear with equal probability.

Now, the only randomness in the input to the NextDouble generation equation is the numerator. The denominator actually ends up costing us bits of randomness because of the division operation. What we really want is to do the division and then as we normalize the result, shift in random bits rather than zeros. However, without implementing our own floating point library, that's not doable, and writing our own floating point implementation is well beyond the scope of the column (and most likely your needs). Instead, we can attempt to optimize what we have, and that means getting the most random bits involved into the equation. Since the numerator is the only place that we can introduce randomness, and since a Double is 64 bits long, rather than generating a UInt32 and dividing by UInt32.MaxValue+1, we could try generating a random UInt64 and dividing by UInt64.Max + 1, thereby doubling the amount of random bits in the input.

This, unfortunately, leads to another problem. Double-precision numbers store an approximation of a real number; System.Double, which complies with the IEEE 754 standard for binary floating-point arithmetic, provides at most 17 decimal digits of precision. Unfortunately, UInt64.MaxValue (18446744073709551615) requires 20 decimal digits of precision. As a result, under .NET floating-point arithmetic, UInt64.MaxValue is equal to UInt64.MaxValue + 1, and, thus, substituting UInt64 for UInt32 in our NextDouble equation will change our range from [0.0, 1.0) to [0.0, 1.0], which violates the design of System.Random's implementation. As a result, we've chosen to keep the implementation shown in Figure 1, but please keep in mind there may be better (and probably more complicated) alternatives out there if any of this bothers you.

Next, we need to implement the parameterless overload of Next, which is used to retrieve a random non-negative Int32 value. We can do this by generating four random bytes, stripping off the high-order bit, and converting the remaining bits to an Int32 with the System.BitConverter class:

_rng.GetBytes(_uint32Buffer);
return BitConverter.ToInt32(_uint32Buffer, 0) & 0x7FFFFFFF;

Generating four random bytes will provide us with random values in the range [Int32.MinValue, Int32.MaxValue], but to match the semantics of Next, we really need a value in the range [0, Int32.MaxValue]. Using only the bottom 31 bits of the 32 random bits provides that mapping by turning any negative value into a non-negative equivalent. Since there are an equal number of negative numbers and non-negative numbers, and since stripping off that one bit provides a one-to-one mapping between them, every value has the same chance of being returned.

Things start to break down when we move on to the overloads of Next that generate a random Int32 within a given interval. The second overload of Next that accepts just a maximum value is simply a subcase of the third overload, which accepts both a minimum and maximum, as the second overload has an implicit minimum of 0. Thus the second overload can just delegate to the third. (The parameterless overload of Next is also a subcase of the third, with both an implicit minimum and an implicit maximum, but it's best handled on its own for reasons that will become apparent later.)

There are two obvious ways to generate a random Int32 within a given interval, and one not so obvious. The first involves multiplying by a random double:

return (int)(minValue + (NextDouble() * (maxValue – minValue)));

The second starts with picking up a random UInt32 and doing the operation:

result = minValue + (randomUInt32 % (maxValue – minValue))

Both of these approaches have problems. The first, where you multiply by a random double, is flawed due to the condition we've already seen where the double that's generated isn't truly cryptographically random. The other method (taking a random integer mod the range) works perfectly when the range is an even power of two (assuming that you have perfectly random input bits that you're selecting from). This is because the UInt32 values generated from GetBytes have 2^32 possible values; to evenly distribute these among all of the values in the target range, the size of that target range must evenly divide the input range, and with a power of two such as 2^32, that will only happen if the size of the target range is also a power of two. When the size of the target range is not a power of two, the "random" results will start to weigh more heavily toward the lower values.

For an example that demonstrates this skewing, let's say we want a number in the range [21, 27]. (For this example, we'll be selecting 8 bits rather than 32 bits just because it's easier for demonstration, but the principle generalizes out to the 32 bits we'd actually be working with.) Following the equation shown earlier, we'll now take a random number between 0 and 255 mod 6 + 21 to get our value. If the random number is in the range [0, 251] this works out great as each of the numbers [0,5] has come up exactly 42 times. However, once we move past 251, we'll get 0 (252), 1 (253), 2 (245), and 3 (255) as values from our mod operation. That means that 0, 1, 2, and 3 have a 43/256 chance of being selected by the mod operation while 4 and 5 only have a 42/256 chance. Therefore, we've got a ~16.80 percent chance of getting 21, 22, 23, or 24 back, but only a ~16.41 percent chance of getting 25 or 26 back. Over the long term, if we were an attacker trying to guess outputs, we would be more successful by always choosing a number at the low end of the range.

In light of this information, we've chosen to use the third option in our implementation in Figure 1. The problem we just explained regarding the mod approach happens because some values in the target range have the potential to be slightly favored over other values. We can fix this by detecting when such favoritism would happen and, in those cases, just trying again. As we saw in the previous example, the favoritism happens when the randomly selected value satisfies the following condition:

RandomValue >= RandomRange - (RandomRange % TargetRange)

In our previous example, the RandomRange is 256 and the TargetRange is 6; therefore, favoritism happens when the RandomValue is >= 252. For small target ranges, the chances of this happening are very small. For huge target ranges (such as a target range of size 2^31+1), the chances of this happening are no worse than 50 percent. Thus, we can generate a random value and check to see if it falls into this favoritism range. If it does, we start over with a new random value and try again. If it doesn't fall into the range, then we just return the result of formula shown for the second approach earlier. While theoretically this algorithm could loop forever, the chances of that happening are infinitely small. In most cases, this will at most take only a few iterations. And even in the worst case, where we have a 50 percent chance of retrieving a bad random value, it shouldn't take too many iterations before we find a good value. For example, after 10 iterations, the chances of not finding a good value are less than 0.1 percent, and after 20 iterations, that drops to less than 0.00001 percent. If for some reason the performance implications of this approach don't meet your needs, you can always revert to one of the previously mentioned approaches.

We have one last observation to make: you may now notice that we can implement the parameterless Next overload in terms of the other overloads with no loss of randomness. The target range required for the parameterless Next is 2^31, which is a power of two and thus does evenly divide the random range of 2^32. As such, no looping would be required. In fact, our bit manipulation in our Next implementation (&'ing with 0x7FFFFFFF) produces the same result as if we had modded the random value by the target range, 2^31. However, the simplicity of the implementation for the parameterless Next compelled us to leave it as is, even though we could have simply delegated and removed the existing code for the method.

With that, CryptoRandom is complete and can be used in scenarios where you're currently using Random. You don't even need to replace declarations of the Random class in your code; since CryptoRandom derives from Random, you only need to replace instantiations of Random with CryptoRandom and everything else will fall into place.

These statements need a disclaimer, however. If Random was meant to be extended, as we claimed toward the beginning of this column, you may be wondering why the designers of RNGCryptoServiceProvider didn't simply do what we've done here and derive from that class instead of creating their own RandomNumberGenerator base class. As we've seen, the interface for RNGCryptoServiceProvider is different because we cannot implement the entire System.Random interface to return true cryptographic random numbers. Instead, we have to start with a set of cryptographically random bits, and massage them to fit the interface, losing some randomness along the way. You as a developer can then take RNGCryptoServiceProvider and the cryptographically random bits it provides and massage them in a way that best fits your needs. If the CryptoRandom class we've provided for you does that in an acceptable way, wonderful. If not, you should now at least understand the issues involved and the tradeoffs that need to be made when you implement your own.

One caveat: CryptoRandom brings with it a performance cost. While Random's implementation does not produce cryptographically strong random numbers, it is extremely fast when compared to RNGCryptoServiceProvider. In our non-scientific tests, Random was over 100 times faster than CryptoRandom. Additionally, if you rely on the reproducibility of the sequences generated by Random, you won't be happy with the non-reproducible sequences generated by CryptoRandom. But after all, that's the point, isn't it?

Send your questions and comments to netqa@microsoft.com.

Stephen Toub is a Senior Program Manager on the Parallel Computing Platform team at Microsoft. He is also a Contributing Editor for MSDN Magazine.

Shawn Farkas is the Software Development Engineer on the CLR team at Microsoft responsible for all of the cryptography classes in the .NET Framework.


{ End Bracket }: Turning the Pages with WPF

{ End Bracket }
Turning the Pages with WPF
Tim Sneath

The Great Library of Alexandria was founded in 300 B.C.E. with the grand objective of collecting the world's knowledge in one place; at its height, the library contained nearly 750,000 scrolls. In the modern world, the British Library contains one of the foremost collections; among its twenty million books and manuscripts are some of the rarest works in existence. It holds the Diamond Sutra, the oldest printed book; Mercator's first atlas of Europe; the Lindisfarne Gospels; Leonardo da Vinci's personal notebook; the Magna Carta; and the Codex Sinaiticus, one of the two earliest Christian Bibles. Such unique items must, of course, be treated with the utmost care. If they are on public display at all, they are well protected behind glass, and direct interaction is limited to a handful of individuals.

Happily, these works are now being digitized for the first time in order to reach a broad audience. Even better, the digitized versions are being turned into a rich interactive experience that adds curatorial content and brings the books to life. In collaboration with a UK-based software developer, the British Library developed a Windows®-based application called Turning the Pages that offers a virtual facsimile in three-dimensional space of a growing number of the library's most precious items (see www.bl.uk/onlinegallery/ttp/ttpbooks.html). Using this application, available to any Microsoft® .NET Framework 3.0-equipped computer at ttpdownload.bl.uk/browserapp.xbap, you can open a book on your desktop and turn each page almost as if you held the book in your hands.

Digitizing these works presented some interesting challenges. Due to the books' fragility and value, each page needed to be scanned using a specialized offset camera that could capture an image without any risk of damage to the item. A curator was then filmed physically turning the pages of each document to create a reference for the virtual model; as you might expect, a heavy vellum page from an eighteenth-century book does not move in the same way as a paper page from the seventeenth century. Such attention to detail is central to creating an experience that feels authentic.

(Click the image for a smaller view)

(Click the image for a larger view)

One feature in Turning the Pages that always generates a reaction is the specular materials used to bring out the gold leaf on the many illuminations that decorate the pages of works like the Sherborne Missal, a fifteenth-century prayer book. As the page is turned, suddenly the light will catch part of the illustration such that it takes on a brilliant radiance.

Turning the Pages is implemented using the 3D class libraries in Windows Presentation Foundation, which in turn are an abstraction of DirectX®. Each page is represented as a geometry mesh, onto which one or more image materials are applied. Images are stored using the HD Photo format for high quality at a given compression rate; for those books that require specularity, a second image is also used for each page that contains a monochrome mask so that only the areas of the page illuminated in gold leaf are reflective. Lastly, frame-based callbacks manually deform the geometry model to create the page-turning effect on a mouse click.

As an evangelist for client platform technologies, I love to see people building great user experiences. The Web has proved a powerful influence in software development: it democratizes the content creation process, it provides new ways for companies to monetize their software, and it provides a friction-free distribution model that allows for minor updates to be made without redistributing an entire application.

But to achieve those advantages, we have been prepared to sacrifice the ability to take full advantage of the tremendous latent power that is available in every personal computer. We need a set of tools more powerful than HTML and CSS so that we can build applications that aren't constrained by lowest common denominator limitations and so that developers and designers can collaborate to deliver user interfaces that are better representations of the environments they model. Applications like Turning the Pages, delivered over the Web but taking advantage of WPF for display rendering, are interesting as an example of how these two worlds are colliding. Over the next few years, I believe we will see huge transformations in the software we use as developers figure out how to offer richer Internet-based interactive applications.

Tim Sneath is the Group Manager for the Silverlight and WPF Technical Evangelism team at Microsoft. His mission is to see developers create stunning applications built on the Microsoft platform and to persuade his mother that computers aren't out to get her.