August 2008 August 2008
Data 2.0: Expose And Consume Data in A Web Services World
The goal of the ADO.NET Data Services Framework is to create a simple REST-based framework for exposing and consuming data-centric services easily. Elisa Flasko and Mike Flasko
BizTalk EDI: Build a Robust EDI Solution with BizTalk Server
We introduce you to the EDI functionality within BizTalk Server 2006 R2, illustrating schema creation, document mapping, EDI delivery and transmission, and exception handling. Mark Beckner
Silverlight: Create Animations with XAML and Expression Blend
In this excerpt from his upcoming book, Laurence Moroney explains the basics of Silverlight animation and the animation tools available in Expression Blend. Lawrence Moroney
Write On!: Create Web Apps You Can Draw On with Silverlight 2
We build a Silverlight 2.0 application using the InkPresenter to let users annotate a pre-defined collection of images, perform handwriting recognition, and save the annotations and recognized text into a server-side database. Julia Lerman
Wicked Code: Craft Custom Controls for Silverlight 2
If you're unfamiliar with Windows Presentation Foundation (WPF), building that first Silverlight custom control can be a daunting experience. This article walks through the process. Jeff Prosise
Columns
Editor's Note: More than Pretty Pictures: Complete freedom in creating the user experience can be a double-edged sword, but a little bit of artistry can transform an application's usability.Howard Dierking | Toolbox: Serializing objects, Scott Allen’s blog, Site Performance, and more: Serialize and Deserialize Fixed Length and Delimited Files, Scott Allen's blog, inspecting Web Pages, and more.Scott Mitchell | CLR Inside Out: Program Silverlight with the CoreCLR: The CoreCLR provides the perfect set of CLR classes and functionality for the Web.Andrew Pardoe
Basic Instincts: Increase LINQ Query Performance: Jared Parsons demonstrates a technique to achieve maximum performance from LINQ queries on large sets of data in order to create a responsive user interface.Jared Parsons | Cutting Edge: Programming AJAX with ASP.NET Partial Rendering: Dino Esposito compares the use of AJAX patterns and DOM manipulations to the use of the ASP.NET partial rendering engine.Dino Esposito | Patterns in Practice: Object Role Stereotypes: Object role stereotypes can help you better understand and clarify the responsibilities of the objects in your application.Jeremy Miller
Team System: Essential Power Tools.: Brian Randell introduces you to some crucial Power Tools for getting the most out of Team System.Brian Randell | Foundations: Workflow Tips and Tricks: Matt Milner addresses reader requests for information about safely sharing a persistence database across applications and using lesser-known parts of the framework such as ChannelManagerService.Matt Milner | Windows with C++: Asynchronous WinHTTP.: This month's column explains how to use Windows HTTP Services, or WinHTTP, the new, powerful API for implementing HTTP clients.Kenny Kerr
Concurrent Affairs: More AsyncEnumerator Features: Jeffrey Richter shows you some additional cool features of his AsyncEnumerator class.Jeffrey Richter | Going Places: What Can a Robot Teach You?: Learn about mobile device programming through WiMo, a Windows Mobile-powered robot.Mike Calligaro | { End Bracket }: U.S. Imagine Cup: With the annual Imagine Cup student competition, competitors were asked to imagine a world where technology enables a sustainable environment, just as Microsoft is doing in its own business practices.Mitch Irsfeld


Data 2.0
Expose And Consume Data in A Web Services World
Elisa Flasko and Mike Flasko
Code download available at: DataServices2008_08.exe (225 KB)
Browse the Code Online

This article is based on a prerelease version of ASP.NET 3.5 and the Microsoft AJAX Library. All information herein is subject to change.
This article discusses:
  • The meaning of data services
  • Exposing and consuming data
  • Describing data with the Entity Data Model
  • Data security
This article uses the following technologies:
ADO.NET Data Services, LINQ, Entity Data Model
Contents
Describing Data with the Entity Data Model
Relational Data
What about Non-Relational Data?
Uniform URI Format
Data Service Security
Client Side: Accessing the Service
.NET Client Library
﻿Silverlight Client Library
AJAX Client Library
Querying a Data Service
Command Mapping
Batching
Learning More
Remember back to the last Rich Internet Application (RIA) you built. How did you get your data? How did you separate that data from presentation and UI information sent to the browser? What if there was a simpler way to do this?
Separation of presentation and data is not a new idea, but with the growing popularity of RIA technologies such as AJAX and Silverlight™, it has become much more prevalent. These technologies are built on the idea of separating presentation and data to enable a more interactive and responsive application.
For example, Silverlight-based RIA applications pre-compile code that drives presentation and is deployed to the client via the Web server. Then, after reaching the Web browser, the code calls back to a Web server to retrieve the data to display in the user interface. Technologies such as these typically eliminate the option of a server-side rendering process that would mix data and presentation code.
In addition to separating presentation and data in order to drive richer, more interactive Web experiences, there is a trend on the Web to expose and consume standalone data, independent of any user interface. The proliferation of data-driven applications such as "mashups" indicates that the broad availability of interesting, readily consumable data is enabling new application scenarios.
Based on observation of these trends, the ADO.NET Data Services Framework began as a way to help developers looking to expose and consume data via services from their RIA applications. When exploring the space, two key ideas surfaced: building general-purpose client libraries and tools with the current approaches to data-centric services is a difficult concept in itself, and creating and maintaining those services requires a significant developer investment. In this article, we'll focus on what data services are and discuss a few of the key features. If you would like more insight into some of the ideas behind data services, see the post titled "Why ADO.NET Data Services?" on the team blog (go.microsoft.com/fwlink/?LinkId=120530).
In general, the goal of the ADO.NET Data Services Framework is to create a simple framework based on Representational State Transfer (REST) for exposing and consuming data-centric services. Such services expose data using a uniform interface to be consumed by Web clients across a corporate intranet or the Internet. The framework consists of a server library to expose data securely as a service and a set of client libraries built for a range of Microsoft applications and technologies (the Microsoft® .NET Framework, Silverlight, and so on) to consume services. Figure 1 illustrates the architecture.
Figure 1 The ADO.NET Data Services Framework Architecture

Describing Data with the Entity Data Model
Each ADO.NET Data Service is described in Entity Data Model (EDM) terms using the Conceptual Schema Definition Language (CSDL). The EDM uses two primary concepts, entities and associations (or relationships). Entities are instances of Entity Types (such as Customer or Employee) which are structured records with a key. An entity key is formed from a subset of properties of the Entity Type. The key (CustomerId and OrderId, respectively) uniquely identifies and allows updates to entity instances and also allows them to participate in relationships. Entities are grouped by EntitySets (Customers is a set of Customer instances). Associations form links between two or more Entity Types (such as Employee WorksFor Department).
Why was the EDM chosen as the data description language for ADO.NET Data Services? As it turns out, the EDM maps very well to the primary Web concepts of resources and links, making it an ideal candidate (entities for resources and associations for links).
The EDM was developed with the goal of becoming the core data model across a suite of developer and server technologies. With only one data model to maintain across numerous applications, application maintenance would be simplified. The EDM could be used to define a model not only for custom applications built on ADO.NET Data Services, but also as the input to reporting and visualization applications, intranet portal applications or workflow applications. ADO.NET Data Services is the second Microsoft technology to build on the EDM concept (the first, of course, was the ADO.NET Entity Framework). For further information on the EDM and the ADO.NET Entity Framework, see msdn.microsoft.com/data and "ADO.NET: ﻿Achieve Flexible Data Modeling with the Entity Framework" in the July 2008 issue of MSDN® Magazine at msdn.microsoft.com/magazine/700331.

Relational Data
By default, ADO.NET Data Services works closely with the ADO.NET Entity Framework to simplify the process of connecting to and exposing a data model over data stored in SQL Server® or other third-party databases (Oracle, DB2, MySQL, to name merely a few).
The Entity Framework raises the level of abstraction at which developers work with data, meaning that rather than coding against rows and columns, a higher-level conceptual model is defined (as an Entity Data Model) over the relational data and the application is then programmed in terms of this model. The application need only understand data in the shapes that make sense for the application and those shapes are expressed using a rich vocabulary that covers concepts such as inheritance, complex types, and explicit relationships.
In general, ADO.NET Data Services works by converting a request to perform an operation over a resource (such as an HTTP request operation over a URI) to the equivalent operation on the data model that the resource represents. In this case, since the data model is backed by a relational database, URIs are converted to Entity Framework Object Services method calls.
An existing data model, created using the Entity Framework, can be exposed with only a few steps in a Visual Studio® 2008 SP1 ASP.NET Web Application project. (Other project types and hosting mechanisms are also supported, such as the Windows® Communication Foundation, or WCF, WebServiceHost.) An ADO.NET Entity Data Model is first created or imported into an ASP.NET Web Application. With the model available, we can then add a new ADO.NET Data Service. The Add New Item wizard will generate a basic service and make it visible in the Solution Explorer. We can then hook up the service to the model by changing the class definition, as shown here:
using NorthwindModel;

public class NorthwindService : DataService <NorthwindEntities>
{
     public static void InitializeService(IDataServiceConfiguration config)
     {

     }

}
Additionally, since the service is initially locked down by default, we must do some work to set service-wide security policy to enable the service to be used. The following code sets security at the EntitySet level when the service is initialized:
public static void InitializeService(IDataServiceConfiguration config)
{
    config.SetEntitySetAccessRule("*", EntitySetRights.All);
}
As you can see, this is a very simple example of setting security for an ADO.NET Data Service. Other security methods and considerations will be discussed later.
At this point we have a working ADO.NET Data Service that we can build and run.
A simple way to test your service without a client app is to just point a Web browser at the entry point of the service, for example <host>/<vdir>/service.svc. The entry point of the service will return the response as XML containing the list of EntitySets exposed by the data service. To view Atom (the default format returned by an ADO.NET Data Service) in Internet Explorer®, you must first ensure that the Feed Reading View in Internet Explorer is turned off. This can be done from the Content tab of Tools | Internet Options.

What about Non-Relational Data?
For all other data sources or to use additional database access technologies (such as LINQ to SQL), a mechanism is provided that enables any data source to be exposed as an ADO.NET Data Service using a plug-in model.
In this case, URIs are converted into LINQ queries. Data Services enables this approach by mapping objects that implement the IQueryable interface to EntitySets. This means that ADO.NET Data Services is able to expose any data source that has an IQueryable provider written for it. In order to enable this, ADO.NET Data Services defines a mapping between CLR objects and artifacts of the EDM-based data model.
Shortly, we will walk through the creation of an ADO.NET Data Service based on a simple CLR object graph. The steps we'll illustrate produce a data service created over an in-memory collection of objects. In a typical production deployment, however, the IQueryable properties shown representing EntitySets would not expose in-memory data, but rather would translate IQueryable expression trees to data source-specific queries. For example, LINQ to Entities translates IQueryable expression trees into SQL statements.
When creating a data service against a non-relational store, the data service is again created within an ASP.NET Web Application project. Note that this time, however, we skip the step used to create the ADO.NET Entity Data Model and begin by adding a new ADO.NET Data Service. The Add New Item wizard will generate a basic service, visible in the Solution Explorer. To continue with this simple example, we must now create the in-memory data that we plan to expose via our service. We do this by creating three classes, two of which are User and Contact, where each User will have a set of Contacts. The third class is a MyDataService class that exposes two public properties (Contacts and Users) as IQueryable<Contact> and IQueryable<User>, as seen in Figure 2.
Figure 2 Creating In-Memory IQueryable Data Source
public class User
{
     public int ID { get; set; }
     public string Name { get; set; }
     public IList<Contact> Contacts { get; set; }
}

public class Contact
{
     public int ID { get; set; }
     public string Name { get; set; }
     public string Email { get; set; }
}

public class MyDataService
{
     static User[] _users;
     static Contact[] _contacts;

     static MyDataService()
     {
        _users = new User[]{ new User{ID=1, Name="Mike"},
                 new User{ID=2, Name="Elisa"} };

         _contacts = new Contact[]{ new Contact{ID=1, Name="Joe",
                              Email="Joe@contoso.com"},
                  new Contact{ID=2, Name="Bob", Email="Bob@contoso.com"},
                  new Contact{ID=3, Name="Sam", Email="Sam@contoso.com"},
                  new Contact{ID=4, Name="Carl", Email="Carl@contoso.com"},
                  new Contact{ID=5, Name="Abby", Email="Abby@contoso.com"},
                  new Contact{ID=6, Name="Annie", Email="Annie@contoso.com"},
                  };

         _users[0].Contacts = new List<Contact>();
         _users[0].Contacts.Add(_contacts[0]);
         _users[0].Contacts.Add(_contacts[1]);
         _users[0].Contacts.Add(_contacts[4]);

         _users[1].Contacts = new List<Contact>();
         _users[1].Contacts.Add(_contacts[2]);
         _users[1].Contacts.Add(_contacts[3]);
         _users[1].Contacts.Add(_contacts[5]);
    }

    public IQueryable<User> Users
    {
         get { return _users.AsQueryable<User>();}
    }

    public IQueryable<Contact> Contacts
    {
         get { return _contacts.AsQueryable<Contact>(); }
    }
}
Now that we have determined a data source, we return to familiar territory, changing the class definition to point to our data source, as seen here:
public class contacts :
  DataService<ADONETDataServiceNonRelSample.MyDataService>
{
    public static void InitializeService(IDataServiceConfiguration config)
    {
          config.SetEntitySetAccessRule("*", EntitySetRights.All);
    }
}
Again, we must do some initial work to set security policy, since the initial service is locked down by default. The code shows the same security policy set at the EntitySet level, as we saw earlier. At this point, we again have a working ADO.NET Data Service that we can build and run. Testing the service can be accomplished by simply using Internet Explorer to browse to the service endpoint.

Uniform URI Format
The uniform Uniform Resource Identifier (URI) is one of the primary concepts that make up ADO.NET Data Services, which defines a relatively simple, yet expressive, URI format allowing applications to query for sets of entities or single entities, as well as traverse the relationships that exist between those entities. The structure of this URI enables agents and controls to understand easily how to navigate the data presented by a service.
When we began testing our initial services, we saw the basic URI format pointing to the data service itself. Building on this basic URI, we are able to query the service using the following format, like so:
http://<host>/<vdir>/<service.svc>/<EntitySet>[(<Key>)
[/<NavigationProperty>[(<Key>)/...]]]
For example, continuing with our earlier Northwind example, if we add /Customers to the end of the data service URI, <host>/<vdir>/NorthwindService.svc/Customers, we return all of the customers in the Customers Entity Set, in this case all of the Customers in the Northwind database. If, on the other hand, we want to return a single customer entity from the Northwind service, we could use the entity's key value, since Customer is a single-key entity. By requesting the URI <host>/<vdir>/NorthwindService.svc/Customers('ALFKI'), we return the single Customer with the key "ALFKI." The navigation property is appended to a single entity query URI in a similar manner. By appending /Customers('ALFKI')/Orders, we navigate the relationship in our model between Customers and Orders, returning all of the Orders associated with the Customer identified by the key "ALFKI."
The URI format described above allows for basic query and traversal of the entities in our store, but does not account for the need to further control the output of queries or place constraints on them. To accomplish those goals, we'll use a set of optional query string parameters supported by ADO.NET Data Services that include $filter, $expand, $orderby, $skip and $top. Query string parameters are appended to the URI following the ? character.
For example, to query all customers in London, we would append the service URI with /Customers?$filter=City eq 'London'. To query a specific customer (with key 'ALFKI') and retrieve the related sales orders, we would append the service URI with /Customers('ALFKI')?$expand=Orders. To sort results by City in ascending (asc) order, we would append the service URI with /Customers?$orderby=City asc (or desc for descending order).
For a complete list of the URI formats supported by ADO.NET Data Services, check out the Data Services documentation, accessible from go.microsoft.com/fwlink/?LinkId=120539.

Data Service Security
Security is always the first concern when discussing a Web-based technology, especially one that provides access to and manipulation of critical data. Therefore, security has been a primary consideration throughout the development cycle of data services.
To provide authentication, ADO.NET Data Services leverages much of the existing ASP.NET and WCF authentication infrastructure to allow for an integrated authentication experience across applications and through the service. If you have an ASP.NET site that uses authentication (either one of the built-in authentication providers or a custom provider that sets the HTTP context principal appropriately), ADO.NET Data Services can leverage your chosen mechanism to establish the current identity (principal) for any given request. Similarly, if your data service is hosted outside ASP.NET (WCF or via a custom host) the host may also choose to use any authentication mechanism so long as it provides APIs for a data service author to access the principal of the request.
By default, any new ADO.NET Data Service is fully locked down, with all entities, Service Operations, and metadata being inaccessible with no read or write access. That said, there are a number of mechanisms for controlling authorization policy and performing per-request validations against your data service.
One of the first steps that a data service developer will take is to open up access to resources exposed by the data service. In our examples, we have shown a simple case of setting read/write access policy for the entire service. This is done for each service using the InitializeService method. The policies set were EntitySet policies allowing or restricting access to the Entity Sets in the model and setting which Entity Sets and related associations are available from the metadata endpoint of the service. Although both policies shown in those examples opened access to the entire model for reading and writing, we could have been much more specific, setting different policy for different EntitySets and including composite policies when necessary, as shown in Figure 3.
Figure 3 Setting Service-Wide Access Policy
public class MyService : DataService<NorthwindEntities>
{
   public static void InitializeService(
      IDataServiceConfiguration config)
   {
       // '/Customers' entities are enabled for all read and write
       // operations
       config.SetEntitySetAccessRule("Customers",
          EntitySetRights.All);

       //  URI '/Orders' is disabled, but '/Orders(1)' is enabled for read
       //  only
       config.SetEntitySetAccessRule("Orders",
           EntitySetRights.ReadSingle);

       //  Can insert and update, but not delete Products
       config.SetEntitySetAccessRule("Products",
           EntitySetRights.WriteInsert |
           EntitySetRights.WriteUpdate);
   }
}
In many scenarios, data services will also be required to run validation logic when entities enter the data service (for inserts, updates, or deletes) and restrict access to entities on a per-request basis. For these scenarios, you can make use of interceptors, enabling a data service developer to plug in custom validation or access policy logic into the request/response processing pipeline of a data service. For example, if we want to enable customers to retrieve only their orders and not orders placed by other customers, we would need to implement a query interceptor, as seen in Figure 4. The query interceptor returns a predicate to be appended to the query that is then pushed to the data store, eliminating the need for additional trips to the data store to retrieve access control information.
Figure 4 Query Interceptor Method
public class nw : DataService<NorthwindModel.NorthwindEntities>
{
     public static void InitializeService(IDataServiceConfiguration config)
     {
         config.SetEntitySetAccessRule("Orders", EntitySetRights.All);
     }

     [QueryInterceptor("Orders")]
     public Expression<Func<Orders,bool>> OnQueryOrders()
     {
         return o => o.Customer.ContactName ==
             HttpContext.Current.User.Identity.Name
     }
}
Similarly, interceptors can also be added for inserts, updates, and deletes as change interceptors. Change interceptors have no return value and take two arguments, an object of the type contained in the EntitySet and an UpdateOperations enumeration that defines the action (update, insert, or delete) being requested on the resource. The interceptor may alter the object passed to it or set the reference to an entirely different instance. If the method throws an exception, the operation is aborted, no change is made in the database, and an error is returned to the client agent.

Client Side: Accessing the Service
In general, we have treated HTTP as the API for data services and thus have spent a lot of time ensuring that it is simple to use directly at the HTTP level. This means that not only is it easy to interpret service payloads, but that any platform with an HTTP stack can easily use data services. That said, if you are using a Microsoft platform to access data services, client-side development is further simplified using the ADO.NET Data Services Client Libraries. This presents a more natural programming model for applications written using the .NET Framework, Silverlight, and ASP.NET AJAX to target data services, manipulate the results in terms of objects, and handle aspects such as association traversal. The client libraries abstract away the details of HTTP requests and responses and also ensure a more uniform experience across each of the client stacks.
The .NET Framework and Silverlight client libraries consist of two primary types, the DataServiceContext class and the DataServiceQuery class. DataServiceContext represents the runtime context for a given data service. Data services themselves are stateless; however, the context in which the developer interacts is not, and state on the client is maintained between interactions in order to support features such as identity resolution and optimistic concurrency. The DataServiceQuery object represents a specific query against the store defined using the URI syntax. To execute a query and obtain the results in the form of .NET objects, you simply enumerate over the query object. You can do so using the standard "foreach" construct in C# or "For Each" in Visual Basic®, for example.
In order to use the entities defined in a data service as .NET objects on the client, corresponding classes need to be defined for the client application. There are three primary methods for accomplishing this. One option is to define them manually. Figure 5 shows a simple hand-written definition for the Region class and a short piece of code that executes a query against regions and prints their IDs and descriptions in the output.
Figure 5 Access an ADO.NET Data Service from a .NET App
namespace TestApplication
{
  public class Region
  {
     public int RegionID { get; set; }
     public string RegionDescription { get; set; }
  }

  class Program
  {
     static void Main(string[] args)
     {
        DataServiceContext ctx = new
           DataServiceContext("http://localhost:25115/NorthwindService.svc");

        DataServiceQuery<Region> regions =
            ctx.CreateQuery<Region>("/Region?$orderby=RegionID");

         foreach (Region r in regions)
         {
            Console.WriteLine(r.RegionID + ", " + r.RegionDescription);
         }
      }
   }
}
A more common approach is to use code generated by Visual Studio. While writing classes manually works well when the service has only a small number of types, as the data service schema grows more complex, the number and size of the classes that must be manually created and maintained also grows significantly.
The most common way to generate the necessary classes is by using the Add Service Reference wizard in Visual Studio. Similar to using Add Service Reference for a WCF service, simply right-click on the Project and choose Add Service Reference (see Figure 6).
Figure 6 Add Service Reference
In the Add Service Reference dialog, enter the URI for the entry point of the service as the Address. In our example, we would enter <host>/<vdir>/NorthwindService.svc. This will generate classes based on the data service definition and add them to the project.
An alternate option is to use the "datasvcutil.exe" command-line tool to generate the classes based on the data service definition. This tool shipped with the release of ADO.NET Data Services and can be found in the directory \Windows\Microsoft.Net\Framework\V3.5. The tool takes as an argument the URI for the data service's entry point as well as the name and location of the output file that is to be generated, like so:
"\Windows\Microsoft.Net\Framework\V3.5\datasvcutil.exe"
    /out:C:\NorthwindSample\northwind.cs /uri:"http://<host>/<vdir>/
    NorthwindService.svc"
The default output from the command-line tool is a C# file that contains a class for each of the entity types described in the data service (Visual Basic types can be generated using the /language:VB switch). Each generated class has members representing the primitive values and associations described in the service. This allows developers to navigate through a graph of associated entities using the object model directly.

.NET Client Library
The ADO.NET Data Services .NET client library presents a programming model that is familiar to developers who write apps using the .NET Framework and data services. Under the covers, the client library uses HTTP and the AtomPub format, so it works naturally over both corporate networks and Internet environments, requiring only simple HTTP-level connectivity to the data service, whether direct or indirect (for example, through proxies).
The client library can be used from any project type, including Windows Forms, Windows Presentation Foundation (WPF), and Web projects. In order to use the client library above, you will need to add a reference to the System.Data.Services.Client.dll assembly.

﻿Silverlight Client Library
The Silverlight Client doesn't ship as a part of ADO.NET Data Services, but rather ships as part of the Silverlight 2 SDK, allowing for a more fully integrated experience when developing Silverlight apps. One difference between the Silverlight client library and the .NET and AJAX client libraries is that Silverlight 2 does not support synchronous development. Therefore, development with the Silverlight client library must make use of the asynchronous APIs, following the common begin/end async pattern. This also means that some of the APIs that you might use in the other two client libraries haven't been enabled in the Silverlight client library.

AJAX Client Library
The ASP.NET AJAX library is currently available at codeplex.com/aspnet/Release/ProjectReleases.aspx?ReleaseId=13357. Similar to the .NET client library discussed earlier, the AJAX client library abstracts the details of HTTP so that the application developer can work directly with JavaScript objects rather than manually parsing and creating HTTP requests and responses. A number of Web pages are also available on the CodePlex site that explain how to use the data service library for AJAX applications (see codeplex.com).

Querying a Data Service
In Figure 5, the queries against the service were built by calling the DataServiceQuery.CreateQuery method and passing in a URI query. As an alternative, the library also allows you to formulate data service queries using LINQ. The client library maps the LINQ statement to a URI in the target data service and retrieves the specified resources as .NET objects. The following code shows how to retrieve all the customers in the city of London with the results ordered by company name:
var q = from c in ctx.Customers
        where c.City == "London"
        orderby c.CompanyName
        select c;

foreach (var cust in q)
{
    Console.WriteLine(cust.CompanyName);
}
When using LINQ to ADO.NET Data Services, the set of queries expressible in the LINQ syntax is broader than those enabled by the REST-based URI syntax of Data Services. If a query cannot be mapped to a valid URI in the target data service, an exception will be thrown.
One way to understand the types of queries that can and cannot be mapped to URIs is to think about hierarchical traversal. Any query that requires two or more pivots, (for example, joins or subqueries that use clauses such as Any), cannot currently be mapped to a URI. Queries with a single pivot and association traversals, however, generally work well.
So far we have queried simple entities. Associations between objects are also tracked and managed by the DataServiceContext and associated objects can be loaded eagerly or as needed, using the URL formats discussed in the Uniform URI Formats section. To load associated entities on an as-needed basis (delay loading), use the LoadProperty method on the DataServiceContext class, as shown in Figure 7.
Figure 7 Delay-Loading Related Entities using LoadProperty
// get a single category
DataServiceQuery<Categories> categories =
               context.CreateQuery<Categories>("/Categories(1)");

foreach (Categories c in categories)
{
     Console.WriteLine(c.CategoryName);

     context.LoadProperty(c, "Products");

     foreach (Products p in c.Products)
     {
          Console.WriteLine("\t" + p.ProductName);
     }
}
In some scenarios, we may also want to avoid the extra round-trip on the wire to fetch the entities, preferring to load them along with the original query. In this case, the expand option is specified on the URI. The client library recognizes that the result includes both top-level and associated entities and will materialize all of them as a single object graph. Figure 8 eagerly loads related products in a single round-trip to the data service.
Figure 8 Using Expand to Eagerly Load Related Entities
public IList<Products> GetProducts(string productName,
    Categories category)
{
     int categoryId = category.CategoryID;
     string uri = serviceUri + "/Categories(" + categoryId + ")/Products?";
     if (!String.IsNullOrEmpty(productName))
     {
         uri = uri + "$filter=ProductName eq '" + productName + "'&";
     }
     uri = uri + "$expand=Categories";
     Uri queryUri = new Uri(uri);
     try
     {
         IEnumerable<Products> products =
            context.Execute<Products>(queryUri);

         return products.ToList();
     }
     catch (Exception)
     {
         return null;
     }
}

Command Mapping
With queries through ADO.NET Data Services mapped through HTTP Get requests, how are Create, Update, and Delete requests executed? Each of the four CRUD operations (Create, Retrieve, Update, Delete) are mapped to a different HTTP verb with Retrieve mapped to GET, Create mapped to POST, Update mapped to PUT, and Delete mapped to DELETE. Just as with queries, the client library used (in this case, the .NET client library) will abstract away the details and use the appropriate HTTP verb, allowing the developer to use the provided APIs quite easily.
To create an instance of an entity in the data service, simply create the .NET object, populate it with the desired data, and then call AddObject on the DataServiceContext object, passing the new object and the name of the EntitySet that the object will be added to, as seen here:
public void AddProduct(Products product)
{
     context.AddObject("Products", product);
     context.AttachTo("Categories", product.Categories);
     context.SetLink(product, "Categories", product.Categories);
     DataServiceResponse r = context.SaveChanges();
}
It's also possible to add or change associations between .NET objects and have the client library reflect those changes as association creation/deletion operations. For example, the previous code creates a Product in the Northwind database and associates it with an existing Category. Categories and products participate in a one-to-many association; so, a given product has one specific category.
After an entity is created, the data service will return a fresh copy of it, including values that have been updated as a result of triggers in the database, autogenerated keys, and so on. The client library will then automatically update the .NET object with the new values.
To modify an existing entity instance, the object first needs to be fetched by the client and tracked by the DataServiceContext. The developer will first query for or attach the object to the context, make the desired changes to its properties, and then call the UpdateObject method. The UpdateObject method instructs the client library to send an update for that object:
public void UpdateProduct(Products product)
{
    Categories newCategory = product.Categories;
    context.AttachTo("Products", product);
    context.AttachTo("Categories", newCategory);
    context.UpdateObject(product);
    context.SetLink(product, "Categories", newCategory);

    context.SaveChanges();
}
To delete an entity instance, the object must again be held by the client and tracked by the DataServiceContext. Once tracked, we can make a simple call to the DeleteObject method on the context instance to mark the object for deletion:
public string DeleteProduct(Products product)
{
     context.AttachTo("Products", product);
     context.DeleteObject(product);
     context.SaveChanges();
}
In all of the above examples, the last method call made on the context has been a call to the SaveChanges method. Changes are tracked in the DataServiceContext instance but not sent to the server immediately. Once all required changes for a given activity are complete, a call to SaveChanges will submit the changes to the data service.

So far, each of the examples has resulted in a single HTTP and sever request for each client operation. This approach (single operation per HTTP request) works well in some scenarios, but it is often a better idea to batch a number of operations and send them to the data service in a single HTTP request. This will reduce the number of round-trips to the data service and enable a logical scope of atomicity for collections of operations. To support these requirements, the client supports sending groups of CUD (Create, Update, Delete) operations and groups of Query operations to the data service in a single HTTP request. The following code shows how two or more queries can be sent to the data service as a single batch:
var q = (DataServiceRequest)from o in context.SalesOrder
                            select o;

// send two queries in one batch request to the data service
DataServiceResponse r = service.ExecuteBatch(
    new DataServiceRequest<Customer>(new
      Uri("http://localhost:25115/NorthwindService.svc/Customers")),
      q);
Sending a group of CUD operations as a batch to the data service is accomplished using a simple call to the SaveChanges method and passing SaveChangesOptions.Batch as the only parameter. This parameter value instructs the client to batch all the pending change operations into a batch and send them to the data service as an atomic group. When sending change operations as a batch using SaveChangesOptions.Batch, either all changes will complete successfully or none of the changes will be applied.

Learning More
Although this article covers just enough to get you started with ADO.NET Data Services, there are still a number of topics that you may be interested in. For more information, go to the Data Services Team blog at blogs.msdn.com/astoriateam and on the MSDN Data Development center at msdn.microsoft.com/data under the topic heading ADO.NET Data Services.

Elisa Flasko is a Program Manager in the Data Programmability team at Microsoft, including the ADO.NET technologies, XML technologies, and SQL Server Connectivity technologies. She can be reached at blogs.msdn.com/elisaj.

Mike Flasko is a Program Manager in the SQL Data Programmability group at Microsoft. You can contact Mike through his blog at blogs.msdn.com/mflasko.


BizTalk EDI
Build a Robust EDI Solution with BizTalk Server
Mark Beckner

This article discusses:
  • Developing an EDI schema
  • Mapping EDI documents
  • Document delivery through a firewall
  • Handling failed documents
This article uses the following technologies:
BizTalk Server 2006 R2
Contents
Developing an EDI Schema
EDI Mapping
Trading Partner Configuration
Transporting EDI Documents
Document Delivery through a Firewall
Handling Failed Documents
EDI and SOA
Electronic Document Interchange (EDI) is a technology standard that has been around for decades. So mixing it with a modern service-oriented architecture (SOA) and the latest release of BizTalk® Server may seem an unlikely combination. Yet EDI encompasses the largest share of real-world business-to-business commerce—nearly 90 percent of the current market—and is growing rapidly year over year. As companies relying on EDI evolve their IT architectures, the capabilities of BizTalk Server 2006 R2 are proving to be a reliable, robust, extensible, supportable, and intuitive way to solve both SOA and EDI infrastructure needs.
Prior to the BizTalk Server 2006 R2 release, EDI support in BizTalk was limited. There were adapters and accelerators that supplied the basic infrastructure for EDI solution implementations, but there were limitations in their capabilities, such as in how documents were validated. With BizTalk Server 2006 R2, the EDI functionality was formalized. It now allows for extensive document validation and numerous methods for transporting documents, including all of the reporting features common to enterprise-level EDI implementations.
BizTalk Server now provides the same level of service that many value-added networks (VANs) provide, with the additional benefit of the underlying BizTalk components that have been essential to enterprise integration solutions and SOAs. These include the development of business workflows through orchestrations, access to a business rule engine, extensive document-tracking capabilities, state management, and other similar functions.
EDI implementations in BizTalk Server 2006 R2 begin with developing the schemas that relate to the documents being traded. Once the documents have been defined, trading partners are created as BizTalk parties and their specifications are configured to ensure the proper processing and routing of EDI documents. Next, the specifics around how documents will be delivered are implemented through a combination of party configurations and BizTalk adapters. When solutions are in place, document flow can be monitored in real time through the use of EDI reports. All of these capabilities ride on top of the BizTalk infrastructure and benefit from all the standard components such as the MessageBox, orchestrations, ports, and pipelines.
The intent of this article is to introduce you to the EDI functionality within BizTalk Server 2006 R2 and to show that this functionality can enable you to integrate your EDI processes more easily with the rest of your enterprise. I will walk through several critical aspects of working with the new BizTalk Server EDI components, illustrating the different aspects of schema creation, document mapping, EDI delivery and transmission, and exception handling.

Developing an EDI Schema
To understand EDI schema development, first look at the specifics of the document structure itself. An EDI document can best be described as a simple text file consisting of three parts: the Header, the Detail, and the Footer. The Header defines from where the document came and for whom it is intended, the document type, and some date information. The Detail contains all of the business information that gives meaning to the document. For example, in the case of an invoice, the Detail consists of such information as line items, descriptions of sold products, pricing, quantities, and totals. The Footer consists of summary information about the Detail rows, such as the total number of lines included within the document.
EDI documents are formatted in segments, with each row of data containing a number of named segments. The format and composition of these segments are dictated by standards such as X12 and Electronic Data Interchange For Administration, Commerce, and Transport (EDIFACT). For X12 documents, the segments ISA and GS are considered to be the Header, the segments GE and IEA correspond to the Footer, and all lines in between the Header and Footer are the Detail (see Figure 1).
Figure 1 An X12 EDI Document (810—Invoice)
For EDIFACT documents, all segments that begin with the characters UN correspond either to the Header (UNA, UNB,) or the Footer (UNT, UNZ), while all segments in between are the Details. Segments and rows are separated from one another with separator characters, and these characters can be different for different trading partners. In both document formats, data is generally separated by the asterisk (*) character, and rows are separated by line feeds, tildes (~), or any other mutually agreed upon character combination.
BizTalk Server 2006 R2 ships with thousands of predefined EDI schemas that function as starting points for all documents exchanged by trading partners. Generally these schemas are altered to reflect specific expected formats. Though EDI has document standards, the reality is that two trading partners who both exchange 810 Invoice documents may still have two different representations of the 810 and therefore require two different schemas. These schemas will be very closely related and may only differ in one or two segments. For example, one may truncate a street address at 50 characters while another allows for 100. But even this small difference requires that the default 810 XML Schema Definition (XSD) be modified and implemented separately for both parties.
The first step in schema development is to define the electronic documents that will be exchanged and to develop the corresponding schemas. Using the invoice example, you would begin by adding a copy of the default 810 Invoice schema to your BizTalk solution. Schema templates can be found in the \Program Files\Microsoft BizTalk Server 2006\XSD_Schema\EDI directory in the file MicrosoftEdiXSDTemplates.exe. Run the executable to extract the templates, then find the file named X12_00401_810.xsd and add it to a BizTalk solution in Visual Studio®. This XSD represents the superset of all data that can be part of an 810 Invoice.
Say you are going to define an invoice document exchange between Company A and Company X. The next step is to create an XSD representation of Company A's XML document; this will act as the source document when creating the EDI instance. In many cases it is unnecessary to modify the default schema instances. But in this case, assume that Company X requires N401 (city name) to have a maximum length of 10 characters instead of the default length of 30. To change the length, click the N401 node and look for the Maximum Length value in the property window. Enter the new value here. This will ensure that if a document tries to pass through the system with more than 10 characters, an EDI error will be thrown indicating an invalid document. During the mapping process, it will be necessary to truncate this field before mapping it into the EDI schema.

EDI Mapping
Suppose Company A has an XML representation of an invoice that needs to be mapped to the EDI standard before delivery. It also needs to map all invoice line items to the IT1 loop and place the total number mapped in the CTT02 node.
Prior to mapping, all of the schemas, whether specifically in EDI format or not, need to be defined and added to the solution. Company A has an XML version of the invoice data that needs to be mapped to 810 Invoice format prior to transmission. This XML data must have an associated BizTalk schema, and in this case, the schema looks like that shown in Figure 2.
Figure 2 Invoice Schema Excerpt
<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns:b="http://schemas.microsoft.com/BizTalk/2003" xmlns="http://schemas.MSDN.com/Common/810/v1" attributeFormDefault="unqualified" elementFormDefault="qualified" targetNamespace="http://schemas.MSDN.com/Common/810/v1" xmlns:xs="http://www.w3.org/2001/XMLSchema">
  <xs:element name="COMMON_810">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="TRANSACTION">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="HEADER">
                <xs:complexType>
                  <xs:sequence minOccurs="1" maxOccurs="1">
                    <xs:element name="GUID" type="xs:string" />
                    <xs:element name="DOCID" type="xs:string" />
                    <xs:element name="DESC" type="xs:string" />
                    <xs:element name="PARTNER" type="xs:string" />
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
              <xs:element name="ADDRESSES">
                <xs:complexType>
                  <xs:sequence minOccurs="0" maxOccurs="unbounded">
                    <xs:element maxOccurs="unbounded" name="ADDRESS">
                      <xs:complexType>
                        <xs:sequence>
                          <xs:element name="TYPE" type="xs:string" />
                          <xs:element name="STREET" type="xs:string" />
                          <xs:element name="CITY" type="xs:string" />
                          <xs:element name="STATE" type="xs:string" />
                          <xs:element name="ZIP" type="xs:string" />
                        </xs:sequence>
                      </xs:complexType>
                    </xs:element>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
              <xs:element name="ITEMS">
              ...
After the schemas have been defined, you will need to create the BizTalk map. In this case it will be a map that transforms the data from the XML version of Company A's invoice information into the standard EDI 810 Invoice instance. Mapping EDI documents is not unlike any other type of BizTalk mapping except that there are several complexities that are unique to EDI. In the case of the invoice, a total count of all line items in the IT1 nodes must be displayed in the CTT01 node. Let's look closer at this to demonstrate what is unique about this type of mapping.
Begin by looking at the IT1 repeating node, which represents line items on the invoice. As shown in Figure 3, there are two types of mapping occurring: simple source to destination (as in the case of PRICE to IT104) and complex (as in the case of TYPE, which determines whether a line item in the source is copied to the target).
Figure 3 Mapping Line Items
In the case of the complex mapping there is a combination of two scripting functoids and a single equal functoid. The first scripting functoid contains logic that determines whether the value in the TYPE field is something that should be mapped across. If the value is equal to a certain string, then it will be added to the IT1 loop.
The logic of interest, though, is in the second functoid, which is connected to the IT101 node. The code in this functoid is actually keeping track of the number of line items that have been mapped across—not the total in the source document, but the total in the target document. This total is used both to populate the value of IT101 and to increment a global variable that can be used eventually to populate the CTT01 node.
The code for the IT101 scripting functoid, which declares and increments a global parameter accessible throughout the map, is shown in Figure 4. A global integer is created, and it's incremented by 1 each time a line item that needs to be mapped is sent across. With each line item that needs to be mapped, this value is incremented, and the resulting value can then be populated in the IT101 node. Once all of the IT1 line items have been mapped, the map can move on to populating the CTT01 node with the total number of line items that occur in the EDI instance. This value is contained in the global integer and can be accessed in a separate scripting functoid specific to the CTT01 node using the code shown here:
public int intAccessTotal()
{
return globalCtr;
}
Figure 4 Functoid to Ignore Certain Line Items
// declare global variable, accessible throughout all other components
// within current map
int globalCtr;
// declare function to increment and return the current value of the
// global count
public int keepCount (bool mapMe)
{
// the value of mapMe is the boolean returned by the equals
// functoid (see map)
if(mapMe)
globalCtr++;
// return the total value to be populated in IT101
return globalCtr;
}

Trading Partner Configuration
Two trading partners must be set up in BizTalk Server, one as the sender and one as the receiver. Trading partners are created as parties within BizTalk and configured through the BizTalk Server Administration Console. Trading partner configuration consists of a number of settings that enable BizTalk to determine what documents belong to which partner. When EDI documents arrive, BizTalk matches documents to trading partners by comparing what is defined in the document Header (or what is in the Applicability Statement 2, or AS2, envelope) with what has been configured for the trading partner.
For example, assume that the following ISA segment showed up in the Header of a document:
ISA*00*          *00*          *01*
BASECOMP12     *ZZ*TRADPART1      *
070407*1555*U*00401*000000025*0*T*>~
The sixth segment (ISA06) has the value of BASECOMP12, while the eighth segment (ISA08) has the value TRADPART1. Remember that segments are delimited by asterisk (*) characters. Here, the third and fifth segments are empty.
Trading partners can be configured as interchange senders or receivers. In this case, BizTalk compares the segments against how parties have been configured and finds that these values correspond to the configuration of trading partner 1 set up as a receiver (see Figure 5). Since the document has been matched to a party, the rest of the document now can be validated against the schemas associated with that trading partner. Documents found to be invalid are processed in one way, while valid documents are sent on to be processed by other EDI components.
Figure 5 Trading Partner Configured

Transporting EDI Documents
Documents can be sent to trading partners via any protocol, including SMTP, File, FTP, HTTP, and many others. However, EDI standards support only VANs and AS2. VANs ensure that documents are valid, that they are routed to the appropriate recipients, and that there is a record of the transaction. AS2 is the technology that allows for the secure transmission of documents directly between trading partners using S/MIME over HTTP/HTTPS. There are robust features in BizTalk Server that enable either standard to be incorporated into a solution so that, whether trading partners want to use AS2 or the VAN, BizTalk can act as the single internal EDI solution.
While BizTalk Server 2006 R2 can eliminate the need for VANs, many trading partners continue to do their business over these networks. BizTalk facilitates the transmission of documents to and from VANs through the use of FTP and secure FTP.
Interacting with VANs via BizTalk may require the use of a third-party adapter, as many VANs require secure FTP. There are some limitations with the standard FTP adapter that may prevent its use in environments such as these. (Often the SYST FTP command is required by VANs, but the standard BizTalk adapter does not support this.) Regardless of the adapter used, however, the idea of connecting to the VAN is a simple process of interacting with an FTP server. EDI documents being shipped to trading partners via the VAN are uploaded via FTP, and EDI documents being retrieved from trading partners are downloaded via FTP. BizTalk is responsible only for the successful delivery and receipt of EDI documents, never for the actual delivery of the document to the trading partner—the VAN takes care of this. It is the duty of an administrator to check the status of documents on the VAN through the use of tools provided specifically for that purpose.
When looking at FTP and VAN communication with EDI in BizTalk, much of the value that the VAN provides, such as document validation, document tracking, and document delivery, is now available within BizTalk itself. In AS2 solutions, BizTalk is the "value-added network," providing all of the functionality without the costs associated with the VAN.
AS2 allows trading partners to interact directly with one another over secure HTTP. When an AS2 implementation is used in an EDI implementation, the port is exposed to the public via the standard HTTP adapter with AS2 properties set on the EDI party, or via a third-party BizTalk adapter. Either way, the core idea is the same: enable a secure transmission and validate the document. The secure transmission is handled through certificates, and the validation of the document is handled through AS2, party, and schema configuration in BizTalk.
An example of a party configured to allow for AS2 transmission over the standard BizTalk HTTP adapter is shown in Figure 6. In this case, AS2 properties are set on the party and act as an extension to the HTTP adapter. When documents arrive over HTTP, BizTalk first compares the information to the AS2 settings, then opens the EDI document. The EDI document is then routed just as if it had arrived through any other method. AS2 simply provides a secure way to transfer the data and any relevant envelope information.
Figure 6 Configuring Base AS2 Properties

Document Delivery through a Firewall
An important concept to illustrate is how documents can be delivered to BizTalk Server on networks where BizTalk is behind a firewall and not available to the public. When a SOAP or HTTP receive port is added to BizTalk, the port will operate on the local IIS server, and all posted data will need to be posted to that local Web server. For example, when using the BizTalk Web Services Publishing Wizard, the Web service can be created only on the local IIS server where BizTalk Server is installed.
In an environment where this IIS server cannot be accessed from outside of the network, some sort of solution must be put into place to allow communications to be routed (see Figure 7). There are several solutions, including placing BizTalk Server in the public portion of the network or creating a reverse proxy that routes requests through a firewall and allows BizTalk to reside in a private portion of the network.
Figure 7 Document Delivery through a Firewall
Communication through the firewall can be made programmatically as it is rarely acceptable to house BizTalk Server in a publicly accessible portion of a network—the security risks are simply too high.
There are several approaches that can be taken in the development of the Web service on the public IIS server in the public network. One is to create a gateway Web service. Another is to create a proxy Web service. The gateway Web service requires custom coding, just like any Microsoft®.NET Framework Web service, and complex transformations of interfaces to match those that BizTalk expects. The gateway accepts a request from the public and maps it to a request to the Web service behind the firewall. A proxy Web service, on the other hand, can be created by modifying what is produced using the BizTalk Web Services Publishing Wizard and placing the copy on the public Web server. Due to its simplicity, modifying a proxy Web service seems like the clear choice.
To create a proxy Web service, launch the BizTalk Web Services Publishing Wizard. Click on the appropriate options in the wizard and define where the Web service will be published (notice that the only option for where the virtual directory is created is on the local IIS server). When the wizard creates the Web service, it is actually creating a proxy that redirects incoming calls from IIS and places them in the BizTalk MessageBox where they can then be routed to the corresponding orchestration. This proxy Web service is coded automatically, and everything is preconfigured to work without additional user changes—as long as the incoming posts come directly to this Web server.
Enabling posts to come in on a public portion of the network and be routed to this Web service requires three basic steps. The first step is to create a proxy-to-the-proxy Web service. When a Web service is exported via the wizard, BizTalk refers to it as a proxy Web service. In order to call this Web service from external clients, a proxy Web service must be created on a public server that can route requests to it. This can be done by creating a virtual directory on the public IIS server with the exact same name and interface as the generated proxy. Once the virtual directory has been created, copy all of the generated files from the original BizTalk IIS directory to a virtual directory on the public server.
The second step is to modify the public-facing Web service so that, instead of posting the incoming SOAP information to the BizTalk engine, it reroutes the information to the internal proxy Web service on the IIS server hosting BizTalk Server. The code to modify sits in one file, which is located in the App_Code subdirectory of the virtual directory to which the Web service was published. The name of the file will depend on the name of the entity being published, but will always end with asmx.cs. Opening this file will show the class and Web method declarations, which provide a public interface to calling entities and enable these requests to be pushed directly to the BizTalk MessageBox.
To see what this generated code looks like, export an orchestration with a two-way SOAP port using the Web Services Publishing Wizard. Once the wizard has completed, open the asmx.cs file in the App_Code directory and look at the code. The name of the Web method will be based on the name of the operation as defined on the port in the orchestration. Within this Web method is code that takes the incoming post and transforms it to a format that can be posted to the MessageBox. Once the Web service pushes it to the MessageBox, it then waits for a response that corresponds to the outbound response on the orchestration's two-way port and posts this back to the calling entity. This code, once copied and placed on the new IIS server in the public network, must be modified so that it forwards the incoming post to the Web service on the BizTalk Server.
The modification to the original proxy Web service code can be accomplished using the code shown in Figure 8. The code overwrites the Web method in the original Web service and, instead of posting to the MessageBox, loads some configurable fields from the web.config file and posts the incoming request to the Web service on the BizTalk IIS server. The routing to the BizTalk engine is still handled by the original proxy Web service.
Figure 8 Proxy-to-the-Proxy Web Service
public BTSRedirect.SyncTransResponse Operation_SyncTrans(
  [System.Xml.Serialization.XmlElementAttribute(
  Namespace="http://GuardianProStar.BizTalk.Schemas.SyncTrans.SyncTransRequest",
  ElementName="SyncTransRequest")] BTSRedirect.SyncTransRequest part)
{
  BTSRedirect.SyncTransRequest objSyncTransReq = part;
  BTSRedirect.SyncTransResponse objSyncTransRes = new BTSRedirect.SyncTransResponse();
  BTSRedirect.GuardianProStar_BizTalk_Orchestrations_SyncTrans_SyncTransactions_Port_SyncTrans
    objWebServiceMethodCall = new
    BTSRedirect.GuardianProStar_BizTalk_Orchestrations_SyncTrans_SyncTransactions_Port_SyncTrans();

  // authentication/credentials
  string strWSUser = System.Configuration.ConfigurationManager.AppSettings["WSUser"];
  string strWSPassword = System.Configuration.ConfigurationManager.AppSettings["WSPassword"];
  string strWSDomain = System.Configuration.ConfigurationManager.AppSettings["WSDomain"];
  string strWSAuthenticationType =
    System.Configuration.ConfigurationManager.AppSettings["WSAuthenticationType"];
  objWebServiceMethodCall.Url =
    System.Configuration.ConfigurationManager.AppSettings["RedirectURL"];

  System.Net.CredentialCache cache = new System.Net.CredentialCache();
  cache.Add(new System.Uri(objWebServiceMethodCall.Url), strWSAuthenticationType,
    new System.Net.NetworkCredential(strWSUser, strWSPassword, strWSDomain));
  objWebServiceMethodCall.Credentials = cache;

  // set the response equal to the return value of the call to the web service
  objSyncTransRes = objWebServiceMethodCall.Operation_SyncTrans(objSyncTransReq);

  return objSyncTransRes;

}
Because both Web services have the identical Web interface, either one can be called using the same client code. During development, therefore, you can work with the original Web service code as published by the wizard to validate that everything is working as expected. The proxy-to-the-proxy can be created during the final stages of testing and deployment without affecting any client code.
The final step involved in working with the proxy-to-the-proxy Web service is to modify the web.config file on the public IIS server so that the configurable fields are available. The fields needed by the new Web service are shown here:
<appSettings>
  <add key="RedirectURL"
  value="[redirection URL]"/>
  <add key="WSUser" value = "UserName"/>
  <add key="WSPassword" value = "Password"/>
  <add key="WSDomain"
    value = "DomainName [Not Always Applicable]"/>
  <add key="WSAuthenticationType" value = "basic"/>
</appSettings>
These settings provide for the authentication values and the redirection URL. The actual list of configurable fields (and whether these are actually stored in the web.config file or elsewhere) is up to the developer, and is shown here for illustration purposes only.
The most complex post routing through the firewall is that involving Web services. HTTP posts (and AS2) can be handled in a similar fashion but do not require copying and modifying generated code from BizTalk. In the case of HTTP posts, these can be routed with fairly simple redirect patterns depending on the nature of the post. Many organizations can get mired in determining the right approach to routing traffic to and from their BizTalk servers, and it is important to understand that these types of conversations should never get overly complex, as there are plenty of simple solutions to this common problem.

Handling Failed Documents
EDI implementations are schema intensive. Every document that passes in or out of BizTalk is validated against a corresponding schema. If the document passes validation, it will be routed to the BizTalk engine and will be processed by the EDI components. If the document does not pass validation, however, it will fail and will, by default, remain unprocessed in BizTalk, flagged as invalid. By enabling error routing on the ports that transport EDI documents, these documents will be able to be routed to any BizTalk component.
For example, if a trading partner were to deliver an invalid document to BizTalk rather than allowing the document to fail and remain hidden, it may be desirable to notify an administrator. This could be accomplished by enabling error routing on the port being used for EDI transport. You could then create an orchestration that subscribes to all documents containing errors. This orchestration could then include all of the logic necessary to determine the reason behind the document's failing and notify an administrator.
To route failed messages to a subscribing entity, the port where the messages are arriving or from where they're being sent must have the Enable Routing for Failed Messages flag set. When this is set, any message that fails will be sent to the MessageBox contained in an envelope with information about why the document failed. One set of promoted properties, the ErrorReport group, will then be available to allow for routing to specific subscribers.
To set up a subscriber to listen for these failed documents, simply set a filter to listen for any document on the MessageBox that had a value in the ErrorReport.ErrorType property.
Say your company wants to subscribe to all failed messages that arrive from external trading partners. The initial receive shape on an orchestration could be set to listen for all documents with a value in ErrorReport.ErrorType. This would be done by dropping the receive shape at the top of the orchestration flow and setting it to subscribe to a port with a generic document type (such as System.Xml.XmlDocument). The filter on the receive shape could then be set by clicking on the Filter property of the shape and setting the Property value to ErrorReport.ErrorType and the Operator value to Exists.
If the filter needed to be further refined to restrict the document being picked up to those that specifically matched the Invoice 810 schema, for example, then the incoming port type could be swapped from the generic System.Xml.XmlDocument to the specific schema desired.

EDI and SOA
I have outlined the core EDI functionality in BizTalk Server, but how does this functionality fit into a service-oriented architecture that ensures the maximum reuse of components?
An EDI solution is really a series of processes that can be reused by numerous trading partners. BizTalk Server provides the infrastructure that enables these processes, and their accompanying data entities, to be easily plugged into an SOA and leveraged by a variety of other workflows and services across the enterprise. Additionally, in the context of EDI all documents can be transferred in a secure manner, are validated for content, and can be tracked. In the event of error, they can be fully recovered. If there is extensive processing that needs to take place, either through the core EDI engine or through orchestrations, state management is ensured. Because the solution is based on BizTalk Server, much of your work on orchestrations, error processing, reporting, and other elements can be leveraged across EDI trading partner relationships.
For more information on this subject, see BizTalk Server 2006 R2 EDI Functionality (available at msdn.microsoft.com/library/bb226440.aspx) and BizTalk Server 2006 R2 EDI and AS2 Tutorials (at msdn.microsoft.com/library/bb259975.aspx).

Mark Beckner runs Inotek Consulting Group, LLC, specializing in technical consulting and enterprise integration infrastructures. He is the author of Pro EDI in BizTalk Server 2006 R2.


Silverlight
Create Animations with XAML and Expression Blend
Lawrence Moroney

This article discusses:
  • Basic and combined transformations
  • Using triggers and events
  • Linear and discrete key frame animation
  • Animation with Expression Blend
This article uses the following technologies:
Silverlight 2, Expression Blend
This article is based on a prerelease version of Silverlight 2. All information herein is subject to change. This article is adapted from the book Introducing Microsoft Silverlight 2, 2nd Edition by Laurence Moroney (Microsoft Press, 2008).
Contents
Transformations
Rotating with the RotateTransform Property
Scaling with the ScaleTransform Property
Moving an Object with the TranslateTransform Property
Skewing an Object with the SkewTransform Property
Define Transforms with MatrixTransform
Combining Transformations
Animation
Using Triggers and Event Triggers
Using BeginStoryboard and Storyboard
Defining the Animation Parameters
Targeting the Animation
Setting Animation Properties
Setting the RepeatBehavior Property
Animating a Value with DoubleAnimation
Animating a Color with ColorAnimation
Animating a Point with PointAnimation
Using Key Frames
Using Linear Key Frames
Using Discrete Key Frames
Using Spline Key Frames
Animation and Expression Blend
Conclusion
One of the neat things about XAML is that you can not only declare your objects using an XML syntax, but that you can define transformations to apply to these objects in the same way. You don't need to be a programmer to rotate, move, and skew your objects. Also, XAML can be used to describe how you can animate your object, with an animation being defined as changing properties on the object over time. First I'll look at transforms. Later, I'll add them to timelines to animate your Silverlight™ content.

Transformations
In the field of graphics, a transform defines how to map points from one coordinate space to another. This is typically described using a transformation matrix, a special mathematical construct that allows for simple mathematical conversion from one system to another. Silverlight XAML abstracts this matrix and supports four set transformations for rotation, scaling, skewing,and translation (movement). Silverlight XAML also has an additional special transformation type that allows you to define and implement your own matrix, which you can then use to combine transformations.
Transformations are applied using transform properties. There are several different types of transform properties, which are applied to different object types.
Thus, when using a Brush type, you define your transformation in different ways. One is to use the Brush.Transform property when you want to affect the brush's content (if you want to rotate an image before using it in an ImageBrush, for example). Another way is to use the Brush.RelativeTransform property, which allows you to transform a brush using relative values (something you might do if you are painting different areas of different sizes using the same brush, for example).
When using a Geometry type, you apply a simple transform using the Geometry.Transform property. Keep in mind, however, that this type does not support relative transforms.
Finally, when using a UI element, you specify the transformation to use with the RenderTransform property. If you are transforming an ellipse, for example, you'll use the Ellipse.RenderTransform to define the desired transform.

Rotating with the RotateTransform Property
RotateTransform allows you to rotate an element by a specified angle around a specified center point. You set the angle of rotation by using the Angle property to set the number of degrees that you want to rotate the item. To orient yourself, consider the horizontal vector pointing to the right to be 0 degrees, and rotation takes place clockwise, so the vertical vector pointing down is the result of a 90-degree rotation.
You set the center of transformation using the CenterX and CenterY properties to specify the coordinates of the pivot. These default to 0.0, which makes the default rotation pivot the upper-left corner of the container.
In this example XAML, a TextBlock is rotated using a RenderTransform that contains a RotateTransform specifying a 45-degree rotation:
<TextBlock Width="320" Height="40"
  Text="This is the text to rotate" TextWrapping="Wrap">
  <TextBlock.RenderTransform>
    <RotateTransform Angle="45" />
  </TextBlock.RenderTransform>
</TextBlock>
As you can see in Figure 1, the text is being rotated around a center point at (0,0)—at the upper-left corner of the screen.
Figure 1 Using the RotateTransform Property
This XAML shows how to use CenterX and CenterY to rotate around a different point. In this case, the rotation is done around the (100,200) point:
<TextBlock Width="320" Height="40"
  Text="This is the text to rotate" TextWrapping="Wrap" >
  <TextBlock.RenderTransform>
    <RotateTransform Angle="45" CenterX="100" CenterY="200" />
  </TextBlock.RenderTransform>
</TextBlock>

Scaling with the ScaleTransform Property
The ScaleTransform property is used to change the size of an object based on the horizontal axis, the vertical axis, or both axes. When scaling an object, you need to specify at least one of the axes around which you want to scale, and by how much you want to scale against that axis.
You use the ScaleX property to scale the object on the horizontal axis, the x-axis, and the ScaleY to scale it on the vertical axis, the y-axis. These are set to a double value, which represents the value by which you multiply the object's current size on the specified axis. Therefore, values greater than 1 will stretch the object by that multiple. For example, using a ScaleX value of 2 will double the size of the object horizontally. Values less than 1, but greater than 0, will shrink the object. Using a setting of 0.5, for instance, will reduce the size of the object by half along the specific dimension.
For example, this XAML creates a red rectangle 96 pixels wide by 88 pixels high:
<Rectangle Fill="#FFFF0404" Stroke="#FF000000"
  Width="96" Height="88"
  Canvas.Left="112" Canvas.Top="72" />
Figure 2 demonstrates what this object looks like when it is rendered in Silverlight.
Figure 2 Rendering the Rectangle
To apply a ScaleTransform to this object, you use a RenderTransform and specify the transform to be a ScaleTransform. Here's the XAML:
<Rectangle Fill="#FFFF0404" Stroke="#FF000000"
  Width="96" Height="88" Canvas.Left="112" Canvas.Top="72">
  <Rectangle.RenderTransform>
    <ScaleTransform ScaleX="2" />
  </Rectangle.RenderTransform>
</Rectangle>
If you're coding along at home, you'll see that the rectangle increased in size horizontally to the right using this ScaleTransform. This was because the center of scaling was not specified. You can specify it with the CenterX property for horizontal scaling or the CenterY property for vertical scaling. These specify the coordinate of the center of scaling. Note that this coordinate is relative to the upper-left corner of the rectangle. Also, the coordinate default is 0, meaning that scaling will take place to the right on the horizontal axis and downward on the vertical axis.
If you set the CenterX property to a positive value (for example, 50), the scaling will be around the X point, 50 pixels to the right of the leftmost side of the rectangle. This will make it look like the rectangle has moved a number of pixels to the left of the one where the CenterX hasn't been changed (the number depends on the size of the scaling factor). This is because the stretching is centered on that point, pushing the left side of the rectangle to the left as well as pushing the right side to the right. You'll get similar effects by setting the ScaleY and CenterY values in the same way:
<Rectangle Fill="#FFFF0404" Stroke="#FF000000"
  Width="96" Height="88"
  Canvas.Left="80" Canvas.Top="80">
  <Rectangle.RenderTransform>
    <ScaleTransform ScaleX="2"
      CenterX="50"/>
  </Rectangle.RenderTransform>
</Rectangle>

Moving an Object with the TranslateTransform Property
A translation is a transform that moves an object in a two-dimensional plane from one position to another. It is defined by setting up vectors that define the object's motion along its x- and y-axes. These are set using the X and Y properties on the transform. To move an item two units horizontally (meaning it will move to the right), you set the X property to 2. To move it to the left, use a negative value, such as -2. Similarly, to move an object vertically, you would use the Y property, and positive values will cause the object to move down the screen, whereas negative values will move it up the screen.
Here's an example of a translate transform that moves the position of the red rectangle that we've been looking at by specifying X and Y values that move it up and to the left. These values effectively make up a vector that determines the transform:
<Rectangle Fill="#FFFF0404"
  Stroke="#FF000000"
  Width="96" Height="88"
  Canvas.Left="80" Canvas.Top="80">
  <Rectangle.RenderTransform>
    <TranslateTransform X="-50" Y="-50"/>
  </Rectangle.RenderTransform>
</Rectangle>
See the results of this in Figure 3. The rectangle has moved upward and to the left relative to its specified position, as compared to the position of the rectangle in Figure 2.
Figure 3 Using the TranslateTransform Property

Skewing an Object with the SkewTransform Property
Skewing an object involves changing it in a progressive, uniform manner along an axis. This has the effect of turning a square or rectangle into a parallelogram. This visual effect is useful in creating the illusion of depth on a two-dimensional surface.
You can apply a skew at a certain angle on either the x- or y-axis and around a center point. These can, of course, be combined so that you can skew on both axes at the same time. The following XAML skews our rectangle on the x-axis by 45 degrees:
<Rectangle Fill="#FFFF0404" Stroke="#FF000000"
  Width="96" Height="88"
  Canvas.Left="80" Canvas.Top="80">
     <Rectangle.RenderTransform>
       <SkewTransform AngleX="45"/>
     </Rectangle.RenderTransform>
   </Rectangle>
You can see the result in Figure 4.
Figure 4 Skewing the Rectangle with SkewTransform
Skewing is useful for simulating three-dimensional effects in graphics. For example, you can apply SkewTransform to three adjacent rectangles—two skewed on the x-axis and one on the y-axis—to create an illusion of a three-dimensional perspective (see Figure 5).
Figure 5 Simulating Perspective with Three Skewed Rectangles

Define Transforms with MatrixTransform
All transformations, at their heart, are performed by multiplying the coordinate space of the object by a transformation matrix. Each of the transforms that you've seen so far is a well-known and well-defined transform. Matrix mathematics and how transforms are implemented are beyond the scope of this article, but for the sake of syntactic completeness, I'll look at how you can define them in Silverlight XAML.
Note that the matrix used in the MatrixTransform is an affine matrix, which means that the bottom row of the matrix is always set to (0 0 1), and as such you set only the first two columns. These are set using the transform's Matrix property, which takes a string containing the first two rows of values separated by spaces:
<Rectangle Fill="#FFFF0404"
     Stroke="#FF000000"
     Width="96" Height="88"
     Canvas.Left="80" Canvas.Top="80">
     <Rectangle.RenderTransform>
       <MatrixTransform Matrix="1 0 1 2 0 1"/>
     </Rectangle.RenderTransform>
   </Rectangle>
The impact of the transform using this matrix is to render a combined stretched and skewed rectangle.

Combining Transformations
As you saw in the previous example, you can create a complex transformation by using a transformation affine matrix and specifying that using the MatrixTransform type. However, if you aren't an expert in matrix mathematics, another technique for using transforms is to combine them by means of the TransformGroup element. This simply allows you to specify multiple transforms, and the combined effect of each will be applied to the object. Here's an example:
<Rectangle Fill="#FFFF0404" Stroke="#FF000000"
  Width="96" Height="88"
  Canvas.Left="80" Canvas.Top="80">
  <Rectangle.RenderTransform>
    <TransformGroup>
      <ScaleTransform ScaleX="1.2" ScaleY="1.2" />
      <SkewTransform AngleX="30" />
      <RotateTransform Angle="45" />
    </TransformGroup>
  </Rectangle.RenderTransform>
</Rectangle>
This example combines a ScaleTransform that increases the size of the shape on both axes by 20 percent, with a 30-degree skew on the x-axis and a rotation of 45 degrees.

The word animation literally means "imparting life onto something." Hence, with animation you can bring your creations to life by changing the attributes of your objects, such as their color, size, opacity, and other properties, over a period of time or in response to user actions.
In XAML, you animate an item by changing one or more of its properties over time. This time is defined using a timeline. For example, to move an item across the screen in five seconds, you'd specify a five-second timeline that animates the Canvas.Left property from 0 to the width of the screen. In the next sections, I'll discuss each type of animation available, as well as the difference in animating these properties using key frames.
Before you look into the different animation types, you should know that there is a framework around animations that involves Triggers, EventTriggers, and Storyboards. First, however, I'll take a look at these basic concepts and then examine the different animation types in more detail.

Using Triggers and Event Triggers
Animations in Silverlight take place in response to an event, which is defined using a trigger. At present, there is only one trigger type supported in Silverlight XAML, the EventTrigger. Each UI property has a Triggers collection that is used to define one or more triggers (meaning one or more EventTriggers).
So, the first step in adding an animation to an element is to define its Triggers collection; then you'll need to add at least one EventTrigger to the collection you've created. For example, if you are animating a rectangle, the first step—specifying the Triggers collection—will look like this:
<Rectangle x:Name="rect" Fill="Red"
  Canvas.Top="100" Canvas.Left="100"
  Width="100" Height="100">
  <Rectangle.Triggers>
  </Rectangle.Triggers>
</Rectangle>
Next, you will need to define an EventTrigger to add to this collection. On this EventTrigger, you will use the RoutedEvent property to specify which event the animation runs in response to. Note that RoutedEvent only supports the Loaded event.
To implement an animation that will begin when the rectangle is loaded, you would specify the EventTrigger as follows:
<EventTrigger RoutedEvent="Rectangle.Loaded">
</EventTrigger>
The XAML snippet to run this animation looks like this:
<Rectangle x:Name="rect" Fill="Red" Canvas.Top="100"
  Canvas.Left="100" Width="100" Height="100">
  <Rectangle.Triggers>
    <EventTrigger RoutedEvent="Rectangle.Loaded">
    </EventTrigger>
  </Rectangle.Triggers>
</Rectangle>
The next step is to define the animation that you want to use. Animations are contained within Storyboards.

Using BeginStoryboard and Storyboard
BeginStoryboard is a trigger action that contains a Storyboard object. Storyboard objects contain the animation definitions. When you define an animation, you simply embed these objects within the EventTrigger definition. The following code shows how this would be accomplished using my rectangle example:
<Rectangle x:Name="rect" Fill="Red"
  Canvas.Top="100" Canvas.Left="100"
  Width="100" Height="100">
  <Rectangle.Triggers>
    <EventTrigger RoutedEvent="Rectangle.Loaded">
      <BeginStoryboard>
        <Storyboard>
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Rectangle.Triggers>
</Rectangle>

Defining the Animation Parameters
Now that the framework for the animation is set up, you can specify the animation that you want to perform. At its most basic level, animation defines changing a property over time. You can animate three different property types. Each of these property types is animated from a value specified in the From attribute (or its current value if this is not set) either to a value specified in the To attribute or by a value specified in the By attribute.
Double Types These are animated using the DoubleAnimation or DoubleAnimationUsingKeyFrames. This method is used to animate properties that contain a double value—for example, dimensions such as Canvas.Left or visual attributes such as Opacity.
Point Types These are animated using a PointAnimiation or PointAnimationUsingKeyFrames type. This particular method is used to animate properties that contain a Point value, such as line segments or curves that are defined using points.
Color Types These are animated using a ColorAnimation or ColorAnimationUsingKeyFrames type. This method is used to animate properties that contain a color value—the background or stroke of an element, for instance.

Targeting the Animation
To define which object you want to apply the animation to, you use the Storyboard.TargetName property on these animation types, and you need to pass it the name of the object in question, which is set on the object using the x:Name property. Additionally, you specify the property that will be animated using the Storyboard.TargetProperty. Note that if you are specifying a complex or attached property (such as Canvas.Left), you place it in parentheses. So, for example, to specify a Double animation to target the Canvas.Left of a rectangle named rect, the resulting XAML will look like this:
<DoubleAnimation Storyboard.TargetName="rect"
  Storyboard.TargetProperty="(Canvas.Left)" />

Setting Animation Properties
To define how long it will take to transition the properties in question from one value to another, you use the Duration property. Note that it is defined in the HH:MM:SS format, wherein a five-second time duration for the animation is specified as 00:00:05, abbreviated to 0:0:5.
<DoubleAnimation Storyboard.TargetName="rect"
  Storyboard.TargetProperty="(Canvas.Left)" Duration="0:0:5" />
If you do not want the animation to begin right away, you can insert a delay using the BeginTime property, using the same syntax:
<DoubleAnimation Storyboard.TargetName="rect"
  Storyboard.TargetProperty="(Canvas.Left)" BeginTime="0:0:5" />
You can also tweak the animation behavior by multiplying the duration by a speed ratio. This is achieved using the SpeedRatio property. For example, in the previous case the duration was set to 5 seconds. You can change the speed ratio to make the animation last 10 seconds by setting SpeedRatio to 2, or, alternatively you can speed the animation up to 1 second by setting SpeedRatio to 0.2.
<DoubleAnimation Storyboard.TargetName="rect"
  Storyboard.TargetProperty="(Canvas.Left)"
  SpeedRatio="2" Duration="0:0:5" />
Silverlight animation provides the facility to reverse the changes made as part of the animation. For example, if you are moving a double value from 0 to 500 over a specific time frame, an AutoReverse will cause the animation to move from 500 back to 0.
Note that if the animation is set to run for 5 seconds as previously, and the AutoReverse is set to true, then the complete animation will take 10 seconds. Following is an example of XAML containing the AutoReverse property:
<DoubleAnimation Storyboard.TargetName="rect"
  Storyboard.TargetProperty="(Canvas.Left)"
  AutoReverse="True"
  Duration="0:0:5" />

Setting the RepeatBehavior Property
When the animation has finished running, you can apply a number of options to control how you want it to behave. You specify these using the RepeatBehavior property. This property can take three different types of values:
  • A time defined in seconds. The timeline will wait for this period and then start the animation again.
  • Repeat behavior set to Forever for constant repetition.
  • A discrete number of repetitions set by specifying a number followed by x. For example, if you want the animation to run three times, you specify the value 3x.
Figure 6 shows the complete XAML for the animated rectangle to move it from 100 to 500 and back to 100 on the x-axis and then repeat that behavior three times:
Figure 6 Reversing and Repeating Animation
<Rectangle x:Name="rect" Fill="Red"
  Canvas.Top="100" Canvas.Left="100"
  Width="100" Height="100">
  <Rectangle.Triggers>
    <EventTrigger RoutedEvent="Rectangle.Loaded">
      <BeginStoryboard>
        <Storyboard>
          <DoubleAnimation RepeatBehavior="3x"
            Storyboard.TargetName="rect"
            Storyboard.TargetProperty="(Canvas.Left)"
            To="500" Duration="0:0:5"
            AutoReverse="True" />
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Rectangle.Triggers>
</Rectangle>
Let's look at each of these animation types in a little more detail. First, I'll examine the attributes needed to animate each of the various types, and then I'll address where the associated key frame type of animation fits into the picture.

Animating a Value with DoubleAnimation
The DoubleAnimation object allows you to specify how a double value will change over a specified timeline. The animation is calculated as a linear interpolation between the property values over time.
When animating a double, you specify the value at the start of the animation using the From value and then change it to either the To value, which is an absolute destination, or the By value, which is a relative destination. For example, if you are moving the Canvas.Left property of an item from 100 (near the left of the screen) to 500, you can set From to 100 and To to 500, or By to 400. Note that if you set both, the To property takes precedence and the By property is ignored. Also, if the rectangle is already located at the desired From position, you do not need to specify the From property.
The previous XAML example displayed this behavior. The rectangle is located with a Canvas.Left value of 100, and the DoubleAnimation specifies the To value as 500. Hence, the animation will move the value from 100 to 500, which will cause the rectangle to move across the screen to the right.

Animating a Color with ColorAnimation
ColorAnimation operates in a manner that is similar to DoubleAnimation. You use it to specify how the color value of an element will change over time. The animation is then calculated as a linear interpolation between the color values over the specified time.
When animating a color, you specify the value at the start of the animation using the From property. If you do not specify this, then the current color is used. You specify the desired end color using the To attribute. You can also specify a By attribute, which will provide the end color that is the result of adding the values of the From color (or the starting color) to the By color.
When you animate a color-based property, you do not animate the contents of the property directly, because the content of the property is usually a brush and not a color. So, if you want to animate the fill color of a rectangle, for example, you don't use the rectangle's Fill property as your target. Instead, you specify that you intend to animate the Color property of the SolidBrush that is used to perform the fill.
Figure 7 shows an example of how to animate the color of a rectangle, changing it from black to white over a time duration of five seconds, using a color animation. As you can see in the code, this XAML snippet specifies the Color property of the SolidColorBrush that is filling the shape as its target property. Note that this is the typical XAML syntax used in addressing complex properties like this scenario.
Figure 7 Animating Color Change
<Rectangle x:Name="rect" Canvas.Top="100"
  Canvas.Left="100" Width="100"
  Height="100" Fill="Black">
  <Rectangle.Triggers>
    <EventTrigger RoutedEvent="Rectangle.Loaded">
      <BeginStoryboard>
        <Storyboard>
          <ColorAnimation Storyboard.TargetName="rect"
            Storyboard.TargetProperty=
            "(Shape.Fill).(SolidColorBrush.Color)"
            To="#00000000" Duration="0:0:5" />
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Rectangle.Triggers>
</Rectangle>

Animating a Point with PointAnimation
To change a value that is defined as a point over time, you use the PointAnimation type. The animation is then calculated as a linear interpolation between the values over the specified time.
In a manner similar to the Color and Double animations, you specify the start value using From and the destination either as a relative direction (using By) or an absolute point (using To). Figure 8 shows an example of how you could animate the end point of a Bezier curve. In this case, the Bezier curve is defined with a start point at (100,100), an end point at (300,100), and a control point at (200,0). An animation is set up to trigger after the path loads, and it animates the end point of the curve (Point2) from (300,100) to (300,600) over a time duration of five seconds.
Figure 8 Animating the End of a Curve
<Path Stroke="Black" >
  <Path.Data>
    <PathGeometry>
      <PathFigure StartPoint="100,100">
        <QuadraticBezierSegment x:Name="seg"
          Point1="200,0" Point2="300,100"  />
      </PathFigure>
    </PathGeometry>
  </Path.Data>
  <Path.Triggers>
    <EventTrigger RoutedEvent="Path.Loaded">
      <BeginStoryboard>
        <Storyboard>
          <PointAnimation Storyboard.TargetName="seg"
            Storyboard.TargetProperty="Point2"
            From="300,100" To="300,600" Duration="0:0:5" />
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Path.Triggers>
</Path>

Using Key Frames
The three animation types that you've just learned about, ColorAnimation, DoubleAnimation, and PointAnimation, all work by changing a defined property over time using linear interpolation. For example, if you are moving a double value from 100 to 500 over five seconds, it will increment by 80 each second.
Each of these three animation types can have this transition defined through a set of milestones called key frames. To change the linear behavior of the animation from the starting property to the ending property, you simply insert one or more key frames. Then you define the style of animation that you want between these various points.
Key frames are defined using key times. These are times that are specified relative to the start time of the animation; they also specify the end time of the key frame. So, for instance, if you need a nine-second animation with three evenly spaced key frames, you can specify the first key frame to end at 0:0:3, the second to end at 0:0:6, and the third to end at 0:0:9. Keep in mind that you do not specify the length of the key time—instead, you specify the end time for each key frame.
As another example, consider a Double animation that you want to span half the range of 100 to 500. The animation should move very quickly in the first half and very slowly in the second half. Overall, it will require a six-second total transition. Since 350 is the midpoint between 100 and 500, you would define a key frame to begin at point 350. You'd tell it to go for one second between the start point and the midpoint, using a key time of 0:0:1, and then set a time duration of five seconds between the midpoint and the end point by using a second key time of 0:0:6. Now the item is set to zip across the screen to the midpoint and then it will seem to crawl the rest of the way.
In the previous example, both animated segments will be linearly interpolated. To provide extra flexibility, two other types of key frames are provided: a discrete key frame that instantly jumps the value between the two values, and a spline key frame that moves the value between the first and end points using a quadratic curve to define the interpolation. (In the following sections, you'll look at how to define an animation using key frames for the Double type. Note that the same principles apply for Point and Color animation types.)
To specify key frames, you use the UsingKeyFrames postfix on your animation. That is, to define Double animations and use key frames, you'll use DoubleAnimationUsingKeyFrames on which you specify your target and property (in the same way you use DoubleAnimation). DoubleAnimationUsingKeyFrames contains the key frame definitions. (And as I mentioned earlier, the same applies to PointAnimationUsingKeyFrames or ColorAnimationUsingKeyFrames.)

Using Linear Key Frames
The default method for animation between two property values is linear interpolation in which the amount is divided evenly over time. You can also define linear steps between frames using the LinearKeyFrame type in which linear interpolation is still used, but it is used between key frames, so you can have an acceleration/deceleration effect.
Consider the animation code shown in the first part of Figure 9. Here, a DoubleAnimationUsingKeyFrames is used, and it defines two key frames. One defines a linear interpolation between 0 and 300 for Canvas.Left changes over one second, and the next defines a linear interpolation between 300 and 600 for Canvas.Left changes over eight seconds. This has the effect of making the rectangle move quickly to the halfway point, then slowly the rest of the way across. Similar principles apply for the LinearPointKeyFrame and LinearColorKeyFrame.
Figure 9 Animation with Key Frames
Linear Interpolation
<Rectangle Fill="#FFFF0000" Stroke="#FF000000"
  Width="40" Height="40"
  Canvas.Top="40" x:Name="rect">
  <Rectangle.Triggers>
    <EventTrigger RoutedEvent="Rectangle.Loaded">
      <BeginStoryboard>
        <Storyboard>
          <DoubleAnimationUsingKeyFrames
            Storyboard.TargetName="rect"
            Storyboard.TargetProperty="(Canvas.Left)" >
            <LinearDoubleKeyFrame KeyTime="0:0:1" Value="300" />
            <LinearDoubleKeyFrame KeyTime="0:0:9" Value="600" />
          </DoubleAnimationUsingKeyFrames>
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Rectangle.Triggers>
</Rectangle>
Discrete Key Frames
<Rectangle Fill="#FFFF0000" Stroke="#FF000000"
  Width="40" Height="40"
  Canvas.Top="40" x:Name="rect">
  <Rectangle.Triggers>
    <EventTrigger RoutedEvent="Rectangle.Loaded">
      <BeginStoryboard>
        <Storyboard>
          <DoubleAnimationUsingKeyFrames
            Storyboard.TargetName="rect"
            Storyboard.TargetProperty="(Canvas.Left)" >
            <DiscreteDoubleKeyFrame KeyTime="0:0:1" Value="300" />
            <DiscreteDoubleKeyFrame KeyTime="0:0:9" Value="600" />
          </DoubleAnimationUsingKeyFrames>
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Rectangle.Triggers>
</Rectangle>

Using Discrete Key Frames
If you want to change the property from one value to another and not use linear interpolation, you can use a discrete key frame. This causes the object to jump to the value at the specified key frame time.
The second part of Figure 9 shows the same example as the previous one, except that it uses a discrete key frame. At one second into the animation, the rectangle will jump halfway across the screen. Then at nine seconds into the animation, it will jump to the right of the screen. Note that similar principles apply for the DiscretePointKeyFrame and DiscreteColorKeyFrame.

Using Spline Key Frames
To change the property from one value to another using a curved value that provides for acceleration and deceleration, you use a spline key frame. To do this, first you define a quadratic Bezier curve and then the speed of the property as it moves from one value to another is determined by a parallel projection of that curve.
If this is hard to visualize, consider the following scenario: the sun is right overhead, and you hit a baseball into the outfield. You look at the shadow of the ball. As it is climbing into the air, the movement of the shadow appears to accelerate. As it reaches its apex, you'll see the shadow decelerate. As the ball falls, you'll see the speed of the shadow accelerate again, until either it is caught or it hits the ground.
Imagine your animation in this case is the ball's shadow and the spline is the curve of the baseball. You define the trajectory of the baseball, a spline, using a KeySpline. The KeySpline defines control points for a quadratic Bezier curve. It is normalized so that the first point of the curve is at 0, and the second is at 1. For a parabolic arc, which is the trajectory the baseball would follow, the KeySpline will contain two comma-separated normalized values.
To define a curve such as the flight of a baseball, you can specify the spline using a KeySpline such as 0.3,0 0.6,1. This defines the first point of the curve at (0.3,0) and the second at (0.6,1). This will have the effect of making the animation accelerate quickly until approximately one-third of the movement of the baseball is complete; then it will move at a uniform speed until approximately two-thirds of the ball's trajectory is reached; and then it will decelerate for the rest of the flight of the animated baseball, as the animation simulates the ball's fall to earth.
Figure 10 shows an example of using a KeySpline to define the spline for this simulation using DoubleAnimationUsingKeyFrames. This example animates the ellipse so that it moves across the screen in a manner similar to the shadow of a baseball, as if you were above the baseball looking down toward the ground as the ball flies through the air.
Figure 10 Defining a Spline with KeySpline
<Ellipse Fill="a#FF444444" Stroke="#FF444444"
  Width="40" Height="40"
  Canvas.Top="40" x:Name="ball">
  <Ellipse.Triggers>
    <EventTrigger RoutedEvent="Ellipse.Loaded">
      <BeginStoryboard>
        <Storyboard>
          <DoubleAnimationUsingKeyFrames
            Storyboard.TargetName="ball"
            Storyboard.TargetProperty="(Canvas.Left)" >
            <SplineDoubleKeyFrame KeyTime="0:0:5"
              KeySpline="0.3,0 0.6,1" Value="600" />
          </DoubleAnimationUsingKeyFrames>
        </Storyboard>
      </BeginStoryboard>
    </EventTrigger>
  </Ellipse.Triggers>
</Ellipse>

Animation and Expression Blend
Animation can be defined graphically in Expression Blend™. This generates the XAML for you to perform the animation, providing the different types of animation for you automatically.
When using Expression Blend, select Animation Workspace from the Window menu. This will give you the tools to graphically design timelines, and when you edit the properties that you want changed using the visual editor, then the XAML code for the animation will be generated.
At the bottom of the screen, you can see the Objects and Timeline view. This allows you to add a timeline and then visually add key frames. To add a new timeline, click the + button in the Objects and Timeline view.
When you click the + button, you'll see a Create Storyboard dialog box that asks you for the name of the storyboard to create. In this case, I've changed the default name from Storyboard1 to Timeline1 and cleared the Create as Resource checkbox.
When using Expression Blend, you can create an animation at the Canvas level or as a Resource. In the case of the former, animations then run in response to triggers on the canvas. Following is an example of the XAML created by Blend from the Create Storyboard dialog box, where the user specified that they did not want to create the animation as a Resource:
<Canvas.Triggers>
  <EventTrigger RoutedEvent="Canvas.Loaded">
    <BeginStoryboard>
      <Storyboard x:Name="Timeline1"/>
    </BeginStoryboard>
  </EventTrigger>
</Canvas.Triggers>
Note that if the user checked the Create as Resource checkbox, then the Storyboard would be created within <Canvas.Resources> and you would have to run it from JavaScript.
The Objects and Timeline view will change to show the timeline that you've just created. You can see the timeline at the bottom of Figure 11.The vertical line at time 0 that you see in the timeline denotes the current time. (In Expression Blend, this line will be yellow.) To add a key frame, you simply drag this line to the time where you want a key frame and click the Record Keyframe button. This button is located just above the timeline to the left of 0:00:000.
Figure 11 Expression Blend Animation Workspace
Drag the line to the four-second mark and add a key frame. You'll see the key frame added as a small oval on the timeline. Now that the timeline is on the four-second frame and you have added a key frame, you can edit the rectangle's color, location, opacity, or shape, and Blend will calculate the correct transformations necessary to facilitate the animation. Finally, you may notice that if you drag the timeline indicator around, you can preview the animation and see how it appears at any particular time.

In this article, I discussed how transformations and animations are defined in Silverlight XAML. I introduced you to different types of transformation used to rotate, scale, or skew an object, as well as to freeform transformations using an affine matrix as applied to a shape. I then offered an overview of animations and showed you how to define an animation to run based on an XAML trigger. You saw how animations change property values over time, and you looked at the XAML types that support animating double, point, and color values. I then explained how to use key frames for finer control over your animations. Finally, I showed you the Expression Blend animation designer so you could see how easily animations can be generated visually using Expression Blend.

Laurence Moroney is a Senior Technology Evangelist with Microsoft, specializing in Silverlight. He is the author of many books on computing topics, including Silverlight, AJAX, Interoperability, and Security. You can find Laurence's blog at blogs.msdn.com/webnext.


Write On!
Create Web Apps You Can Draw On with Silverlight 2
Julia Lerman

This article discusses:
  • Silverlight InkPresenter control
  • Ink in a Web application
  • Handwriting recognition
  • Creating transparent and image backgrounds
This article uses the following technologies:
Silverlight 2, Expression Blend, Visual Studio 2008
This article is based on prerelease versions of Visual Studio 2008 SP1, Silverlight 2, and Expression Blend. All information herein is subject to change.
Contents
Introducing the InkPresenter
Tablet PC Not Required
InkPresenter 101
Adding Events and a Background
A More Stylish InkPresenter
Getting That Silverlight Look
Add a Background Image or Video to the InkPresenter
Stroke Design Basics
Handwriting Recognition
The InkAnalyzer Class
Analyzing on the Server Side
Hooking Up the Service to the Silverlight Client
Persisting Annotations Using Services
Store and Retrieve Annotations
Putting It All Together
Silverlight is one of the most exciting new Web technologies to come out of Microsoft in the past few years. It's such a big deal that the annual MIX conference now seems to center around Silverlight™ and its many capabilities. Silverlight 1.0 was initially released in 2007 and each new release of Silverlight 2 prior to its final release has been filled with impressive new features. One of the very cool features of Silverlight that hasn't gotten the attention it deserves is the InkPresenter control. The InkPresenter control enables Internet users to draw directly onto Silverlight applications from their browser.
Because Silverlight can be used on a variety of OSs and in a variety of browsers, so too can the InkPresenter, which removes another set of limitations around browser, OS and hardware. The sample application I'll present here uses the Microsoft® .NET Framework programming features of Silverlight 2 combined with the InkPresenter to let users annotate a predefined collection of images, perform handwriting recognition, save the annotations and recognized text to a server-side database, retrieve annotations for a selected image, and filter images based on their associated text. The database and handwriting recognition functionality are provided by a Windows® Communication Foundation (WCF) service. Figure 1 shows the completed application.
Figure 1 The Completed Application
It should be noted that everything described in this article can also be accomplished in Silverlight 1.0. In fact, prior to the availability of Silverlight 2, I wrote a similar application in Silverlight 1.0 with all of the same features. However, without the .NET-targeted code at your disposal on the client-side, you would need to leverage .NET Web services for more of the features.

Introducing the InkPresenter
The InkPresenter control that makes this application possible is a container for a collection of strokes. Each stroke is made up of a collection of StylusPoints. Note that there is another Stroke property in Silverlight that is a member of the Shape class, so be careful not to confuse the two.
If you think in terms of pen and paper, each time you touch the pen to the paper, you begin a stroke, and you then proceed to move the pen around whether you are drawing a circle or writing a word. Lifting the pen from the paper signals the end of that stroke. Placing the pen down again begins a new stroke.
Strokes have DrawingAttributes that define characteristics such as the stroke's color, width and a few other attributes. The individual points in a stroke also have properties: X and Y coordinates for location and a PressureFactor. PressureFactor is interpreted for computers with a digitizer and allows you to programmatically impact the stroke based on how hard the user is pressing the stylus to the digitizer. Figure 2 illustrates the class hierarchy.
Figure 2 The InkPresenter Class
As with other visual elements in Silverlight, the InkPresenter object and its children can be represented as XAML. Figure 3 shows three small strokes in an InkPresenter and Figure 4 shows the XAML representation of one stroke in the StrokeCollection. Even though the strokes are small, the digitizer collects quite a lot of data. If you were to do the same test using a mouse, much less data would be collected, as the Stylus points would only be doubles and points would be collected at less frequent intervals.
Figure 3 Some Strokes
Figure 4 First Stroke in an InkPresenter Control
<StrokeCollection>
<StrokeCollection xmlns="http://schemas.microsoft.com/client/2007">
  <Stroke>
    <Stroke.DrawingAttributes>
      <DrawingAttributes Color="#FF000000"         OutlineColor="#00000000" Width="3" Height="3" />
    </Stroke.DrawingAttributes>
    <Stroke.StylusPoints>
      <StylusPoint X="81.4583358764648" Y="96.5833282470703" />
      <StylusPoint X="81.4583358764648" Y="96.5833282470703" />
      <StylusPoint X="81.0833358764648" Y="96.4166717529297" />
      <StylusPoint X="81.0833358764648" Y="96.4166717529297" />
      <StylusPoint X="81.0833358764648" Y="96.4166717529297" />
      <StylusPoint X="81.0833358764648" Y="96.4166717529297" />
      <StylusPoint X="81.0833358764648" Y="96.4166717529297" />
      <StylusPoint X="80.4583358764648" Y="96.8333282470703" />
      <StylusPoint X="80.4583358764648" Y="96.8333282470703" />
      <StylusPoint X="80" Y="97.2916717529297" />
      <StylusPoint X="80" Y="97.2916717529297" />
      <StylusPoint X="79.625" Y="97.75" />
      <StylusPoint X="79.625" Y="97.75" />
      <StylusPoint X="79.625" Y="97.75" />
      <StylusPoint X="79.625" Y="97.75" />
      <StylusPoint X="79.625" Y="96.5416717529297" />
      <StylusPoint X="79.8333358764648" Y="95.7083358764648" />
      <StylusPoint X="80.25" Y="94.7916641235352" />
      <StylusPoint X="80.7916641235352" Y="93.5416641235352" />
      <StylusPoint X="81.5" Y="92.125" />
      <StylusPoint X="82.4166641235352" Y="90.4583358764648" />
      <StylusPoint X="83.4583358764648" Y="88.5833358764648" />
      <StylusPoint X="84.75" Y="86.5416641235352" />
      <StylusPoint X="86.1666641235352" Y="84.3333358764648" />
      <StylusPoint X="87.7083358764648" Y="82.1666641235352" />
      <StylusPoint X="89.25" Y="79.9166641235352" />
      <StylusPoint X="90.75" Y="77.9583358764648" />
      <StylusPoint X="92" Y="76.0833358764648" />
      <StylusPoint X="93.1666641235352" Y="74.8333358764648" />
      <StylusPoint X="94" Y="73.625" />
      <StylusPoint X="94.7083358764648" Y="73.1666641235352" />
      <StylusPoint X="95.125" Y="73.1666641235352" />
      <StylusPoint X="95.125" Y="73.1666641235352" />
      <StylusPoint X="95.125" Y="73.1666641235352" />
      <StylusPoint X="94.7083358764648" Y="73.5" />
    </Stroke.StylusPoints>

  </Stroke>
...
</StrokeCollection>
Working with the InkPresenter is actually a matter of creating and interacting with its strokes. However, the InkPresenter does none of this by default. It provides Events and Methods; it allows you to add and remove StrokeCollections and it allows you to access the Strokes in order to interact with them. But it is up to you to programmatically keep track of the mouse or stylus activity within the boundaries of an InkPresenter control and build the strokes.

Tablet PC Not Required
Before Windows Presentation Foundation (WPF) and Silverlight, developers were dependent on the Tablet PC SDK to create custom programs that took advantage of the Tablet PC drawing features. The SDK is set of COM APIs with .NET wrappers that enable development in .NET, Visual Basic® 6.0, and C++. The Windows XP Tablet PC Edition was the required OS.
Beginning with version 1.7 of the Tablet PC SDK, the critical control, InkOverlay, no longer required Full Trust on a system in order to run. It became possible to create ink-enabled Windows Forms controls and embed them into Web pages, as you would with any ActiveX® control. And although ActiveX controls were limited to use in Internet Explorer® only, this provided a first step at enabling developers to bring drawing and other inking capabilities to the Web.
In Windows Vista®, Tablet PC functionality is built into the OS as a first-class citizen and the development APIs for Tablet features have been incorporated into the WPF InkCanvas object. This means that any computer with the .NET Framework 3.0 installed can support Tablet functionality even if it isn't a Tablet PC. But it's important to recognize that using a stylus on a Tablet's digitizer results in much higher resolution (by orders of magnitude) than what a user will achieve using a mouse.
The ink capabilities in Silverlight are a subset of what's available in WPF. One major difference, however, is that while the WPF InkCanvas has a property called InkPresenter that displays the ink on the InkCanvas, in Silverlight there is no InkCanvas and you must work with the InkPresenter directly.
Most importantly, because Silverlight is not limited to Windows or Internet Explorer, more environments can use ink-enabled Web sites.

InkPresenter 101
Silverlight Tools for Visual Studio® 2008 embed a Silverlight XAML designer into Visual Studio 2008, but the design surface itself is read-only, so you may find it more convenient to work in Expression Blend™ 2.5 for some of the InkPresenter design work. The two applications are very well integrated, so this is quite simple to do and once you become familiar with Expression Blend, it's also a lot of fun.
I prefer to start my project in Visual Studio because the default project templates are very helpful. Then when I want to do some design work in XAML, I open the project in Expression Blend. You can do this easily from the Visual Studio IDE by right-clicking a XAML file in the Solution Explorer and selecting the option to open the file in Expression Blend. If you don't have Expression Blend 2.5, you can edit the XAML by hand right in Visual Studio 2008 and instantly see the results of your changes.
When creating a Silverlight project, you have the option to use Visual Basic or C#. Even though my Visual Basic skills are better, for this project, I choose to work in C# because I had a stockpile of JavaScript code from work I did in Silverlight 1.0, which readily ported to C#.
After creating the project in Visual Studio, the first task is to open the project in Expression Blend so that you can start designing. Once you have the XAML file open in the Expression Blend designer, you can add the InkPresenter control to the design surface. To find the InkPresenter control, you'll need to open the Expression Blend Asset Library by clicking the >> icon at the bottom of the toolbox. Then you'll have to click the Show All checkbox to see the InkPresenter as well as number of other less commonly used controls. Alternatively, you can use the search box to search for controls.
Drag the InkPresenter onto the XAML design surface; then, using its properties window, give the control a name. I've chosen "inkP" which will be used frequently in the code later in this article.
When the InkPresenter is selected on the design surface, you can see its boundaries, but when it is not selected it seems to disappear. The InkPresenter control is a container that is able to render ink and while it does have a background property, it does not have Fill, Stroke (for borders) or many other properties you will find in other controls. Therefore you'll want another control to provide the visual boundaries.
For a simple example, add a Rectangle to the canvas with the same placement and dimensions as the InkPresenter. By default, a Rectangle will have a black border (the Stroke property) with a StrokeThickness of 1. The Rectangle, which I've named "inkBorder," and InkPresenter should be siblings in the Canvas. The z-Order of the InkPresenter must be higher (on top of) the Rectangle—otherwise the Rectangle will hide the InkPresenter.
Because it is difficult to find the InkPresenter control on the design surface, when you need to select it, the easiest way is to select it in the Objects and Timeline panel. This action will then highlight the control on the design surface. If you were to test the solution so far (by pressing F5), you would see the Rectangle border, but if you tried to draw inside of it with a mouse or a stylus, nothing would happen yet.

Adding Events and a Background
As I mentioned above, the InkPresenter is merely a container for Stroke Collections. On its own, it does not have the ability to create strokes. You will have to do that programmatically by responding to events on the InkPresenter. The key events for capturing strokes are MouseLeftButtonDown, MouseMove, and MouseLeftButtonUp. When the InkPresenter receives a MouseLeftButtonDown event, you will need to create a new stroke in memory and add it to the InkPresenter's StrokeCollection. As the mouse moves around inside the InkPresenter creating MouseMove events, you will need to add StylusPoints to that Stroke. When the user causes the MouseLeftButtonUp event to fire, either by lifting the stylus from the digitizer or by releasing the mouse, you will then need to complete the stroke.
Expression Blend 2.5 makes it easy to wire up events to controls using the Properties window. Select the inkP control, then click the event icon in the top of the Properties window to see the control's events. Next, double-click the text box for each of the three aforementioned event handlers. For each event handler, a corresponding method will be added to the codebehind of the XAML control in Visual Studio. The integration between Visual Studio and Expression Blend works in both directions. Visual Studio will require you to confirm each change as it is made; so watch for the blinking Visual Studio icon on the taskbar.
Once the three handlers have been created, switch to Visual Studio to add the code in Figure 5 that will allow the InkPresenter to collect and display stroke data as the user interacts with the control. This code performs the tasks laid out previously: creating a new stroke and adding stylus points. The stylus points are accessed through the MouseEventArgs of both the MouseLeftButtonDown and MouseMove events.
Figure 5 Collect and Display Stroke Data
System.Windows.Ink.Stroke newstroke;

void inkP_MouseLeftButtonDown(object sender, MouseEventArgs e)
{
  inkP.CaptureMouse();
  newStroke = new System.Windows.Ink.Stroke();
  newStroke.StylusPoints.Add(e.StylusDevice.GetStylusPoints(inkP));
  inkP.Strokes.Add(newStroke);
}
void inkP_MouseMove(object sender, MouseEventArgs e)
{
  if (newStroke != null)
  {
    newStroke.StylusPoints.Add(e.StylusDevice.GetStylusPoints(inkP));
  }
}
void inkP_MouseLeftButtonUp (object sender, MouseEventArgs e)
{
  newStroke = null;
  inkP.ReleaseMoustCapture();
}
There is one more piece to this puzzle. The InkPresenter must have a Background property in order to receive the mouse events. The background is similar to the Fill property of other controls but without the additional customization that you can perform on Fill. Depending on your design scenario, you could get creative with the background, but for now, simply set the Background property to "Transparent."
You are able to set this background in Expression Blend using the Properties window for the InkPresenter control by choosing a solid color brush for the background and then setting its Alpha value to 0. As an alternative to this, you can simply type Background="Transparent" directly into the XAML.
The following is the XAML for the two controls after the events have been wired up and the Background property has been designated:
<Rectangle Margin="20,30,35,24"
  x:Name="inkBorder" Stroke="#FF000000"/>
<InkPresenter Margin="20,30,35,24"
  x:Name="inkP"
  MouseLeftButtonDown=
    "inkP_MouseLeftButtonDown"
  MouseLeftButtonUp=
    "inkP_ MouseLeftButtonUp"
  MouseMove="inkP_MouseMove"
  Background="Transparent"
  Opacity="1"/>
Now when you run the project from Expression Blend or from Visual Studio, you can see the strokes on the InkPresenter as they are drawn (see Figure 6).
Figure 6 Strokes on the InkPresenter

A More Stylish InkPresenter
As you have seen, the InkPresenter is a container, but more like a canvas than other visual elements, such as Rectangle. You will need to combine the InkPresenter with other elements in order to make it visually interesting; otherwise you would be doing Silverlight an injustice. Therefore, open up the existing XAML in Expression Blend so you can spruce things up a bit more by using the Rectangle that you added earlier to create the border for the InkPresenter.
First let's round out the corners of the border Rectangle. Select the Rectangle and change its RadiusX and RadiusY properties to 25. The Rectangle now has nice rounded edges; however, the InkPresenter's corners stick out of the visual border and will accept ink. The solution is to change, or clip, the InkPresenter's boundary to match the visual border. You can accomplish this using the Silverlight clipping feature to reshape the InkPresenter.
Expression Blend makes it easy to clip an element to match the shape of another element. Before doing this, however, create a copy of the inkBorder Rectangle for later use. In the Objects and Timeline window, right-click the new Rectangle. From its context menu, select Path and then Make Clipping Path. Expression Blend will then pop up a window telling you to choose which object will be clipped by the path—in other words, which object will adopt the shape of the Rectangle. Select the InkPresenter. Two things happen as a result of this: the InkPresenter now has the shape of the Rectangle and the new Rectangle is gone. When the Rectangle became the clipping path for the InkPresenter, it ceased to be an object. Now you see why you copied the Rectangle.
The resulting XAML looks like Figure 7. Run the project to test the new edge of the InkPresenter. It will look like Figure 8. You can use any shape as a clipping path in Silverlight and thus you can create any shape you like for an InkPresenter. Figure 9 shows an example of a random shape being used to clip an InkPresenter.
Figure 7 XAML for the Clipped Object
<Rectangle x:Name="inkBorder" Width="346" Height="234"
  Stroke="#FF000000" Canvas.Top="25" Canvas.Left="25"
  RadiusX="25" RadiusY="25"/>
<InkPresenter x:Name="inkP"
  Width="607" Height="408" Canvas.Left="25" Canvas.Top="34"
  MouseLeftButtonDown="inkP_MouseLeftButtonDown"
  MouseLeftButtonUp="inkP_MouseLeftButtonUp"
  MouseMove="inkP_MouseMove"
  Background="Transparent"
  Clip="M0.5,25.5 C0.5,11.692881 11.692881,0.5 25.5,0.5 L581.5, 0.5 C595.30712,0.5 606.5,11.692881 606.5,25.5 L606.5, 382.5 C606.5, 396.30712 595.30712,407.5 581.5,407.5 L25.5,407.5 C11.692881, 407.5 0.5,396.30712 0.5,382.5 z" >
</InkPresenter>
Figure 8 Clipping the Edges of the InkPresenter
Figure 9 This Randomly Drawn Shape Was Used to Clip an InkPresenter

Getting That Silverlight Look
Silverlight allows you to create interesting visual layers using transparency. Your InkPresenter can have the look of a semi-transparent surface as well. To achieve this effect, you should start by adding a background image to the canvas. I've used the background from the Silverlight.net Web site (see silverlight.net). You can easily set the background by dragging the image onto the canvas and setting its Stretch property to Fill. It's important to make sure that the z-Order of the Image is the first control listed in the design surface. Otherwise it will be positioned on top of the Rectangle and the InkPresenter, hiding both.
Next, modify the Rectangle to give it a black background. One way to do this is to select the Fill Brush in the Properties window and set its R, G, and B values to 0. Change the Opacity of the Rectangle to 10 percent to make this semi-transparent.
While it is possible to set the Background color on the InkPresenter and give it transparency, this transparency will impact the ink as well. I prefer to leave the Background of the InkPresenter completely transparent and use some other control to provide the effect. You may find it interesting to experiment with changing the Alpha value of the control's Background color and compare it to the effect of changing the control's Opacity. You can also impact the ink's transparency directly using the Alpha property of the DrawingAttribute's Color and OutlineColor properties. This has the exact same effect as the DrawingAttribute.Transparency property that exists in the WPF InkCanvas and in the Tablet PC SDK. Figure 10 shows the InkPresenter combined with a semi-transparent Rectangle to provide a nice visual effect for your drawing background.
Figure 10 Creating a Semi-Transparent Background

Add a Background Image or Video to the InkPresenter
The semi-transparent background is a very appealing solution for some scenarios, but it is also possible to use images and even videos as the background. To create such a background, the actual Background property of the InkPresenter does not change. Images or videos are added as child elements of the InkPresenter object. If the child element has no height, width, left, or top properties, it will inherit the properties of the parent InkPresenter.
Give it a try—add an image element to your XAML design surface using Expression Blend and then drag that image into the InkPresenter. Alternatively, you can add the XAML directly:
<InkPresenter x:Name="inkP"
  Width="607" Height="426" Canvas.Left="25" Canvas.Top="34"
  MouseLeftButtonDown="inkP_MouseLeftButtonDown"
  MouseLeftButtonUp="inkP_MouseLeftButtonUp"
  MouseMove="inkP_MouseMove"
  Background="Transparent">
  <Image Source="Assets/Leaves.jpg" Stretch="Fill" />
</InkPresenter>
Now it's possible to draw directly on the image. Alternatively, you could lower the Opacity value of the image to make it semi-transparent as well, as you can see in Figure 11.
Figure 11 Creating Semi-Transparency Using the Opacity Property
Adding a video is just as easy. Rather than an Image, you will use a MediaElement. Expression Blend treats videos differently than images. While you can drag and drop the video from the Silverlight project onto the XAML design surface, the file won't be found when you run the project. Instead, you will need to put the video inside the host Web project. Then the source attribute for the MediaElement needs to refer to the URL of the file. The Media Element will not automatically fill the InkPresenter as the Image did. You will have to manually adjust the size or add Stretch="Fill" directly into the XAML. Here's an example of a video that plays in the background of an InkPresenter:
<InkPresenter x:Name="inkP" . . .   >
  <MediaElement Height="246" x:Name="Butterfly_wmv" Width="345"
    Source="http://localhost:52476/MSDNMagAnnotationClient_Web/Assets/Butterfly.wmv"
    Stretch="Fill"/>
</InkPresenter>
You can learn more about using and interacting with MediaElements in the Silverlight documentation.

Stroke Design Basics
By default, the stroke's default drawing attributes create a black stroke with a height and width of 3. The value represents Device Independent Pixels (DIPs) and cannot be set to a value that is smaller than 2.
In the code, you can create methods and event handlers to impact the various stroke attributes such as penWidth and penColor. For example, a variable called currentColor can have its value changed by a control's click event and then currentColor can be used in the inkP_MouseLeftButtonDown event when a new stroke is created.
To try this out, add a variable declaration to the class, setting the default value to black, like so:
System.Windows.Media.Color currentColor =        Colors.Black;
In the following example, I created a single method that can be used as the MouseLeftButtonDown event for any number of colored Rectangles. The method will determine the color of the Rectangle, then use the color as the value of the currentColor variable:
private void ChangeColor(object sender, MouseButtonEventArgs e)
{
  Rectangle rec = (Rectangle)sender;
  SolidColorBrush scb = (SolidColorBrush)rec.Fill;
  currColor = scb.Color;
}
In the inkP_MouseLeftButtonDown method, add code to set the DrawingAttributes to the currentColor variable:
newStroke.DrawingAttributes.Color = currentColor;
Finally, you'll need a way to trigger the change. Add two Rectangle elements setting the Fill of one to black and the Fill of the other to red. Each Rectangle will need a MouseLeftButtonDown event to call the ChangeColor method. This XAML creates two Rectangle objects that render as circles:
<Rectangle MouseLeftButtonDown="ChangeColor"
  Width="24" Height="22" Fill="#FF000000"
  Stroke="#FF000000" RadiusX="25" RadiusY="25"
  Canvas.Left="-10" Canvas.Top="282"/>
<Rectangle MouseLeftButtonDown="RedInk"
  Width="24" Height="22"
  Fill="#FFCE0C0C" Stroke="#FF000000"
  RadiusX="25" RadiusY="25"
  Canvas.Left="25" Canvas.Top="282"/>
Alternatively, you could write code that drills into a StrokeCollection in order to change the color of the existing strokes.
One of the DefaultDrawingAttributes for the Stroke object is OutlineColor. If you plan to draw on a multicolor background, such as an image, it is helpful to have a consistent outline color for the ink being drawn. You can add code to the inkP_MouseLeftButtonDown event to set the OutlineColor of the newStroke:
newStroke.DrawingAttributes.OutlineColor =
  Colors.White;
Figure 12 illustrates the concept.
Figure 12 Ink with a Border

Handwriting Recognition
One of the very cool features of Ink in applications is handwriting recognition. While this has been part of the Tablet PC SDK since the beginning and is a feature of WPF, it is not part of Silverlight. But you can still use handwriting recognition in your Silverlight applications by sending the stroke data to an ASP.NET Web Service or a WCF Service which will perform the recognition and return the results.
Microsoft Research used handwriting samples from more than one million people to create the algorithms for handwriting recognition. It may surprise you to know that the recognition works better with cursive than with block letters. There are also handwriting recognition engines for many languages including a variety of Asian languages.
The engine will analyze the collection of strokes. It can determine whether the strokes represent a word, a sentence, a paragraph, or a drawing. The recognition engine can identify individual words within a set of strokes that has multiple words and then recognize the word as a unit. Then, using its samples and algorithms, it responds with a selection of possibilities for what that word might be. If you send a set of words (such as a sentence), then it will return a set of words as well as a series of alternates. In the past, recognition was limited to Tablet PCs. Now it can be performed on any computer/browser combination that supports Silverlight!

The InkAnalyzer Class
System.Windows.Ink.InkAnalyzer performs the handwriting recognition. It's surprisingly simple to use this class. You merely pass a collection of strokes to an InkAnalyzer object, call its Analyze method, and determine if the Analyze succeeded using a Boolean property called Successful. If Successful is true, the GetRecognizedString method will return the best guess while the GetAlternates method returns an array of alternates strings.
Although the InkAnalyzer class is not part of the Silverlight API, you can still provide handwriting recognition to your Silverlight apps using Web or WCF services. The Web server can host the WPF APIs and provide the functionality for performing the recognition. This, however, will require a few conversions.
First, you will need to reference the APIs in Figure 13 in any component that will perform Ink Recognition. The first two APIs are readily available through the Add References interface in Visual Studio. The last two are installed with the Tablet PC SDK and can be found in Program Files\Reference Assemblies\Microsoft\Tablet PC\v1.7. You will want to watch out for ambiguous references between the IACore and IAWinFX namespaces when you are coding. Finally, you need to additionally reference the IALoader.dll that is installed with the Tablet PC SDK. You can find this file in C:\Program Files\Microsoft SDKs\Windows\v6.0\Bin if you're running Windows Vista.
Figure 13 WPF APIs
API | Function
PresentationCore.dll | Contains the System.Windows.Ink APIs.
WindowsBase.dll | Contains functionality used by Collections.
IAWinFX.dll | Adds the InkAnalyzer class and functionality to System.Windows.Ink.
IACore.dll | Provides the InkAnalyzerBase class and functionality via System.Windows.Ink.AnalysisCore.
In order to transmit the Strokes from the InkPresenter across the wire to a service, the strokes need to be serialized. However these objects are not serializable. Therefore, you will need to create a string-based XAML representation of the StrokeCollection. The string can then be serialized and sent to the service.
In Silverlight 1.0, you needed to use JavaScript to create this string representation. In Silverlight 2, you have the benefit of being able to use LINQ to XML. (If you are using Visual Basic, you get the added benefit here of XML literals.)
The code shown in Figure 14 drills into the StrokeCollection object, reading the DrawingAttributes, the StylusPoints and their details, and uses LINQ to XML to create its XAML representation. This code will be used for a number of purposes, including the handwriting recognition, even though the recognition functions will ignore the DrawingAttributes. If you were creating this method solely for the purpose of recognition, you could leave out the code that collects the DrawingAttributes.
Figure 14 Creating XAML Representation from Strokes
public XElement StrokestoXAML(StrokeCollection mystrokes)
{
  //this method uses LINQ to XML
  //be sure to add the namespace to each element in order to load back
  //into a new StrokeCollection later with the XAMLReader

  string xmlnsString = "http://schemas.microsoft.com/client/2007";

  XNamespace xmlns = xmlnsString;
  XElement XMLStrokes = new XElement(xmlns + "StrokeCollection",
      new XAttribute("xmlns", xmlnsString));

  //create stroke, then add to collection element
  XElement mystroke;
  foreach (Stroke s in mystrokes)
  {
    mystroke = new XElement(xmlns + "Stroke",
      new XElement(xmlns + "Stroke.DrawingAttributes",
        new XElement(xmlns + "DrawingAttributes",
           new XAttribute("Color", s.DrawingAttributes.Color),
           new XAttribute("OutlineColor",
                          s.DrawingAttributes.OutlineColor),
           new XAttribute("Width", s.DrawingAttributes.Width),
           new XAttribute("Height", s.DrawingAttributes.Height))));

    //create points separately then add to mystroke XElement
    XElement myPoints = new XElement(xmlns + "Stroke.StylusPoints");
    foreach (StylusPoint sp in s.StylusPoints)
    {
      XElement mypoint = new XElement(xmlns + "StylusPoint",
        new XAttribute("X", sp.X.ToString()),
        new XAttribute("Y", sp.Y.ToString()));
      //add the new point to the points collection of the stroke
      myPoints.Add(mypoint);
    }
    //add the new points collection to the stroke
    mystroke.Add(myPoints);
    //add the stroke to the collection
    XMLStrokes.Add(mystroke);
  }
  return XMLStrokes;
}
Notice the use of the Namespace to build the XAML. The Silverlight 2 XMLReader requires this namespace to be used in the root of a XAML element to Load the XAML into an object at a later time.

Analyzing on the Server Side
Now the string can be passed as a parameter to a service operation, which will return the results as a string. The method used by the operation must recreate a StrokeCollection from the XAML string. Then the StrokeCollection can be sent to the InkAnalyzer. On the server, you no longer have access to the Silverlight APIs. Instead, you'll be using the WPF APIs.
While WPF also has a XAMLReader.Load method, the WPF StrokeCollection is slightly different from the Silverlight StrokeCollection, so the schema will not be recognized and XAMLReader.Load will fail. Instead, you can use LINQ to XML to drill easily into the XAML string and read the data from individual Stroke elements, then build up new WPF Stroke objects.
The differences between the WPF and Silverlight StrokeCollection are minimal. For example, WPF Stroke.DrawingAtrributes do not have an Outline Color and the StylusPoints in a WPF stroke are based on a device-independent coordinate system rather than the pixel values used in Silverlight. Even with these differences, the Stroke, StrokeCollection, and StylusPoint classes are in the same namespaces in Silverlight as they are in WPF.
The CreateWPFStrokeCollectionfromXAML method in Figure 15 uses LINQ to XML to create a collection of Stroke elements, then iterates through those elements creating a new Stroke object for each one. The method uses LINQ again to create a collection of the StylusPoints and then creates the StylusPointsCollection for each stroke. Note that this is not a complete recreation of the StrokeCollection as you do not need DrawingAttributes, such as color, in order to perform the recognition. These two methods rely on the PresentationCore and WindowsBase APIs discussed previously. Once the StrokeCollection has been created, it can be passed to a method that will perform the analysis.
Figure 15 The CreateWPFStrokeCollectionfromXAML Method
private StrokeCollection CreateWPFStrokeCollectionfromXAML
  (string XAMLStrokes)
{
  //because namespace was used to create this
  // (for Silverlight to reuse the XAML),
  //you need to insert the namesace into the Descendent's parameter

  var xmlElem = XElement.Parse(XAMLStrokes);
  XNamespace xmlns = xmlElem.GetDefaultNamespace();
  StrokeCollection objStrokes = new StrokeCollection();
  //Query the XAML to extract the Strokes
  var strokes = from s in xmlElem.Descendants(xmlns+ "Stroke") select s;
  foreach (XElement strokeNodeElement in strokes)
  {
    //query the stroke to extract its StylusPoints
    var points = from p
      in strokeNodeElement.Descendants(xmlns + "StylusPoint") select p;
    //create Stylus points collection from point element values
    StylusPointCollection pointData =
      new System.Windows.Input.StylusPointCollection();
    foreach (XElement pointElement in points)
    {
      double Xpoint = Convert.ToDouble(pointElement.Attribute("X").Value);
      double Ypoint = Convert.ToDouble(pointElement.Attribute("Y").Value);
      pointData.Add(new StylusPoint(Xpoint, Ypoint));
    }
    //create a new Stroke from the StylusPointCollection
    System.Windows.Ink.Stroke newstroke = new
       System.Windows.Ink.Stroke(pointData);
    //add the new stroke to the StrokeCollection
    objStrokes.Add(newstroke);
  }
  return objStrokes;
}
The method in Figure 16 can be used in a WCF or ASMX Web service to accept a StrokeCollection from an InkPresenter and return a string representing the best guess. This functionality relies on the IAWinFX and IACore assemblies.
Figure 16 Passing String to Recognizer
public string RecognizeStrokes(string XAMLStrokes)
{
  try
  {
    //custom method to create WPF StrokeCollection from the string-based XAML
    var strokeColl = CreateWPFStrokeCollectionfromXAML(XAMLStrokes);
    var IA = new System.Windows.Ink.InkAnalyzer();
    IA.AddStrokes(strokeColl);
    var status = IA.Analyze();
    if (status.Successful)
      return IA.GetRecognizedString();
    else
      return "Not Recognized";
  }
  catch (Exception ex)
  {
    //trap and display errors at design time, not in production code
    return "error:" + ex.Message;
  }
}

Hooking Up the Service to the Silverlight Client
With the RecognizeStrokes method wrapped inside an ASMX or WCF service, you can easily call the Web service method from within the Silverlight application. I have used a WCF service to provide the handwriting recognition in my solution. If you need any help with WCF, there is a simple QuickStart on the Silverlight.net Web site that shows how to create a WCF service that can be accessed by Silverlight.
While it is common in Windows-based applications to have recognition performed on the fly as the ink is being drawn, in a distributed application it is more sensible to have the user explicitly request recognition. Therefore you will need a control on the design surface to start the process. Create a control such as a button in XAML. You'll need an event handler for the button's Click event that will trigger the GetXAMLfromStrokes method; then send the resulting XAML to a Web service for recognition. In my Web service, the operation is called Recognize. You will also need a TextBlock control to display the returned string. I've called it RecoText.
When adding WCF Service references to a Silverlight app, the proxy only implements asynchronous calls for the service operations. Therefore, you'll need a method for handling RecognizeCompleted (see Figure 17). If the annotation is written on multiple lines, it will be recognized with hard returns, so I've used a ScrollViewer control in place of a TextBox for RecoText, as you see in Figure 18.
Figure 17 Recognize Operations
{
  StrokeCollection sc = inkP.Strokes;
  XElement sXML = StrokestoXAML(sc,true);
  string ss = sXML.ToString();
  GetRecoString(ss);

  //the next two lines are to test the validity of XAML created
  //(this would make a great unit test for this application)
  //StrokeCollection sc2 = new StrokeCollection();
  //sc2 = (StrokeCollection)System.Windows.Markup.XamlReader.Load(ss);
}

private void GetRecoString(string inkStrokes)
{
  Binding binding = new BasicHttpBinding();
  EndpointAddress endpoint = new
    EndpointAddress("http://myserver/SilverlightInkService.svc");
  var svc = new ServiceReference1.SilverlightInkServiceClient(binding,
    endpoint);
  svc.RecognizeCompleted += new
    EventHandler<ServiceReference1.RecognizeCompletedEventArgs>
    (svc_RecognizeCompleted);
  svc.RecognizeAsync(inkStrokes);
}

private void svc_RecognizeCompleted(object sender,
  ServiceReference1.RecognizeCompletedEventArgs e)
{
  RecoText.Text = e.Result.ToString();
}
Figure 18 Using a ScrollViewer Control for Recognized Text
For a better testing experience, you may want to add one more button on the XAML page that will clear the ink and the recognized text so that you can experiment with a variety of annotations. As its name indicates, InkPresenter.Strokes.Clear will remove the strokes from the InkPresenter.

Persisting Annotations Using Services
Another important use of Web services in this solution is for persisting annotations. Whether the StrokeCollection is handwritten text, a drawing, or some other markup, in most cases, you will want to retain it in a database or some other type of data store. It is also possible to save the annotation data on the user's computer using IsolatedFileStorage which is available in Silverlight. However, in this article I will focus on server-side persistence.
In the case of persistence, most often you will save the entire StrokeCollection, which can be added to another InkPresenter at any time. And as I've mentioned, because the StrokeCollection object must be serialized, the simplest method is to create the XAML string representation using the GetXAMLfromStrokes method.
With the XAML string in hand, the rest of the job is no different than storing any other text in a database. Keep in mind that the XAML could get quite large in the case of drawings, so you will have to consider that possibility when defining the database field that will store the XAML as well as client and service settings to accommodate the transfer of large amounts of data.
With WCF, you may also want to consider using JavaScript Object Notation (JSON) serialization, which is a more compressed format than XML serialized data. While it is possible to take advantage of the XML data type in SQL Server® 2005 and later, unless you plan to leverage the benefits of the XML data type (which can be queried and indexed, has schema support, and allows for data modification) the SQL Server types such as nvarchar or even nvarchar(MAX) for anticipated large drawings should do the trick.
In the sample app, users will have a variety of images to select from. The annotations for each image will be stored in a database with the image's unique file name. When an image is selected, its annotation can be retrieved from the database and displayed.

Store and Retrieve Annotations
In the case of the recognition, only a single string was being passed to and from the service. However, when storing and retrieving the data, you will be passing in the XAML and possibly other metadata relating to the annotation such as date created, the user, or references to an image or video to which the annotation belongs. WCF manages these different components of a message with DataContracts. Using the example of an annotation that belongs to an image file I'll create an operation that can store the path of an image along with the XAML and then retrieve the XAML for a specified file path.
This WCF service uses a combination of interfaces and attributes to define the operation and the data contracts. You can see in Figure 19 that three separate service operations are defined: StoreImageXAML, RetrieveImageXAML, and Recognize. The StoreImageXAML takes a parameter of type ImageXAMLComposite, which is defined by the ImageXAMLComposite class as a DataContract. The other two operations are much less complex as they only accept and return a single string. The service class implements these three methods which call helper methods for the recognition and database interaction. The methods, which work with the database, use LINQ to SQL (see Figure 20).
Figure 19 Defining OperationContracts
namespace SilverlightInkWCFService
{
  [ServiceContract]
  public interface ISLInk
  {
    [OperationContract]
    void StoreImageXAML(ImageXAMLComposite value);

    [OperationContract]
    string RetrieveImageXAML(string imageName);

    [OperationContract]
    string Recognize(string XAMLString);
  }

  [DataContract]
  public class ImageXAMLComposite
  {
    string imagePath;
    string xamlString;

    [DataMember]
    public string XAMLString
    {
      get { return xamlString; }
      set { xamlString = value; }
    }

    [DataMember]
    public string ImagePath
    {
      get { return ImagePath; }
      set { ImagePath = value; }
    }
  }
}
Figure 20 Calling Recognition Helper Methods
public void StoreImageXAML(ImageXAMLComposite value)
{
  //ExistingRows, InsertRow and updateRow use LINQ to the SQL
  //SubmitChanges
  if (ExistingRows(value.ImagePath) == true)
    insertRow(value.ImagePath, value.XAMLString);
  else
    updateRow(value.ImagePath, value.XAMLString);
}

public ImageXAMLComposite RetrieveImageXAML(string imagePath)
{
  //getXAML method performs a LINQ to SQL query
  return getXAML(imagePath);
}

public string Recognize(string XAMLString)
{
  return RecognizeStrokes(XAMLString);
}

Putting It All Together
Here I've covered all of the critical pieces for creating and interacting with an InkPresenter, performing handwriting recognition, and storing and retrieving the XAML representation of the annotations through services. If you do not have a Tablet PC or are not running Windows Vista, you will still be able to use the application, even with a mouse, though the resolution won't be as fine.
Another interesting possibility is to use videos. While you can use the Silverlight animation and triggers to play back strokes, coordinating the timing with a video will be challenging, but fun.
You can download Silverlight, the SDK, Silverlight Tools Beta2 for Visual Studio 2008 and Expression Blend 2.5 June 2008 Preview from go.microsoft.com/fwlink/?LinkId=122132.
Special thanks to Stefan Wick from Microsoft, who offers help in the Notebook, Tablet PC, and UMPC Development MSDN® forum. He provided important guidance for this article.

Julia Lerman, a .NET consultant, has been building software for more than 20 years. She is well-known in the .NET community as a conference speaker, author, Microsoft.NET MVP, and leader of the Vermont .NET User Group. Her upcoming book is titled Programming Entity Framework. Julia blogs at thedatafarm.com/blog.


Wicked Code
Craft Custom Controls for Silverlight 2
Jeff Prosise

Code download available at: WickedCode2008_08.exe (585 KB)
Browse the Code Online
This article discusses:
  • The WPF control model
  • Creating control templates
  • Deriving controls
  • Adding events
This article uses the following technologies:
Silverlight 2
This article is based on the Beta 2 version of Silverlight 2. All information herein is subject to change.
Contents
Step 1: Create a New Silverlight Project
Step 2: Derive from Control (or ContentControl)
Step 3: Create a Control Template
Step 4: Create a Default Control Template
Step 5: Add Template Bindings
Step 6: Replace TextBlock with ContentPresenter
Step 7: Add a Click Event
Step 8: Add Visual States
The Completed Control
One of the many features that distinguishes Silverlight™ 2 from Silverlight 1.0 is support for controls. Silverlight 2 features a rich and robust control model that is the basis for the controls included in the platform and for third-party control packages. You can also use this control model to build controls of your own. But for developers unfamiliar with the Windows® Presentation Foundation (WPF) control model, building that first Silverlight custom control can be a daunting experience. At the time of this writing—just before the release of Silverlight 2 Beta 2—there was precious little documentation available, and a quick Web search turned up few tutorials to show the way. While I'm on that topic, it's a good time to note that although I am using Beta 2, there will likely be further changes before the final release.
When learning to write custom controls for a new platform, I often begin by trying to duplicate some of the built-in controls: buttons, listboxes, and so on. They may seem simple on the outside, but controls such as these invariably expose key features of the control model and test one's knowledge of the same. Plus, you can't build a super-duper-multicolor-multithreaded-all-in-one-do-it-all widget control if you can't first build a simple push button.
The best way to learn about Silverlight 2 custom controls is to build one—step by step, piece by piece. You not only learn about the parts that make up a control, but how the parts fit together. The following tutorial describes how to build a SimpleButton control that duplicates important aspects of the built-in Button control's look and behavior and provides a helpful first look at control development, Silverlight style.

Step 1: Create a New Silverlight Project
The first step in creating a custom control is to fire up Visual Studio® 2008 (make sure you've installed the Silverlight add-in for Visual Studio so you can create Silverlight projects) and create a project. Normally you'd create a Silverlight class library project so the control can be compiled into its own assembly and added as a reference to projects that will use it. I'll take a slightly different path and create a Silverlight application project so the control can be built and used from within the same project. Therefore, let's begin by creating a new Silverlight application project named SimpleButtonDemo, as shown in Figure 1. Answer "yes" when Visual Studio offers to create a Web project to go with the Silverlight project.
Figure 1 Creating the SimpleButtonDemo Project

Step 2: Derive from Control (or ContentControl)
The next step is to add a C# class representing the control. At a minimum, the control class should derive from the Silverlight System.Windows.Controls.Control class in order to inherit basic control functionality. However, it can also derive from Control derivatives such as ContentControl and ItemsControl. Many of the built-in controls derive directly or indirectly from ContentControl, which adds a Content property that allows the control's content—for example, the content on the face of a push button—to be customized. The ListBox control, in contrast, derives from ItemsControl, which implements the basic behavior of a control that presents a collection of items to the user. We'll derive from ContentControl since we're implementing a button.
Use the Visual Studio Add New Item command to add a new C# class to the SimpleButtonDemo project. Name the file SimpleButton.cs. Then open SimpleButton.cs and modify the SimpleButton class so that it derives from ContentControl:
namespace SimpleButtonDemo
{
    public class SimpleButton : ContentControl
    {
    }
}
At this point, you've implemented a bare-bones custom control—one that can be instantiated with a declaration in a XAML document. To demonstrate, add the following statement to Page.xaml:
<custom:SimpleButton />
In order for Silverlight to make sense of this declaration, you also need to add the following attribute to Page.xaml's root UserControl element:
xmlns:custom="clr-namespace:SimpleButtonDemo; assembly=SimpleButtonDemo"
As you can see here, clr-namespace identifies the namespace in which the SimpleButton class is defined, and assembly identifies the assembly containing the control. In this example, the control assembly and the application assembly are one and the same. If SimpleButton were implemented in a separate assembly named MyControls.dll, you'd set assembly equal to "MyControls" instead. The code in Figure 2 shows the contents of Page.xaml after these modifications are made. Incidentally, you don't have to use custom as the prefix for custom controls; you could just as easily use foo or your company name instead.
Figure 2 Page.xaml
<UserControl x:Class="SimpleButtonDemo.Page"
    xmlns="http://schemas.microsoft.com/client/2007"
    xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
    xmlns:custom="clr-namespace:SimpleButtonDemo;assembly=SimpleButtonDemo"
    Width="400" Height="300">

    <Grid x:Name="LayoutRoot" Background="White">
      <custom:SimpleButton />
    </Grid>

</UserControl>
You can see what your efforts have wrought so far by launching either of the test pages (SimpleButtonDemoTestPage.aspx or SimpleButtonDemoTestPage.html) within the SimpleButtonDemo_Web project that Visual Studio added to the solution. Figure 3 shows how SimpleButtonDemoTestPage.html looks in the browser. Granted, there's not much to write home about just yet, but that will change in the next step.
Figure 3 Presenting the SimpleButton Control

Step 3: Create a Control Template
The reason you found yourself staring at a blank browser window in the previous step is that even though SimpleButton was instantiated, it didn't render a UI. You can remedy that by modifying the SimpleButton declaration in Page.xaml to include a control template. The code in Figure 4 shows the modified control declaration.
Figure 4 Modified Control Declaration
<custom:SimpleButton>
  <custom:SimpleButton.Template>
    <ControlTemplate>
      <Grid x:Name="RootElement">
        <Rectangle x:Name="BodyElement" Width="200" Height="100"
          Fill="Lavender" Stroke="Purple" RadiusX="16" RadiusY="16" />
        <TextBlock Text="Click Me" HorizontalAlignment="Center"
          VerticalAlignment="Center" />
      </Grid>
    </ControlTemplate>
  </custom:SimpleButton.Template>
</custom:SimpleButton>
The declaration now initializes the control's Template property, which defines the control's visual tree, to include a Rectangle and a TextBlock positioned inside a 1-row, 1-column Grid. Launch SimpleButtonDemoTestPage.html in your browser again and the output will look very different (see Figure 5). SimpleButton now has a face!
Figure 5 The SimpleButton Control

Step 4: Create a Default Control Template
It's unreasonable to require developers using your control to define their own control templates. A custom control should have a default template so even a simple declaration like the one in Figure 2 will display something on the page. Providing a default template doesn't preclude someone from overriding it with a template like the one in Figure 4, but it does make your control a lot friendlier by removing the requirement that a template be provided.
The mechanism used to define a default template for a custom control is borrowed from WPF. You begin by adding a file named Generic.xaml to the control project. (Yes, it has to be named Generic.xaml. Case doesn't matter, but the name itself does.) Then you define a style in Generic.xaml that uses a property setter to assign a value to the control's Template property. The Silverlight runtime automatically looks for Generic.xaml in the control assembly (where it is embedded as a resource) and applies the style to control instances. In addition to defining a default template, the style also can assign default values to other control properties such as Width and Height.
To see for yourself, use the Visual Studio Add New Item command to add a text file named Generic.xaml to the SimpleButtonDemo project. Then replace the contents of Generic.xaml with the code in Figure 6. Generic.xaml now includes an unnamed style that is applied to all instances of SimpleButton (note the TargetType attribute). That style includes a default value for SimpleButton's Template property that is identical to the Template value explicitly assigned to the control in Figure 5.
Figure 6 Generic.xaml
<ResourceDictionary
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:custom="clr-namespace:SimpleButtonDemo;assembly=SimpleButtonDemo">
  <Style TargetType="custom:SimpleButton">
    <Setter Property="Template">
      <Setter.Value>
        <ControlTemplate TargetType="custom:SimpleButton">
          <Grid x:Name="RootElement">
            <Rectangle x:Name="BodyElement" Width="200" Height="100"
              Fill="Lavender" Stroke="Purple" RadiusX="16" RadiusY="16" />
            <TextBlock Text="Click Me" HorizontalAlignment="Center"
              VerticalAlignment="Center" />
          </Grid>
        </ControlTemplate>
      </Setter.Value>
    </Setter>
  </Style>
</ResourceDictionary>
With a default template in place, go back to SimpleButton.cs and add the following statement to the class constructor:
this.DefaultStyleKey = typeof(SimpleButton);
Then open Page.xaml and modify the control declaration so that it looks again like this:
<custom:SimpleButton />
Launch the test page in your browser and the control should look exactly as it did before. But this time, obtaining that look was much simpler.

Step 5: Add Template Bindings
One problem with SimpleButton as it exists right now is that property values assigned to the control aren't honored by the control template. In other words, if the control were declared as follows, it would still have a width of 200 and height of 100 because these values are hardcoded into the control template:
<custom:SimpleButton Width="250" Height="150" />
One of the most important features of Silverlight 2 from a control developer's perspective is template binding. Template bindings allow property values assigned to a control to flow down into the control template and are declared in XAML using the {TemplateBinding} markup extension. Rather than define the Width and Height properties of the Rectangle that forms the body of SimpleButton using hardcoded values like this:
Width="200" Height="100"
You should define them this way instead:
Width="{TemplateBinding Width}" Height="{TemplateBinding Height}"
Now the width and height assigned to the control will also be the width and height assigned to the Rectangle.
Figure 7 shows a revised version of Generic.xaml that assigns default values to the Width, Height, and Background properties inherited from the base class, and that uses template bindings to reference these property values in the control template.
Figure 7 Generic.xaml Revised
<ResourceDictionary
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:custom="clr-namespace:SimpleButtonDemo;assembly=SimpleButtonDemo">
  <Style TargetType="custom:SimpleButton">
    <Setter Property="Width" Value="200" />
    <Setter Property="Height" Value="100" />
    <Setter Property="Background" Value="Lavender" />
    <Setter Property="Template">
      <Setter.Value>
        <ControlTemplate TargetType="custom:SimpleButton">
          <Grid x:Name="RootElement">
            <Rectangle x:Name="BodyElement"
              Width="{TemplateBinding Width}"
              Height="{TemplateBinding Height}"
              Fill="{TemplateBinding Background}"
              Stroke="Purple" RadiusX="16" RadiusY="16" />
            <TextBlock Text="Click Me"
              HorizontalAlignment="Center"
              VerticalAlignment="Center" />
          </Grid>
        </ControlTemplate>
      </Setter.Value>
    </Setter>
  </Style>
</ResourceDictionary>
Test the modified control template by editing the control declaration in Page.xaml as follows:
<custom:SimpleButton Width="250" Height="150" Background="Yellow" />
The output is illustrated in Figure 8. The TemplateBindings are a huge, important step in the right direction because now you have instances of SimpleButton honoring the property values that you assign to them.
Figure 8 The Improved SimpleButton Control

Step 6: Replace TextBlock with ContentPresenter
The fact that SimpleButton derives from ContentControl means it has a Content property that developers can use to customize the content on the face of the button—at least in theory. The built-in Button control can be customized with the XAML in Figure 9 to produce the look shown in Figure 10.
Figure 9 Customizing the Button Control
<Button Width="250" Height="150">
  <Button.Content>
    <StackPanel Orientation="Horizontal" HorizontalAlignment="Center">
      <Ellipse Width="75" Height="75" Margin="10">
        <Ellipse.Fill>
          <RadialGradientBrush GradientOrigin="0.25,0.25">
            <GradientStop Offset="0.25" Color="White" />
            <GradientStop Offset="1.0" Color="Red" />
          </RadialGradientBrush>
        </Ellipse.Fill>
      </Ellipse>
      <TextBlock Text="Click Me" VerticalAlignment="Center" />
    </StackPanel>
  </Button.Content>
</Button>
Figure 10 Button with Customized Content
But try the same with SimpleButton and you'll quickly find that the content is still simple text. In fact, you can't even set Content="Test" to change the button text because the control template currently contains a hardcoded TextBlock with hardcoded text.
You can fix this deficiency by replacing the TextBlock in SimpleButton's default control template with a ContentPresenter, as shown in Figure 11. A TextBlock can only render text, but a ContentPresenter can render any XAML assigned to the control's Content property. With these changes in place, the XAML SimpleButton declaration in Figure 12 produces the output in Figure 13. SimpleButton now supports two levels of customization. Its entire visual tree can be redefined with a custom template, or just its content can be redefined using the Content property. Moreover, you can change the button text with a simple Content attribute. SimpleButton is beginning to behave more and more like a real button control.
Figure 11 Generic.xaml Once Again
<ResourceDictionary
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:custom="clr-namespace:SimpleButtonDemo;assembly=SimpleButtonDemo">
  <Style TargetType="custom:SimpleButton">
    <Setter Property="Width" Value="200" />
    <Setter Property="Height" Value="100" />
    <Setter Property="Background" Value="Lavender" />
    <Setter Property="FontSize" Value="11" />
    <Setter Property="Template">
      <Setter.Value>
        <ControlTemplate TargetType="custom:SimpleButton">
          <Grid x:Name="RootElement">
            <Rectangle x:Name="BodyElement"
              Width="{TemplateBinding Width}"
              Height="{TemplateBinding Height}"
              Fill="{TemplateBinding Background}"
              Stroke="Purple" RadiusX="16" RadiusY="16" />
            <ContentPresenter Content="{TemplateBinding Content}"
              HorizontalAlignment="Center" VerticalAlignment="Center"
              FontSize="{TemplateBinding FontSize}" />
          </Grid>
        </ControlTemplate>
      </Setter.Value>
    </Setter>
  </Style>
</ResourceDictionary>
Figure 12 SimpleButton with Custom Content
<custom:SimpleButton Width="250" Height="150">
  <custom:SimpleButton.Content>
    <StackPanel Orientation="Horizontal" HorizontalAlignment="Center">
      <Ellipse Width="75" Height="75" Margin="10">
        <Ellipse.Fill>
          <RadialGradientBrush GradientOrigin="0.25,0.25">
            <GradientStop Offset="0.25" Color="White" />
            <GradientStop Offset="1.0" Color="Red" />
          </RadialGradientBrush>
        </Ellipse.Fill>
      </Ellipse>
      <TextBlock Text="Click Me" VerticalAlignment="Center" />
    </StackPanel>
  </custom:SimpleButton.Content>
</custom:SimpleButton>
Figure 13 SimpleButton with Customized Content

Step 7: Add a Click Event
The next step in bringing SimpleButton to life is to have it fire Click events. Implementing events in Silverlight controls is generally no different than implementing events in ordinary Microsoft® .NET Framework classes: you simply declare the events in the control class and then write code to fire them.
One aspect of event firing that is slightly different in Silverlight, however, is the fact that Silverlight supports routed events. In WPF, event routing has a richer connotation in that events can travel up and down the visual tree; in Silverlight, events only travel up, an action known as "bubbling." Routed events are defined by the RoutedEventHandler delegate, and routed event handlers receive a RoutedEventArgs object that contains a Source property identifying the object that fired the event (which will be different from the sender parameter passed to the event handler if the event was originally fired by an object deeper in the visual tree). The built-in Button control's Click event is a routed event, so SimpleButton's should be, too.
Figure 14 shows the modifications you must make to the control class in SimpleButton.cs to fire routed Click events. SimpleButton's constructor now registers a handler for MouseLeftButtonUp events. The handler fires a Click event provided there is at least one registered listener. Traditional button controls only fire Click events if button-up events are preceded by button-down events. That logic was omitted from SimpleButton to keep the source code as simple as possible.
Figure 14 SimpleButton with Support for Click Events
public class SimpleButton : ContentControl
{
    public event RoutedEventHandler Click;

    public SimpleButton()
    {
        this.DefaultStyleKey = typeof(SimpleButton);
        this.MouseLeftButtonUp += new MouseButtonEventHandler
            (SimpleButton_MouseLeftButtonUp);
    }

    void SimpleButton_MouseLeftButtonUp(object sender,
        MouseButtonEventArgs e)
    {
        if (Click != null)
            Click(this, new RoutedEventArgs());
    }
}
To test Click events, you can modify the control declaration in Page.xaml as follows:
<custom:SimpleButton Content="Click Me" Click="OnClick" />
Then add the event handler shown next to Page.xaml.cs. Clicking the SimpleButton should now produce an alert box containing the word "Click!"—proof positive that the event was fired and handled correctly:
protected void OnClick(Object sender, RoutedEventArgs e)
{
    System.Windows.Browser.HtmlPage.Window.Alert("Click!");
}

Step 8: Add Visual States
Two key components of Silverlight controls are visual states and visual state transitions. Visual states define a control's appearance in various states: how it looks when it's pressed, when the mouse enters it, when it's disabled, and so on. Visual state transitions define how a control transitions from one visual state to another: for example, from the "normal" state to the "pressed" state.
Silverlight 2 Beta 2 introduced a new component known as the Visual State Manager (VSM) to simplify states and state transitions and to facilitate better tool support. With the VSM, you use VisualState objects encapsulating Storyboards to define states, and VisualTransition objects to define transitions. Then you use the VisualStateManager class's static GoToState method to transition the control to the specified state in response to user stimuli.
Figure 15 shows how to modify Generic.xaml to leverage the VSM. SimpleButton defines two states: Normal and MouseOver. These are declared with <vsm:VisualState> elements. The Normal state doesn't require a Storyboard because it represents the control in its default state. The MouseOver state uses a ColorAnimation to change the control's background color—actually, the Fill property of the rectangle representing the body of the control—to pink. The ColorAnimation's Duration property is 0 because the length of the animation is driven by the Duration property of the corresponding VisualTransition. Note the xmlns:vsm attribute added to the <ResourceDictionary> element to enable declarative use of VisualStateManager.
Figure 15 The Completed Generic.xaml
<ResourceDictionary
  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"
  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"
  xmlns:custom="clr-namespace:SimpleButtonDemo;assembly=SimpleButtonDemo"
  xmlns:vsm="clr-namespace:System.Windows;assembly=System.Windows">
  <Style TargetType="custom:SimpleButton">
    <Setter Property="Width" Value="200" />
    <Setter Property="Height" Value="100" />
    <Setter Property="Background" Value="Lavender" />
    <Setter Property="FontSize" Value="11" />
    <Setter Property="Template">
      <Setter.Value>
        <ControlTemplate TargetType="custom:SimpleButton">
          <Grid>
            <vsm:VisualStateManager.VisualStateGroups>
              <vsm:VisualStateGroup x:Name="CommonStates">
                <vsm:VisualStateGroup.Transitions>
                  <vsm:VisualTransition To="Normal" Duration="0:0:0.2"/>
                  <vsm:VisualTransition To="MouseOver" Duration="0:0:0.2"/>
                </vsm:VisualStateGroup.Transitions>
                <vsm:VisualState x:Name="Normal" />
                <vsm:VisualState x:Name="MouseOver">
                  <Storyboard>
                    <ColorAnimation Storyboard.TargetName="BodyElement"
Storyboard.TargetProperty="(Rectangle.Fill).(SolidColorBrush.Color)"
To="Pink" Duration="0" />
                  </Storyboard>
                </vsm:VisualState>
              </vsm:VisualStateGroup>
            </vsm:VisualStateManager.VisualStateGroups>
            <Rectangle x:Name="BodyElement"
              Width="{TemplateBinding Width}"
              Height="{TemplateBinding Height}"
              Fill="{TemplateBinding Background}"
              Stroke="Purple" RadiusX="16" RadiusY="16" />
            <ContentPresenter Content="{TemplateBinding Content}"
              HorizontalAlignment="Center" VerticalAlignment="Center"
              FontSize="{TemplateBinding FontSize}" />
          </Grid>
        </ControlTemplate>
      </Setter.Value>
    </Setter>
  </Style>
</ResourceDictionary>
Figure 16 shows the completed SimpleButton class, now modified to use the states and transitions declared in Generic.xaml. The class itself is also decorated with [TemplateVisualState] attributes indicating which visual states the control supports. (These attributes aren't required, but they do enable tools such as Expression Blend™ to provide a richer design-time experience.) The class constructor wires MouseEnter and MouseLeave events to a pair of handlers, and the handlers transition the control state utilizing VisualStateManager.GoToState. Now the control turns pink when the mouse enters and goes back to its original color when the mouse leaves.
Figure 16 The Completed SimpleButton Class
[TemplateVisualState(Name = "Normal", GroupName = "GroupCommon")]
[TemplateVisualState(Name = "StateMouseOver", GroupName = "GroupCommon")]
public class SimpleButton : ContentControl
{
    public event RoutedEventHandler Click;

    public SimpleButton()
    {
        DefaultStyleKey = typeof(SimpleButton);
        this.MouseLeftButtonUp +=
            new MouseButtonEventHandler(SimpleButton_MouseLeftButtonUp);
        this.MouseEnter +=
            new MouseEventHandler(SimpleButton_MouseEnter);
        this.MouseLeave +=
            new MouseEventHandler(SimpleButton_MouseLeave);
    }

    void SimpleButton_MouseLeftButtonUp(object sender,
        MouseButtonEventArgs e)
    {
        if (Click != null)
            Click(this, new RoutedEventArgs());
    }

    void SimpleButton_MouseEnter(object sender, MouseEventArgs e)
    {
        VisualStateManager.GoToState(this, "MouseOver", true);
    }

    void SimpleButton_MouseLeave(object sender, MouseEventArgs e)
    {
        VisualStateManager.GoToState(this, "Normal", true);
    }
}

The Completed Control
The finished source code for SimpleButton is included with the download that accompanies this article. You could refine the control even further by implementing additional properties and state animations and perhaps sprucing up the UI with a glassed appearance. Nonetheless, SimpleButton contains all the key elements of a Silverlight control, and if you can embrace what you've read here, you should have no problem building controls of your own.

Jeff Prosise is a contributing editor to MSDN Magazine and the author of several books, including Programming Microsoft .NET. He's also cofounder of Wintellect (wintellect.com), a software consulting and education firm that specializes in the .NET Framework. Have a comment on this column? Contact Jeff at wicked@microsoft.com.


Editor's Note
More than Pretty Pictures
Howard Dierking
I'm a terrible artist. Whatever that genetic data is that allows an individual to make aesthetically pleasing decisions about things like shapes, sizes, and colors, it completely skipped over me. And to make matters worse, because I'm a musician, I am just right-brained enough to know how this lack of ability in the visual arts cripples me as a developer of software with which humans need to be able to respond and interact. So to make up for my lack of natural talent, I have recently been spending a good bit of time focusing on the area of usability and data visualization. I am continually amazed at some of the research that's being done in this area.
In the past few years, we have witnessed some terrific advances in the availability of information. Server-side technologies such as Web services, RSS, and RESTful APIs have simplified both intra- and inter-application access to data. And new technologies are continually being released to enable even more seamless, more transparent access to information (you're definitely going to want to read Elisa Flakso and Mike Flasko's article "Expose and Consume Data in a Web Services World"). Along with this increased availability of information comes the continued need to structure and present it in a way that is easily understandable by its ultimate consumers.
On the client side, technologies such as Windows Presentation Foundation (WPF), AJAX, and now Silverlight are enabling richer interactions between human and computer that are much more seamless and collaborative in nature, as opposed to those applications that felt more like inputting commands into a command processor. In fact, these types of technologies have served as a foundation for creating "mash-up" applications that, even at the presentation layer, are an aggregation of several different applications.
However, one of the challenges that we face yet again is that of consistency. In regard to Web applications, the complete and total freedom around everything related to the user experience can be a double-edged sword. And while WPF enables a shift away from this dynamic, the freedom offered to Web application developers stands in sharp contrast to the standardization, both by platform and general heuristics, applied to Windows-based applications. Although traditional Win32 applications and their Windows Forms descendants didn't have the stunning visual effects of a modern WPF application, there was predictability in terms of where UI elements were found and in how they behaved within the context of the larger application window.
Fortunately, in the world of Web applications, standard presentation patterns did start to emerge and were bolstered by technologies such as CSS and DHTML. Interestingly, it seems that the standard look for most modern Web sites, including large corporate sites, has emerged out of the standard CSS templates used in blogs. As such, one of the reasons that I am so excited about XAML, whether in the form of WPF or Silverlight, is that this declarative style of UI design enables the same maturity to develop naturally, even in rich-client applications.
As we continue to mature as application developers, we do so in a world with enormous amounts of data. And we do so with a continually evolving toolset—tools that enable us both to collect and process that data, and tools that enable us to better distill and present that data to our users. So while my sketches may always look like they were drawn by a fourth grader, I believe that I—and all of us who are artistically challenged—must learn some specific techniques for ensuring that our users are not simply impressed but also able to be more efficient.

Visit us at msdn.microsoft.com/magazine. Questions, comments, or suggestions for MSDN Magazine? Send them to the editor: mmeditor@microsoft.com.

Thanks to the following Microsoft technical experts for their help with this issue: Tim Cahill, Pablo Castro, Barak Cohen, Matthew Cox, Anish Desai, Mike Flasko, Matt Gibbs, Bertrand Le Roy, Laurence Moroney, Ari Pernick, Edward Psyk, Dave Reed, Mark Rideout, Don Smith, Stephen Toub, Stefan Wick, Matt Winkler, and Robert Wlodarczyk.


Toolbox
Serializing objects, Scott Allen's blog, Site Performance, and more
Scott Mitchell
Contents
Serialize and Deserialize Fixed-Length and Delimited Files
Blogs of Note
Build Faster Web Pages with YSlow and Firebug
From the Bookshelf

Serialize and Deserialize Fixed-Length and Delimited Files
I was recently working on a project where we needed to import several thousand records stored in a comma-delimited file into our Microsoft® .NET Framework application. Not only did our application need to import this data, but it also needed the ability to export data in the same format for use by external programs that still rely on the data in a delimited format.
If you've ever used the .NET Framework built-in XML or binary serialization capabilities, you know how easy it is to save an object's state to disk as an XML or binary file, as well as how easy it is to reload the object from disk back into memory. In a nutshell, you decorate a class with attributes, indicating how the class's properties are to be serialized. Once the class has been decorated, serializing or deserializing an object requires just one line of code.
While the .NET Framework includes serialization formatter classes for serializing objects to XML and binary formats, it does not include formatters for working with comma-delimited files. The good news is that serializing objects to fixed-length and delimited text files is as easy as XML and binary serialization thanks to FileHelpers version 2.0 by Marcos Meli.
To use FileHelpers to serialize an array of class instances to a fixed-length or delimited text file, start by decorating the class's properties with the applicable attributes. When serializing to fixed-length files, there is an attribute to indicate the length of each field; when serializing to delimited files, there's an attribute to indicate the delimiter.
There are also attributes that you can use to indicate the data type and format of the data, whether white space should be trimmed, and whether the field is optional. An appropriately decorated class is referred to as a record class. (In addition to using attributes to specify serialization specifics, FileHelpers also includes an API for programmatically defining the serialization and deserialization rules at run time.)
As you may know, creating the record class can be a tedious process. To help streamline the process and reduce the chance of a syntax error or typo, FileHelpers ships with a Record Class Wizard program. The wizard is a standalone GUI application that prompts for the precise fields in the fixed-length or delimited file and builds the corresponding record class with the appropriate properties and attributes.
As with XML and binary serialization in .NET, once you've created a record class—either by decorating its properties with attributes or by programmatically defining the record class—persisting a collection of such objects to disk or populating a collection of such objects from an existing text file is as easy as calling the library's WriteFile or ReadFile methods.
As the FileHelpers library serializes or deserializes the object or text file, it raises an event immediately before and after each record is processed. These events give you the opportunity to inspect or modify the data or cancel the operation prior to each read or write. FileHelpers includes asynchronous versions of the WriteFile and ReadFile methods, which are useful when working with especially large amounts of data. There's also a file transform engine that assists in transforming the structure of a fixed-length or delimited file.
In addition to classes for serializing the record class to disk, there are also classes for reading and writing records to and from Microsoft Access®, SQL Server®, and Microsoft Excel® data stores. Using these classes requires a bit more code than when serializing to disk. For example, when importing a record class to a SQL Server database you must specify the server's connection information and provide the INSERT statement and values. However, using these classes can shave off a lot of boilerplate data access code.
Price: Free, open source.
filehelpers.com
Serialize and Deserialize Fixed-Length and Delimited Files

Blogs of Note
A common challenge developers face when building data-driven apps centers around database versioning. After the app's initial deployment, and as new feature enhancements are being added, developers will need to make changes to the data model. What's the best way to implement these changes and propagate them through the development, testing, and deploying phases? If you're looking for guidance to project-management questions such as this one, check out the blog by author, speaker, and consultant Scott Allen.
The advice Scott dispenses comes directly from the trenches. Whether it's an entry about unit testing, shrink wrapping an ASP.NET Web application, or getting the most out of Visual Studio®, Scott shares his experiences, what he tried, what worked, what didn't, and traps to avoid. Alongside architecture and project management-focused blog entries, you'll find more nitty-gritty technical posts about LINQ, JavaScript and JavaScript Object Notation (JSON), Windows® Workflow, and Silverlight™.
My favorite aspect of Scott's blog is his ongoing "What's Wrong With This Code" series. In each installment, Scott shares a snippet of code with some faulty logic and invites his readers to try to find and fix the problem. It's a great way to hone your debugging and general problem-solving skills, as well as a way to see the techniques and thought processes used by other developers as they tackle the same problem.
odetocode.com/blogs/scott
Hone Your Debugging Skills and See Other Techniques on Scott Allen's Blog

Build Faster Web Pages with YSlow and Firebug
In the March 2008 Toolbox column I reviewed Firebug, a free, open source browser extension for Mozilla Firefox that displays information about the visited page's HTML, CSS, JavaScript, and images. Firebug makes it easy for a Web developer to visually inspect and modify the various elements of a Web page. It also displays the size and time required to download any external resources such as images or JavaScript files.
For a more detailed analysis of your page's load time, check out YSlow version 0.9.5, a Mozilla Firefox browser add-on created by Yahoo! that integrates with Firebug. Once installed, YSlow appears as a tab in the Firebug user interface. From the YSlow tab you can view detailed information about the external resources (just as you can from Firebug), statistical information about the page's components, and tips for improving the page's performance.
YSlow's Stats view breaks down the bandwidth consumed by the page's HTML, IFrame content, Flash content, JavaScript files, CSS files, and images, and sums the total size and total number of HTTP requests needed. Furthermore, it details this breakdown when the data has been cached by the browser. The cookies sent to the page, along with the sizes and contents, also are enumerated here.
The most indispensable feature of YSlow is the Performance view, which lists 13 performance tips, such as "Make fewer HTTP requests" and "gzip components." The loaded Web page is evaluated against these 13 metrics and assigned a letter grade for each. If there are specific changes that can improve the page's performance, YSlow lists them here. You can click any of these tips to be whisked to a page with a more detailed explanation of the impact on performance and information on how to implement the performance enhancement.
In addition to its Stats and Performance views, YSlow includes a number of helpful tools. From the YSlow menu you can run the page's JavaScript through Douglas Crockford's JSLint program, which parses the script for common problems. There are also tools to view the page's entire JavaScript or CSS content on a single page, regardless of how many external files into which the JavaScript and CSS are split. And there's a tool for displaying YSlow's Performance, Stats, and external resources views in a single, formatted Web page, which makes it easy to print or e-mail the results.
Price: Free.
YSlow: developer.yahoo.com/yslow
Firebug: getfirebug.com
YSlow's Detailed Analysis of Web Pages

From the Bookshelf
As a freelance developer and consultant, I work with clients who have existing apps but need help in adding new features. Some applications are written in C#, others in Visual Basic®, so I need to have a mastery of both languages. Visual Basic is easy to pick up; its prose-like syntax and straightforward semantics have helped make it one of the most popular programming languages in the world. But there's a wide gulf between knowing Visual Basic and mastering it.
To help bolster my Visual Basic skills, I picked up Accelerated VB 2008 (Apress, 2008) by Guy Fouché and Trey Nash. This book dispenses with the typical page-filling content found in most computer trade books and replaces it with distilled information and code examples that get right to the point. Because of this terse style, however, Accelerated VB 2008 is not for beginners who are interested in learning Visual Basic, but it's ideal for intermediate developers who are either looking to bolster their understanding of Visual Basic or who are in need of a good desk reference.
The first several chapters focus on core Visual Basic language concepts, including types and typing rules; classes, methods, properties, and fields; interfaces; delegates and events; and exception handling. Subsequent chapters look at working with higher-level constructs, such as strings, arrays, collections, and generics. There's also chapters on threading and LINQ.
I especially liked the Visual Basic 2008 Best Practices chapter, which offers guidance for developers creating new classes. The chapter includes discussions on designing classes that can be cloned, converted to other types, and ordered.
If you' are an intermediate developer looking to hone your Visual Basic skills, or are in need of a Visual Basic language desk reference, pick up Accelerated VB 2008.
Price: $39.99.
apress.com

Send your questions and comments for Scott to toolsmm@microsoft.com.

Scott Mitchell, author of numerous books and founder of 4GuysFromRolla.com, is an MVP who has been working with Microsoft Web technologies since 1998. Scott is an independent consultant, trainer, and writer. Reach him at Mitchell@4guysfromrolla.com or via his blog at ScottOnWriting.NET.


CLR Inside Out
Program Silverlight with the CoreCLR
Andrew Pardoe
Contents
Inside the CoreCLR Engine
The CoreCLR Security Model
The Base Class Library
Working Cross Platform
Silverlight™ 2 includes a number of changes in the Windows® Presentation Foundation (WPF) UI framework: new controls, rich networking APIs, and Digital Rights Management (DRM) support. One major change in Silverlight 2 is the ability to use Microsoft® .NET-compliant languages to program the Web client. Here I will focus on the development core of Silverlight: CoreCLR.
The past dozen or so years have given us many different Web programming technologies ranging from CSS to variants of ECMAScript. Most of them are specific to the task of Web programming—skills learned while programming CSS aren't applicable in other domains. In contrast, Silverlight 2 allows you to take the same .NET Framework skills that you use for desktop programming, such as the Base Class Libraries, XAML, and C#, and apply those skills directly to Web client applications. Moreover, we didn't have to create a separate CoreCLR development environment: you can simply use Visual Studio® to design, develop, debug and profile C# or Visual Basic® the same way you would a desktop app. We created Silverlight 2 CoreCLR precisely to make Web programming as rich as desktop programming.
While it's good for developers to have a rich programming environment, users don't want to download big browser plug-ins. In order for Silverlight to be successful with users, we had to make the installation fast. We were able to get the Beta 1 install down to 4.3MB—about 6 to 10 seconds to install over a broadband connection. This is an amazing accomplishment when you consider that the two major core pieces of the .NET Framework 2.0 CLR—mscorwks.dll and mscorlib.dll—are each about the same size as the Silverlight 2 coreclr.dll and mscorlib.dll together.

Inside the CoreCLR Engine
The design of CoreCLR began right after version 2.0 of the CLR shipped in October 2005. The two main design goals were size and compatibility: from a programmer's point of view, coding against the CLR should always be the same while from a user's point of view, the download needed to be very small. Because Silverlight is intended to target a different set of scenarios from the desktop CLR, we were able to make some changes that simplified CoreCLR and allowed us to reduce the size of the Silverlight installation. But consistency at the bottom of the stack is of key importance. Behavioral differences—even if they are correct—manifest themselves as bugs higher in the stack.
In order to ensure compatibility we used the same code for components at the bottom of the stack. The execution engine and virtual machine are the same. This includes the type system and metadata, the garbage collector (GC), the JIT compiler, and the thread pool, as well as other core parts of the runtime engine.
However, some changes were made to fit the Web application scenario. For example, because rich Internet applications are normally simple and short-running, the JIT compiler focuses on decreasing start-up time instead of performing more complex optimizations. Likewise, the server garbage-collection mode, which is tuned for multiple worker threads that use similar allocation patterns, doesn't make sense for Web-hosted applications. So Silverlight only includes the standard workstation GC, which is tuned for interactive applications. But the Microsoft intermediate language (MSIL) and metadata used in Silverlight applications is the exact same as used in managed applications for the desktop, and the behavior of your application will be consistent from the user's desktop to the browser.
The fact that Silverlight is not intended to replace the desktop CLR caused the biggest change in the core engine: CoreCLR will run side-by-side in process with the desktop CLR. In the past, we've never been able to run two versions of the CLR from within the same process. There are a few reasons this is a difficult problem. One is managing process-wide state: each instance of the CLR assumes that it's the only one in the process and thus it is the only one touching its static data. If a variable staticFoo exists in versions 1.1 and 2.0 of the CLR, and both versions of the CLR are loaded in the same process at the same time, neither version can write to the variable staticFoo without affecting the other CLR's state.
While process-wide state is the most obvious issue, other problems can result from running two CLRs side-by-side in a single process. For example, if you have two GCs running at once, how do you keep one GC from suspending the other GC's thread? Additionally, there is a problem with footprint: when you load multiple CLRs into a process they each have to load code that may be common, and they each have their own space for static variables and managed heaps.
There are some key scenarios that require the ability to host CoreCLR side-by-side with the desktop runtime. If CoreCLR and the desktop CLR couldn't run next to each other, it would be impossible to write a desktop Windows Forms or WPF app that hosts a Web Browser Control, which could navigate to a Web page that uses Silverlight. To get around this potential problem, we could have just had Silverlight depend on the CLR installed on your Windows machine: every install of Windows XP SP2 and Windows Vista® has a reasonably recent CLR installed with the OS. But having all Silverlight code run on CoreCLR guarantees absolute compatibility no matter which CLR you have installed on your machine (or in the case of Mac OS X, even if you don't have any CLR on your machine!) So we did the work to make CoreCLR run side-by-side in process with the desktop CLR, and we think that users' Silverlight experience will be much better for our efforts.

The CoreCLR Security Model
Another big change in the core engine has to do with the new security model. Note that .NET developers have traditionally used Code Access Security (CAS) to prevent untrusted code from performing privileged operations. CAS is very capable, but rather complicated. It allows the user or administrator to define various sandboxes for code using permission sets and then map individual assemblies to those sandboxes. For Silverlight applications, we just need one sandbox that's equivalent to the sandbox that Internet Explorer® uses for running script in a Web page. This simplified scenario allowed us to remove all of the CAS policy.
We also simplified the model of security enforcement. The new model is based on security transparency, a concept introduced in the version 2.0 of the CLR. At the core of the transparency model is a categorization of your code into one of three buckets: Transparent, SafeCritical, or Critical code. Transparent, the lowest trust level for code, cannot elevate privilege or access sensitive resources or information on the computer. In Silverlight 2, all application code is Transparent. Critical code, the most trusted level of code, can interact with the system through P/Invokes or even contain unverifiable code. For Silverlight 2, all Critical code must be part of the Silverlight platform. SafeCritical code, then, acts as the bridge that allows Transparent code to access system resources by calling Critical code. Think of Critical code as the kernel APIs of Windows, Transparent code as user-application code, and SafeCritical code as the API between user code and kernel code.
Transparent code can only call other Transparent or SafeCritical code. SafeCritical code can then call Critical code on behalf of the user code. It's the responsibility of SafeCritical code to canonicalize, or put into a standard format, the inputs and sanitize the outputs of the Critical code to protect the security of the system (see Figure 1).
Figure 1 Security Enforcement in the CoreCLR
The case for canonicalizing inputs to Critical code is a bit more self-evident than the case for sanitizing outputs. For example, if my Web application wants to write to a file on the local disk, it can do so using Isolated Storage. However, you don't want my application asking to write to a file named "..\..\..\..\bootmgr," so it's important to make sure the input is in a regular, canonical format. It's a little more unusual to think that the outputs of the Critical code are a security risk. The key security concept is that controlling disclosure of information is of great importance in reducing the overall surface area for an attacker. Say I try to access some bit of user information on your system and get the response "permission denied." When I repeat the same access operation, but for a different user, I get the response "user Bob does not exist." If I know that I get both responses I can repeatedly attempt invalid accesses to garner a list of user names on the system.
A simplified security policy is a clear win for developers working in .NET code, but it also helps developers working on .NET code. We've tried to mark as little code Critical and SafeCritical as necessary. Having most of our code be Transparent helps us to decrease the amount of code that needs in-depth security reviews. We still have to review our Transparent code for correctness and security, but at least we know it can't perform any privileged operations. Large pieces of Silverlight, including the Dynamic Language Runtime (DLR), are written entirely in Transparent code. Limiting the privileged parts of Silverlight allows us to ship a more secure product by focusing our attention on the areas that really need careful review.

The Base Class Library
The .NET Framework has evolved on the desktop to address both user and server scenarios. Therefore, there's a lot of functionality in the Base Class Library (BCL) that doesn't make sense in Web client scenarios. For example, because Silverlight doesn't support CAS, much of System.Security is not necessary. Many other classes, like System.Console, don't make sense on the Web. (Why, then, do we include a stripped-down System.Console class? It helps us test the product.)
We had the same goal with the libraries that we had with the core engine: to pare down to the smallest set of functionality that would enable .NET developers to be successful without having to learn an entirely new technology. We garnered some inspiration and guidance from the .NET Compact Framework, which had dealt with the same problem applied to a different scenario. While trimming the BCL for Silverlight we also maintained compatibility between the .NET Compact Framework and Silverlight. Sharing a single library between all the platforms in this way maximizes the reusability of .NET skills.
There are a number of places in the BCL where you can find duplicated functionality. Sometimes the functionality is duplicated within the BCL itself—such is the case with generic collections and non-generic collections. Sometimes the functionality already exists in the base OS, as in the case of globalization support. Not only is it unnecessary to support every alternative in the Silverlight BCL, but we also can provide wins in performance and consistency by eliding this duplicated behavior.
Since we introduced support for generic collections in version 2.0 of the .Net Framework, we've been advocating that people move to generics. In version 1.x of the runtime, general-purpose data structures had to be based on objects so that the same core data structure class could be used to create collections of different types. With generic-type parameters, the compiler is able to extend these general-purpose data structures to provide type safety, making code easier to write and maintain. Additionally, generic collections generally perform better on value types than non-generic collections because there's no need to box items. Overall, generics provide all the functionality that non-generic collections provide. And because the duplication is unnecessary, we don't include non-generic collections like ArrayList in the Silverlight BCL.
Everyone is familiar with at least some of the problems of globalization: many European cultures use a comma as a decimal point; in Chinese numbers are grouped into fours (1000,0000), and so forth. The .NET Framework implements globalization functionality internally so it can work properly in multiple cultures. To do this, it includes globalization data for all supported cultures, allowing .NET–targeted applications to behave consistently across all supported versions of Windows. There are drawbacks, however. The CLR must include large data tables, and the data often becomes stale over time. In addition, the data is Windows-centric, so the data for some .NET cultures is different for the same cultures in Mac OS X. For these reasons, the CoreCLR does not include its own globalization data. Instead, System.Globalization.CultureInfo uses the globalization functionality provided by the host OS. Thus, Silverlight applications will behave more like Mac applications on Mac OS X and like Windows applications on Windows.
Overall we've tried to maintain a similar API surface area among the CLR, the .NET Compact Framework, and Silverlight, but there are other small differences scattered throughout the BCL. For example, because Silverlight has a single UI thread, it has a single Dispatcher object that holds a queue of work items for the UI. Using the Dispatcher will allow you to update the UI from a non-UI thread. This code will allow you to update a UI element—MyListBox—with a collection created on another thread (say, from a background thread):
MyListBox.Dispatcher.BeginInvoke(() => MyListBox.ItemsSource = MyItems);
We recommend using System.ComponentModel.BackgroundWorker in Silverlight because it encapsulates the updating of the UI on completion, but for compatibility we still include low-level threading APIs such as System.Threading.ThreadPool.QueueUserWorkItem and System.Threading.Monitor.Enter.
Like the Security Transparency model, some functionality that is new in the Silverlight BCL actually appeared in previous .NET Framework releases. A good example of this is isolated storage, which provides a virtualized file system to sandboxed applications. It's been around since the .NET Framework 1.0, but it has always addressed limited scenarios. Silverlight focuses on sandboxed applications and thus can make full use of isolated storage:
using (IsolatedStorageFile isoStore =
    IsolatedStorageFile.GetUserStoreForApplication())
{
    using (StreamWriter writer = new StreamWriter(isoStore))
    {
         writer.Write("This is an isolated storage file.");
    }
}
Like cookies in a Web browser, isolated storage allows Silverlight applications to maintain state across invocations. However, isolated storage offers a full virtualized file system that supports creation of directories and files. Although isolated storage is not intended for storage of high-value data such as passwords, the storage location is obfuscated and access is limited to the application that owns the store.
Quotas for isolated storage are defined by application groups, which are based on the domain name of the Silverlight application. For example, two Microsoft applications, located in directories under microsoft.com, would share an application group, meaning that both applications share the same quota. By default, an application group is given 1MB of storage.
However, if an application needs additional storage space it can request a larger quota by prompting the user with a dialog box that would specify, for example, that microsoft.com wants to increase its quota to 8MB. Users can enable or disable isolated storage as well as delete current uses of it in the Silverlight configuration dialog (it is called Application Storage in the dialog). Application groups also can have shared stores, which enables related applications to share data among them.
While isolated storage has been around for some time, its uses have never been as compelling as they are with Silverlight. A configurable, secure file system for interactive Web applications allows development of traditional office applications, such as word processors, or applications that maintain large quantities of data, such as stock-tracking systems.

Working Cross Platform
Silverlight will run on non-Windows platforms. We have a partnership with Novell to support Linux through the Mono project's Moonlight runtime. Microsoft also is working on a version of Silverlight for the industry-leading Symbian OS and Windows Mobile®. Moonlight runs on Mono, and the mobile version of Silverlight will run on the .NET Compact Framework (which has a much lower memory footprint than CoreCLR). But the Mac OS X version of Silverlight runs on the exact same CoreCLR as on Windows.
We accomplished this with the help of a Platform Adaptation Layer (PAL). The PAL is an API written to work against different platforms. It provides abstractions for error handling, file handling, networking services, threading semantics, and so forth. Functions in the PAL share the names of Win32® APIs but differ in implementation. Some of the APIs just pass through the PAL function's parameters to an OS X function, whereas others need to use custom logic to match up OS X functionality with Windows API signatures. Some Windows functionality used by CoreCLR doesn't exist on the Mac and thus had to be implemented entirely in the PAL (see Figure 2).
Figure 2 Platform Adaptation Layer
A lot of the Silverlight PAL benefits from lessons learned when we developed the Shared Source Common Language Infrastructure (SSCLI), also known as Rotor. The SSCLI ran on a number of UNIX-style platforms as well as Windows. The base OS functionality varies widely on UNIX-style platforms. The SSCLI PAL had to work on both microkernels (such as the Mach kernel in Mac OS X) and monolithic kernels and had to cope with different OS services such as threading, exception handling, and networking stacks. Because Silverlight only targets Windows and Intel Mac machines we were able to write Mac-specific implementations for many of the functions in the PAL, which helps with the size and performance of the PAL.
The PAL only supports the subset of Win32 necessary to enable Silverlight to run. There is no need to support the registry, GDI+, or COM. We didn't implement Windows on top of the OS X, nor did we implement enough of Windows to support the full capabilities of the desktop CLR. Limiting the PAL to support only Silverlight allows it to be small and fast.
Hiding the differences between operating systems is a tricky problem when you consider just how different OS X is from Windows. Much of OS X is written in Objective C and exposes an exception handling system which is incompatible with C++. The CLR creates I/O threads, which are separate from worker threads. These are based on I/O Completion Ports that were introduced in Windows NT® 3.5 and don't exist on OS X. Even something as simple as locating a file is different on Mac because of the backslash directory delimiter in Windows.
Throughout the design and development of the CoreCLR we've focused on providing an environment that allows developers to reuse existing skills and tools to develop rich content for a small, secure runtime. Most of our decisions were driven by the reduced scenarios of rich Internet applications, but some designs also benefited from looking at some of the work we have done in the past. Some of the decisions we made with CoreCLR will eventually make their way back into the desktop. For example, you can look forward to the next version of the desktop CLR running side-by-side in process with other versions of the CLR. Also, most of the changes made for the improved Security Transparency model will appear in our next CLR.
We carefully considered what made sense for Web-based scenarios and what didn't need to be in the runtime. We hope we've made the right choices and trust that you'll tell us how we can continue to improve. Have fun coding against Silverlight 2, and keep your eye on this space for deeper dives into CoreCLR in the future.

Send your questions and comments to clrinout@microsoft.com.

Andrew Pardoe is a program manager for CLR at Microsoft. He works on all aspects of the execution engine for both Silverlight and the desktop runtime. He can be reached at Andrew.Pardoe@microsoft.com.


Basic Instincts
Increase LINQ Query Performance
Jared Parsons
Code download available at: BasicInstincts2008_08.exe (2,291 KB)
Browse the Code Online
Contents
The Problem
Building a Demo Application
Basic User Interface
Methods for Improving Performance
Incremental Search
Sorting and Lazy Evaluation
Making Incremental Search more Flexible
Sorting Anonymous Types
LINQ is a powerful tool enabling quick filtering data based on a standard query language. It can tear through a structured set of data using a simple and straightforward syntax. This makes it ideal for filtering data that is to be plugged into a data-binding scenario.
There is a catch, though. While it is true that LINQ is powerful and very efficient, large sets of data can still cause unexpected performance problems. This column will focus on a technique to achieve maximum performance from LINQ queries on large sets of data in order to create a responsive UI.

The Problem
In a recent project, I devised a word wheel search UI for a set of XML data using a LINQ-to-XML query for filtering the data. A word wheel search allows users to do a word-based search over a set of data. It will display results that contain a substring matching the text the user has typed in. The results are updated as the user types, providing immediate feedback.
I designed and implemented the word wheel UI using generated XML files. Users could type into a textbox, and the results would be narrowed to those elements whose name contained a substring of the typed-in text. The UI met the design specification and responded efficiently in my test environment. When the rest of the project was completed, my code began producing production XML files, I checked in the new code, and went home. Unfortunately, the next morning I found that QA had gifted me with many UI bugs. The UI had become unresponsive to the point of being unusable.
Figuring I had a bottleneck in one of my query conditions, I thought this would be an easy fix. But after profiling the UI and optimizing my query, I found that I couldn't improve the performance to an acceptable level. The efficiency of the query wasn't the issue; I was being blocked by the sheer amount of data. The test XML files only consisted of 5,000 to 10,000 elements, yet the production XML files had around 40 times as much data. Even with an ultra-fast query, searching several hundred thousand rows with no built-in indexing gets very sluggish.
I needed to find a way to speed up the search process, so I decided to take advantage of some special features of LINQ to improve performance: deferred execution and lazy evaluations. With these features, LINQ queries are not executed at the point at which they are defined. Instead, they are executed as you enumerate through the results. This gives a great deal of flexibility in how you use query results to efficiently scroll through data.

Building a Demo Application
In order to demonstrate my use of deferred execution and lazy evaluations, I'm going to walk you through building a simplified version of my word wheel search using a set of XML files and LINQ-to-XML queries. To make sure that you fully utilize the XML integration for Visual Basic®, it is necessary for you to add an XML schema to the project for the documents that you plan to process. XML schemas are similar to Microsoft® .NET Framework metadata in that they provide an outline for how the XML must look. This allows the Visual Studio® IDE to provide IntelliSense® for LINQ-to-XML queries.
Getting Visual Studio to provide IntelliSense for a schema is straightforward. Visual Basic will recognize all XML schemas that are in the current project. Simply add the schema to your project and then add an import statement for the XML namespace in any file where you will be defining a LINQ-to-XML query:
Imports <xmlns="http://tempuri.org/MsdnWheelSearch.xsd">
It is perfectly possible to use Xlinq without a schema. However, it's kind of like using Visual Studio with IntelliSense turned off. It's possible, but not any fun.
In the word wheel search application, the XML files contain information concerning assemblies, types, and members from the base class libraries (BCLs). Each XML element, or row, will contain an attribute named Id that uniquely identifies the row in its respective parent. This will be a generated number for the assembly rows. For all other rows, it will be the underlying metadata token. In addition, each row will also contain the unique ID of its parent. See Figure 1 for the XML schema of this data.
Figure 1 XML Schema
<?xml version="1.0" encoding="utf-8"?>
<xs:schema id="MsdnWheelSearch"
    targetNamespace="http://contoso.com/MsdnWheelSearch.xsd"
    elementFormDefault="qualified"
    xmlns="http://contoso.com/MsdnWheelSearch.xsd"
    xmlns:mstns="http://contoso.com/MsdnWheelSearch.xsd"
    xmlns:xs="http://www.w3.org/2001/XMLSchema"
>
  <xs:element name="Info">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="Assemblies" minOccurs="0" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
               <xs:element name="Assembly" minOccurs="0" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:attribute name="Name" type="xs:string" />
                  <xs:attribute name="CodeBase" type="xs:string" />
                  <xs:attribute name="Id" type="xs:int" />
                  <xs:attribute name="FullName" type="xs:string" />
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="Types" minOccurs="0" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="Type" minOccurs="0" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:attribute name="Name" type="xs:string" />
                  <xs:attribute name="Namespcae" type="xs:string" />
                  <xs:attribute name="Id" type="xs:int" />
                  <xs:attribute name="AssemblyId" type="xs:int" />
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
        <xs:element name="Members" minOccurs="0" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="Member" minOccurs="0" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:attribute name="Name" type="xs:string" />
                  <xs:attribute name="Kind" type="xs:string" />
                  <xs:attribute name="Id" type="xs:int" />
                  <xs:attribute name="TypeId" type="xs:int" />
                  <xs:attribute name="AssemblyId" type="xs:int" />
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>
You can download the sample project associated with this column to get several XML files that fit this schema. Included are both a small and a very large data set. You can use the UI to easily perform the same queries across both sets of data to get an idea of the relative performance impact they have.
The main query on which I am going to focus for this column returns all members where the name contains the specific substring currently entered by the user. It also will display the type information for the type that owns this member. This is accomplished by filtering the method rows based on a name match and then joining the results with all types having the identical Id and AssemblyId attributes, as shown here:
Private Function GetQueryMethodsOfName( _
  ByVal doc As XDocument, ByVal name As String) As IEnumerable
  Return From method In doc...<Member> _
    Where method.@Name.Contains(name) _
    Where 0 = String.Compare(method.@Kind, "Method") _
    Join type In doc...<Type> _
    On type.@Id Equals method.@TypeId _
    And type.@AssemblyId Equals method.@AssemblyId _
    Select Type = type.@Name, Method = method.@Name
End Function
The sample code has other queries that explore joining the rows to probe for data in the BCL, and I encourage you to try them out.

Basic User Interface
Figure 2 shows the example application. The primary interface for the user is the textbox. As the user types, the application filters the query results to include only rows that contain data matching the input. As the user types, the application constantly updates this value to produce a more refined result for the user. Figure 3 shows the code for my wheel search class.
Figure 2 Sample Word Wheel Application
Figure 3 Wheel Search Class
Public Class WheelSearch
  Private m_doc As XDocument = New XDocument()
  Private m_query As Query = Query.MethodsOfName

  Public Property Query() As Query
    Get
      Return m_query
    End Get
    Set(ByVal value As Query)
      m_query = value
      OnSearchTextChanged(Nothing, EventArgs.Empty)
    End Set
  End Property

  Public Property Xml() As XDocument
    Get
      Return m_doc
    End Get
    Set(ByVal value As XDocument)
      If value Is Nothing Then
        value = New XDocument
      End If
      m_doc = value
      OnSearchTextChanged(Nothing, EventArgs.Empty)
    End Set
  End Property

  Public Sub New()

    ' This call is required by the Windows Form Designer.
    InitializeComponent()

    ' Add any initialization after the InitializeComponent() call.
    m_dataGridView.AutoGenerateColumns = True
  End Sub

  Protected Overrides Sub OnLoad(ByVal e As System.EventArgs)
    MyBase.OnLoad(e)
    OnSearchTextChanged(Nothing, EventArgs.Empty)
  End Sub

#Region "Event Handlers"

  Private Sub OnSearchTextChanged(ByVal sender As Object, _
    ByVal e As EventArgs) Handles m_searchTb.TextChanged
    If m_doc Is Nothing Then
      Return
    End If
    m_dataGridView.IncrementalSearch = _
      QueryContainer.Search(m_query, m_doc, m_searchTb.Text)
  End Sub

#End Region

End Class
The results of the query will be shown in a DataGridView. And while the results of a LINQ query typically work great with data binding, it's necessary for you to convert the results to a type such as List(Of Object) when using the DataGridView. It will be the responsibility of each individual query to return an enumeration of a type that is easily plugged into the data-binding infrastructure.
Now you should load up the UI and test out the performance of the word wheel search. For the smaller data set, you will notice that the performance is snappy. With the larger data set, the performance becomes very sluggish and almost unusable.
The reason for the poor performance can be traced directly to the conversion of the LINQ query into a List(Of Object). This can be verified with a profiler. What happens under the hood here is the List(Of Object) must add each result from the query immediately, forcing the query to fully process each piece of data it encompasses. With the smaller data set, this occurs quickly enough not to be noticed. With the larger data set, this is just too much to process in between keystrokes.

Methods for Improving Performance
Assuming that the structure of the data cannot be modified, how can you increase the responsiveness of the UI? One option is to not search as the user types but instead wait until he explicitly invokes a search (for example, by pressing the Enter key or clicking a button). This solution, however, provides a much poorer user experience because it doesn't provide any immediate feedback. In addition, it doesn't solve the problem that any given search could take a significant amount of time.
Could you improve performance by capping the number of rows returned by the query? Sure, but this still detracts from the user experience because it's not showing all of the data. You'd have to design more UI to tell the user you didn't finish the search and provide a mechanism to do a full search. The full search would leave you right back where you started—with a slow experience.
So you need to show all the data, and you can't modify the structure. But do you have to show all of the data at once? What if you only give the user a small subset of the data at a time, and if a key press is detected you abandon the current search and start a new one? This would be close to other instant-search technologies, such as that found in Microsoft Outlook®.

Incremental Search
A great feature of LINQ is that it supports deferred execution. Queries are not executed at the point that they are written in code; they are merely defined at that point. They are not actually executed until the call to MoveNext on the returned enumerator. You can take advantage of this to search the data incrementally and only spend a minimal amount of time possibly blocking the UI.
I defined a class called IncrementalSearch that works on any object implementing IEnumerable. All LINQ queries implement this interface. IncrementalSearch supports the concept of stopping a search after a certain time period expires. The algorithm is to get the next value by calling MoveNext until the specified time period has expired. At that point, the data found is returned in a new Result object (see Figure 4).
Figure 4 IncrementalSearch Class
Public Enum SearchStatus
  InProgress
  Completed
End Enum

Public Class Result
  Private m_allValues As IList
  Private m_incrementalValues As IList
  Private m_status As SearchStatus

  Public ReadOnly Property AllValues() As IList
    Get
      Return m_allValues
    End Get
  End Property

  Public ReadOnly Property IncrementalValues() As IList
    Get
      Return m_incrementalValues
    End Get
  End Property

  Public ReadOnly Property SearchStatus() As SearchStatus
    Get
      Return m_status
    End Get
  End Property

  Sub New(ByVal allValues As IList, _
    ByVal incrementalValues As IList, _
    ByVal status As SearchStatus)
    m_allValues = allValues
    m_incrementalValues = incrementalValues
    m_status = status
  End Sub
End Class

Public Class IncrementalSearch
  Private m_enumerable As IEnumerator
  Private m_allFound As New List(Of Object)
  Private m_status As SearchStatus = SearchStatus.InProgress
  Private m_delay As TimeSpan
  Public ReadOnly Property SearchStatus() As SearchStatus
    Get
      Return m_status
    End Get
  End Property

  Public Property Delay() As TimeSpan
    Get
      Return m_delay
    End Get
    Set(ByVal value As TimeSpan)
      m_delay = value
    End Set
  End Property

  Public Sub New(ByVal e As IEnumerable)
    m_delay = TimeSpan.FromSeconds(0.1)
    m_enumerable = e.GetEnumerator()
  End Sub

  Public Function Search() As Result
    If m_status = SearchStatus.Completed Then
      Return New Result(m_allFound, New List(Of Object), m_status)
    End If

    Dim found As New List(Of Object)
    Dim start = DateTime.Now
    Do
      If Not m_enumerable.MoveNext() Then
        m_enumerable = Nothing
        m_status = SearchStatus.Completed
        Exit Do
      End If

      found.Add(m_enumerable.Current)
    Loop Until DateTime.Now - start > m_delay

    m_allFound.AddRange(found)
    Return New Result(m_allFound, found, m_status)
  End Function

End Class
The Search method returns a Result object based on the data it finds, including the values found in the current call to Search, the list of all values found in all previous calls to Search, and the current status of the search. This means that the UI can now return data in chunks. The maximum time you can delay will be for getting the next element instead of the entire set of rows:
Dim found As New List(Of Object)
Dim start = DateTime.Now
Do
  If Not m_enumerable.MoveNext() Then
    m_enumerable = Nothing
    m_status = SearchStatus.Completed
    Exit Do
  End If

  found.Add(m_enumerable.Current)
Loop Until DateTime.Now - start > m_delay
The goal is for the UI to display the data through data binding and to provide an efficient search. Displaying the data should not cause the UI to hang. Return values are displayed in a custom DataGridView using the incremental search pattern. The example implementation is called DataGridViewIncrementalSearch and derives from DataGridView (see Figure 5). This implementation works great with data binding and is designed to perform well with large sets of data.
Figure 5 DataGridViewIncrementalSearch
Public Class DataGridViewIncrementalSearch
  Inherits DataGridView

  Private m_search As IncrementalSearch
  Private WithEvents m_timer As New Timer

  Public Property IncrementalSearch() As IncrementalSearch
    Get
      Return m_search
    End Get
    Set(ByVal value As IncrementalSearch)
      m_search = value
      m_timer.Start()
    End Set
  End Property

  Public Sub New()
    m_timer.Interval = CInt(TimeSpan.FromSeconds(0.2).TotalMilliseconds)
  End Sub

  Protected Overrides Sub Dispose(ByVal disposing As Boolean)
    If disposing Then
      m_timer.Dispose()
    End If
    MyBase.Dispose(disposing)
  End Sub

  Private Sub OnTick(ByVal sender As Object, ByVal e As EventArgs) _
    Handles m_timer.Tick
    m_timer.Stop()
    If m_search Is Nothing OrElse m_search.SearchStatus = _
      SearchStatus.Completed Then
      Return
    End If

    Dim result = m_search.Search()
    DataSource = Nothing
    DataSource = result.AllValues
    m_timer.Start()
  End Sub

End Class
There is one extra property of the same type called IncrementalSearch. Whenever this value is set, the control will call the Search method and continually update the displayed set of values. To achieve this, the control will have to call the Search method at regular intervals until the data source is exhausted.
A Windows® Forms timer is ideal for this scenario. It can be configured to call a method on the UI thread at a particular interval and can be enabled or disabled. When the IncrementalSearch property is updated, it enables the timer. The handler function will call Search. Once the search is complete, the timer will be disabled.
One issue with a Windows Forms timer is that the interval is calculated independently of how long it takes a handler to perform an action on the Tick event. That means that if the time it takes to execute the handler is longer than the interval, there will be another Tick event waiting to go off almost immediately. In the worst case scenario, these events will stack up and make the UI unresponsive. In order to work around this, you can disable the timer while calling Search and re-enable it if there is more data to process (see Figure 6). No explicit column selection functionality was created for this DataGridView. For this example, I'll let data binding choose the columns.
Figure 6 Enabling and Disabling the Search Timer
Private Sub OnTick(ByVal sender As Object, ByVal e As EventArgs) _
  Handles m_timer.Tick
  m_timer.Stop()
  If m_search Is Nothing _
    OrElse m_search.SearchStatus = SearchStatus.Completed Then
    ' Search is completed don't restart the timer
    Return
  End If

  ' Get the next update from the incremental search
  Dim result = m_search.Search()
  DataSource = Nothing
  DataSource = result.AllValues

  ' Restart the timer to perform another search
  m_timer.Start()
End Sub
One quirk in the code is the updating of the data source. The underlying IncrementalSearch object creates a List(Of Object) to hold the return values from the query. Every Result object contains a reference to this list instead of a copy. When updating the Data source property, the DataGridView will see that it is the same Object and will not refresh the displayed results. Setting the reference to Nothing in between will force it to reload the data.
While this is a nice, easy-to-use solution, it won't work in all scenarios. The underlying problem here is user delay. Because this is a single-threaded solution, there is no way to break the execution once you call MoveNext. Therefore, the longest you will delay the user from typing is now the DelayTime value for the IncrementalSearch class plus the longest time to find one element in the search.

Sorting and Lazy Evaluation
Most LINQ queries operate with lazy evaluation, which means that only a single element is processed each time MoveNext is called on the enumerator. The time it takes to find one element in the search is the same time it takes to find the next element. This allows for the result to be efficiently broken up into usable blocks of data:
Dim e = From method In doc...<Member> _
        Where method.@Name.Contains(name) _
        Where String.Compare(method.@Kind, "Method") = 0 _
        Select MethodName = method.@Name
' Returns when the first matching <Member> is found
Dim first = e.First()
This model will break down if the underlying LINQ query processes a substantial portion or all of the values during a single call to MoveNext. This form of query is known as eager evaluation. An example of this is the Order By operator, which returns the results in a sorted fashion. Because it is sorted, the first value cannot be returned until all parts of the query have finished executing. Otherwise it couldn't know which was the first element to return because the last element returned could be the first in terms of sorting. If you try the incremental search pattern with a query using eager evaluation, you are right back at the original problem:
Dim e = From method In doc...<Member> _
        Where method.@Name.Contains(name) _
        Where String.Compare(method.@Kind, "Method") = 0 _
        Order By method.@Name _
        Select MethodName = method.@Name
' The following line won't finish until the entire query is finished
' processing
Dim first = e.First()
Once again, the advantage here is in only dealing with a subset of the rows returned. Since the code is only displaying a subset of the data to the user, that is the only data that needs to be sorted at any given time. The underlying LINQ query can return the data unsorted, and then the code can sort it as more comes along.
It is easy to modify the IncrementalSearch class to sort the data at the end of each Search call. There are two approaches available to searching the data. The first approach is to take advantage of the built-in List(Of T).Sort method. This has an average complexity of O(N LogN) and is, generally speaking, fast enough for most applications:
Dim search As New List(Of String)(SearchForMethodNames)
search.Sort(StringComparer.OrdinalIgnoreCase)
The second method takes advantage of data being sorted at the start of the call. Instead of appending the data to the end of the existing data list, you could search for the proper place to insert and place the newly found data there. Finding the proper index is a relatively quick operation when compared to fully sorting the list, so at first this seems like the more logical choice. But there are two reasons why you cannot keep the list ordered in this manner.
First, while the IncrementalSearch class itself keeps the list sorted, there is nothing stopping the caller from reordering the values. So you cannot assume this is sorted. Second, there is an added cost to inserting into the middle of a List(Of T). Under the hood it is backed by an Array, so every element to the right of the value being inserted has to be shifted one element to the right. For large data structures, the time to do the shift is non-trivial and eventually makes this method of sorting much less efficient.
To use the collection's Sort method, you need a method for comparing the values. Ideally, you will be able to use existing comparers in the framework to sort well-known data types. However, the IncrementalSearch class is weakly typed, and most comparers are strongly typed to their object. You could not use String.Compare for a query that returned Strings, for instance, because they would be stored in a weakly typed object collection. To allow for searching and reuse of existing classes, you will need to make the IncrementalSearch class more flexible by providing both a weakly and strongly typed implementation.

Making Incremental Search more Flexible
Right now the IncrementalSearch class is very flexible because it works against untyped collections. This limits the usefulness of the results, though, because everything is typed as object. LINQ queries typically return collections of anonymous types, and IntelliSense makes it much easier to use the types. So you must make IncrementalSearch strongly typed in order to allow for more processing on the results.
There is a problem with making IncrementalSearch itself generic. If IncrementalSearch is generic, then the control wrapping IncrementalSearch must be generic as well, or you'll have to create an instance of IncrementalSearch that is bound to a particular type. However, the Windows Forms designer does not support unbound generic controls, meaning that you would lose all designer support; hence, it is not an option. Having the control bind IncrementalSearch to a specific type works well, but this will limit the reusability of the control to searches of that specific type.
The solution is a hybrid approach. The IncrementalSearch class remains non-generic and exposes the search results through non-generic collection interfaces such as IList. It then is made into an abstract class where the Search method becomes abstract. The Result class likewise will only expose non-generic collections. Keeping IncrementalSearch non-generic and providing abstract methods allows it to be reusable in a Windows Forms control and still have strongly typed collections.
IncrementalSearch(Of T) will be strongly typed on the search result and will derive from IncrementalSearch. It will override the Search method and provide the underlying strongly typed collection classes. See Figure 7 for the full definition of the refactored IncrementalSearch class.
Figure 7 Refactored IncrementalSearch Code
Public MustInherit Class IncrementalSearch
  Protected m_status As SearchStatus = SearchStatus.InProgress
  Protected m_delay As TimeSpan

  Public MustOverride ReadOnly Property AllValues() As IList

  Public ReadOnly Property SearchStatus() As SearchStatus
    Get
      Return m_status
    End Get
  End Property

  Public Property Delay() As TimeSpan
    Get
      Return m_delay
    End Get
    Set(ByVal value As TimeSpan)
      m_delay = value
    End Set
  End Property

  Public MustOverride Function Search() As Result

  Public Shared Function Create(Of T)(ByVal e As IEnumerable(Of T)) _
    As IncrementalSearch(Of T)
    Return New IncrementalSearch(Of T)(e)
  End Function
End Class

Public NotInheritable Class IncrementalSearch(Of T)
  Inherits IncrementalSearch
  Private m_enumerator As IEnumerator(Of T)
  Private m_allFound As New List(Of T)
  Private m_comparer As IComparer(Of T)

  Public Overrides ReadOnly Property AllValues() _
    As System.Collections.IList
    Get
      Return m_allFound
    End Get
  End Property

  Public Property Comparer() As IComparer(Of T)
    Get
      Return m_comparer
    End Get
    Set(ByVal value As IComparer(Of T))
      m_comparer = value
    End Set
  End Property

  Public Sub New(ByVal e As IEnumerable(Of T))
    m_delay = TimeSpan.FromSeconds(0.1)
    m_enumerator = e.GetEnumerator()
  End Sub

  Public Overrides Function Search() As Result
    If m_status = SearchStatus.Completed Then
      Return New Result(m_allFound, New List(Of T), m_status)
    End If

    Dim found As New List(Of T)
    Dim start = DateTime.Now
    Do
      If Not m_enumerator.MoveNext() Then
        ' IEnumerable(Of T) must be disposed
        m_enumerator.Dispose()
        m_enumerator = Nothing
        m_status = SearchStatus.Completed
        Exit Do
      End If

      found.Add(m_enumerator.Current)
    Loop Until DateTime.Now - start > m_delay

    m_allFound.AddRange(found)
    If m_comparer IsNot Nothing Then
      m_allFound.Sort(m_comparer)
    End If
    Return New Result(m_allFound, found, m_status)
  End Function
End Class

Public NotInheritable Class IncrementalObjectSearch
  Inherits IncrementalSearch
  Private m_enumerator As IEnumerator
  Private m_allFound As New ArrayList
  Private m_comparer As IComparer

  Public Overrides ReadOnly Property AllValues() As _
    System.Collections.IList
    Get
      Return m_allFound
    End Get
  End Property

  Public Property Comparer() As IComparer
    Get
      Return m_comparer
    End Get
    Set(ByVal value As IComparer)
      m_comparer = value
    End Set
  End Property

  Public Sub New(ByVal e As IEnumerable)
    m_enumerator = e.GetEnumerator()
  End Sub

  Public Overrides Function Search() As Result
    If m_status <> SearchStatus.InProgress Then
      Return New Result(m_allFound, New List(Of Object), m_status)
    End If

    Dim found As New List(Of Object)
    Dim start = DateTime.Now
    Do
      If Not m_enumerator.MoveNext() Then
        m_enumerator = Nothing
        m_status = SearchStatus.Completed
        Exit Do
      End If
      found.Add(m_enumerator.Current)
    Loop Until DateTime.Now - start > m_delay

    m_allFound.AddRange(found)
    If m_comparer IsNot Nothing Then
      m_allFound.Sort(m_comparer)
    End If
    Return New Result(m_allFound, found, m_status)
  End Function
End Class
The IncrementalSearch(Of T) class will now have an additional property named Comparer of type IComparer(Of T). When this is present, the results will be sorted after each call to Search:
Public NotInheritable Class IncrementalSearch(Of T)
  Inherits IncrementalSearch
  Private m_enumerator As IEnumerator(Of T)
  Private m_allFound As New List(Of T)
  Private m_comparer As IComparer(Of T)
Now it's possible to reuse existing mechanisms to sort a query without introducing eager evaluation:
Private Function GetFullMethodNameSorted( _
  ByVal doc As XDocument, ByVal name As String) _
  As IncrementalSearch

  Dim e = From method In doc...<Member> _
    Where method.@Name.Contains(name) _
    Where 0 = String.Compare(method.@Kind, "Method") _
    Join type In doc...<Type> _
      On type.@Id Equals method.@TypeId _
      And type.@AssemblyId Equals method.@AssemblyId _
    Select type.@Namespace & "." & type.@Name & "." & method.@Name
  Dim search = IncrementalSearch.Create(e)
  search.Comparer = StringComparer.Ordinal
  Return search
End Function
For handwritten queries, this solution often is enough. However, this is primarily designed to hold the result of LINQ queries that often contain anonymous types.
In code you cannot refer to the type of an anonymous type. Instead you must depend on type inference to create strongly typed generic classes. This can be accomplished by adding a factory method named Create onto the IncrementalSearch base class. Since there is only one generic parameter and the argument satisfies this parameter, the compiler can infer the argument and there is no need to explicitly write it out:
Dim e = From method In doc...<Member> _
  Where String.Compare(method.@Kind, "Method") = 0 _
  Select MethodName = method.@Name

' What is the type name of the element?
Dim s1 As New IncrementalSearch(Of ???)(e)

' No need to write the type name
Dim s2 = IncrementalSearch.Create(e)
To continue supporting weakly typed collections, you can implement another child of IncrementalSearch named IncrementalObjectSearch. This will be similar to the current implementation except that it will be based on an ArrayList instead of a List(Of Object):
Public NotInheritable Class IncrementalObjectSearch
  Inherits IncrementalSearch
  Private m_enumerator As IEnumerator
  Private m_allFound As New ArrayList
  Private m_comparer As IComparer

Sorting Anonymous Types
Queries often return a collection of anonymous types. Because you cannot describe the type of an anonymous type in code, it is impossible to write an implementation of IComparer(Of T).
Luckily you're working in Visual Basic here. You know what members are present on the underlying values returned by the query. Using late binding you can access these directly. Figure 8 contains an example of how to create a sorted version of the original query. Since you are using a Comparer instead of an order by statement, this will be a lazy evaluation and will perform well in the UI.
Figure 8 Custom Sorting
Private Function GetQueryMethodsOfNameSorted( _
  ByVal doc As XDocument, ByVal name As String) As IncrementalSearch
  Dim e = From method In doc...<Member> _
          Where method.@Name.Contains(name) _
          Where 0 = String.Compare(method.@Kind, "Method") _
          Join type In doc...<Type> On type.@Id Equals method.@TypeId _
          And type.@AssemblyId Equals method.@AssemblyId _
          Select Type = type.@Name, Method = method.@Name
  Dim x = New IncrementalObjectSearch(e)
  x.Comparer = New MethodsOfNameComparer()
  Return x
End Function

Private Class MethodsOfNameComparer
  Implements IComparer

  Public Function Compare(ByVal x As Object, _
    ByVal y As Object) As Integer Implements _
    System.Collections.IComparer.Compare

    Dim comp = StrComp(x.Type, y.Type)
      If comp <> 0 Then
        Return comp
      End If

    Return StrComp(x.Method, y.Method)
  End Function
End Class
This was a fun pattern to implement and write about, but I only have so much space in a column. There are still many other ways to make the UI even more efficient. Here are some ideas that you might want to experiment with on your own:
  • Reorganizing the source data to be more efficient for queries.
  • Using LINQ to create more efficiently organized data sets in memory, letting further queries operate against this better-structured data.
  • Offloading the query onto another thread.
  • Providing UI to indicate to the user a search is in progress.
  • Putting the DataGridView into VirtualMode.
  • Searching only against previous results rather than the entire document as the user types more text.
  • Introducing Parallel LINQ queries.
Hopefully I'll have time to expand on these in my blog. Also, be sure to check out the Visual Basic Team Blog (blogs.msdn.com/vbteam) and the MSDN® Visual Basic Developer Center (msdn.com/vbasic) for more information.

Send your questions and comments to instinct@microsoft.com.

Jared Parsons is a Software Engineer at Microsoft working on the Visual Basic Debugger and IDE. His passion lies in code, coding, and pretty much anything related to programming. He blogs regularly at blogs.msdn.com/jaredpar.


Cutting Edge
Programming AJAX with ASP.NET Partial Rendering
Dino Esposito

Code download available at: CuttingEdge2008_08.exe (470 KB)
Browse the Code Online
Contents
Pages with a Menu
Navigation URLs
Async Postbacks
Menu and Partial Rendering
Give Me a Placeholder
Dynamically Loaded Controls
Which Control Posted Back?
Caching Output for User Controls
It's about Pragmatism
The heart of AJAX is the XMLHttpRequest object. The user experience opportunities that AJAX offers are contingent on the availability of this object on a broad number of browsers over multiple platforms. A lot of things have happened since 2004, when component vendors first began showcasing AJAX applications, but at the core, AJAX would not be possible without the ability to execute an out-of-band call using the XMLHttpRequest object.
As the complexity of AJAX applications has increased, developers have come to realize that plain script-driven, out-of-band calls are not enough to build a new generation of Web applications in a cost-effective way. Subsequently, the demand for a set of more powerful tools is growing. These tools are expected to add AJAX capabilities to pages and applications, while using the same development paradigm as classic ASP.NET. So, to make a long story short, developers need more in order to build AJAX sites than plain calls to the XMLHttpRequest object for fetching data and handcrafted JavaScript functions for manipulating the Document Object Model (DOM).
In this month's column, I'll look at a pragmatic approach for AJAX that leverages the ASP.NET partial rendering engine. As you will see, AJAX involves trade-offs between application performance and developer productivity. Real-world AJAX sites are not built entirely with either partial rendering or manual scripting of the XMLHttpRequest object. They require a powerful mix of techniques that are often well synthesized by custom controls.

Pages with a Menu
AJAX is mostly about minimizing the number of full page reloads. Both partial rendering and manual script-driven calls allow you to fetch server-side data and avoid a full page refresh. For many content types, back-end services and DOM manipulation may be enough. But what if you have to support navigation?
Classic hyperlinks break the AJAX magic, telling the browser to place a request to another URL. As a result, the current page is frozen until the new chunk of HTML is downloaded. When the fresh data arrives, the page is dismissed and the browser's client is then fully redrawn.
Top-level links placed on a Master Page that point users to different areas of the site may really be implemented as traditional hyperlinks. Depending on your users' expectations, a full page reload may be acceptable in this case.
Whenever you build a menu, you make a choice about how to handle user clicks. You may assign each menu item a URL or just post back to the same page. From an AJAX perspective, you choose between dismissing the current page and loading a brand new one, and asynchronously loading some new content from the server.
Take a look at the following code snippet, which shows a fragment of the ASP.NET Menu control:
<asp:Menu runat="server" ID="Menu1">
<asp:MenuItem Text="Products" Value="Products">
    <asp:MenuItem Text="By price" NavigateUrl="..." />
    <asp:MenuItem Text="All" Value="Products-All" />
</asp:MenuItem>
...
</asp:Menu>
The first MenuItem child element of the Products menu item features a NavigateUrl property. The property gets or sets the URL to navigate to when the menu item is clicked. The second MenuItem element features a Value property. The Value property is used to store additional data about the menu item, and it is passed to the MenuItem's postback event. Menu items at the same level must each have a unique value for the Value property. When no explicit navigation URL is specified, clicking a menu item results in a classic postback. On the server, you simply handle the MenuItemClick event and process the content of Value property.
Obviously, if you choose an approach using the Value property, you can use AJAX techniques and avoid a full page reload. The overall model of this type of page is a single-page interface.
The single-page interface reduces the burden on the user interface because it decreases the number of reloads and eliminates flickering. However, the single-page interface also means that your application uses less-distinct URLs, which results in diminished support from search engines. The single-page interface also means that the development team writes fewer pages, but they're richer. Such an approach may also reduce parallelism within the team.

Navigation URLs
When the menu takes users to a brand new page, you don't really need a Menu control to implement the solution. In theory, a list of hyperlinks would suffice. There are dozens of DHTML and AJAX menu frameworks available out there, but, at the end of the day, they differ based on the animation, graphics, and predefined skins they offer. Functionally speaking, these menus are plain collections of hyperlinks, and hyperlinks are natively handled by browsers in a non-AJAX way.
Generally, large Web sites that contain hundreds of total pages are likely designed around a few entry pages that then direct users to different subsites. The home page, therefore, just needs links to these subsites. In this case, navigation is the requirement and AJAX would not be necessary. But what if you need to bring content down to the currently displayed page? How would you organize this content to maximize performance of the application and productivity of the team?

Async Postbacks
Last month I demonstrated how to use Windows® Communication Foundation (WCF) services to return either raw data or HTML to the client. Both solutions have pros and cons. Sending raw data optimizes the bandwidth but requires you have some DOM manipulation logic on the client implemented with JavaScript. Sending server-generated HTML increases the quantity of data that is moved around, but it allows you to maintain most of your rendering logic on the server.
Both approaches fall into the category of the so-called "true" AJAX solutions, where the application is clearly designed according to a two-tier model—Web browser and service layer. Any state and logic that involves the user interface is maintained and controlled by the client; no view state or postbacks are needed.
Server-generated HTML is most useful if you have read-only markup to return. It works well if you need to return a static grid of data or a panel with some information on the selected customer or invoice. It's much less enticing if you need to bring up an interactive panel full of controls that fire events and require handlers. In a pure AJAX approach, the displayed markup—whether generated on the client or the server—must contain JavaScript function calls.
Last month I used services to implement the HTML Message and Browser-Side Template patterns. But, as you may have noticed, my examples were not really interactive. It turns out that both approaches may work in some cases, but not in all. My sample services returned stock quotes, but the final grid was not pageable. To support paging, for example, I had to insert hyperlinks pointing to JavaScript functions and then make sure that I downloaded the referenced JavaScript.

Menu and Partial Rendering
Figure 1 shows a sample page with a top-level menu that allows users to navigate in the set of application features. The menu has been created using the ASP.NET Menu control, as shown in Figure 2. As you can see, all menu items avoid specifying the NavigateUrl attribute, which would point each to a physical URL, possibly for a totally different page. Having specified the Value attribute, a postback occurs and the MenuItemClick event is fired to the page whenever the user selects any of the menu items:
void Menu1_MenuItemClick(object sender, MenuEventArgs e)
{
    LoadContent(e.Item.Value);
}
Figure 1 An ASP.NET AJAX Page with a Top-Level Menu
Figure 2 Menu of the Sample Page
<asp:Menu ID="Menu1" runat="server" Orientation="Horizontal"
        onmenuitemclick="Menu1_MenuItemClick">
    <Items>
        <asp:MenuItem Text="Customers">
            <asp:MenuItem Text="All"
                 Value="Customers-All" />
            <asp:MenuItem Text="By Country"
                 Value="Customers-by-country" />
        </asp:MenuItem>
        <asp:MenuItem Text="Orders">
            <asp:MenuItem Text="1997"
                  Value="Orders-1997" />
            <asp:MenuItem Text="By Customer"
                  Value="Orders-by-customer" />
        </asp:MenuItem>
        <asp:MenuItem Text="Products">
            <asp:MenuItem Text="All"
                 Value="Products-All" />
        </asp:MenuItem>
    </Items>
</asp:Menu>
The MenuEventArgs class features an Item property that represents the clicked menu item. The Value property on the MenuItem object informs the event handler about the identity of the clicked item. By using a local LoadContent function, the page can dynamically load the required content. But what about the postback?
The ASP.NET Menu control fully supports partial rendering. To swallow the postback and load new content smoothly, all that you have to do is set up a ScriptManager control and insert an UpdatePanel control, as shown here:
<asp:UpdatePanel runat="server" ID="UpdatePanel1"
          UpdateMode="Conditional">
    <ContentTemplate>
       ...
    </ContentTemplate>
    <Triggers>
        <asp:AsyncPostBackTrigger ControlID="Menu1"
            EventName="MenuItemClick" />
    </Triggers>
</asp:UpdatePanel>
The updateable content is bound to the MenuItemClick of the Menu. Through the UpdateMode property, the panel is also configured for control-specific updates. This means that the updateable area is refreshed only when the user clicks on a menu item.
In most Web applications, you use a menu to point users to different functional areas of the site or the page. Having distinct ASPX pages to implement specific functionalities is helpful for a number of reasons. It keeps your code clean and easier to maintain and test. But more importantly, it allows you to assign the implementation of distinct functionalities to distinct developers on the team. Thus, you can improve the productivity by parallelizing some development tasks.
The single-page interface is a typical AJAX pattern that suggests you have a main page in your application whose user interface is rearranged every time an event is fired. The single-page interface model is great at minimizing postbacks, but it requires some care when it comes to parallelizing development tasks on a team. You should consider it to be a page architecture based on the concept of plug-ins. In other words, the page defines placeholders with a contracted interface where dynamically loaded components could be plugged in quickly and easily.

Give Me a Placeholder
In ASP.NET, the simplest way to achieve this page architecture oriented toward plug-ins involves using the PlaceHolder control and ASCX user controls to define on-demand components. Here's how you define an updateable area:
<asp:UpdatePanel runat="server" ID="UpdatePanel1"
    UpdateMode="Conditional">
    <ContentTemplate>
        <asp:PlaceHolder runat="server" ID="PlaceHolder1"
            EnableViewState="false" />
    </ContentTemplate>
    <Triggers>
        <asp:AsyncPostBackTrigger ControlID="Menu1"
            EventName="MenuItemClick" />
    </Triggers>
</asp:UpdatePanel>
You associate each menu item with an ASCX user control and load it into the placeholder whenever the item is clicked. Figure 3 shows the source code of the aforementioned LoadContent function, which you invoke from the MenuItemClick event handler.
Figure 3 Dynamically Loading Content into the Page
public partial class _Default : System.Web.UI.Page
{
    public string TrackedUserControl
    {
        get { return ViewState["TrackedUserControl"] as string; }
        set { ViewState["TrackedUserControl"] = value; }
    }

    protected void Menu1_MenuItemClick(object sender, MenuEventArgs e)
    {
        LoadContent(e.Item.Value);
    }

    private void LoadContent(string menuItemName)
    {
        string ucUrl = menuItemName + ".ascx";

        UserControl uc = null;
        try
        {
            uc = this.LoadControl(ucUrl) as UserControl;
        }
        catch(Exception ex)
        {
        }
        if (uc != null)
        {
            Placeholder1.Controls.Add(uc);
            TrackedUserControl = ucUrl;
        }
    }
    ...
}
The LoadControl method on the Page class takes the URL of an ASP.NET user control and loads it into memory as an instance of a class that derives from UserControl. If the control is loaded correctly, the method adds it to the Controls collection of the PlaceHolder. Note that the ASP.NET PlaceHolder control doesn't output any markup, so don't be afraid of cluttering your user interface markup if you use it.
Using ASCX user controls in the context of an AJAX site with dynamic and interactive content doesn't reduce your chance of developing multiple blocks of content at the same time—an ASCX control is a sort of ASP.NET page and can be developed independently from the rest of the site.

Dynamically Loaded Controls
In ASP.NET, dynamically loaded controls require a bit of attention because of their behavior across postbacks. Suppose you create a user control that contains a pageable grid of customers, and then you load the user control into the placeholder (see Figure 4). Everything works just fine until you click one of the bottom hyperlinks to navigate to a new page. When this happens, the page partially refreshes, but the whole content of the user control disappears. Why?
Figure 4 Interactive Control Dynamically Loaded into Page
In ASP.NET, each page request is treated independently from any other previous or successive requests. So a brand new instance of the page class is created to service each incoming request. This new page contains fresh instances of all server controls that are statically referenced in the ASPX source code.
But what about the dynamically added controls? The page doesn't know about them unless the code in the page uses some persistence mechanism to track dynamic controls. The page view state is an excellent storage medium for this type of information. In Figure 3, I added the TrackUserControl property to the page class to remember which user control is currently displayed.
In the Page_Load event, you use the content of the TrackUserControl property to figure out which user control has to be manually reloaded in order to restore exactly the same collection of controls the page contained last time:
void Page_Load(object sender, EventArgs e)
{
    if (!IsPostingFromMenu() && IsPostBack)
        ReloadContent();
}
Note, though, that the postback may have one of two causes—either the user triggered an action from within the displayed user control or the user clicked another menu item. The if statement in Page_Load determines whether the user is posting from the menu. If so, nothing happens because some new content has to be loaded; otherwise, the tracked user control is reloaded so that it can handle the postback event:
void ReloadContent()
{
    UserControl uc = null;
    try
    {
        uc = this.LoadControl(TrackedUserControl) as UserControl;
    }
    catch
    {
    }
    if (uc != null)
        Placeholder1.Controls.Add(uc);
}
What about the view state for the dynamically added user control? And what about any data that HTML elements may have posted to it? When the user is served a page, there is no distinction between statically and dynamically added controls. Any input the user enters (for example, text in an input field) is automatically packed into the HTTP request when the form is submitted. On the server, therefore, any information to be consumed by the dynamically loaded control is available. However, you should know that when the page's Init event fires, no such control is yet available. This is because the page's control tree is only populated with statically referenced controls.
It is up to the page author to add any dynamically referenced controls to the tree. The page author, though, may need to read about added controls from a persistent storage medium. If this information is stored in Session or Cache, then you can safely access it from the Init event handler. If that information is stored in the view state bag, then you have to wait for the Load event. In the ASP.NET page lifecycle, in fact, the view state is unpacked and processed between the Init and Load events.
At this stage, any posted data is also examined and mapped to existing controls. But what about the data that is destined for dynamically added controls? Basically, posted data is processed in one of two passes—once before or once after the Page__Load event. Data for which no matching control is found in the first round is cached and processed again after the Load event.
In the Load event, developers should check which controls were dynamically added the previous time and reload them in the page control tree. Nicely enough, the Add method of the Controls collection automatically updates the state of the control with any data found for that control in the view state.

Which Control Posted Back?
A page that loads some of its content from external resources needs to know why a postback is raised. In the sample code you see in Figure 4, the page may post back because the user selected a given element from the menu or because the user triggered a postback while working with another component on the user interface.
In ASP.NET, there's no property anywhere that can tell you which control caused the postback. The ID of the posting control, however, is not difficult to find. If the user posted back by clicking a Submit button, then the ID of the control is listed in the HTTP request. If no button control exists in the body of the HTTP request, then the user posted back by interacting with a Link button or an auto-postback control.
In any case, the ID of the postback source is in the __EVENTTARGET hidden field. In ASP.NET AJAX, when partial rendering is used, things are easier, as the ScriptManager control exposes a custom property—the AsyncPostBackSourceElementID property. The following code, which is an excerpt of the code that produced Figure 4, shows how to determine whether the user clicked a menu item or posted back to the server through another control:
bool IsPostingFromMenu()
{
    ScriptManager sm = ScriptManager.GetCurrent(this);
    string ctlID = sm.AsyncPostBackSourceElementID;
    Control c = this.FindControl(ctlID);
    if (c == null)
        return false;

    return (c.ID == "Menu1");
}
When the user clicks repeatedly on the menu, a given ASCX user control may be loaded again. Loading a user control means downloading the control markup and going through the server lifecycle. However, since continuously reloading a user control could possibly have a negative impact on performance, you can use an ASP.NET feature such as output caching to help boost the overall performance of the page.

Caching Output for User Controls
A user control that provides some content to a site may be expensive to create because it could require one or more database calls and potentially a few million CPU cycles. Why then would you want to regenerate the same user control several times per second, especially if your content is not continuously changing?
A better strategy is to create the user control once, cache its output, and give it a maximum duration. Once the cached snapshot of the user control becomes stale, the first incoming request will be served again in the standard way, running the control's code and caching the generated markup again.
The ASP.NET page output caching feature allows you to cache page and user control responses so that subsequent requests can be satisfied without executing code but rather by simply returning the cached output. To make a user control cacheable, you declare the @OutputCache attribute in the source of the ASCX, as shown in the following code. The code snippet caches the output of the control for one minute:
<% @OutputCache Duration="60" VaryByParam="None" %>
Output caching is a nice feature, but it is far from being a silver bullet. It is simply a way to have the application serve more pages and user controls more quickly. It won't necessarily make the application more efficient or scalable. With output caching, however, you certainly reduce the workload on the server, as pages and resources are cached downstream.
Additionally, output caching is only a viable option for anonymous content. Requests for cached content are served by IIS directly and never make it to the ASP.NET pipeline, where the caller can be authenticated.
Finally, pages and user controls that post back require some extra work. In particular, you need to instruct ASP.NET to save multiple copies of the output that vary by one or more parameters. For this purpose, you use the VaryByParam attribute on the @OutputCache directive.

It's about Pragmatism
While partial rendering is often perceived as the poor man's AJAX and reserved for those who can't exert the energy required to create "true" AJAX applications, it is one of the best options at the moment for developers using ASP.NET. In the end, remember that AJAX involves many trade-offs, and effective AJAX results from the combination of a variety of techniques and patterns.

Send your questions and comments for Dino to cutting@microsoft.com.

Dino Esposito is currently an architect at IDesign and the author of Programming ASP.NET 3.5 Core References. Based in Italy, Dino is a frequent speaker at industry events worldwide. You can join his blog at weblogs.asp.net/despos.


Patterns in Practice
Object Role Stereotypes
Jeremy Miller
Contents
What's the Problem?
Responsibility-Driven Design
Information Holder
Structurer
Service Provider
Interfacer
Coordinator
Controller
Using Object Role Stereotypes Effectively
Object-oriented systems are composed of many objects, each one fulfilling some set of responsibilities. Objects often need to collaborate with other objects in order to fulfill these responsibilities. You can better understand the responsibilities of objects and the collaborations between them by applying a concept known as object role stereotypes. This article will discuss the most common object role stereotypes by applying them to real-world examples and well-known design patterns.

What's the Problem?
Neolithic programmers lived in a state of simplicity. Programs were composed of a singular straight, unbroken line of instructions to the computer. Even today, many of us initially learn to program by writing standalone functions:
public static class Program {
  public static void Main(string[] args) {
    Console.WriteLine("Look at me World!  I'm coding!");
    // And about 30-50 other lines of code
  }
}
The single method style of program construction is the easiest form of programming to learn, but it breaks down quickly as the program becomes larger. Early programmers and computer scientists soon realized that they needed some way of managing complexity as software projects increased in scope and ambition.
While it certainly didn't have the same impact on mankind as the discovery of fire, the ability to decompose a single program into multiple subroutines, classes, or services gave programmers some fantastic advantages over the monolithic block of code: divide and conquer. You can turn a single big problem into a series of easily achievable tasks. The human mind simply can't juggle so many different variables at one time. Decomposing a system allows you to deal only with one issue at a time, whether that be data validation, retrieval, or display.
Ideally, the system becomes much easier to understand but there﻿'s plenty of potential for getting things wrong as you split up the system between developers, teams, or even different systems. Adding to this potential for trouble is the fact that a divide-and-conquer strategy enables you to more easily deliver the system's functionality in increments, which means that your code will likely have to work seamlessly with code delivered in a previous release.
As I discussed in the June issue of MSDN® Magazine, the Open Closed Principle can make code much easier and less risky to extend over time by dividing the responsibilities of the system into separate modules (see msdn.microsoft.com/magazine/cc546578). Those are significant benefits, but they come with a cost. You need to distribute the responsibilities of the system in a beneficial way. Fortunately there's an entire software design method built around heuristic tools for determining and distributing the responsibilities of a system.

Responsibility-Driven Design
CRC Cards
Responsibility-Driven Design is closely linked to a lightweight modeling technique known as CRC cards (Class or Candidate/Responsibility/Collaborators). These are actual cards used for exploratory design.
To draw up a CRC card, simply take a 3" x 5" note card and write the proposed name of a class across the top of the lined side of the card. On the unlined side of the card, write a sentence or two that serves as a purpose statement for the class. The purpose statement should reflect the Object Role Stereotypes of the class.
Divide the lined side of the note card into two columns. In the left column, list the responsibilities of the class (something along the lines of "knows the invoice header information" or "decides whether to send an e-mail message" or "sends an e-mail message"). Again, each of these responsibilities should fit in with the object role stereotypes of the class. In the right-hand column, list other classes that must be interacted with in order to fulfill the duties in the left-hand column.
CRC cards seem to have been passed over in popularity by UML modeling, but the technique is very effective, much simpler, and all you need is a stack of 3" x 5" note cards. See c2.com/doc/oopsla89/paper.html for more information on CRC cards.
Responsibility-Driven Design (RDD) is an informal design method developed in the Smalltalk community in the late 1980s and early 1990s. Rebecca Wirfs-Brock, who conceived the theory, sums it up like this: "Objects are not just simple bundles of logic and data. They are responsible members of an object community."
As its name implies, RDD starts by breaking down a system or an individual feature into the various actions and activities that the system must perform, then proceeds to assign these responsibilities to objects within the system. Object responsibilities are described as either knowing, doing, or deciding.
Once you have a list of responsibilities, you need to consider what the objects are going to be and how the responsibilities are going to be distributed between the objects. Objects within a system take on a role consisting of closely related responsibilities. You can employ object role stereotypes to help define the role of a single object in the system and to serve as a guide for assigning responsibilities to the various objects in the system.
RDD identifies the six common stereotypes listed in Figure 1. Before I go on to describe these six in detail, I should point out that you can blend stereotypes within a single class. For example, a single class will frequently be both an information holder and a structurer. You might also find some value in defining your own stereotypes within a project. (See the "CRC Cards" sidebar for another idea for modeling elements of your application.)
Figure 1 RDD Stereotypes
Stereotype | Description
Information Holder | Knows things and provides information. May make calculations from the data that it holds.
Structurer | Knows the relationships between other objects.
Controller | Controls and directs the actions of other objects. Decides what other objects should do.
Coordinator | Reacts to events and relays the events to other objects.
Service Provider | Does a service for other objects upon request.
Interfacer | Objects that provide a means to communicate with other parts of the system, external systems or infrastructure, or end users.

Information Holder
Last year I worked on an energy trading system. In that system users might enter a trade to purchase 1,000 barrels of gasoline. In order to make that trade, we had to check the on-hand inventory, which was frequently tracked in metric tons, and compare the desired quantity with the on-hand quantity. Several other times we had to compare two quantities that were in different units of measure.
Let's say that you were building a simplistic energy trading system from scratch. One of the most obvious responsibilities in the system is to represent a quantity of something being traded. Another is to make all of those "less than" or "greater than" determinations. As you probably already know, in order to compare two quantities in the energy system, you have to know the numeric amount and unit of measure of both quantities (you'll probably also need to know the density of the commodity being traded, but I'm skipping over that). One approach would be to create an Information Holder to "know" facts about unit of measure and quantity conversions. The desired behavior might lead you to two different classes like the ones shown in Figure 2.
Figure 2 Handling Units of Measure and Quantity
// UnitOfMeasure is a strongly typed enumeration that "knows" about,
// and can answer questions about a logical Unit of Measure
public class UnitOfMeasure {
  public static readonly UnitOfMeasure MT = new UnitOfMeasure(2500);
  public static readonly UnitOfMeasure LB = new UnitOfMeasure(1);

  // In a real system you'd have to do the conversion by using
  // the density from the real commodity in the barrel
  public static readonly UnitOfMeasure BBL = new UnitOfMeasure(100);

  private readonly double _weightInPounds;

  protected UnitOfMeasure(double weightInPounds) {
    _weightInPounds = weightInPounds;
  }

  // To make the system easier to extend, let's make sure that the
  // responsibility for converting quantities between various units of
  // measure in the UnitOfMeasure class
  public double ConvertIntoThisUOM(Quantity quantity) {
    return quantity.Amount*quantity.Uom._weightInPounds/_weightInPounds;
  }
}

public class Quantity {
  private readonly double _amount;
  private readonly UnitOfMeasure _uom;

  public Quantity(double amount, UnitOfMeasure uom) {
    _amount = amount;
    _uom = uom;
  }

  public double Amount {
    get { return _amount; }
  }

  public UnitOfMeasure Uom {
    get { return _uom; }
  }

  public bool IsLessThan(Quantity other) {
    double otherAmount = _uom.ConvertIntoThisUOM(other);
    return _amount < otherAmount;
  }

  // Also implement
  // bool IsGreaterThan, Equals, Subtract, Add, etc.
}
The Quantity class is an example of the Money pattern (also known as a Whole Value). Quantity is an information holder. Any time the rest of the application has a question about difference or sum of two quantities, the application just has to ask a Quantity object. Quantity doesn't know where its data comes from or how it's going to be consumed. Because of its simplicity, the Quantity isn't all that useful on its own, but that very same simplicity makes it easy to reuse in different ways.
Two of the other domain concepts in an energy trading system are trades and allocations (see Figure 3), both of which contain a surprising number of quantities. The Trade class is strictly responsible for providing data and making determinations about an energy trade. The Trade uses the information in the Allocation class, and both the Trade and Allocation classes use the Quantity class to make decisions.
Figure 3 Trades and Allocations
public class Trade {
  private Quantity _originalQuantity;
  private List<Allocation> _allocations = new List<Allocation>();

  // Creates a new allocation. If the requested quantity
  // is greater than the remaining quantity, return the remaining
  // quantity
  public Allocation DrawDown(Quantity requestedQuantity) {
    Quantity remainingQuantity = GetRemainingQuantity();
    Quantity allocationQuantity =
       requestedQuantity.IsLessThan(remainingQuantity)
      ? requestedQuantity
      : remainingQuantity;

    Allocation allocation = new Allocation(allocationQuantity, DateTime.Today);
    _allocations.Add(allocation);

    return allocation;
  }

  // How much of the original Trade quantity is left?
  public Quantity GetRemainingQuantity() {
    Quantity returnValue = _originalQuantity;
    foreach (Allocation allocation in _allocations) {
      returnValue = returnValue.Subtract(allocation.Quantity);
    }

    return returnValue;
  }
}
The entity and value objects in a rich domain model are a common example of information holders. An information holder may collaborate with service providers like data access classes or configuration classes to fetch more information on demand, or it might be completely built externally by something else like an object relational mapping (ORM) tool.
There's an important point here about designing around behavior rather than designing from a data-centric viewpoint. By starting from a single responsibility, a design evolves where the function of converting values between different units of measure is largely encapsulated by a single class (Quantity). For example, if you look at the database behind a financial system, you'll see many fields named something like XXX_qty and XXX_uom. If you had started from a data-centric perspective and created objects as a faithful representation of the database, you might have ended up duplicating quite a bit of the quantity comparison and arithmetic functionality.
In fact, that's exactly what I observed in the real-life energy trading system this example was based on. Avoid the primitive obsession. Don't be afraid to create small objects instead of wallowing in the mud of primitive variables.
The job of a structurer is to track, store, and maintain relationships between objects. The humble Dictionary<T, U> class introduced in the Microsoft® .NET Framework 2.0 is an example of a simple structurer. If you have some sort of many-to-many relationship among entities in your domain model, you will often need to use some sort of structurer object to model the connections.
In some situations it's valuable to keep the structurer's responsibility completely separate from the business processes that consume the objects held by the structurer. For example, in the early days of a system you may use a simple, naive data structure to cache data. In this situation, you probably don't have enough time to design the ultimate data structure because you need to make your first release.
As your system grows in scope, the volume of data that it needs to process will grow along with it, potentially making your original data structure a performance and scalability bottleneck. If you have created a separate object to fulfill the structurer role, you should be able to replace that object with a different data structure without requiring a change to the consumers of the original data structure

Service Provider
A service provider knows how to perform a task on behalf of another object, but it's generally passive while it waits to be activated by some other class.
I work on an open source tool called StoryTeller that creates, manages, and runs automated acceptance tests authored for the FitnesseDotNet engine. Test data is stored in a Test class (an information holder stereotype). StoryTeller executes tests with the FitnesseDotNet engine, which expects its test data to be expressed as HTML tables. A couple other screens also display HTML views or export HTML reports of test data.
Needless to say, I needed to be able to convert the Test objects into HTML in several different contexts. I therefore created a service provider class called HTMLWriter that is responsible for converting a Test object into an HTML string:
public interface IHTMLWriter {
  // Converts a Test (Test : ILeaf) into HTML form
  string GetHTML(ILeaf leaf, bool withStyle);

  // Writes out the Test into HTML form, prepended with the tables for
  // SetUp and TearDown from the Test's containing Suite
  string GetHTMLWithSetupAndTearDown(Test test, bool withStyle);
}
HTMLWriter was easy to build because it simply takes in a Test object and returns a formatted string. I didn't have to consider how to retrieve the Test object or deal with the rest of the system. Unit testing, as shown in Figure 4, was easy because I could just build up a Test object, run it through HTMLWriter, and check that the output was formatted as expected.
Figure 4 Testing HTMLWriter
[Test]
public void Write_out_a_single_table() {
  Test test = new Test();
  test.AddTable()
    .AddRow("type1")
    .AddRow("a,b,c");

  HTMLWriter writer = new HTMLWriter();
  string expectedHTML =
    "<table><tr><td colspan=\"3\">type1</td></tr>" +
    "<tr><td>a</td><td>b</td><td>c</td></tr></table>";

  string actualHtml = writer.GetHTML(test, false);
  Assert.IsTrue(actualHtml.Contains(expectedHTML));
}
If I hadn't isolated the responsibility for converting a Test into HTML into a single service provider, I would have gotten into trouble. One of the consumers of HTMLWriter is the TestRunner class, partially shown here:
public class TestRunner : MarshalByRefObject, ITestRunner {
  public TestResult ExecuteTest(Test test) {
    HTMLWriter writer = new HTMLWriter(test);
    string html = writer.GetHTMLWithSetupAndTearDown(test, true);
    TestResult result = ExecuteTest(html);
    result.ReadPropertiesFromTest(test);

    return result;
  }
}
As you can see, TestRunner is a very busy class in and of itself. (During the course of writing this article, I've realized that TestRunner has some additional unrelated responsibilities that do not fit the class.) If I had added the responsibility for converting a Test into HTML, TestRunner would be far too complicated for my comfort. Moreover, the system itself would have been much harder to test because there would have been no way to test the HTML conversion independently from executing a Test against the FitnesseDotNet engine.
In enterprise development these days, your code rarely lives in isolation. Some of the most difficult tasks involve integrating with other systems—and these other systems are going to need to interface with your code. Even within your own system, your code will need to interact with other subsystems. The interfacer role is effectively a mediator used to simplify communication with another system or subsystem.
A classic example of an interfacer is the Facade pattern. In the energy trading system I discussed earlier, when new trades were created we needed to invoke the pricing rules to determine a price for the new trade. Just by itself, pricing is a very complicated process. It's a subsystem in its own right that probably contains dozens of individual classes. In order to make the pricing functionality easily consumable by the team down the hall, we could have created a Facade class that hides the complexity of the underlying pricing structure:
public class PricingSystem {
  public void Price(Trade trade) {
    // PricingSystem is going to call on several underlying
    // services in the Pricing subsystem in order to
    // add pricing information to the trade
  }
}
Now, when we need to price a trade as part of submitting a new trade, the interaction goes like this:
public class TradeService {
  public void SubmitTrade(Trade trade) {
    // pre-processing

    PricingSystem pricingSystem = new PricingSystem();
    pricingSystem.Price(trade);

    // persist the Trade and send the proper notifications
  }
}
The value of the Facade is that the pricing functionality is easy to consume because there is just a single class and method to call without forcing the consumer of the subsystem to be an unmanageable monolithic block of code. If every dependency on pricing in the system goes through PricingSystem, you have much more freedom to change the pricing subsystem without having to worry about breaking or changing other parts of the system.

Coordinator
A coordinator reacts to events and relays commands to other objects. A coordinator is valuable when you have a process that is event driven and it is convenient to decouple listening or detecting events from the processing that occurs when an event is triggered.
As an example, I worked on a shipping system that automated the flow of boxes and activities on a factory floor. The system would direct other actions and process data when it received socket messages from physical scanners along the floor. Let's break that problem down into the responsibilities:
  1. One way or another, you need to listen for socket messages from the physical scanners.
  2. You need to route boxes throughout the system once a box reaches a certain point on the factory floor.
  3. It may not be obvious, but you must translate between the IP address of the incoming socket communication and the physical or logical location of the scanner on the factory floor.
The real system from this example combined all three responsibilities in the same class—to everyone's sorrow. The business processing couldn't be tested without having the scanners replicated in a testing lab. You've got to test everything together at some point, but it would have been much more efficient to have the business processing tested in isolation before bothering with the integration testing. Reading the code was a nightmare because we had to translate IP addresses in the code into their logical meaning.
The main reason not to write code this way is that all three responsibilities identified above change independently of each other. I shouldn't have to touch business logic code when the factory changes to a different type of physical scanner or has to change the IP address when a scanner is replaced.
Instead, I'll separate the responsibilities into three classes as shown in Figure 5. The advantage of this design is that I've completely decoupled the shipment routing functionality from the physical infrastructure of the factory floor. This design would have given the shipping system team far greater abilities to simulate factory scenarios and enabled them to push automated testing out of the specialized testing lab.
Figure 5 Decoupling Responsibilities
// Information Holder class that maps the IP address of the
// physical scanner to the logical "Position" within
// the factory
public interface IFactorySensorMap {
  Position FindPositionFromScannerAddress(IPAddress address);
}

// The business processing code
public interface IShippingSystem {
  void BoxDetectedAt(string barCode, Position position);
}

// The Coordinator class that listens for socket requests
public class ScannerListener {
  private IFactorySensorMap _sensorMap;
  private IShippingSystem _shippingSystem;

  // scanDetected is called as the result of detecting an
  // incoming Socket call
  private void scanDetected(string barCode,
    System.Net.IPAddress ipAddress) {
    Position position =
      _sensorMap.FindPositionFromScannerAddress(ipAddress);
      _shippingSystem.BoxDetectedAt(barCode, position);
  }
}
A multithreaded business process is another great example of the value of a coordinator. Getting thread management code right is hard enough without having to wade through intermingled business logic. From experience, business logic is much harder to understand when it's embedded directly inside of thread management code. In this case, you can make both responsibilities easier to code, understand, and test by using a coordinator class to manage the threads and simply tell a separate business processing object when events occur. The business logic can then be written and tested independently of the thread management code.
A controller object directs the actions of other classes. A controller is differentiated from a coordinator by the level of control. A controller doesn't just tell other objects that an event happened; the controller decides what the other objects should do based on runtime conditions.
Going back to the energy trading system example, let's say you need to create a screen in a user interface to purchase a quantity of gasoline. Before the new purchase trade can be entered into the system, the screen needs to check whether there is enough gasoline in inventory. If there is enough inventory to fulfill the requested quantity, the system will decrement the on-hand inventory by the requested quantity and submit the new trade. If there is not enough inventory, the screen should show a message box indicating that the requested trade cannot be made.
Using object role stereotypes, you would first define the responsibilities of this screen as something like this:
  1. You need an interfacer to capture the user input and show warning messages.
  2. You have to process a new trade, so you could build a service provider to process that trade.
  3. You need to fetch and update the on-hand inventory, so create another service provider to access and update the on-hand inventory.
  4. You have to compare and also subtract two quantities of gasoline, which may not be expressed in the same units of measure (the inventory may be stored in metric tons and the requested quantity might be expressed in barrels). The Quantity class discussed earlier can handle this responsibility.
  5. Some class needs to be the controller for the screen and proceed to govern the workflow of the requested trade.
Once you see these responsibilities, you might start creating three new classes. The first class is an EnergyTradingScreen that is the actual view to display the warning messages and capture the user data entry. This could be done with either Windows® Presentation Foundation (WPF) or a Windows Forms control. The second class is an InventoryRepository service provider to fetch and update inventory data. The third class is a TradeService service provider to submit a new trade into the system.
This solves some of the tasks in the system, but there is still something missing. You still need to assign the controller responsibilities to some class. Let's examine the existing candidates. The TradeService and InventoryRepository classes are a poor fit. The workflow responsibility doesn't fit in with the responsibilities of a service provider class. The most common design would be to make the EnergyTradingScreen responsible for the workflow, but that has some negative consequences. Instead, you should create a whole new class. Using the Supervising Controller pattern (a form of Model View Presenter), you can put the workflow in a new controller class called EnergyTradingController, shown in Figure 6.
Figure 6 EnergyTradingController
public class EnergyTradingController {
  private readonly EnergyTradingScreen _screen;
  private readonly InventoryRepository _repository;
  private readonly TradeService _service;

  public EnergyTradingController(
    EnergyTradingScreen screen,
    InventoryRepository repository,
    TradeService service) {

    _screen = screen;
    _repository = repository;
    _service = service;
  }

  public void Purchase(Customer customer, Quantity requestedQuantity) {
    // First, we need to decide if the trade could be fulfilled
    Quantity onhandQuantity = _repository.GetOnHandQuantity();
    if (requestedQuantity.IsLessThan(onhandQuantity)) {
      // There is enough inventory to fulfill this request, so place
      // the new trade
      createTrade(requestedQuantity, customer);
    }
    else {
      // Cannot fulfill the trade, so tell the screen to show a user
      // message
      _screen.DisplayQuantityUnavailableMessage(requestedQuantity);
    }
  }

  private void createTrade(Quantity requestedQuantity, Customer customer) {
    Trade newTrade = _service.StartTrade(customer, requestedQuantity);

    // Decrement the onhand inventory so we don't over allocate
     _repository.ReserveQuantity(newTrade);

    // Now we can go ahead and make the trade
    _service.SubmitTrade(customer, requestedQuantity);
  }
}
There are a lot of advantages to extracting a separate controller class. The workflow is easy to understand by just scanning EnergyTradingController because the workflow is the only thing happening in this class. The details of how user messages are displayed, how inventory is really stored in the system, or how trades are processed within the system are managed by other classes. More importantly, pulling the workflow out of the other classes allows those other classes to be simpler and more focused on providing a single service. When I remove workflow responsibilities from the service provider classes, I greatly improve my ability to reuse these services in other contexts besides this one energy trade screen.

Using Object Role Stereotypes Effectively
Designing software is often an exercise in managing complexity. While the overall solution is necessarily as complex as the problem that you're trying to solve, you can take steps to limit the complexity of any given class by only assigning it a discrete set of responsibilities.
When you're deciding where a new responsibility should go, you should ask yourself what object role stereotype fits the new responsibility and look for suitable classes that match that stereotype.
It's not just a matter of the initial creation of the code. Your objects will change over time as bugs are fixed and new functionality is added to existing code. It's often the later releases where the design of a system goes irretrievably wrong as the initially clean structure is corrupted and obscured by layers of cruft. Understanding the object role stereotypes of existing classes can help you know where to assign new responsibilities and give you additional clues for when a completely new class should be added to the system.

Send your questions and comments to mmpatt@microsoft.com.

Jeremy Miller is a Microsoft MVP for C#. He is the author of the open source StructureMap (structuremap.sourceforge.net) tool for Dependency Injection with .NET and the forthcoming StoryTeller (storyteller.tigris.org) tool for supercharged FIT testing in .NET. Visit his blog, "The Shade Tree Developer," at codebetter.com/blogs/jeremy.miller, part of the CodeBetter site.


Team System
Essential Power Tools.
Brian Randell
Contents
Team System Web Access
Team Foundation Server Power Tools
One Last Thing
Microsoft has a long history of introducing new features to already shipped products as a way to add value to what you have purchased. Many teams have released such enhancements under the banner of Power Toys.
The first release to be deemed a Power Toy came from the Windows team for Windows® 95 (see Raymond Chen's blog at blogs.msdn.com/oldnewthing/archive/2005/02/02/365432.aspx for more details). Other teams have followed suit, including the Visual Studio® Team System (VSTS) team. What you need to understand is that, unlike many of the Power Toys released by other teams at Microsoft, I find many of the tools released in this fashion by the VSTS team to be mandatory.
In this column, I'll provide you with a tour of the major tools that are available today and describe the ones I believe you really should download and use. While many items I'll discuss exist for VSTS 2005, I'll be focusing on the current releases available for the VSTS 2008 versions of the tools.

Team System Web Access
The very first item on your download list should be the VSTS Web Access 2008 Power Tool (go.microsoft.com/fwlink/?LinkId=120834). Team System Web Access, or TSWA, provides a rich Web-based access mechanism to your Team Foundation Server (TFS) installation, as shown in Figure 1. TSWA allows you to manage work items and builds, view and run reports, access SharePoint® document libraries, and view source files under version control. Microsoft acquired this tool and added it to its stable of available tools in March of 2007.
Figure 1 Team System Web Access
The team changed the name from Power Toy to Power Tool after the first few releases. Brian Harry, Technical Fellow and "godfather" of TFS, discusses this in his blog (blogs.msdn.com/bharry/archive/2006/09/04/739645.aspx and blogs.msdn.com/bharry/archive/2006/11/30/team-foundation-server-roadmap.aspx). Brian points out that the features the VSTS team is providing aren't really toys. They are out-of-band or off-cycle product updates or enhancements.
Note that the VSTS team and Microsoft both support these tools. Thus, you can use them with confidence. In fact, over time, many of the tools will find their way into the core products. For example, TSWA is planned to be a core component of the next release of Team System.
While I personally tend to prefer a rich client experience, having a rich Web interface is really a core feature needed by most development teams using TFS. When you install TSWA, you get the flexibility of using integrated authentication or forms-based authentication. Thus, you have the flexibility of installing TSWA for internal use only on your intranet or you can deploy it in an extranet environment.
Even though you can install TSWA on your application tier, typically you should deploy it on its own server. You just need Windows Server® 2003 or Windows Server 2008, IIS, and the Team Foundation Client 2008 installed. The client must be installed because currently there is no redistributable runtime for the TFS libraries. TSWA, along with any custom code you write, uses the client libraries to access TFS programmatically.
Once you deployed TSWA, you have a rich Web interface that duplicates most of the features found in the Windows-based client (checking in and out source code being the major missing feature). TSWA also adds some new features of its own, such as data access restrictions, text search, and workflow diagrams on work items.
You may be wondering how the SharePoint portal that the New Team Project wizard creates relates to TSWA. In general, you can view the SharePoint site as the primary Web interface for your internal team and stakeholders. The SharePoint site focuses on artifacts related to the software solution you are creating, overall project process, and reporting. TSWA provides supplemental features not available today via SharePoint, such as work items. In addition, TSWA provides an easier way to bring in external stakeholders and users—in particular, those users who don't need access to SharePoint but might want to file a bug and track its progress.
This last feature is significant. With the release of Team Foundation Server 2008, Microsoft modified the license agreement to allow any person to file a bug, track its status, or defect without the need for a client access license. However, TSWA doesn't offer an easy way to restrict access to features that could allow the user to accidently violate the license agreement (for example, managing another user's work items). To address this, Microsoft has published plans to provide an update for VSTS 2008, currently known as the TFS Bug Submission Portal Power Tool. This update grants access to just the limited feature set allowed by the license change. You can read the spec at go.microsoft.com/fwlink/?LinkId=120835. This is just one example of updates planned under the Power Tools umbrella.

Team Foundation Server Power Tools
The next item is a set of utilities of great value to just about every team: Visual Studio Team System 2008 Team Foundation Server Power Tools (go.microsoft.com/fwlink/?LinkId=120836). As of press time, the March 2008 release is the most current.
The collection of tools includes TFPT.exe, the Visual Studio Team System Process Editor, Team Foundation Server Best Practice Analyzer, work item templates, custom check-in policies, a Build Notification applet, and the TFSServerManager client. Included in the download is a detailed Microsoft® Word document that describes the contents and how to use the various tools.
TFPT.exe is a command-line tool found under %Program Files%\Microsoft Team Foundation Server 2008 Power Tools. It extends the built-in TF.exe command-line tool that is primarily used for working with the version control subsystem of TFS. Currently there are 19 supported commands (see Figure 2).
Figure 2 TFPT.exe Commands
Command | Description
tfpt annotate | Display line-by-line change information for a file.
tfpt bind | Convert VSS-bound solutions into TFS-bound solutions.
tfpt changedocurl | Update the server information on TFS-bound office documents.
tfpt createteamproject | Create a team project.
tfpt destroygl | Destroy a global list.
tfpt destroywi | Destroy one or more work items.
tfpt destroywitd | Destroy a work item type and its work items.
tfpt getcs | Get only the changes in a particular changeset.
tfpt history | Show the history of an item.
tfpt online | Pend adds, edits, and deletes to writable files.
tfpt query | Query for work items.
tfpt review | Review (diff/view) workspace changes.
tfpt rollback | Pend changes to roll back a changeset.
tfpt treeclean | Delete files that are not under version control.
tfpt tweakui | Enhance how client connects to TFS.
tfpt unshelve | Unshelve into workspace with pending changes.
tfpt uu | Undo changes to unchanged files in the workspace.
tfpt workitem | Create, update, or view work items.
tfpt workspace | Update a workspace.
Depending upon the command you choose, you might be required to provide additional parameters. Also, some of the commands, such as the rollback command, provide a GUI experience once you execute the command. Be aware that many of the commands require you to execute the tool from a folder that you've mapped to a TFS workspace. You'll want to modify your Path so you can easily execute TFPT.exe from any of your local workspaces.
For those of you who've used Team System for a long time, you might notice one command jumping out at you: createteamproject. For some users, this has been a long time coming. The big caveat I want to mention is that you must apply SP1 to your Visual Studio 2008 client installation in order for this command to work. Other commands that I find very useful in my work include rollback, treeclean, and workspace. Check them out for yourself.
There's more goodness to be found related to version control in the Team Foundation Server Power Tools. First, there's the custom check-in policies pack. I described check-in policies in detail in the November 2007 issue of MSDN® Magazine (msdn.microsoft.com/magazine/cc163320). This adds a number of useful check-in policies to your installation. Custom Path Policy and Changeset Comments Policy are two I can't live without.
Second, you'll find a number of enhancements inside Visual Studio. Microsoft added some enhancements from the Team Foundation Server 2005 Power Tools in TFS 2008. Annotate is one example. However, some other goodies still require the Power Tools. For example, you get the Find in Source command, Quick Label, and a simple but welcome feature that opens the selected folder in the Source Control Explorer in Windows Explorer (see Figure 3).
Figure 3 Open Folders in Windows Explorer
Each time you create a team project, you specify a process template. Microsoft includes two out of the box. That said, I don't have a single customer who doesn't want to modify the base process templates. Most teams need to add additional fields to some of their work items, or they want to add their company logo to their reports, change the name of the Bug work item to Defect, or add their own docs to the SharePoint process guidance.
Microsoft did a great job of exposing extensibility points to enable these kinds of customizations. However, the out-of-the-box experience does leave something to be desired, unless you're editing process templates for a living. Without the Power Tools, you must download the process template to your local disk, put it under version control, and start editing the files.
You can edit some files, such as your report definition files, using nice GUI designers like those provided in Business Intelligence Development Studio. Other files types, however, like the ever-popular work item type definition files, require you to fire up Notepad or your favorite XML editor and begin hacking angle brackets. This is not the best experience if it's your first time modifying process templates.
Thankfully, the Power Tools include a process template editor. Once installed, you'll find the Process Editor menu command off the Visual Studio Tools menu. This tool allows you to edit your downloaded template's components on disk (or, against my general advice, you can edit some items such as work item type definitions online). For example, you can open the Scenario work item type from the Microsoft Solutions Framework for Agile Software Development version 4 template, define a new field for tracking your beverage of choice to drink once you ship a feature described by the scenario, then add this field to the work item layout, and get a preview of the change (see Figure 4), all without seeing any XML.
Figure 4 A Modified Scenario Work Item Type Definition
That said, as you become more comfortable with process template modifications, you might find that editing the XML files directly is more convenient. The good news is the Process Editor helps there, too. On most screens, you'll find a View XML button to show the XML fragment related to the particular feature you're editing. A big enhancement in the March 2008 release is updated support in the editor for adding custom controls to your work item type definitions. Prior to this release, you would receive an error if you opened a work item type definition that used custom controls.
Another feature related to work items in the Power Tools is work item templates. You create a work item template containing common default items for a particular work item type. Then when you need to open a new bug, for example, you select the template, and it will prepopulate the work item for you. Enhancements in the March 2008 release include work item templates having their own node in the Team Explorer window.
Over the years, Brian Harry has posted statistics related to the usage of TFS by Microsoft to his blog. These posts include data such as the number of items under version control, the server uptime, and the number of users. In the March 2008 release, Brian has published the tool he's been using for this under the guise of the TFSServerManager client. This tool provides an easy way to gather your own statistics on TFS usage for your development team.
The Team Foundation Server Best Practices Analyzer is a diagnostic tool that gathers configuration information from your TFS deployment and performs tests on it. It then analyzes the information using specific rules and gives you the results in the form of information, warning, or error messages. Beyond that it provides links to articles to help you resolve any warning and error situations.
This tool is useful both as a preinstallation step as well as for helping you maintain and monitor your TFS installation. You'll find documentation is included with it in the form of a Windows Help file (TfsBpa.chm). But you must have Windows PowerShell™ installed before you install the Best Practices Analyzer.
Finally, related to Team Build, you'll find the Build Notification Power Tool. This tool runs in the Windows task bar's notification area. You configure it to monitor a specific set of build definitions on your TFS installation. You then tell it to notify you based on various events including when a build is queued, started, or completed. When one of the events occurs, such as a build is queued up, you'll receive a message in the lower-right corner of your screen (see Figure 5).
Figure 5 Notification for a New Build

One Last Thing
All of the tools that I've referenced are great. Some, however, can adversely affect your local machine or server installation (anything with destroy comes to mind). Wouldn't it be nice to try things out where you wouldn't affect your real work environment? On the other hand, maybe you don't own TFS and want try it out. Or possibly you're interested in Visual Studio Team System 2008 Database Edition but don't currently own it?
One of the jobs I perform for Microsoft is to build a Microsoft Virtual PC image containing a complete Team System installation along with sample data and hands-on labs. You can download the compressed image from go.microsoft.com/fwlink/?LinkId=120876. This image contains a full TFS installation as well as Visual Studio 2008 Team Suite.
Assuming you have a machine that can run Virtual PC 2007 (or Virtual Server 2005 R2 SP1) and has enough free disk space, RAM, and a modern processor, you can download the image and try things out in your own private sandboxed environment. Microsoft posted the most recent version of the image at the end of March 2008. And this image will work until December 31, 2008. You should note that Microsoft restricts the use of this image to evaluation only—no production work, please.
Check the download site every few months, since Microsoft will post updates to the image containing Windows updates, additional Power Tools, and more. This is a great way to try things out, like the Power Tools, as well as to evaluate products you don't currently own.
The members of the Microsoft Team System team are committed to delivering the best products possible. Knowing they can't get everything in the box and still ship, they deliver these great additional products and features that add value to your existing purchase. That said, I only had space to cover the two biggest groups of Power Tools. You can also download Visual Studio Team System 2008 Database Edition Power Tools (go.microsoft.com/fwlink/?LinkId=120940) and Visual Studio Team System Architecture Edition Power Tools (go.microsoft.com/fwlink/?LinkId=120941), as well as the core Visual Studio product. I hope you get a chance to download and start getting the most out of your VSTS installation.

Send your questions and comments for Brian to mmvsts@microsoft.com.

Brian A. Randell is a Senior Consultant with MCW Technologies LLC. He spends his time speaking, teaching, and writing about Microsoft technologies. Brian is the author of Pluralsight's Applied Team System course and is a Microsoft MVP. Contact Brian via his blog at mcwtech.com/cs/blogs/brianr.


Foundations
Workflow Tips and Tricks
Matt Milner
Code download available at: Foundations2008_08.exe (187 KB)
Browse the Code Online
Contents
Creating Template Activities
Managing Client Endpoints with ChannelManagerService
Sharing Persistence Stores across Applications
As an author and instructor, I often get asked questions about Windows® Workflow Foundation (WF) that do not require enough depth to warrant an entire column. However, the responses and techniques have proven to be useful to many developers and often can be used in a wide variety of scenarios. This month I will cover several different topics, some that are intended to address specific reader questions, such as how to safely share a persistence database across apps, and others that show how to leverage parts of the framework that do not get much coverage, such as the ChannelManagerService.

Creating Template Activities
One common request from developers is to be able to create activity templates. People are looking for a way to create most of the logic in an activity but leave some extensibility points for consumers of the activity to fill in with their own activities. Unfortunately, Windows WF does not support that model directly, but there are two alternative options that work in many scenarios.
The first option is to create a Visual Studio® new-item template that contains the template structure of your activity, and then let the users add that new item to their project and customize it. In this scenario, developers are the expected user because the solution requires Visual Studio to create an activity from the template and a compiler to build the new activity so that it can be used.
I wrote about creating custom templates in Visual Studio in the January 2006 issue of MSDN® Magazine (msdn.microsoft.com/magazine/cc188697), so I will not spend time on the details here. The benefit of this model is that it leverages the tools available and provides a quick way to ensure a consistent base structure for your activities.
The second option for creating a template is to build a custom toolbox item that can create the template when the user drags an item onto the Workflow Designer surface. In the December 2006 issue of MSDN Magazine (msdn.microsoft.com/magazine/cc163504), I built a custom switch activity that was a composite activity with multiple child branches.
Without a custom toolbox item, when a user dragged my switch activity from the toolbox to the design surface, the default toolbox behavior was to simply add an instance of the switch activity. Someone who did not know about the activity would likely think there was a problem with it, as there would be no indication of where to put his custom logic. The toolbox item allowed me to initialize the switch activity with two branches so the user would know where to begin customizing the activity, as shown in Figure 1.
Figure 1 Switch Activity with and without a Custom Toolbox Item
The switch activity is a composite activity that provides control flow logic but not any custom business logic. Simple composite activities, the kind you get when you choose to add a new activity to your project, generally derive from the SequenceActivity class and rely on contained activities to define business process logic.
An example of this type of composite activity would be a manager-approval process where the general process is reusable, but the individual steps can vary for each process instance. You might, for example, want to create a template that consists of resolving manager details such as e-mail or instant messenger address, sending him a message, and handling all errors around those steps. You might also want to allow someone to customize this process so he can more easily manipulate the message being sent or the parameters around how it should be sent.
Included in the code download for this column are a few sample activities including simple activity stubs for getting manager details and sending an instant message (IM). The included NotifyManagerActivityToolboxItem creates activities, adds error handling, and binds properties on several activities together. The key overrides in the ActivityToolboxItem base class are CreateComponentsCore and OnComponentsCreated.
CreateComponentsCore provides access to the IDesignerHost that can be used to retrieve a number of services and expects, in return, an array of components to be added to the design surface. Figure 2 shows the code to create the hierarchy for the custom activity to notify a manager.
Figure 2 Creating an Activity Hierarchy
protected override System.ComponentModel.IComponent[]
  CreateComponentsCore(System.ComponentModel.Design.IDesignerHost host) {

  //configure the activities
  SequenceActivity parent = new SequenceActivity();
  GetManagerActivity getMgr = new GetManagerActivity();
  SendIMActivity sendIM = new SendIMActivity();

  //create the hierarchy
  parent.Activities.Add(getMgr);
  parent.Activities.Add(sendIM);

  //add fault handling
  FaultHandlersActivity handlers = new FaultHandlersActivity();
  FaultHandlerActivity handler = new FaultHandlerActivity();
  handler.FaultType = typeof(System.ArgumentException);

  WriteLineActivity errorWriteLine = new WriteLineActivity();

  handler.Activities.Add(errorWriteLine);
  handlers.Activities.Add(handler);
  parent.Activities.Add(handlers);

  //return the activities to be used
  return new System.ComponentModel.IComponent[] { parent };
}
It is important to note that in this example I created a single parent activity because my goal was to include error handling. By using a composite activity as a parent activity, I was able to add the fault-handling logic into the hierarchy. I could just as easily have returned an array containing the GetManagerActivity and the SendIMActivity with no root parent activity, and both would get added to the designer. Another option would be to have a custom parent activity that provides the ability to carry the customization beyond the initial adding of activities to the design surface.
Also note that I did not actually set up any activity binding in this method. I only created the activity tree. Because I have not given my activities names yet and instead rely on the designer services to do that for me, I have delayed any configuration among the activities for the OnComponentsCreated method. In this overridden method, I gain access to all of the activities created by my toolbox item, and I can then configure them with activity bindings, as shown in Figure 3.
Once the custom toolbox item class has been written, the ToolboxItem attribute is used to link it to a specific activity:
[ToolboxItem(typeof(NotifyManagerActivityToolboxItem))]
public partial class NotifyManager: SequenceActivity
{ ... }
Figure 3 Configuring Activity Binding
protected override void OnComponentsCreated(
  System.Drawing.Design.ToolboxComponentsCreatedEventArgs args) {

  base.OnComponentsCreated(args);

  //get a handle to the sequence
  SequenceActivity parent = args.Components[0] as SequenceActivity;

  if (parent != null) {
    GetManagerActivity getMgr = null;
    SendIMActivity sendIM = null;
    FaultHandlerActivity faultHandler = null;

    //get activity references
    foreach (Activity a in parent.Activities) {
      if (a is FaultHandlersActivity)
        faultHandler =
          ((FaultHandlersActivity)a).Activities[0] as FaultHandlerActivity;

      if (a is GetManagerActivity)
        getMgr = a as GetManagerActivity;

      if (a is SendIMActivity)
        sendIM = a as SendIMActivity;
    }

    //bind send IM to get manager
    if (getMgr != null && sendIM != null) {
      ActivityBind sipBind = new ActivityBind(
        getMgr.QualifiedName, "ManagerSIPAddress");
      sendIM.SetBinding(SendIMActivity.SIPAddressProperty, sipBind);
    }

    //bind write line activity to fault message
    if (faultHandler != null) {
      ActivityBind faultBind = new ActivityBind(
        faultHandler.QualifiedName, "Fault.Message");
      WriteLineActivity errorWrite =
        faultHandler.Activities[0] as WriteLineActivity;

      errorWrite.SetBinding(WriteLineActivity.OutputTextProperty, faultBind);
    }
  }
}
Notice that because, in this case, the toolbox item actually creates the structure of the activity, the definition of my activity structure is empty. The activity acts only as a shim for the toolbox item so that it can create the template.
In addition to being able to build and configure the activities themselves, it is often useful to work with the services available in the IDesignerHost that is passed to the CreateComponentsCore method. One service interface that is especially helpful in this regard is the IExtendedUIService, shown in Figure 4. This service provides useful methods for selecting or manipulating properties and adding assembly or Web service references.
Figure 4 IExtendedUIService Interface Definition
public interface IExtendedUIService {
  void AddAssemblyReference(AssemblyName assemblyName);
  void AddDesignerActions(DesignerAction[] actions);
  DialogResult AddWebReference(out Uri url, out Type proxyClass);
  Type GetProxyClassForUrl(Uri url);
  ITypeDescriptorContext GetSelectedPropertyContext();
  Uri GetUrlForProxyClass(Type proxyClass);
  Dictionary<string, Type> GetXsdProjectItemsInfo();
  bool NavigateToProperty(string propName);
  void RemoveDesignerActions();
  void ShowToolsOptions();
}
One flaw with both solutions to providing templates is that the created template is "white box" because once you create it, the user can easily change it. The best way to guard against this is to write a custom activity validator, as discussed in my column on building custom activities. The validator will execute both in the designer and at compilation, and will enforce any structure or relationships required by the template.

Managing Client Endpoints with ChannelManagerService
In the Launch 2008 issue of MSDN Magazine (msdn.microsoft.com/magazine/cc164251), I discussed how to use the new features in the Microsoft® .NET Framework 3.5 to integrate workflow with Windows Communication Foundation (WCF). One optional but very important class that was new in the .NET Framework 3.5 is ChannelManagerService. The ChannelManagerService class, as its name suggests, is a runtime service that you add into WorkflowRuntime in order to provide better support for the Send activity. Because it is an optional service, it does not get as much coverage as the activities it supports. But as you begin building real workflow applications with the Send and Receive activities, you may find its features to be very compelling. Two specific features that the service provides are client channel caching across workflow instances and the ability to add configured client endpoints through code, rather than through the standard WCF configuration file entries.
When using the Send activity without ChannelManagerService, a new ChannelFactory and new security session are negotiated for each call. Keep in mind that opening channels and renegotiating security is expensive. If you do not need new security contexts for each call, then the ideal solution would be to have a cache of client channels that the workflows can use, which is what ChannelManagerService provides. It keeps a pool of client channels for endpoints and allows the Send activity to use a client channel from the pool rather than having to create a new channel for each call. The following code creates the service and adds it to the workflow runtime:
ChannelManagerService channelMgr =
  new ChannelManagerService();
workflowRuntime.AddService(channelMgr);
To understand the difference between using the Send activity with and without ChannelManagerService, I created a simple WCF echo service that returns the data sent and the session ID from the operation context. The service itself is configured to require sessions. In the client application, I added a service reference and then created a workflow with two Send activities configured to use the same ChannelToken. The client program runs the workflow twice in parallel, then once more after the first two have completed.
As you can see in Figure 5, when running without ChannelManagerService, each call, whether in the same workflow instance or a different one, uses a new channel factory and, therefore, gets a new session ID. (You also can use the tracing facilities of WCF to see that multiple channels are opened and secure sessions negotiated.) With the ChannelManagerService added to the runtime, the first two workflows each get a unique channel from the pool and use that channel for both of the service calls in the instance. The third workflow then is assigned one of the existing channels from the pool and uses that channel to call the service.
Figure 5 Service Calls with and without ChannelManagerService
When managing a pool of resources, it is important to have some control over the pool itself. ChannelManagerService can be constructed with a parameter of type ChannelPoolSettings that lets you configure three key throttling settings. The LeaseTimeout property indicates the amount of time after a channel is returned to the pool that it is removed from the pool (default value is 10 minutes). The IdleTimeout property indicates how long a channel can be idle before being closed (default value is 2 minutes). Finally, the MaxOutboundChannelsPerEndpoint property indicates how many channels can be in the cache for a given remote endpoint (default value is 10). The default values are intentionally set low and should be reviewed prior to deploying a solution in a test or production environment.
The other feature of ChannelManagerService is the ability to manage endpoints in code much like you can with service endpoints when hosting the service. SendActivity requires you to configure ChannelToken, which in turn requires the name of an endpoint. By default, the framework will attempt to find the endpoint configuration information in the configuration file. ChannelManagerService allows you to pass a collection of named ServiceEndpoints into the constructor, and it will then use those names when looking for the endpoints configured on SendActivity.
This kind of freedom enables parity in the client and server programming model by allowing endpoints to be arbitrarily added to the runtime. If conflicting endpoint names are found between code and configuration, the configuration will take precedence. It's important to note that when using the context bindings to communicate with workflow services, ChannelToken helps manage the context identifiers, so it is not required that you use ChannelManagerService.

Sharing Persistence Stores across Applications
One powerful feature of Windows WF is that workflows can be persisted, their state stored, and the process resumed later. The framework ships with a SQL Server® persistence service and the scripts to create the related database structures and stored procedures. The simple database structure that's defined can either be created in a standalone database or included in an existing application database.
In addition to providing simple state persistence, one of the common use cases for persistence is to share a persistence database across processes to enable workflows to run in multiple hosts. For example, when creating a load-balanced Web application, both Web servers should be configured with the same connection string to allow either server to access the state of a workflow and resume processing when the user interacts with the app.
In some cases it is desirable to have a single persistence database and share it across instances of WorkflowRuntime, usually running in different applications. For example, in a business with several Web applications sharing databases for user login, sessions, and other infrastructure-related stores, it would be nice to be able to share a single database for storing workflow state.
The problem with the workflow persistence is that it does not inherently understand the notion of different applications using the same database. This leads to problems because a workflow that belongs to one application might get loaded by the persistence service that is running in another application.
To see how workflows might get loaded into the wrong application, it is important to understand how the persistence services work with regard to timers or delays. When a workflow has a delay activity in it that causes the workflow to become idle, the persistence service can be configured to indicate that the workflow should be unloaded from memory after being persisted to the data store. To correctly resume the workflow, the persistence service needs to know when the delay or timer expires in order to reload the workflow at that time. It is the job of the persistence service to track when the next timer expires for the workflow instance and to load that workflow into memory when the timer has, in fact, expired.
The SQL persistence service accomplishes this by saving the next timer in the database table along with the workflow state, then polling the database table for expired timers on a configurable interval. In the scenario where two applications are sharing the same database, there is a persistence provider in each application polling for those expired timers. A workflow can be started and go idle in the first application, and it is possible, and in fact likely, that some of the time the persistence service in the other application will query the database for expired timers and find the persisted workflow instance and attempt to load it.
Several problems can occur when the persistence service attempts to load a workflow into an application that does not know about it. First, when a workflow is loaded from the database it must be deserialized. For that to succeed, the type information for the workflow and the activities within it must be discoverable in the host process. Since the workflow was built for another application, the load generally will fail because the workflow type information cannot be found in the second application. You can see this by registering for the ServicesExceptionNotHandled event on the WorkflowRuntime class:
workflowRuntime.ServicesExceptionNotHandled +=
  delegate(object sender,
  ServicesExceptionNotHandledEventArgs snhe) {

  Console.WriteLine(
  "Runtime Service exception: {0}",
  snhe.Exception.Message);
};
In addition to failing to load the workflow and taking the hit of processing an exception each time it tries to load the workflow, the persistence provider actually locks the workflow at the database level for its configured OwnershipTimeout. This means the workflow cannot be loaded by any other persistence provider or runtime until the configured time has passed, essentially making the workflow inaccessible and the original workflow instance's delay activity last longer than expected.
To address this scenario and allow multiple applications to share the same persistence database, it is essential that the persistence service have some knowledge of a logical application where the workflow is running. Fortunately, SQL Server provides the answer in a way with which almost all .NET developers are familiar: the connection string. One of the many parameters that can be included in the connection string is an application name, which can be used by SQL Server tools and, more important, queries to identify the client application when executing queries. When configuring the persistence service for a workflow runtime, the connection string can be augmented to include the logical application name, as shown here:
SqlWorkflowPersistenceService sql =
  new SqlWorkflowPersistenceService(
  "Application Name=HostA;database=wfstate;Integrated Security=SSPI");
Unfortunately, providing the application name is only half the answer because the tables and stored procedures that are included as part of the framework do not use the application information that is sent as part of the connection. To support the application separation, it is necessary to modify the structure and logic of the persistence database slightly.
First, the InstanceState table must be updated to include a column for the application name. In the simplest form, this is an NVarChar column added to the table, though it also would be possible to create a separate Applications table with an ID and name that would allow storing only the application ID in the InstanceState table. For ease of demonstration, I have chosen to show the simple solution in the code download.
Once the table has the ability to store the application information, the stored procedures that save and load workflow state need to be updated to use the application name. SQL Server makes the programming changes simple by providing the APP_NAME function, which returns the current application name for the session that will match the name specified in the connection string.
The primary stored procedure that needs to be updated is InsertInstanceState. This procedure is responsible for inserting the current state of the workflow into the InstanceState table, and it must be updated to also insert or update the ApplicationName column. This procedure actually has four different statements where the SQL needs to be updated, as it handles inserts, updates, and locking in both cases.
The other stored procedures that need to be updated are shown in Figure 6 and are all queries to get the workflows for the current runtime. Each needs to be updated to filter the results to include only those rows from the InstanceState table where the application name is equal to the result of the APP_NAME function.
Figure 6 Stored Procedures that Must Filter Based on Application Name
Stored Procedure | Purpose
RetrieveExpiredTimerIds | Called to get the expired timers. This is the most important query to update, as it is most likely to cause problems.
RetrieveAllInstanceDescriptions | Gets the metadata about all instances stored in the persistence database. This should be filtered so that, when querying the runtime for instances, only those in the current application are returned.
RetrieveANonblockingInstanceStateId | Gets an individual identifier for a workflow that has work to do.
RetrieveNonblockingInstanceStateIds | Retrieves a list of workflow IDs for workflows that have work to do.
The combination of the application name being set on the connection string and the use of the APP_NAME function in the database provides the application concept and isolation needed to share a persistence database across applications. Now when a particular host queries for expired timers, it will only receive those results for workflows that belong to its logical application and that it should be able to successfully load and process. Notice that this solution continues to support the notion of load balancing, as a Web application hosted on two different servers need only use the same connection string, including the application name, to be able to share the workflow state across processes.

Send your questions and comments to mmnet30@microsoft.com.

Matt Milner is an independent software consultant specializing in Microsoft technologies including .NET, Web services, Windows Workflow Foundation, Windows Communication Foundation, and BizTalk Server. As an instructor for Pluralsight, Matt teaches courses on Workflow, BizTalk Server, and Windows Communication Foundation. Matt lives in Minnesota with his wife, Kristen, and his two sons. Contact Matt via his blog at pluralsight.com/blogs/matt.


Windows With C++
Asynchronous WinHTTP.
Kenny Kerr

Contents
WinHTTP Overview
Session Objects
Connection Objects
Request Objects
Request Notifications
Request Cancellation
Sending Request Data
What's Next
As a result of advances in distributed programming, most modern Windows®-based applications must be able to make HTTP requests. Although HTTP is relatively simple, modern HTTP handling is not at all trivial. Asynchronous processing requires buffering potentially large requests and responses, authentication, automatic proxy server detection, persistent connections, and much more. Sure, you can ignore many of these issues but the quality of your application will suffer. And HTTP is not simple enough to imitate with TCP sockets. So what is a C++ developer to do?
A common misconception is that you need to use the Microsoft® .NET Framework if you want your application to access the Web. The truth is that developers using managed code still must deal with many of the same issues I just mentioned, and many of new styles of Web services, such as Representational State Transfer (REST), are just as easy to program using native code.
In this month's column, I am going to show you how to use the Windows HTTP Services, or WinHTTP, API for implementing HTTP clients. On Windows Vista® and Windows Server® 2008, WinHTTP additionally include support for file uploads larger than 4GB, improvements to certificate-based authentication, the ability to retrieve source and destination IP addresses, and more.
WinHTTP provides both a C API as well as a COM API. The C API is naturally harder to use at first, but with a little help from C++ it provides a very powerful and flexible API for building HTTP clients. It also provides both synchronous and asynchronous programming models. I'm going to focus on the asynchronous model for a few reasons. First, parallel programming is critical in today's concurrency-aware world. Second, too much emphasis in documentation and training in general is placed on single-threaded programming to the point where little room is left to discuss parallel programming. What little prescriptive guidance is provided tends to be unrealistic in its simplicity. Many developers shy away from parallel programming, but you'll soon discover that it can be quite natural and even fun! (OK, so your definition of fun might differ from mine, but don't be discouraged.)

WinHTTP Overview
Although the C API doesn't make it obvious, the WinHTTP API is logically modeled as three separate objects: session, connection, and request. A session object is required to initialize WinHTTP. Only one session is required per application. The session helps to create connection objects. One connection object is required for each HTTP server with which you wish to communicate. The connection object, in turn, helps to create individual request objects. An actual network connection is established the first time a request is sent. The relationship between these objects is illustrated in Figure 1. A session may have multiple active connections and a connection may have multiple concurrent requests.
Figure 1 Sessions, Connections, and Requests
Session, connection, and request objects are represented by HINTERNET handles. Although different functions are used to create these objects, they are all destroyed by passing their respective handles to the WinHttpCloseHandle function. Additionally, functions such as WinHttpQueryOption and WinHttpSetOption use the handles to query and set a variety of different options supported by the WinHTTP objects.
Figure 2 lists the WinHttpHandle class that you can use to manage the lifetime of all three types of WinHTTP handles as well as query and set options on them. So, given a request object, you can retrieve its full URL with the WINHTTP_OPTION_URL option:
DWORD length = 0;

request.QueryOption(WINHTTP_OPTION_URL, 0, length);

ASSERT(ERROR_INSUFFICIENT_BUFFER == ::GetLastError());
CString url;

COM_VERIFY(request.QueryOption(WINHTTP_OPTION_URL,
   url.GetBufferSetLength(length / sizeof(WCHAR)), length));
url.ReleaseBuffer();
Figure 2 WinHttpHandle Class
class WinHttpHandle
{
public:
    WinHttpHandle() :
        m_handle(0)
    {}

    ~WinHttpHandle()
    {
        Close();
    }

    bool Attach(HINTERNET handle)
    {
        ASSERT(0 == m_h);
        m_handle = handle;
        return 0 != m_handle;
    }

    HINTERNET Detach()
    {
        HANDLE handle = m_handle;
        m_handle = 0;
        return handle;
    }

    void Close()
    {
        if (0 != m_handle)
        {
            VERIFY(::WinHttpCloseHandle(m_handle));
            m_handle = 0;
        }
    }

    HRESULT SetOption(DWORD option,
                      const void* value,
                      DWORD length)
    {
        if (!::WinHttpSetOption(m_handle,
                                option,
                                const_cast<void*>(value),
                                length))
        {
            return HRESULT_FROM_WIN32(::GetLastError());
        }

        return S_OK;
    }

    HRESULT QueryOption(DWORD option,
                        void* value,
                        DWORD& length) const
    {
        if (!::WinHttpQueryOption(m_handle,
                                  option,
                                  value,
                                  &length))
        {
            return HRESULT_FROM_WIN32(::GetLastError());
        }

        return S_OK;
    }

    HINTERNET m_handle;
};
I use the COM_VERIFY macro in my code snippets to clearly identify where functions return an HRESULT that must be checked. You can replace this with appropriate error handling, whether that is throwing an exception or returning the HRESULT yourself.

Session Objects
A session object is created using the WinHttpOpen function. The first parameter specifies an optional agent string for your application. The next three parameters specify how WinHTTP will resolve server names. This is useful for controlling whether servers are accessed directly or through a proxy server (read more about handling proxies in the sidebar entitled "Determining Proxy Settings" below). The last parameter contains flags of which there is currently only one defined. WINHTTP_FLAG_ASYNC indicates that WinHTTP functions should operate asynchronously:
HINTERNET session = ::WinHttpOpen(0, // no agent string
                                  WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
                                  WINHTTP_NO_PROXY_NAME,
                                  WINHTTP_NO_PROXY_BYPASS,
                                  WINHTTP_FLAG_ASYNC);
Figure 3 lists the WinHttpSession class that derives from the WinHttpHandle in Figure 2 and simplifies the creation of session objects. WinHttpSession creates the session with certain default values that should suffice in most cases.
Figure 3 WinHttpSession Class
{
public:
    HRESULT Initialize()
    {
        if (!Attach(::WinHttpOpen(0, // no agent string
                                  WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
                                  WINHTTP_NO_PROXY_NAME,
                                  WINHTTP_NO_PROXY_BYPASS,
                                  WINHTTP_FLAG_ASYNC)))
        {
            return HRESULT_FROM_WIN32(::GetLastError());
        }

        return S_OK;
    }
};

Connection Objects
A connection object is created using the WinHttpConnect function. The first parameter specifies the session to which the new connection will belong. The next one lists the name or IP address of the server. The third parameter specifies the server's port number. For this parameter, you can alternatively use INTERNET_DEFAULT_PORT, and WinHTTP will use port 80 for regular HTTP requests and port 443 for secure HTTP requests (HTTPS):
HINTERNET connection = ::WinHttpConnect(session,
                                        L"example.com",
                                        INTERNET_DEFAULT_PORT,
                                        0); // reserved

if (0 == connection)
{
    // Call GetLastError for error information
}
The code in Figure 4 shows the WinHttpConnection class, which also derives from the WinHttpHandle and simplifies the creation of connection objects.
Figure 4 WinHttpConnection Class
{ public: HRESULT Initialize(PCWSTR serverName, INTERNET_PORT portNumber, const WinHttpSession& session) { if (!Attach(::WinHttpConnect(session.m_handle, serverName, portNumber, 0))) // reserved { return HRESULT_FROM_WIN32(::GetLastError()); } return S_OK; } };
Request Objects
The request object is where things start to get interesting. A request object is created using the WinHttpOpenRequest function:
HINTERNET request = ::WinHttpOpenRequest(connection,
                                         0, // use GET as the verb
                                         L"/developers/",
                                         0, // use HTTP version 1.1
                                         WINHTTP_NO_REFERRER,
                                         WINHTTP_DEFAULT_ACCEPT_TYPES,
                                         0); // flags
The first parameter specifies the connection to which the request will belong. The second parameter specifies the HTTP verb to use in the request; if this parameter is 0 then a GET request is assumed. The third parameter specifies the name or relative path of the resource that is being requested. The fourth parameter specifies the HTTP protocol version to use; if 0, then HTTP version 1.1 is assumed. The fifth parameter specifies any referring URL. This parameter is usually WINHTTP_NO_REFERRER, indicating no referrer. The second-to-last parameter specifies the media types that you as the client will accept. If this parameter is WINHTTP_DEFAULT_ACCEPT_TYPES, no types are specified. Servers typically take this to mean that the client only accepts text responses. The last parameter specifies flags that can be used to control the behavior of the request. For example, you might want to specify the WINHTTP_FLAG_SECURE flag for making an SSL request.
Here the synchronous and asynchronous programming models diverge. The synchronous model calls a function to send the request and then calls a function that blocks until a response has been received. The asynchronous model, on the other hand, uses a callback function allowing the remaining function calls to operate asynchronously and thus not block the calling thread.
The callback function is associated with a request object using the WinHttpSetStatusCallback function:
if (WINHTTP_INVALID_STATUS_CALLBACK == ::WinHttpSetStatusCallback(request,
    Callback, WINHTTP_CALLBACK_FLAG_ALL_NOTIFICATIONS,
    0)) // reserved
{
    // Call GetLastError for error information
}
The first WinHttpSetStatusCallback parameter specifies the request about whose progress you would like to be notified. The second parameter specifies the address of a callback function. And the third parameter specifies the notifications that you are interested in receiving. WINHTTP_CALLBACK_FLAG_ALL_NOTIFICATIONS is simply a bitmask that includes all possible notifications.
The callback function should be prototyped as follows:
void CALLBACK Callback(HINTERNET handle,
                       DWORD_PTR context,
                       DWORD code,
                       void* info,
                       DWORD length);
The first parameter provides the handle for the object to which the callback relates. Technically, you can have a callback to handle notifications for session and connection objects, but for the most part this will be a request object. The second parameter is an application-defined value associated with a request when it is sent. The third parameter provides a code indicating the reason for the callback. The last two parameters specify a buffer that may contain any additional information depending on the code provided by the previous parameter. For example, if the WINHTTP_CALLBACK_STATUS_REQUEST_ERROR notification code is received, then the info parameter points to a WINHTTP_ASYNC_RESULT structure.
A request is sent using the WinHttpSendRequest function:
if (!::WinHttpSendRequest(request,
                          WINHTTP_NO_ADDITIONAL_HEADERS,
                          0, // headers length
                          WINHTTP_NO_REQUEST_DATA,
                          0, // request data length
                          0, // total length
                          0)) // context
{
    // Call GetLastError for error information
}
The first parameter identifies the request to send. The second and third parameters specify a string that may contain any additional headers to include the request. If WINHTTP_NO_ADDITIONAL_HEADERS is specified, then no additional headers are initially included. As an alternative, you can add headers to the request before it is sent using the WinHttpAddRequestHeaders function. The fourth and fifth parameters specify an optional buffer containing any data to be included as the content, or body, of the request. This is commonly used for POST requests. The second-to-last parameter specifies the total length of the request's content. This value is used to create the Content-Length header included with the request. If it is greater than the length of the data provided by the previous parameters, then additional data may be written to complete the request using the WinHttpWriteData function.
Since I'm using the asynchronous programming model, the function will return without waiting for the request to be sent, and WinHTTP will provide status updates via the callback function passing to it the context value specified in the last parameter to WinHttpSendRequest.
Let's stop for a moment and consider how we can use C++ to bring together what I've covered thus far. In the previous sections, I introduced the rather straightforward WinHttpSession and WinHttpConnection classes for modeling sessions and connections. I need a WinHttpRequest class that not only wraps the various request-related function calls but also provides a simple way to author specific types of requests for an application. Included in that is efficiently managing a buffer that will store both request and response data as it is transferred. Figure 5 provides the outline of the WinHttpRequest class template that I'll build on during the rest of this article. Much of the interesting logic happens inside of the OnCallback member function. However, let me first describe how the WinHttpRequest class template is designed.
Figure 5 WinHttpRequest Class Template
class WinHttpRequest : public WinHttpHandle { public: HRESULT Initialize(PCWSTR path, __in_opt PCWSTR verb, const WinHttpConnection& connection) { HR(m_buffer.Initialize(8 * 1024)); // Call WinHttpOpenRequest and WinHttpSetStatusCallback. } HRESULT SendRequest(__in_opt PCWSTR headers, DWORD headersLength, __in_opt const void* optional, DWORD optionalLength, DWORD totalLength) { T* pT = static_cast<T*>(this); // Call WinHttpSendRequest with pT as the context value. } protected: static void CALLBACK Callback(HINTERNET handle, DWORD_PTR context, DWORD code, void* info, DWORD length) { if (0 != context) { T* pT = reinterpret_cast<T*>(context); HRESULT result = pT->OnCallback(code, info, length); if (FAILED(result)) { pT->OnResponseComplete(result); } } } HRESULT OnCallback(DWORD code, const void* info, DWORD length) { // Handle notifications here. } SimpleBuffer m_buffer; };
Figure 6 lists the DownloadFileRequest class. It derives from WinHttpRequest and uses itself as the template parameter. This is a common technique used by the Visual C++® libraries to implement efficient compile-time virtual function calls. Like the WinHttpSession and WinHttpConnection classes, WinHttpRequest provides an Initialize member function that initializes the WinHTTP request object. It starts by creating a buffer that will be used for the request and response data. This buffer's implementation isn't important. It just needs to manage the lifetime of a BYTE array. Initialize then calls WinHttpOpenRequest to create the WinHTTP request object and WinHttpSetStatusCallback to associate it with the static Callback member function inside the WinHttpRequest class.
The SendRequest member function just wraps the WinHttpSendRequest function and passes its "this" pointer as the context value for the request. Recall that this value is passed to the callback function and enables it to determine which request object the notification is for. SendRequest uses a static_cast to adjust the "this" pointer to the derived class specified by the template parameter. This is how the compile-time polymorphism is implemented in this case.
Finally, the static Callback member function casts the context value back into a pointer to a request object and calls the OnCallback member function to handle the notification. Since network errors are inevitable, there needs to be some way of handling them naturally. Derived classes implement an OnResponseComplete member function that takes an HRESULT. If it is S_OK, then the request completed successfully. But if there was some failure in the callback, then an appropriate HRESULT indicates the request has failed.
Before we look at the implementation of the OnCallback member function, take another look at the DownloadFileRequest concrete class in Figure 6 to put WinHttpRequest into perspective. The beauty here is that DownloadFileRequest can focus on the specifics involved in downloading a file. It is not bogged down by the complexities of HTTP request management. Its Initialize member function accepts the path to the source of the file to download as well as a destination stream to which to write it. The OnReadComplete member function will be called as each chunk of the response is received; OnResponseComplete is called when the request is completed.
Figure 6 DownloadFileRequest Class
{
public:
    HRESULT Initialize(PCWSTR source,
                       IStream* destination,
                       const WinHttpConnection& connection)
    {
        m_destination = destination;

        return __super::Initialize(source,
                                   0, // GET
                                     connection);
    }

    HRESULT OnReadComplete(const void* buffer,
                           DWORD bytesRead)
    {
        return m_destination->Write(buffer,
                                    bytesRead,
                                    0); // ignored
    }

    void OnResponseComplete(HRESULT result)
    {
        if (S_OK == result)
        {
            // Download succeeded
        }
    }

private:
    CComPtr<IStream> m_destination;
};

Request Notifications
Now I would like to discuss how the OnCallback member function is implemented. As you can see in Figure 7, OnCallback once again uses a static_cast in order to adjust the "this" pointer to point to the derived class. The switch statement is then used to handle various notifications. It will return S_FALSE if there is a particular notification that has not been handled. This may come in handy if you want to override the OnCallback member function in a derived class.
Figure 7 OnCallback Implementation
                   const void* info,
                   DWORD length)
{
    T* pT = static_cast<T*>(this);

    switch (code)
    {
        case <some notification code>:
        {
            // Handle specific notification here.

            break;
        }
        default:
        {
            return S_FALSE;
        }
    }

    return S_OK;
}
The first notification of interest when sending the request, assuming the server is reachable, is WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE. This indicates that the WinHttpSendRequest function call completed successfully. Assuming that you're sending a simple GET request, you can now call the WinHttpReceiveResponse function to instruct WinHTTP to start reading the response
case WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE:
{
    if (!::WinHttpReceiveResponse(m_handle,
                                  0)) // reserved
    {
        return HRESULT_FROM_WIN32(::GetLastError());
    }

    break;
}
:
If a response is received, the WINHTTP_CALLBACK_STATUS_HEADERS_AVAILABLE notification arrives indicating that the response headers are available to read. You can query for various headers that may be relevant to your application. At the very least, you should query for the HTTP status code returned by the server. Header information may be retrieved using the WinHttpQueryHeaders function. A simple implementation might just check for the HTTP_STATUS_OK status code, indicating that the request was successful (see Figure 8).
Figure 8 Check for Successful Request
{
    DWORD statusCode = 0;
    DWORD statusCodeSize = sizeof(DWORD);

    if (!::WinHttpQueryHeaders(m_handle,
                   WINHTTP_QUERY_STATUS_CODE | WINHTTP_QUERY_FLAG_NUMBER,
                   WINHTTP_HEADER_NAME_BY_INDEX,
                   &statusCode,
                   &statusCodeSize,
                   WINHTTP_NO_HEADER_INDEX))
    {
        return HRESULT_FROM_WIN32(::GetLastError());
    }

    if (HTTP_STATUS_OK != statusCode)
    {
        return E_FAIL;
    }

    if (!::WinHttpReadData(m_handle,
                   m_buffer.GetData(),
                   m_buffer.GetCount(),
                   0)) // async result
    {
        return HRESULT_FROM_WIN32(::GetLastError());
    }

    break;
}
WinHttpQueryHeaders is a powerful function. Rather than simply returning the text value of a particular named header, it provides a set of information flags that, for the most part, map to headers. But these flags also provide access to other aspects of an HTTP request/response state.
In Figure 8, I'm using the WINHTTP_QUERY_STATUS_CODE flag in order to retrieve the status code returned by the server. And I'm also using the WINHTTP_QUERY_FLAG_NUMBER modifier to tell WinHttpQueryHeaders that I want the value returned as a 32-bit number rather than as a string.
The third parameter specifies the name of the header to query. If this parameter is WINHTTP_HEADER_NAME_BY_INDEX, the flag passed to the previous parameter identifies the information to return. The next two parameters specify a buffer that will receive the information. The last parameter is useful for reading multiple headers with the same name. If this parameter is WINHTTP_NO_HEADER_INDEX, only the first header's value is returned.
Once you're satisfied that the request was successful, you can call the WinHttpReadData function to begin reading the response. The second and third parameters specify a buffer that will receive the data. In this particular case, I'm using the buffer owned by the WinHttpRequest class template. The last parameter must be zero when using the asynchronous programming model.
When enough response data is received, the WINHTTP_CALLBACK_STATUS_READ_COMPLETE notification arrives, indicating that data may be available in the buffer. The amount of data read is indicated by OnCallback's length parameter. It can be zero, which would indicate that no more data is available, or it can have any value up to the size of the buffer specified in the WinHttpReadData function call (see Figure 9).
Figure 9 Checking Notification
{
    if (0 < length)
    {
        HR(pT->OnReadComplete(m_buffer.GetData(),
                              length));

        if (!::WinHttpReadData(m_handle,
                               m_buffer.GetData(),
                               m_buffer.GetCount(),
                               0)) // async result
        {
            return HRESULT_FROM_WIN32(::GetLastError());
        }
    }
    else
    {
        pT->OnResponseComplete(S_OK);
    }

    break;
}
Now you should be able to see exactly where the "events" in the DownloadFileRequest class in Figure 6 come from. The WINHTTP_CALLBACK_STATUS_READ_COMPLETE handler calls the OnReadComplete method on the derived class each time data is received. This is followed by another call to WinHttpReadData to read the next block of data. Finally, when there is no more data available, it calls the OnResponseComplete method on the derived class with S_OK indicating that the response has been read successfully.

Request Cancellation
WinHTTP provides a relatively error-prone model for asynchronous completion since your application is always notified of an operation's completion via the callback function. Since worker threads are used to execute the callback function, however, canceling a request does require some attention to detail.
A request is canceled simply by closing the request handle using the WinHttpCloseHandle function, but you need to be prepared for subsequent callbacks to arrive after closing the request handle. You can call the WinHttpSetStatusCallback function to indicate that no more callbacks should be made prior to closing the request handle, but WinHTTP does not synchronize this with its worker threads.
You should thus ensure that any shared state is suitably guarded. In particular, any unprotected data that is being shared with the callback should not be freed until the WINHTTP_STATUS_CALLBACK_HANDLE_CLOSING notification arrives.
In addition, you need to synchronize calls to WinHttpCloseHandle with other functions that may be operating on the request object such as WinHttpReadData, WinHttpWriteData, and WinHttpSendRequest. Request cancellation is one of the more problematic areas of asynchronous WinHTTP so be sure to carefully review your code as it relates to cancellation.

Sending Request Data
What about POST requests? Apart from using a different HTTP verb, such requests also have additional data that is included as content within the body of the request. Figure 10 illustrates an updated WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE handler, which can easily handle both GET and POST requests.
Figure 10 Handling GET and POST Requests
case WINHTTP_CALLBACK_STATUS_WRITE_COMPLETE:
{
    HRESULT result = pT->OnWriteData();

    if (FAILED(result))
    {
        return result;
    }

    if (S_FALSE == result)
    {
        if (!::WinHttpReceiveResponse(m_handle,
                                      0)) // reserved
        {
            return HRESULT_FROM_WIN32(::GetLastError());
        }
    }

    break;
}
When the WINHTTP_CALLBACK_STATUS_SENDREQUEST_COMPLETE notification arrives, it gives the derived class an opportunity to write data as part of the request by calling its OnWriteData member function. The WinHttpRequest class template provides a default implementation for GET requests that simply returns S_FALSE. Data can be written using the WinHttpWriteData function:
if (!::WinHttpWriteData(request,
                       buffer,
                       count,
                       0)) // async result
{
    // Call GetLastError for error information
}
The second and third parameters specify a buffer containing the data to write. Once the data has been written, the WINHTTP_CALLBACK_STATUS_WRITE_COMPLETE notification arrives. The OnWriteData member function on the derived class is once again called to write the next block of data. Once it has completed, it returns S_FALSE and the base class handler proceeds to call WinHttpReceiveResponse to receive the response as usual.

What's Next
Of course, there's a lot more that I could say about writing HTTP client applications with C++, but unfortunately I'm out of space here. However, I hope this detailed introduction has given you enough information to get you started.
As you can see, WinHTTP provides a modern and powerful API for writing HTTP client applications. Over and above all of the functionality I've described in this article, it also provides a few other important facilities including support for a variety of authentication schemes as well as supplying and validating certificates for SSL. Furthermore, it also provides functions for building and parsing URLs. If you are looking for a rich platform for building efficient and scalable HTTP client applications, WinHTTP may be just what you need.
Determining Proxy Settings
One of the challenges in writing a reliable HTTP client application is the lack of a consistent way to retrieve proxy settings. What happens when the server cannot be reached? How do you determine which proxy server to use? Although there isn't a universal standard in place, with a little effort you can usually figure out which proxy server to use. Unfortunately, many applications instead simply prompt the user for the name of a proxy server. But I think that software should be smart enough not to require the user to supply information that software can usually figure out.
There are a few places to look for proxy settings. I'll start with the most direct approach. In my discussion on Session objects I introduced the WinHttpOpen function for creating a new session object:
HINTERNET session = ::WinHttpOpen(0, // no agent string
                                  WINHTTP_ACCESS_TYPE_DEFAULT_PROXY,
                                  WINHTTP_NO_PROXY_NAME,
                                  WINHTTP_NO_PROXY_BYPASS,
                                  WINHTTP_FLAG_ASYNC);
It had a few parameters for proxy settings that I skimmed over initially. Here I'll describe what they do. The second parameter specifies the access type. Figure A lists the values that may be used.
Figure A Session Access Types
Session Object Parameters | Description
WINHTTP_ACCESS_TYPE_NO_PROXY | Don't use a proxy server by default
WINHTTP_ACCESS_TYPE_DEFAULT_PROXY | Use WinHTTP proxy settings stored in the registry.
WINHTTP_ACCESS_TYPE_NAMED_PROXY | Use the specified proxy settings.
WINHTTP_ACCESS_TYPE_NO_PROXY indicates that proxy settings should not be retrieved or used. You can override this after the session has been created and I'll describe how to do that in a moment.
WINHTTP_ACCESS_TYPE_DEFAULT_PROXY indicates that any proxy settings stored by WinHTTP in the registry should be used. This is useful in allowing administrators to control the proxy settings. It is also independent of Internet Explorer's proxy settings and is thus ideal for server scenarios. The proxycfg.exe utility may be used on Windows XP and Windows Server 2003 to set and query the WinHTTP proxy settings. The proxycfg.exe utility has been replaced on Windows Vista and Windows Server 2008 and the functionality is now provided by the netsh.exe utility. You can also set the WinHTTP proxy settings directly using the WinHttpSetDefaultProxyConfiguration function and query it using the WinHttpGetDefaultProxyConfiguration function. These settings persist across reboots and are shared by all logon sessions. Both functions use the WINHTTP_PROXY_INFO structure which maps directly to the three proxy-related parameters of the WinHttpOpen function.
WINHTTP_ACCESS_TYPE_NAMED_PROXY indicates that WinHTTP should use the proxy settings in the next two parameters. WinHttpOpen's third parameter specifies the name of a proxy server and the fourth parameter specifies an optional list of HTTP servers that should not be routed through the proxy server specified in the previous parameter.
If you're writing a Windows service or server application then this may be all that you need. On the other hand, end users don't typically know how or want to configure proxy settings so a little more effort may be required to make it as seamless as possible. Another approach is to retrieve Internet Explorer's proxy settings since most users will have their browsers already configured to access the Web through a Web proxy. You can find the Internet Explorer dialog for configuring proxy settings by clicking the "LAN settings" button on the Connections tab of the Internet Explorer Options window. As this is a common requirement, WinHTTP provides the WinHttpGetIEProxyConfigForCurrentUser function for directly retrieving these very Internet Explorer proxy settings for the current user:
WINHTTP_CURRENT_USER_IE_PROXY_CONFIG ieProxyConfig = { 0 };

if (!::WinHttpGetIEProxyConfigForCurrentUser(&ieProxyConfig))
{
    // Call GetLastError for error information.
}
Here's what WINHTTP_CURRENT_USER_IE_PROXY_CONFIG looks like:
struct WINHTTP_CURRENT_USER_IE_PROXY_CONFIG
{
    BOOL    fAutoDetect;
    LPWSTR  lpszAutoConfigUrl;
    LPWSTR  lpszProxy;
    LPWSTR  lpszProxyBypass;
};
The non-null string member variables must be freed using the GlobalFree function. The fAutoDetect member maps to the "Automatically detect settings" option in the LAN settings dialog. The lpszAutoConfigUrl member maps to the "Use automatic configuration script" address and is only populated if this option is checked. The lpszProxy member maps to the "Use a proxy server..." address and port and is only populated if this option is checked. Finally, lpszProxyBypass maps to the list of servers specified in the Exceptions area of the window that appears when you click the Advanced button and is only populated if the "Bypass proxy server for local addresses" option is checked.
If lpszProxy is populated you're in luck as you can simply use it as the proxy server. Figuring out what to do in the case of the other options takes a bit more work.
If lpszAutoConfigUrl is populated then it means that Internet Explorer downloads a Proxy Auto-Configuration (PAC) file to determine the proxy server for a particular connection. Reproducing this behavior can be challenging since the downloaded PAC file often contains JavaScript that may be used to direct the client to a particular proxy server depending on the destination server. Fortunately WinHTTP provides the WinHttpGetProxyForUrl function that abstracts this for you:
WINHTTP_AUTOPROXY_OPTIONS autoProxyOptions = { 0 };
autoProxyOptions.dwFlags = WINHTTP_AUTOPROXY_CONFIG_URL;
autoProxyOptions.lpszAutoConfigUrl = ieProxyConfig.lpszAutoConfigUrl;

WINHTTP_PROXY_INFO proxyInfo = { 0 };

if (!::WinHttpGetProxyForUrl(session,
                             L"http://example.com/path",
                             &autoProxyOptions,
                             &proxyInfo))
{
    // Call GetLastError for error information.
}
The non-null string member variables of the WINHTTP_PROXY_INFO structure must also be freed using the GlobalFree function. The first WinHttpGetProxyForUrl parameter specifies the WinHTTP session handle. The second parameter specifies the URL of the request you would like to send. The third parameter is an input parameter and specifies a WINHTTP_AUTOPROXY_OPTIONS structure which controls the behavior of the WinHttpGetProxyForUrl function. This is where you specify the URL of the PAC file retrieved from Internet Explorer proxy settings. The last parameter is an out parameter and specifies the WINHTTP_PROXY_INFO structure that receives the results. The results can then be used with the WinHttpSetOption function along with the WINHTTP_OPTION_PROXY option to set the proxy settings for a particular request object.
Finally, if WINHTTP_CURRENT_USER_IE_PROXY_CONFIG's fAutoDetect member variable is true it means that Internet Explorer uses the Web Proxy Auto-Discovery (WPAD) protocol to discover proxy settings. WPAD relies on updates to the network's DNS or DHCP servers to provide the URL of a PAC script so that clients don't have to be preconfigured with a specific PAC script. This is analogous to the relationship between static and dynamic IP addresses. You can instruct WinHttpGetProxyForUrl to use WPAD to discover the URL of the PAC file automatically by changing the WINHTTP_AUTOPROXY_OPTIONS initialization as follows:
WINHTTP_AUTOPROXY_OPTIONS autoProxyOptions = { 0 };
autoProxyOptions.dwFlags = WINHTTP_AUTOPROXY_AUTO_DETECT;
autoProxyOptions.dwAutoDetectFlags = WINHTTP_AUTO_DETECT_TYPE_DHCP
  | WINHTTP_AUTO_DETECT_TYPE_DNS_A;
Naturally you can use the WinHTTP "AutoProxy" capabilities without a dependency on Internet Explorer. Using Internet Explorer settings is mainly a stopgap until WPAD becomes more ubiquitous in corporate network environments. If WPAD is not available on your network then WinHttpGetProxyForUrl may take a few seconds to return and GetLastError will return ERROR_WINHTTP_AUTODETECTION_FAILED.

Send your questions and comments to mmwincpp@microsoft.com.

Kenny Kerr is a software craftsman specializing in software development for Windows. He has a passion for writing and teaching developers about programming and software design. Reach Kenny at weblogs.asp.net/kennykerr.


Concurrent Affairs
More AsyncEnumerator Features
Jeffrey Richter
Code download available at: ConcurrentAffairs2008_08.exe (155 KB)
Browse the Code Online
Contents
Joining Concurrent Asynchronous Operations
APM Support
Return Values
Callback Thread Control with Synchronization Contexts
Synchronizing Access to Shared Data
Discard Groups
Cancellation
Wrap-Up
In my November 2007 Concurrent Affairs column, I discussed the idea of using C# language features to simplify asynchronous programming (see msdn.microsoft.com/magazine/cc163323). In particular, I focused on anonymous methods, lambda expressions, and iterators. Then in my June 2008 column, I introduced my AsyncEnumerator class and explained how it can be used to drive a C# iterator (see msdn.microsoft.com/magazine/cc546608). I also walked through AsyncEnumerator's architecture and explained how it works internally.
In this column, I want to show you some additional features that AsyncEnumerator provides, such as joining across multiple concurrent asynchronous operations, Asynchronous Programming Model (APM) support, return values, callback thread control, synchronizing access to shared data, automatic discarding of uncompleted operations, and cancellation/timeout support. Along the way, I'll also walk you through some common programming patterns made possible with AsyncEnumerator.

Joining Concurrent Asynchronous Operations
One of the great things about performing asynchronous operations is that you can perform several of them concurrently, greatly improving your application's performance. For example, if you initiate three asynchronous Web service requests concurrently, and if each request takes 5 seconds to complete, then the total time your program spends waiting is just 5 seconds. On the other hand, if you perform synchronous Web service requests, then your application waits for each one to complete before initiating the next. So performing three synchronous Web service requests, each of which takes 5 seconds, means that your application waits at least 15 seconds.
With AsyncEnumerator, it is extremely easy to initiate several concurrent asynchronous operations. Your code can then process the completed operations after all of them complete or as each one completes. The iterator in Figure 1 shows both techniques for processing completed operations. When I ran it, I got the following output (note that you must wait for the Web request to timeout):
All the operations completed:
   Uri=http://wintellect.com/        ContentLength=41207
   Uri=http://www.devscovery.com/    ContentLength=13258
   Uri=http://1.1.1.1/  WebException=Unable to connect to remote server
An operation completed:
   Uri=http://wintellect.com/        ContentLength=41208
An operation completed:
   Uri=http://www.devscovery.com/    ContentLength=13258
An operation completed:
   Uri=http://1.1.1.1/   WebException=Unable to connect to remote server
Figure 1 Coordinating Multiple Asynchronous Operations
public static class AsyncEnumeratorPatterns {
  public static void Main() {
    String[] urls = new String[] {
      "http://Wintellect.com/",
      "http://1.1.1.1/",   // Demonstrates error recovery
      "http://www.Devscovery.com/"
    };

    // Demonstrate process
    AsyncEnumerator ae = new AsyncEnumerator();
    ae.Execute(ProcessAllAndEachOps(ae, urls));
  }

  private static IEnumerator<Int32> ProcessAllAndEachOps(
       AsyncEnumerator ae, String[] urls) {
    Int32 numOps = urls.Length;

    // Issue all the asynchronous operation(s) so they run concurrently
    for (Int32 n = 0; n < numOps; n++) {
      WebRequest wr = WebRequest.Create(urls[n]);
      wr.BeginGetResponse(ae.End(), wr);
    }

    // Have AsyncEnumerator wait until ALL operations complete
    yield return numOps;

    Console.WriteLine("All the operations completed:");
    for (Int32 n = 0; n < numOps; n++) {
      ProcessCompletedWebRequest(ae.DequeueAsyncResult());
    }

    Console.WriteLine(); // *** Blank line between demos ***

    // Issue all the asynchronous operation(s) so they run concurrently
    for (Int32 n = 0; n < numOps; n++) {
      WebRequest wr = WebRequest.Create(urls[n]);
      wr.BeginGetResponse(ae.End(), wr);
    }

    for (Int32 n = 0; n < numOps; n++) {
      // Have AsyncEnumerator wait until EACH operation completes
      yield return 1;

      Console.WriteLine("An operation completed:");
      ProcessCompletedWebRequest(ae.DequeueAsyncResult());
    }
  }

  private static void ProcessCompletedWebRequest(IAsyncResult ar) {
    WebRequest wr = (WebRequest)ar.AsyncState;
    try {
      Console.Write("   Uri=" + wr.RequestUri + "    ");
      using (WebResponse response = wr.EndGetResponse(ar)) {
        Console.WriteLine("ContentLength=" + response.ContentLength);
      }
    }
    catch (WebException e) {
      Console.WriteLine("WebException=" + e.Message);
    }
    }
}
The code at the beginning of the iterator issues several asynchronous operations and then executes a yield return numOps statement. This statement tells AsyncEnumerator not to call back into your code until the number of operations indicated by the value in the numOps variable have completed. The code just below the yield return statement then executes a loop to process all of the completed operations.
Note that the operations can complete in a different order than they were issued. In order to correlate each result with its request, I passed wr, which identifies the WebRequest object I'm using to initiate the request, in BeginGetResponse's last argument. Then in the ProcessCompletedWebRequest method, I extract the WebRequest object used to initiate the request from the IAsyncResult's AsyncState property.
The code at the bottom of the iterator also issues several asynchronous operations and then enters a loop to process each operation as it completes. However, the iterator must first wait until each operation completes. This is accomplished by the yield return 1 statement which tells AsyncEnumerator to call back into the iterator code as soon as a single operation completes.

APM Support
In my previous column, I explained how calling AsyncEnumerator's Execute method starts the execution of an iterator's code. However, I also explained that the thread that calls Execute will block until the iterator exits or executes a yield break statement.
Having the thread block may hurt an application's scalability and is something you'd really want to avoid, especially in server applications. It also hurts responsiveness if called by a GUI thread because the iterator takes an indefinite amount of time to execute, and during this time a Windows® Forms or Windows Presentation Foundation (WPF) application will stop responding to input. You'd really want to call Execute only when writing test code or when experimenting with an iterator method.
For production code, you should call AsyncEnumerator's BeginExecute and EndExecute methods. Internally, when you call BeginExecute, the AsyncEnumerator object constructs an instance of the AsyncResultNoResult class, which I discussed in the March 2007 issue of MSDN® Magazine (msdn.microsoft.com/magazine/cc163467). When you call BeginExecute, you can pass a reference to your own AsyncCallback method, and the AsyncEnumerator object will invoke this method when the iterator completes its execution. This method should then call AsyncEnumerator's EndExecute method to get the iterator's result. I'll show some examples later where I take advantage of the BeginExecute and EndExecute methods. Here is what the methods look like:
public class AsyncEnumerator<TResult>: AsyncEnumerator {
  public IAsyncResult BeginExecute(
    IEnumerator<Int32> enumerator,
    AsyncCallback callback, Object state);

  public void EndExecute(IAsyncResult result);
}
Also, since AsyncEnumerator supports the APM, it can be integrated with all of the existing Microsoft® .NET Framework application models since they already support the APM. This means that you can use AsyncEnumerator with ASP.NET Web Form applications, ASP.NET XML Web services, Windows Communication Foundation (WCF) services, Windows Forms applications, WPF applications, console applications, Windows services, and so on.
Another thing worth pointing out is that, since AsyncEnumerator supports the APM, it can be used inside another iterator and allow for composition of asynchronous operations. For example, you could implement an iterator that knows how to asynchronously make a database request and process the results when they come in. I call this the subroutine iterator. Then, inside another iterator, you could initiate several database requests by invoking the subroutine iterator in a loop. For each loop iteration, you would construct an AsyncEnumerator and call its BeginExecute method, passing in the name of the subroutine iterator and any additional arguments that you want.
Note that there is an important benefit you get with this model: all the subroutine iterators run concurrently without blocking any threads (unless the underlying implementation of the APM blocks threads, such as if BeginXxx were implemented to queue a delegate to the ThreadPool that blocked until some operation completed). This allows you to create a simple iterator that encapsulates a single asynchronous operation and invoke it from within other iterators while maintaining scalability and responsiveness.

Return Values
In many scenarios, it is useful to have the iterator return a result after all of its processing has completed. However, an iterator cannot return a value when it is finished because an iterator cannot have a return statement in it. And yield return statements return a value for each iteration, not a final value.
If you want to have an iterator return a final value after processing, I have created the helper class AsyncEnumerator<TResult>. The model for this class is shown here:
public class AsyncEnumerator<TResult>: AsyncEnumerator {
  public AsyncEnumerator();
  public TResult Result { get; set; }

  new public TResult Execute(IEnumerator<Int32> enumerator);
  new public TResult EndExecute(IAsyncResult result);
}
Using AsyncEnumerator<TResult> is straightforward. First, change your code to instantiate an AsyncEnumerator<TResult> instead of the normal AsyncEnumerator. For TResult, specify the type that you want the iterator to ultimately return. Now, modify the portion of your code that calls the Execute or EndExecute method (which used to return void) to obtain the return value and use this value however you want.
Next, modify your iterator code so that it accepts an AsyncEnumerator<TResult> instead of an AsyncEnumerator. Of course, you must specify the same data type for the generic TResult parameter. Finally, inside your iterator code, set the AsyncEnumerator<TResult> object's Result property to whatever value you want returned from the iterator.
To help put all this together, Figure 2 shows code that implements a simple asynchronous ASP.NET Web service that simultaneously requests the HTML for several different Web sites (passed as a comma-separated string). After all the Web site data has been received, the Web service returns an array of strings with each element showing the Web site's URL and the number of bytes downloaded from that Web site, or an error if an error occurred.
Figure 2 Retrieving Multiple Web Sites Concurrently
[WebService(Namespace = "http://tempuri.org/")]
[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1)]
public class Service : System.Web.Services.WebService {
  private static List<String[]> s_recentRequests = new List<String[]>(10);
  private static SyncGate s_recentRequestsSyncGate = new SyncGate();

  private AsyncEnumerator<String[]> m_webSiteDataLength;

  [WebMethod]
  public IAsyncResult BeginGetWebSiteDataLength(
    String uris, AsyncCallback callback, Object state) {

    // Construct an AsyncEnumerator that will eventually return a String[]
    m_webSiteDataLength = new AsyncEnumerator<String[]>();

    // NOTE: The AsyncEnumerator automatically saves the ASP.NET
    // SynchronizationContext with it ensuring that the iterator
    // always executes using the correct IPrincipal,
    // CurrentCulture, and CurrentUICulture.

    // Initiate the iterator asynchronously.
    return m_webSiteDataLength.BeginExecute(
      GetWebSiteDataLength(m_webSiteDataLength, uris.Split(',')),
                          callback, state);
    // NOTE: Since the AsyncEnumerator's BeginExecute method returns an
    // IAsyncResult, we can just return this back to ASP.NET
  }

  private IEnumerator<Int32> GetWebSiteDataLength(
             AsyncEnumerator<String[]> ae, String[] uris) {

    // Issue several web request simultaneously
    foreach (String uri in uris) {
      WebRequest webRequest = WebRequest.Create(uri);
      webRequest.BeginGetResponse(ae.End(), webRequest);
    }

    yield return uris.Length;  // Wait for ALL the web requests to complete

    // Construct the String[] that will be the ultimate result
    ae.Result = new String[uris.Length];

    for (Int32 n = 0; n < uris.Length; n++) {
      // Grab the result of a completed web request
      IAsyncResult result = ae.DequeueAsyncResult();

      // Get the WebRequest object used to initate the request
      WebRequest webRequest = (WebRequest)result.AsyncState;

      // Build the String showing the result of this completed web request
      ae.Result[n] = "URI=" + webRequest.RequestUri + ", ";

      using (WebResponse webResponse = webRequest.EndGetResponse(result)) {
        ae.Result[n] += "ContentLength=" + webResponse.ContentLength;
      }
    }

    // Modify the collection of most-recent queries
    s_recentRequestsSyncGate.BeginRegion(SyncGateMode.Exclusive, ae.End());
    yield return 1;   // Continue when collection can be updated (modified)

    // If collection is full, remove the oldest item
    if (s_recentRequests.Count == s_recentRequests.Capacity)
      s_recentRequests.RemoveAt(0);

    s_recentRequests.Add(ae.Result);
    s_recentRequestsSyncGate.EndRegion(ae.DequeueAsyncResult());
  // Updating is done //
  }

  // ASP.NET calls this method when the iterator completes.
  [WebMethod]
  public String[] EndGetWebSiteDataLength(IAsyncResult result) {
    return m_webSiteDataLength.EndExecute(result);
  }

  private AsyncEnumerator<String[][]> m_aeRecentRequests;

  [WebMethod]
  public IAsyncResult BeginGetRecentRequests(AsyncCallback callback,
                                             Object state) {
    m_aeRecentRequests = new AsyncEnumerator<String[][]>();
    return m_aeRecentRequests.BeginExecute(GetRecentRequests(m
       _aeRecentRequests), callback, state);
  }

  private IEnumerator<Int32> GetRecentRequests(
     AsyncEnumerator<String[][]> ae) {
    // In a shared way, read the collection of most-recent requests
    s_recentRequestsSyncGate.BeginRegion(SyncGateMode.Shared, ae.End());
    yield return 1;   // Continue when collection can be examined (read)

    // Return a copy of the collection as an array
    ae.Result = s_recentRequests.ToArray();
    s_recentRequestsSyncGate.EndRegion(ae.DequeueAsyncResult());
// Reading is done
  }

  [WebMethod]
  public String[][] EndGetRecentRequests(IAsyncResult result) {
    return m_aeRecentRequests.EndExecute(result);
  }
}

Callback Thread Control with Synchronization Contexts
As asynchronous operations complete, various thread pool threads wake up to notify your AsyncEnumerator object. If AsyncEnumerator used these threads to call back into your iterator, then your iterator's code can be executed by different threads even though an iterator's code will only ever have one thread at a time executing inside it. For some scenarios, having different threads execute the iterator's code is problematic. For example, in a Windows Forms or WPF application, a control must be manipulated by the thread that created the control, and that cannot be a thread pool thread.
Having the iterator code execute via arbitrary thread pool threads can pose another problem. With an ASP.NET application, for example, when a client request first comes in, ASP.NET associates the client's IPrincipal (for impersonation) as well as the Culture information with the thread pool thread's CurrentPrincipal, CurrentCulture, and CurrentUICulture properties. However, if you use this thread to call some BeginXxx method, then when the new thread pool thread executes to notify you of the operation's completion, the new thread pool thread will not, by default, have these properties set correctly.
To help solve these problems, the CLR allows each thread to have a SynchronizationContext-derived object associated with it. This object is used to help maintain the threading model employed by an application model. For Windows Forms and WPF, their SynchronizationContext-derived objects know how to marshal a function call (made by a thread pool thread) to the GUI thread. Regarding ASP.NET, its SynchronizationContext-derived object knows how to initialize the principal and culture properties onto each thread pool thread used to process a single request.
To maintain your application's proper threading model, AsyncEnumerator offers a SyncContext property. This property is initialized to the value returned by SynchronizationContext's static Current property inside AsyncEnumerator's constructor. If it is null—as it normally would be for a console or Windows service application—then whenever a thread pool thread calls into the AsyncEnumerator object, the object just uses that thread to call into your iterator. However, if the SyncContext property is not null, then the AsyncEnumerator object has the thread pool thread call your iterator via the specified SynchronizationContext-derived object.
So for Windows Forms and WPF application, this means that your iterator code will always execute via the GUI thread, and, therefore, you can just execute code in your iterator to update controls on the form. There is no need to call Control's Invoke/BeginInvoke methods or Dispatcher's Invoke/BeginInvoke methods. This makes it simple to have your iterator do progress reporting in your UI as asynchronous operations complete. For ASP.NET, this means that the principal and culture properties will always be set correctly when your iterator code executes. The code in Figure 2 and the Windows Forms example I will show you later both take advantage of this feature.

Synchronizing Access to Shared Data
In some scenarios, especially in server-side scenarios, you may have multiple AsyncEnumerator objects (one per client request) each processing their own iterators simultaneously. For example, imagine a Web site that accesses some database and then updates a set of objects in memory. You'll certainly want to use the APM for the database access (calling SqlCommand's BeginExecuteReader, for example), and then you might need to update the in-memory objects in a thread-safe way. Normally you might use methods of the Monitor class or the C# lock statement, or perhaps the ReaderWriterLockSlim that ships with the .NET Framework 3.5. However, all of these locks can block the calling thread, hurting your scalability and responsiveness. To avoid blocking threads, I tend to use my ReaderWriterGate, which I describe in my May 2006 column (see msdn.microsoft.com/magazine/cc163532).
When I started to use ReaderWriterGate with AsyncEnumerator, I realized that the object model of ReaderWriterGate could be improved to make integration with AsyncEnumerator better. So I created a new class, called SyncGate, that behaves very similar to ReaderWriterGate. Here is its model:
public sealed class SyncGate {
     public SyncGate();
     public void BeginRegion(SyncGateMode mode,
       AsyncCallback asyncCallback);
     public void EndRegion(IAsyncResult ar);
   }
   public enum SyncGateMode { Exclusive, Shared }
If you have multiple iterators running that want to access shared data, first construct a SyncGate and store a reference to it in a static field or somehow get the reference to it out to the various iterators. Then, inside the iterators, just before the code that touches the shared data, call SyncGate's BeginRegion method, indicating whether the code needs exclusive (write) or shared (read) access to the data. Then have the iterator yield return 1. At this point your iterator will give up its thread, and when your iterator can access the data safely, AsyncEnumerator will call back into your code automatically. This means that no threads are blocked while waiting to get access to the shared data.
In your iterator, just after the code that touches the shared data, call EndRegion. This tells SyncGate that your code is finished touching the shared data and allows other iterators the ability to access it if they desire. In Figure 2, the bottom of the GetWebSiteDataLength iterator uses SyncGate to access a static collection exclusively. Also in Figure 2, the GetRecentRequests iterator shows how to gain shared access to the same collection.

Discard Groups
Another feature offered by AsyncEnumerator class is discard groups. This feature allows your iterator to issue several concurrent asynchronous operations and then later decide that it doesn't care about some (or all) of them, causing the AsyncEnumerator object to discard the results automatically as the remaining operations complete.
For example, imagine code that wants to get a city's temperature. There are many Web services that you could query to acquire this information. You could write an iterator that queries three Web services to get this information, and as soon as any of them returns the temperature, you can discard the other two Web services' results. Some people use this pattern to improve the performance of their applications.
Another example is when an iterator performs a sequence of operations and the user just wants to cancel them because he gets tired of waiting or he just changes his mind. A similar scenario exists where you start an asynchronous operation, but if it doesn't complete in some fixed amount of time, you want to discard all operations that haven't completed yet. For example, some Web services process a client's request and if all the processing can't complete in, say, two seconds, the service would rather notify the client that his request failed than have the client wait indefinitely for the response.
Here's how to use discard groups: inside your iterator, you group a bunch of related operations together as part of a discard group. A discard group is just an Int32 value ranging from 0 to 63, inclusive. For example, you can issue a bunch of BeginXxx methods indicating that they are all part of discard group 0. Then your iterator can process some of them as they complete. When you decide in your iterator code that you don't want to process anymore of the operations that are part of this discard group, you call AsyncEnumerator's DiscardGroup method:
public void DiscardGroup(Int32 discardGroup);
This method tells the AsyncEnumerator object to discard any remaining operations that were issued as part of the specified discard group so that your iterator code will never see any of these operations when it calls DequeueAsyncResult. Unfortunately, this is not quite good enough, as the .NET APM mandates that EndXxx methods be called for every BeginXxx method, or resources may possibly be leaked.
To address this requirement, AsyncEnumerator must call the right EndXxx method for any operation it discards. Since there is no way that AsyncEnumerator could figure out the correct EndXxx method to call on its own, you must tell it which one. When you call a BeginXxx method, instead of simply passing ae.End for the AsyncCallback argument, you must pass one of these methods:
AsyncCallback End(Int32 discardGroup, EndObjectXxx callback);
AsyncCallback EndVoid(Int32 discardGroup, EndVoidXxx callback);
EndObjectXxx and EndVoidXxx are delegates defined as follows:
delegate Object EndObjectXxx(IAsyncResult result);
delegate void EndVoidXxx(IAsyncResult result);
If a BeginXxx method has a corresponding EndXxx method that returns a value, then you would call the End method just shown. If you call a BeginXxx method that has a corresponding EndXxx method that returns void (the unusual case), then you would call the EndVoid method. Now whenever you tell AsyncEnumerator to discard a group, it will know what EndXxx method to call.
Note that if the EndXxx method throws any exception at all, AsyncEnumerator catches it and swallows it. It does this because if you are discarding the operation, you are indicating that you don't care about whether the operation succeeded or failed.
I should also point out that when your iterator exits or executes a yield break statement, AsyncEnumerator automatically discards all discard groups—if you exit the iterator, then you are indicating that you don't care about any of the operations that have not been processed yet. This can be very convenient because it allows your iterator to issue some asynchronous operations, process however many completed operations it needs, and then just exit.
AsyncEnumerator automatically cleans up any remaining operations that complete in the future. Note, however, that discarding operations doesn't cancel them. If one of the pending asynchronous operations was writing to a file or updating a database, discarding the relevant groups will not prevent those operations from completing. It will simply allow your code to continue without regard for the operations' completions.

Cancellation
AsyncEnumerator allows external code to cancel the iterator during its processing. This feature is especially useful for Windows Forms and WPF applications as it allows the impatient user to cancel an ongoing operation and regain control of the application. AsyncEnumerator also has the ability to cancel itself after a specified amount of time. This feature is useful to server applications that want to set a limit on the amount of time it takes to respond to a client's request. The methods related to cancellation are shown here:
public class AsyncEnumerator {
  // Call this method from inside the iterator
  public Boolean IsCanceled(out Object cancelValue);

  // Call this method from inside the iterator
  public Boolean IsCanceled();

  // Call this method from code outside the iterator
  public Boolean Cancel(Object cancelValue};

  // Call this method from code inside or outside the iterator
  public void SetCancelTimeout(Int32 milliseconds,
    Object cancelValue);
}
To take advantage of cancellation, inside your iterator, issue each asynchronous operation as part of a discard group. This allows the AsyncEnumerator to automatically discard any operations that complete after the cancellation request. Then, to detect a cancellation request, include code similar to that shown in Figure 3 after each yield return statement.
Figure 3 Detecting Cancellation
IEnumerator<Int32> MyIterator(AsyncEnumerator ae, ...) {
  // obj refers to some object that has BeginXxx/EndXxx methods
  obj.BeginXxx(...,         // Pass any arguments as usual
    ae.End(0, obj.EndXxx),  // For AsyncCallback indicate
                            // discard group 0 & proper End method
                            //to call for cleanup
    null);                  // BeginXxx's AsyncState argument

  // Make more calls to BeginXxx methods if desired here...

  yield return n; // Resume iterator after 'n' operations
                  // complete or if cancelation occurs

  // Check for cancellation
  Object cancelValue;
  if (ae.IsCanceled(out cancelValue)) {
    // The iterator should cancel due to user request/timeout
    // Note: It is common to call "yield break" here.
  } else {
    // Call DequeueAsyncResult 'n' times to
    // process the completed operations
  }
}
Now, when you want to start executing a cancelable iterator, you construct an AsyncEnumerator object and call BeginExecute on it just like you normally would. Then, when some part of your application wants to cancel the iterator, it calls the Cancel method. When calling Cancel, you can pass it a reference to an object that is then passed to the iterator by way of the IsCanceled method's out parameter. This object gives the code canceling the iterator a way to communicate to the iterator why it is being canceled. If the iterator doesn't care to know why it is being canceled, then it can call the overloaded IsCanceled method that takes no parameters.
The SetCancelTimeout method can be called by code both inside and outside the iterator. When this method is called, it sets up a timer that will call Cancel automatically (passing the value you specify via the CancelValue argument) when the time expires.
The code in Figure 4 shows a Windows Forms application that uses many of the features discussed in this column. The UI for the application is shown in Figure 5. It uses the AsyncEnumerator's SyncContext feature to ensure that all the iterator code executes via the GUI thread allowing UI controls to be updated. This code also shows how to use the AsyncEnumerator's APM support so as not to block the GUI thread allowing the UI to remain responsive.
Figure 4 WindowsFormsViaAsyncEnumerator.cs
namespace WinFormUsingAsyncEnumerator {
  public partial class WindowsFormsViaAsyncEnumerator : Form {
    public static void Main() {
      Application.Run(new WindowsFormsViaAsyncEnumerator());
    }

    public WindowsFormsViaAsyncEnumerator() {
      InitializeComponent();
    }

    private AsyncEnumerator m_ae = null;

    private void m_btnStart_Click(object sender, EventArgs e) {
      String[] uris = new String[] {
        "http://Wintellect.com/",
        "http://1.1.1.1/",   // Demonstrates error recovery
        "http://www.Devscovery.com/"
      };

      m_ae = new AsyncEnumerator();

      // NOTE: The AsyncEnumerator automatically saves the
      // Windows Forms SynchronizationContext with it ensuring
      // that the iterator always runs on the GUI thread;
      // this allows the iterator to access the UI Controls

      // Start iterator asynchonously so that GUI thread doesn't block
      m_ae.BeginExecute(GetWebData(m_ae, uris), m_ae.EndExecute);
    }

    private IEnumerator<Int32> GetWebData(AsyncEnumerator ae, String[] uris) {
      ToggleStartAndCancelButtonState(false);
      m_lbResults.Items.Clear();

      if (m_chkAutoCancel.Checked)
        ae.SetCancelTimeout(5000, ae);

      // Issue several Web requests (all in discard group 0) simultaneously
      foreach (String uri in uris) {
        WebRequest webRequest = WebRequest.Create(uri);

        // If the AsyncEnumerator is canceled, DiscardWebRequest cleans up
        // any outstanding operations as they complete in the future
        webRequest.BeginGetResponse(ae.EndVoid(0, DiscardWebRequest),
                                                          webRequest);
      }

      yield return uris.Length;  // Process the completed Web requests
                                 // after all complete

      String resultStatus; // Ultimate result of processing shown to user

      // Check if iterator was canceled
      Object cancelValue;
      if (ae.IsCanceled(out cancelValue)) {
        ae.DiscardGroup(0);
        // Note: In this example calling DiscardGroup above is not mandatory
        // because the whole iterator is stopping execution; causing all
        // discard groups to be discarded automatically.

        resultStatus = (cancelValue == ae) ? "Timeout" : "User canceled";
        goto Complete;
      }

      // Iterator wasn't canceled, process all the completed operations
      for (Int32 n = 0; n < uris.Length; n++) {
        IAsyncResult result = ae.DequeueAsyncResult();

        WebRequest webRequest = (WebRequest)result.AsyncState;

        String s = "URI=" + webRequest.RequestUri + ", ";
        try {
          using (WebResponse webResponse = webRequest.
                 EndGetResponse(result)) {
                    s += "ContentLength=" + webResponse.ContentLength;
          }
        }
        catch (WebException e) {
          s += "Error=" + e.Message;
        }
        m_lbResults.Items.Add(s);  // Add result of operation to listbox
      }
      resultStatus = "All operations completed.";

    Complete:
      // All operations have completed or cancellation occurred, tell      // user
      MessageBox.Show(this, resultStatus);

      // Reset everything so that the user can start over if they desire
      m_ae = null;   // Reset since we're finished
      ToggleStartAndCancelButtonState(true);
    }

    private void m_btnCancel_Click(object sender, EventArgs e) {
      m_ae.Cancel(null);
      m_ae = null;
    }

    // Swap the Start/Cancel button states
    private void ToggleStartAndCancelButtonState(Boolean enableStart) {
      m_btnStart.Enabled = enableStart;
      m_btnCancel.Enabled = !enableStart;
    }

    private void DiscardWebRequest(IAsyncResult result) {
      // Get the WebRequest object used to initate the request
      // (see BeginGetResponse's last argument)
      WebRequest webRequest = (WebRequest)result.AsyncState;

      // Clean up the async operation and Close the WebResponse (if no       // exception)
      webRequest.EndGetResponse(result).Close();
    }
  }
}
Figure 5 The Sample Application
Inside the iterator, many Web requests are issued as part of a discard group, and, since the UI stays responsive, the user can click the Cancel button if she gets tired of waiting for the result. If this happens, the AsyncEnumerator will autocomplete any of the operations so that your iterator code doesn't have to worry about doing any cleanup. Note that the form also demonstrates how to set up a timer so that the AsyncEnumerator will cancel itself automatically after five seconds if all the operations have not completed.
This sample makes Web requests using the .NET WebRequest class. When calling WebRequest's BeginGetResponse method, cleanup requires more than just calling EndGetResponse. You also must call Close (or Dispose) on the WebResponse object that EndGetResponse returns.
For this reason, the code passes the DiscardWebRequest method to the EndVoid method when calling BeginGetResponse. The DiscardWebRequest method ensures that the WebResponse object is closed if making the Web request was successful and doesn't throw an exception.

Many developers know that asynchronous programming is the key to increasing the performance, scalability, responsiveness, and reliability of their applications, servers, and components. Unfortunately, many developers refuse to fully embrace asynchronous programming because the programming model has been so much more tedious and difficult than the tried-and-true synchronous programming model.
Using the C# iterator feature and my AsyncEnumerator class allows developers to embrace asynchronous programming from within a synchronous programming model. AsyncEnumerator also integrates easily with other parts of the .NET Framework and offers many features that allow developers to extend their applications beyond what is possible with the usual synchronous programming model.
I've been using AsyncEnumerator for more than a year and have helped many companies integrate it into their software with excellent results. Download the code from wintellect.com/PowerThreading.aspx. I hope you gain as much from it as I have.

Send your questions and comments for Jeffrey to mmsync@microsoft.com.

Jeffrey Richter a cofounder of Wintellect (www.Wintellect.com), an architecture review, consulting and training firm. He is the author of several books, including CLR via C#. Jeffrey is also a contributing editor to MSDN Magazine and has been consulting with Microsoft since 1990.


Going Places
What Can a Robot Teach You?
Mike Calligaro
Contents
I, WiMo
WiMo Sings the Blues
WiMo Text Home
You Put Your Left Tread In ...
WiMo's Electric Eye
WiMoFi
Do WiMos Dream of Electric Sheep?
What can a robot teach you? If you're a fan of the best show on television, you're probably thinking, "Well, it could probably teach me how to walk, talk, and beat the frak out of my opponents." If, however, you're also a fan of the best mobile operating system on the planet, you might be thinking, "Well, it could probably teach me how to programmatically take pictures, use SMS as a data channel, send data over Bluetooth or WiFi, and be notified at various points in a playing song." I can't help you with beating the frak out of your opponents. But if you're interested in the other stuff, read on.

Although I've been a software developer at Microsoft for more than 14 years, I started life as a hardware design engineer. I graduated with a degree in Electrical Engineering and spent my career's formative years designing mainframes. This was back in the days when we had to lay out the logic gates by hand—none of that fancy Verilog stuff those young whippersnappers use today. (We also walked miles to work in the smog. Uphill. Both ways. And we liked it.)
This is something of a contrast to my friend and coworker, Brian Cross. Brian's a software guy to the core, always has been and always will be. He's been at Microsoft his whole career and speaks C# almost as well as English. (It was Brian—and Charles Petzold, of course—who taught me the wonders of managed code.) So it's somewhat ironic that, between the two of us, Brian is the guy who built a robot.
You may have heard of WiMo, Brian's Windows Mobile®-based robot, an illustration of which is shown in Figure 1. WiMo has gained some notoriety for being a fully programmable robot that you don't need to be an Electrical Engineer to build. Brian, in classic Microsoft style, sees the whole thing as a software problem. In its simplest form, you can build a functional WiMo out of nothing more than a Windows Mobile phone and some Legos—no soldering iron required.
Figure 1 Artist's Rendering of WiMo
But, easy as that may be, this is a column about mobile development, not robotics. So today's column isn't really going to be about building a WiMo. Instead I'll explore some of the things that WiMo's code base does and discuss how you can apply those same things to your mobile apps. If, along the way, you're inspired to do something really off the wall with your phone, all the better.
First things first. WiMo is basically a simple set of motors and sensors whose main purpose is to carry around a Windows Mobile device that acts as its brain. All of the real work is done by the phone, and all of the code that does this work is available for free on WiMo's Web site, wimobot.com. If you're interested in building a WiMo, you'll want to follow the Building WiMo link to wimobot.com/Building.aspx. If you're just interested in the source code, though, go to wimobot.com/SourceCode.aspx. We're going to be spending a lot of time talking about that source code, so you might as well get it now.
Go ahead. I'll wait.
What's that? Yes, click the disclaimer that says you won't sue Brian if you use his technology to build a machine that eventually turns on its creator. That'll never happen. Trust us.
After you've unpacked the code, open the file WiMo.sln in Visual Studio®. Don't mind the XML document errors; Part of WiMo's charm is that it's being done by Brian the "We can do anything" developer, not Brian the "We can't even think about releasing something without a proper test plan" corporate suit. WiMo.sln has a number of projects, but you should start at Form1.cs in the WiMo project. In that file, you'll see the usage of many of the more interesting classes defined in the solution.

WiMo Sings the Blues
The Windows Mobile part of WiMo communicates with the wheels and sensors via Bluetooth. Humorously, Brian chose this connection style because he was worried about accidentally ruining his phone if he tried to solder wires to it. All the better, because now you can lift this code and use it for non-robotic things. If you're looking for a more official venue for source code, you can find Bluetooth samples on MSDN® at msdn.microsoft.com/embedded/aa714519. You'll find a lot of similarity between the WiMo Bluetooth classes and the MSDN samples, though.
Most Windows Mobile phones support Bluetooth, and if yours is one that uses the Microsoft® Bluetooth stack (unfortunately, some don't), the WiMo code will work on it. Because Bluetooth is so common, if you want to write an app that communicates with nearby phones, using Bluetooth as your communication channel is a reasonable way to do it.
The Brain behind WiMo—a Phone
So say you've copied over the Bluetooth classes from WiMo or the MSDN samples, and you've paired your phone with a device called "MyDevice." Here's the code to connect with the device:
using Microsoft.WindowsMobile.SharedSource.Bluetooth;

BluetoothRadio radio = new BluetoothRadio();
NetworkStream bthStream = null;
foreach (BluetoothDevice device in radio.PairedDevices) {
  if (device.Name == "MyDevice") {
    try {
      bthStream =
        device.Connect(StandardServices.SerialPortServiceGuid);
    }
    catch (SocketException) {
      // do error recovery stuff here
    }
    break;
  }
}
At that point, you have a standard NetworkStream you can use to send and receive data with bthStream.Read and bthStream.Write. And as easy as that, you're now talking over Bluetooth. You could use this for any situation where you've got your app running on two phones and you want them to communicate with each other. But there are a ton of other possibilities as well.
This is the basic method for talking to any Bluetooth device that speaks serial. For instance, if you want to connect to a Bluetooth-enabled GPS unit, you'd do something very similar. A poster on the WiMo blog said that he used code like this to collect engine performance data from his Bluetooth-enabled car. Lego Mindstorms speaks Bluetooth, so this is how the Lego version of WiMo works. And, if you're not afraid of a little soldering, you can connect an inexpensive Bluetooth modem to any serial device. (Brian has the details on the WiMo site under the "Hardware to buy to build a WiMo" section.) There's a huge array of sensors available in the world that you could connect to one of these modems. As mobile developers, you know what to do with the data once you get it onto your phone. The problem was traditionally that getting it onto the phone was hard. Not anymore.

WiMo Text Home
Because WiMo's brain is a phone, it can receive commands from a number of different sources, including SMS. This means that WiMo can receive commands from most of the cell phones on the planet, Windows Mobile or otherwise, and can even talk to desktop IM clients that support SMS. (Windows Live™ Messenger is one of these.)
This is handled by the MessageInterceptor class, which you can see being used in WiMo's DesktopSms.cs file. This file is under the WiMo.Common.Device project. It's also documented in full on MSDN at go.microsoft.com/fwlink/?LinkId=120888. MessageInterceptor basically says, "Whenever the phone receives an SMS with these conditions, send it to my app." You can even have it launch your app if it's not already running.
Message interception, like most things C#, is easy to implement. Figure 2 shows the basic code you need.
Figure2 Message Interception
using Microsoft.WindowsMobile.PocketOutlook;
using Microsoft.WindowsMobile.PocketOutlook.MessageInterception;

const string smsCommandId = "MyAppName";
const string smsComparisonValue = "MyAppSmsIdentfier";

MessageInterceptor intercept =
  new MessageInterceptor(InterceptionAction.NotifyAndDelete);
intercept.MessageCondition = new MessageCondition();
intercept.MessageCondition.CaseSensitive = true;
intercept.MessageCondition.ComparisonType =
  MessagePropertyComparisonType.StartsWith;
intercept.MessageCondition.ComparisonValue = smsComparisonValue;
intercept.EnableApplicationLauncher(smsCommandId);

intercept.MessageReceived +=
  new MessageInterceptorEventHandler(intercept_MessageReceived);

void intercept_MessageReceived(object sender,
  MessageInterceptorEventArgs e) {
  if (e.Message is SmsMessage) {
    SmsMessage sms = (SmsMessage)e.Message;
    // sms.Body contains the text of the message we received
  }
}
You may think of SMSs as short e-mail messages, but these classes allow you to use them as a full data channel. Imagine writing a turn-based game like Chess (or Shogi, the far superior Japanese version of Chess) where you move by sending and receiving SMS messages.
Or imagine having an app on your phone that can be remotely configured. Do you frequently misplace your phone? (I'll bet it ran off with your car keys.) You could log into your IM client and send it a command to play a WAV file that proclaims, "Hey Mike, I'm over here." (Although telling me where your phone is would be kind of weird.) Or, if you want to be really hip, you could have your phone proclaim, "WiMo 5 is alive!" No one would get the joke, but that's part of the appeal.

You Put Your Left Tread In ...
This is a little embarrassing but ... how do I put this? OK, WiMo dances the Hokey Pokey. There, I said it. Like the three laws of robotics, this is actually a protection for humanity. If WiMo ever evolves to the point where it can pick up a weapon and say, "I'm sorry, Brian I'm afraid I can't do that," Brian will respond, "WiMo, dance!" Then, after completing the Hokey Pokey, WiMo will slink off into a corner and pretend that the whole sordid affair never happened.
Not that I'd ever suggest you code your Windows Mobile apps to interact with the Hokey Pokey, but the idea of being notified at certain points in a song is pretty nifty. And the actual infrastructure that the WiMo Hokey Pokey uncovers is really very cool.
The infrastructure I'm talking about is the Windows Mobile State and Notifications Broker (SNAPI). The managed version is in the Microsoft.WindowsMobile.Status namespace, and the class you really care about is SystemState.
A good resource for additional detail on the SystemState class can be found at go.microsoft.com/fwlink/?LinkId=120893. Click the link for SystemState Properties. See that list of a hundred or so system properties that you can access? Signal strength, battery level, Bluetooth states, whether or not the user is talking on the phone, the device's phone number, and so on? What could you do with access to that kind of power? Use it wisely. SNAPI can notify you whenever one of these values changes. It can even launch your app if your chosen value changes while your app isn't running.
WiMo dances as a result of being notified whenever the MediaPlayerTrackTimeElapsed property changes. It knows what to do at each point in the song and acts accordingly. Here's the code:
using Microsoft.WindowsMobile.Status;

SystemState mediaTimeElapsed =
  new SystemState(SystemProperty.MediaPlayerTrackTimeElapsed);
mediaTimeElapsed.Changed +=
  new ChangeEventHandler(mediaTimeElapsed_Changed);

void mediaTimeElapsed_Changed(object sender, ChangeEventArgs args) {
  int time = (int)args.NewValue;
  // Do something with the new value
}
If you're interested in having your app launch when one of these states changes, look at the EnableApplicationLauncher method in SystemState. There you'll find the WiMo code that does this in the main Form1.cs.

WiMo's Electric Eye
Most cell phones today have cameras, and WiMo makes good use of the one built into its brain. The part of WiMo that holds the phone is on a disk attached to a motor. One of the things WiMo knows how to do is spin the phone around to point the camera at you, take a picture, and then spin the phone back around so that you can see the screen. The phone can also act as a rudimentary video camera, allowing you to drive WiMo remotely and watch what it's seeing.
In Windows Mobile 5.0 (which WiMo is based on), the camera API is somewhat limited. There's no managed interface to it, and the native interface is heavily geared toward taking a still image and saving it on a storage card. To use it as a video camera, frame rate is limited by how fast the device can store and retrieve images from that storage card. No one would ever claim that this is ideal, but it gets the job done.
There are, however, some silver linings in this situation. First, if you've ever wanted to write a native DLL that you could call from your managed application, WiMo provides a very nice example of how to do it. Look at the WiMoNative project, and then look at how the rest of the WiMo solution interacts with it. If you've ever pInvoked something from managed code, you're already halfway there. Calling into a DLL that you wrote is just a matter of pInvoking the functions you exported. For example, look at the end of the Camera.cs file under the Wimo.Common.Device project. Here are two of the pInvoke declarations you'll find there:
[DllImport("WiMoNative.dll")]
private static extern int OpenCamera(int id, ref IntPtr hCamera);

[DllImport("WiMoNative.dll")]
private static extern int CloseCamera(IntPtr hCamera);
And here are the native declarations of the same functions in WiMoNative.cpp:
HRESULT OpenCamera(DWORD dwResolutionId, HANDLE* hCamera);
HRESULT CloseCamera(HANDLE hCamera);
In addition, WiMoNative.cpp is probably the simplest introduction to DirectShow that you'll ever find. The OpenCamera function builds a filter graph and hooks up the appropriate pins in around sixty lines of code. Over the years I've had multiple experts try to teach me how DirectShow works. And each time my eyes have glazed over in the first five minutes. But when it comes to understanding code, there's nothing like stepping through a solid example that does something useful.

WiMo started with Brian saying, "I feel like learning something new. I'll tackle electronics and build a robot." But that's really only half the story. Brian also said, "I wrote all these Windows Mobile managed classes. I should do something that shows people how to use them." If WiMo feels like it uses every technology under the sun, that's because it does. Intentionally.
TCP/IP is one of those technologies. WiMo can talk to a desktop PC via WiFi, which uses the same managed TcpClient that wired Ethernet connections use. The code is available in WimoNetTcp.cs under the Wimo.Common project. The MSDN page for the TcpClient is msdn.microsoft.com/system.net.sockets.tcpclient.
Communicating over TCP is similar to communicating over Bluetooth. You first make a connection, and then you use a NetworkStream to send and receive data. If you're connecting to a server, this is as easy as:
using System.Net.Sockets;
using System.Net;

const string myServer = "www.myserver.com";
const int myPort = 11000;

TcpClient client = new TcpClient();
client.Connect(myServer, myPort);

NetworkStream stream = client.GetStream();

// From here you can call the stream.Read and stream.Write APIs

stream.Close();
If you are talking to another device, you can use the version of client.Connect that takes an IP Address instead of a host name. Of course, then you need some way to get that IP Address from the remote machine. Getting your own addresses is easy, though. For example, let's say you want your IPv4 address.
string myHostName = Dns.GetHostName();
IPHostEntry hostEntry = Dns.GetHostEntry(myHostName);
foreach (IPAddress ip in hostEntry.AddressList) {
  if (ip.AddressFamily == AddressFamily.InterNetwork) {
    // ip is the IPv4 address
  }
}
If you're writing an app that communicates via TCP to the same app on another phone, you might get your IP address, send it to the other phone via SMS, and then have the other phone use that IP Address to connect to you. For that, you should look at the TcpListener class at msdn.microsoft.com/system.net.sockets.tcplistener. The general idea is this:
TcpListener listener =
  new TcpListener(IPAddress.Any, port);
listener.Start();
TcpClient client = listener.AcceptTcpClient();
listener.Stop();
That's a whirlwind tour of TCP/IP, but I believe it's enough to get you going.

Do WiMos Dream of Electric Sheep?
There's a lot more to WiMo's codebase than I touched on here. WiMo is actually a fleet of robots with a number of different wheel and sensor modules. The early ones were built from robot parts (motors, microcontrollers, and circuit boards). But later WiMos were based on other things, including a remote control monster truck and Lego Mindstorms. Brian's currently working on turning a Roomba vacuum cleaner into a WiMo (you'll find the start of that code in the sources). There's a lot of neat code there that abstracts these different bases, as well as some good stuff for talking to an inexpensive microcontroller.
My goal in this column was to show you some of the technologies you have available to you as you develop your Windows Mobile apps. But if you want to go beyond that, don't hold back. If you've ever wanted to dive into electronics, WiMo is a great place to start, even if you don't know the difference between an inductor and a capacitor. The Wimobot site will show you all steps needed, everything from which components to buy to how to hook them together to how to write the code that talks to them.
If you're not interested in a device that can dance the Hokey Pokey, you've still got a huge range of options before you. The typical Windows Mobile phone is made of hardware that is better in all ways than the PC I used when I started at Microsoft. It's got a faster processor, more RAM, more storage, and a screen that shows more pixels. And you can carry it around with you.
Did you replace a Windows Mobile 5.0 device with a Windows Mobile 6 one? What's your WM5 device doing now? You probably disabled the cell phone service, but everything else should still work. If this is the case, you've got a fully programmable mobile computer collecting dust in a closet—you should do something with it. Build a robot. Make a security camera (WiMo takes pictures and sends them to a PC, and it doesn't need any wheels or sensors to do it). Build or buy a weather station that talks Bluetooth, and use your phone to gather and analyze data from it. Maybe even start working on that Astromech Droid. ("Come along, WiMo.")
I'd say "The sky's the limit" but that's too ... limiting. If you're reading this, you're probably an engineer. You can do anything. You just have to get started. I hope that WiMo and this column have inspired you to do something exciting with your Windows Mobile devices. Happy coding.

Send your questions and comments to goplaces@microsoft.com.

Mike Calligaro is a Senior Development Lead with the Windows Mobile team at Microsoft and a contributor to the Windows Mobile team blog, which you can see at blogs.msdn.com/windowsmobile.


{ End Bracket }
U.S. Imagine Cup
Mitch Irsfeld

Imagine a world where technology enables a sustainable environment—that's what Microsoft called on students to consider for its 2008 U.S. Imagine Cup competition. It's a challenge that Microsoft has taken on as the company strives to become a leader in environmental sustainability, an issue that is gaining momentum as more technologists and users inquire about green IT initiatives and carbon footprints. As a result, the relationship between environment sustainability and business sustainability has become more apparent.
The success of Imagine Cup subtantiates this trend, as more than 185,000 young software developers and artists from more than 100 countries participated in this year's international competition. In the U.S. alone the competition drew more than 16,000 entries, up from 4,000 last year. Their solutions, presented at the competition in Los Angeles this past spring, reflected this shift in environmental awareness and displayed a fresh blend of technology and unconventional thinking. Many of the solutions mirrored the commitment Microsoft has made to preserve and improve the environment through software innovation, strategic partnerships, and utilizing sound environmental practices.
Participants were asked to give particular attention to the business sustainability of their solutions, which is essential to the success of many green initiatives. Several teams already had plans to bring their solutions to market, including Team Carbon Cart from Seattle Pacific University, which developed an e-commerce site with an eco-conscience. CarbonCart.com secured a drop-ship partnership with Amazon.com enabling CarbonCart.com to offer Amazon's entire product inventory. But the interesting part is that CarbonCart.com further partnered with the Carbon Fund to allow shoppers to offset the carbon footprint of their orders, including the carbon emissions of shipping the merchandize.
The winner of the U.S. Software Design competition, Team Sparx from the Rochester Institute of Technology, showed how technology can help consumers and businesses save energy and money today. Its Reactivity technology programs a network of sensors for homes or businesses to take readings of environmental variables such as electrical current, temperature, humidity, light, sound, carbon dioxide, carbon monoxide, and motion. It then stores the data in SQL Server® for analysis.
Other software design entries included an interactive drawing tablet to reduce paper use in elementary school education, carbon footprint calculators, and an online solution that integrates with Facebook for matching commuters to share transportation.
To help winners turn their technical solutions into viable businesses, Imagine Cup offers the Imagine Cup Innovation Accelerator program. The program provides technical and business coaching from cosponsoring companies to help students build a business plan and bring their ideas to market while retaining the intellectual property rights.
For its part, Microsoft continues to expand the environmental sustainability initiative on three fronts: internal efforts to reduce its own carbon footprint, providing environment-friendly enhancements to software, and rolling out new solutions to help customers use energy more efficiently. Specifically, Microsoft offers an Environmental Sustainability Dashboard to Dynamics AX customers. Also, there are now power management capabilities in Windows Vista® and Windows Server® 2008. Furthermore, by investing in technologies such as Live Meeting, Microsoft has greatly reduced the need to travel for meetings.
Microsoft also participates in The Green Grid consortium of IT companies seeking to reduce power consumption and improve efficiency in their datacenters. The company has also made a sustained effort to provide digital downloads, thus reducing the need to manufacture and ship CDs and DVDs.
With these steps toward environmental sustainability and the incentives offered by the Imagine Cup competition, Microsoft hopes to instill a commitment to the environment in the next generation of developers and IT professionals. See more about the Imagine Cup at imaginecup.com/About/WhatIs.aspx.

Mitch Irsfeld is Managing Editor of the U.S. MSDN Flash and TechNet Flash newsletters at Microsoft and is based in Redmond, Washington.